From patchwork Wed Aug 27 09:07:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 24247 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id F20B6C332D for ; Wed, 27 Aug 2025 09:08:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3B4C6692F3; Wed, 27 Aug 2025 11:08:09 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="h1wuLYUG"; dkim-atps=neutral Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B1F40692EF for ; Wed, 27 Aug 2025 11:07:52 +0200 (CEST) Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-45a20c51c40so50402225e9.3 for ; Wed, 27 Aug 2025 02:07:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1756285672; x=1756890472; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Cvoka5m82n+rWiPnVl3CH3ZzB0QBvy8hzVw8dQpv3g8=; b=h1wuLYUG5vmjLZzR5slJFJJohe5JS6T/VtMGXIPH73XVsQkDMG3VXt79qA0G/sEGJR N/wJ2FGAvqv0LBLzYJFtdL4Fk8W7p8Mzownu6pxIjdOzjif2CPQA//ZJvUGlP5X5S7dr 3kGKl6tt5ksyG/U9t0QDZtI/ohLjA2Q7cbORXdHao/0APBc2+B3/8wpoO+VxPs368EcP grKyt+kVl5tXZZ8RcWJKcQS63m23xH9Zx2fShcKUKPB9u12JfhBDtrqZSH3gIeC3jkhj 32oU4xKuANhaZK3u13W8ZeSgTN0MPukcZc29c7atmR8REhQfogaXY8a+OiCW7Q7+E1AI z7ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756285672; x=1756890472; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Cvoka5m82n+rWiPnVl3CH3ZzB0QBvy8hzVw8dQpv3g8=; b=cDINhtEU6DK0k9JeqRpVMzogERZbBoec4wWV9QGsdV7HLSnLf6tm/uiVNRzeEcd5od K6vIVE26Jk5Lb5Evf4QWWsgMx99o/GbkBiih8JJMoMGZXvStTtadc10CsJ8py9XWzuSf 8BkA/ZtSB7Wp5jKqIErYoa4XeQeD52OgW1UASw6rFvT0WoNKMo/YAsgIxH8tGK0HxmZu 8cJAwymx2q36K+OfkLyEScR1xRafl/oOa/n4WVOjGEX0bwyVDxaL3LxL/GyIir4dFumF SvxH+2kkyhlsUsd2UcPC6RNb10KASHI7vuBLizoP24o/KfCvLTK5jwz1KwgQSUTV7sJH FTvg== X-Gm-Message-State: AOJu0YwGXAT5S5ZXFJBKg2+XoYFR28vF6StmT/0VKO0q3yyesXu2S/AR 2IfgcujifWfQw+2oX2NCgvAROnaq8H2jvgwiK10XeBpKfDhN3nUWUEQF7WxTQSraC4k1vT5QOyQ wCKLG X-Gm-Gg: ASbGncvhO1s+FRKB8Bphoyi7r1ok5jTIPSoXZmlbLWfuiyuHXXDojcoC+w2RBoCT3o5 n+P27F13h+a2oVk6Z0XrFHbIxGdvTAJxOVes1LHlg1Cn547j4vZ1mBq0Gta3r/5vbRV9VPK+gg5 bEK0jR0BXM8wjPmOcGv5G3xPRHIFXzdG2WopDBCBjwe9z0ypEGFbCrn2J2w2u9eaWUsHZ3bqE43 VVv8W4L9IQB07bNAQRCYWZ9KWX61IcanwE0omnO/XqvBkUhMGI4cZ79yAK7mcXYv0U8+xEneUP5 vqje0pbGhph4pPfp+roov5mcHGn0P7uFcLEDXGcka6r2Hp5WRp4Aek4BJVp4i9uWAkqWsOIfvVr iAPE7XJ8VYekzcLHISEzzAPBY9+k2WGmv9MdCBgl4NjnAYrnMJbOygKrW05llYua4IQm/GLCJ7z ADFCnX/n6oxi4cgssaxkRDLWsQPkhJYu/8fyMXvFyKb3u/D2Ryrw== X-Google-Smtp-Source: AGHT+IE+Ne/Sh4AIanjMEq2l5OyGWsralWvadk7iMtuz3JO8IOaj2MOsX91nLBvKiKdIYpVEeJxhvQ== X-Received: by 2002:a05:600c:1ca5:b0:459:ddd6:1cc7 with SMTP id 5b1f17b1804b1-45b69616f89mr33246145e9.0.1756285670764; Wed, 27 Aug 2025 02:07:50 -0700 (PDT) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:ffc9:aff6:7f7f:893b]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-45b6f30fe02sm21498675e9.18.2025.08.27.02.07.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Aug 2025 02:07:50 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman Subject: [RFC PATCH 10/12] ipa: rpi: Support memory cameras Date: Wed, 27 Aug 2025 10:07:37 +0100 Message-Id: <20250827090739.86955-11-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250827090739.86955-1-david.plowman@raspberrypi.com> References: <20250827090739.86955-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add support for "memory cameras" (for Bayer reprocessing). Mostly very minor changes, but the way we fill in certain bits of metadata before runningthe IPAs changes significantly (see prepareIspFillMetadata). For example, regular sensors get device status information from the the sensor (or possibly from values we wrote to the sensor a few frames back), whereas "memory cameras" assume that the exposure/gain values passed in with the request controls will be the ones for setting up the tuning algorithms. Signed-off-by: David Plowman --- src/ipa/rpi/common/ipa_base.cpp | 135 +++++++++++++++++++++++++------- src/ipa/rpi/common/ipa_base.h | 5 ++ 2 files changed, 111 insertions(+), 29 deletions(-) diff --git a/src/ipa/rpi/common/ipa_base.cpp b/src/ipa/rpi/common/ipa_base.cpp index f2c51f80..d33da113 100644 --- a/src/ipa/rpi/common/ipa_base.cpp +++ b/src/ipa/rpi/common/ipa_base.cpp @@ -194,6 +194,8 @@ int32_t IpaBase::init(const IPASettings &settings, const InitParams ¶ms, Ini int32_t IpaBase::configure(const IPACameraSensorInfo &sensorInfo, const ConfigParams ¶ms, ConfigResult *result) { + isMemoryCamera_ = sensorInfo.model == "memory"; + sensorCtrls_ = params.sensorControls; if (!validateSensorControls()) { @@ -308,10 +310,13 @@ void IpaBase::start(const ControlList &controls, StartResult *result) /* * SwitchMode may supply updated exposure/gain values to use. * agcStatus_ will store these values for us to use until delayed_status values - * start to appear. + * start to appear. Initialise digital gain to 1, as we'll otherwise end up with + * completely black output if AGC isn't running (which is unusual, but does + * happen with memory cameras). */ agcStatus_.exposureTime = 0.0s; agcStatus_.analogueGain = 0.0; + agcStatus_.digitalGain = 1.0; metadata.get("agc.status", agcStatus_); if (agcStatus_.exposureTime && agcStatus_.analogueGain) { @@ -415,6 +420,75 @@ void IpaBase::unmapBuffers(const std::vector &ids) } } +void IpaBase::prepareIspFillMetadata(const PrepareParams ¶ms, unsigned int ipaContext, + Span &embeddedBuffer, bool &hdrChange) +{ + RPiController::Metadata &rpiMetadata = rpiMetadata_[ipaContext]; + + rpiMetadata.clear(); + hdrChange = false; + + if (!isMemoryCamera_) { + /* + * Regular cameras need to get the device status from the sensor, find the + * appropriate embedded data buffers (if any), and so forth. + */ + fillDeviceStatus(params.sensorControls, ipaContext); + fillSyncParams(params, ipaContext); + + if (params.buffers.embedded) { + /* + * Pipeline handler has supplied us with an embedded data buffer, + * we must pass it to the CamHelper for parsing. + */ + auto it = buffers_.find(params.buffers.embedded); + ASSERT(it != buffers_.end()); + embeddedBuffer = it->second.planes()[0]; + } + + /* + * AGC wants to know the algorithm status from the time it actioned the + * sensor exposure/gain changes. So fetch it from the metadata list + * indexed by the IPA cookie returned, and put it in the current frame + * metadata. + * + * Note if the HDR mode has changed, as things like tonemaps may need updating. + */ + AgcStatus agcStatus; + RPiController::Metadata &delayedMetadata = rpiMetadata_[params.delayContext]; + if (!delayedMetadata.get("agc.status", agcStatus)) { + rpiMetadata.set("agc.delayed_status", agcStatus); + hdrChange = agcStatus.hdr.mode != hdrStatus_.mode; + hdrStatus_ = agcStatus.hdr; + } + } else { + /* + * Memory cameras make the device status from exposure/gain information passed + * in the request controls. Making a "fake" device status like this will cause + * the correct values to be used to set up all the tuning algorithms for this + * frame. + */ + DeviceStatus deviceStatus = {}; + + const auto exposureTime = params.requestControls.get(controls::ExposureTime); + if (exposureTime) + deviceStatus.exposureTime = Duration(*exposureTime * 1000); + else + deviceStatus.exposureTime = Duration(1ms); + + const auto analogueGain = params.requestControls.get(controls::AnalogueGain); + if (analogueGain) + deviceStatus.analogueGain = *analogueGain; + else + deviceStatus.analogueGain = 1.0; + + LOG(IPARPI, Debug) << "Exposure time " << deviceStatus.exposureTime + << " AG " << deviceStatus.analogueGain; + + rpiMetadata.set("device.status", deviceStatus); + } +} + void IpaBase::prepareIsp(const PrepareParams ¶ms) { applyControls(params.requestControls); @@ -428,37 +502,14 @@ void IpaBase::prepareIsp(const PrepareParams ¶ms) unsigned int ipaContext = params.ipaContext % rpiMetadata_.size(); RPiController::Metadata &rpiMetadata = rpiMetadata_[ipaContext]; Span embeddedBuffer; - - rpiMetadata.clear(); - fillDeviceStatus(params.sensorControls, ipaContext); - fillSyncParams(params, ipaContext); - - if (params.buffers.embedded) { - /* - * Pipeline handler has supplied us with an embedded data buffer, - * we must pass it to the CamHelper for parsing. - */ - auto it = buffers_.find(params.buffers.embedded); - ASSERT(it != buffers_.end()); - embeddedBuffer = it->second.planes()[0]; - } + bool hdrChange; /* - * AGC wants to know the algorithm status from the time it actioned the - * sensor exposure/gain changes. So fetch it from the metadata list - * indexed by the IPA cookie returned, and put it in the current frame - * metadata. - * - * Note if the HDR mode has changed, as things like tonemaps may need updating. + * This call fills in certain metadata items that subsequent calls to prepare + * will expect to find. Note that different things happen here for regular + * sensors and "memory cameras". */ - AgcStatus agcStatus; - bool hdrChange = false; - RPiController::Metadata &delayedMetadata = rpiMetadata_[params.delayContext]; - if (!delayedMetadata.get("agc.status", agcStatus)) { - rpiMetadata.set("agc.delayed_status", agcStatus); - hdrChange = agcStatus.hdr.mode != hdrStatus_.mode; - hdrStatus_ = agcStatus.hdr; - } + prepareIspFillMetadata(params, ipaContext, embeddedBuffer, hdrChange); /* * This may overwrite the DeviceStatus using values from the sensor @@ -648,6 +699,18 @@ void IpaBase::setMode(const IPACameraSensorInfo &sensorInfo) */ mode_.sensitivity = helper_->getModeSensitivity(mode_); + if (isMemoryCamera_) { + /* + * For memory cameras, the min/max gain/exposure doesn't make much + * sense (and our "sensor" won't have any V4L2 controls), but we do + * need to set the camera mode information that we have so far, so + * that the control algorithms will see it. + */ + helper_->setCameraMode(mode_); + + return; + } + const ControlInfo &gainCtrl = sensorCtrls_.at(V4L2_CID_ANALOGUE_GAIN); const ControlInfo &exposureTimeCtrl = sensorCtrls_.at(V4L2_CID_EXPOSURE); @@ -688,6 +751,10 @@ void IpaBase::setCameraTimeoutValue() bool IpaBase::validateSensorControls() { + /* Don't need any of these controls with memory cameras. */ + if (isMemoryCamera_) + return true; + static const uint32_t ctrls[] = { V4L2_CID_ANALOGUE_GAIN, V4L2_CID_EXPOSURE, @@ -953,6 +1020,11 @@ void IpaBase::applyControls(const ControlList &controls) case controls::EXPOSURE_TIME: { RPiController::AgcAlgorithm *agc = dynamic_cast( controller_.getAlgorithm("agc")); + + /* Don't print a "no AGC" warning if this is a memory camera. */ + if (!agc && isMemoryCamera_) + break; + if (!agc) { LOG(IPARPI, Warning) << "Could not set EXPOSURE_TIME - no AGC algorithm"; @@ -979,6 +1051,11 @@ void IpaBase::applyControls(const ControlList &controls) case controls::ANALOGUE_GAIN: { RPiController::AgcAlgorithm *agc = dynamic_cast( controller_.getAlgorithm("agc")); + + /* Don't print a "no AGC" warning if this is a memory camera. */ + if (!agc && isMemoryCamera_) + break; + if (!agc) { LOG(IPARPI, Warning) << "Could not set ANALOGUE_GAIN - no AGC algorithm"; diff --git a/src/ipa/rpi/common/ipa_base.h b/src/ipa/rpi/common/ipa_base.h index 39feb7ef..9460ccd4 100644 --- a/src/ipa/rpi/common/ipa_base.h +++ b/src/ipa/rpi/common/ipa_base.h @@ -45,6 +45,9 @@ public: void mapBuffers(const std::vector &buffers) override; void unmapBuffers(const std::vector &ids) override; + void prepareIspFillMetadata(const PrepareParams ¶ms, unsigned int ipaContext, + Span &embeddedBuffer, bool &hdrChange); + void prepareIsp(const PrepareParams ¶ms) override; void processStats(const ProcessParams ¶ms) override; @@ -143,6 +146,8 @@ private: utils::Duration manualPeriod; } flickerState_; + bool isMemoryCamera_; + bool cnnEnableInputTensor_; bool awbEnabled_;