From patchwork Wed Dec 10 16:15:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 25474 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 64454C326B for ; Wed, 10 Dec 2025 16:41:13 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 7F7C9614BB; Wed, 10 Dec 2025 17:41:12 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="GW77hlHX"; dkim-atps=neutral Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id CCACD614CD for ; Wed, 10 Dec 2025 17:41:06 +0100 (CET) Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-47796a837c7so68795e9.0 for ; Wed, 10 Dec 2025 08:41:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1765384866; x=1765989666; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=blvqyUtkLPoAGDVhsamTfHUttwbtDGPkwxR46PSscrY=; b=GW77hlHXXzbYBO/sGZCby0w8ecSfCPqUg1MSjhn315fDuo5jvZbtyolPVZLu2X3aj5 IagC4g+bnvhHhJbYaG8EGn7ukqmjWb7CBwdwkxY2AioY4CkAyXdnDjKsZY427SX4o3Qa bdy/0k/6g8XY/azQQMm3tF0wOOhrE7w4m1rgqExBRt9PBdKDWaaS5Ncd3BRSC1k8FZ2D rmCCJ1h90gOB5d5LLHUPp9YuChXy+s3XC1Kc7voZ007ZO7772cSuD0ryxteAnnCBSL9j QfGOPc8mBo4rJRshIzO34VcQ25jxt7WKV/bPtW5EKQjUiYnpy3nbmOIcUu+dEj/Bli6q qAlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765384866; x=1765989666; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=blvqyUtkLPoAGDVhsamTfHUttwbtDGPkwxR46PSscrY=; b=QfCGaJGz4W2Q8hHMhhOC13iNmzry4+5nHz05c9veuenxHzOhMuQrcXg2coe99uCmzN c9NuI5sij4AIBztyEl64GvEI5k8MauZ+5mErkOFVv0Dk0QNfGlrLUGAjXRVN96kTRPFI VXfuxKnlmUY4Q70iolvoqTP+qCfMpyXHPRcYJvTpxbtlAD1YliXQfex9a21zn6Zln4OF cROTSTbruAVPUX0MFTjjjtlWPKYKvlZZXa8TlNIp8EmOI5psVFopbeMFRpgfVcrDx/ZX Amrm9kmVQhxbJVyNuteeB/J0QwZ7ejAFEbJQejRdiK5BQEbhOx3VzuJLJyawmojQyjp+ qgPg== X-Gm-Message-State: AOJu0Ywnb3ZhOPvWAlvniLa8IhneMFPR2vaTmZ4WFVR9k1JCSjrErIOS hM5y0mo6F+2+MpmUyrTYhqjJMDuEZ+rvDhIVHi8Y6HP+AgejwwkEdRUwHc7Ri4tTa14zmJERxPE usZSY X-Gm-Gg: ASbGncu8mvm5UpHU7yq1VJZXoIkIRqdpHN3EbkiuGBDauBWjQS5af6YpE+ZCdP2T85c YH+tlWgJn/ZlsyzgonLf8ZcAzHhxmYTks2MeCnEdwUlgMdIeh6iP+EP5lJsVcvJr37v+jDCaDke 4T8t8zS0dN0Q3hL60o61Xab/EgA+zPgwLQ8gwJc0oH6I2l4k2tsm0H98aeaA84QM1x+VvagSR2z ZQJ7naYSe2zxaR0PB12CP2DWtVOICddJeDY6kc54hzN79PP+8UWCZHYlFrOWMsz9vqR2ABcRgN+ lgeob/oq0enAYuRhTdvW8ixAdmqSUiVockME8x4yVYJU2V9dIISe2Z8jkTSk8vQOKrkgzNm4xnN 2mhRTGNbvtMrpw6iPo7IOHstIrYWOmLYNN4giCdRwFq6W31Jm/T5MVZzNOviLL6qSKwH1ga3hlh +v0t2idRYximM3wh/6e/1Ore5XSY3dmr0bnHqb+/kA1vE1CzWsDm/qQELljHN2jsfrt8jl2wRNm puCapprR7RQNwAMx8J8E+GnEk0lX8sbtE87bNxz X-Google-Smtp-Source: AGHT+IGuo/hbgCB3jxXHWsxACNjKjgMny7K6/XQAAIvwKeKqpvoQNNfYUwx3/r0IKpA5dUb6UyfXnQ== X-Received: by 2002:a05:600c:3421:b0:479:1348:c61e with SMTP id 5b1f17b1804b1-47a83cadf28mr19865365e9.20.1765384865950; Wed, 10 Dec 2025 08:41:05 -0800 (PST) Received: from davidp-pi5.pitowers.org ([2a00:1098:3142:1f:88ea:c658:5b20:5e46]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-47a88371e13sm1270415e9.11.2025.12.10.08.41.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Dec 2025 08:41:05 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman Subject: [PATCH 09/11] ipa: rpi: Support memory cameras Date: Wed, 10 Dec 2025 16:15:24 +0000 Message-ID: <20251210164055.17856-10-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20251210164055.17856-1-david.plowman@raspberrypi.com> References: <20251210164055.17856-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add support for "memory cameras" (for Bayer reprocessing). Mostly very minor changes, but the way we fill in certain bits of metadata before running the IPAs changes significantly (see prepareIspFillMetadata). For example, regular sensors get device status information from the the sensor (or possibly from values we wrote to the sensor a few frames back), whereas "memory cameras" assume that the exposure/gain values passed in with the request controls will be the ones for setting up the tuning algorithms. Signed-off-by: David Plowman --- src/ipa/rpi/common/ipa_base.cpp | 135 +++++++++++++++++++++++++------- src/ipa/rpi/common/ipa_base.h | 5 ++ 2 files changed, 111 insertions(+), 29 deletions(-) diff --git a/src/ipa/rpi/common/ipa_base.cpp b/src/ipa/rpi/common/ipa_base.cpp index 04e737f6..7413c086 100644 --- a/src/ipa/rpi/common/ipa_base.cpp +++ b/src/ipa/rpi/common/ipa_base.cpp @@ -195,6 +195,8 @@ int32_t IpaBase::init(const IPASettings &settings, const InitParams ¶ms, Ini int32_t IpaBase::configure(const IPACameraSensorInfo &sensorInfo, const ConfigParams ¶ms, ConfigResult *result) { + isMemoryCamera_ = sensorInfo.model == "memory"; + sensorCtrls_ = params.sensorControls; if (!validateSensorControls()) { @@ -310,10 +312,13 @@ void IpaBase::start(const ControlList &controls, StartResult *result) /* * SwitchMode may supply updated exposure/gain values to use. * agcStatus_ will store these values for us to use until delayed_status values - * start to appear. + * start to appear. Initialise digital gain to 1, as we'll otherwise end up with + * completely black output if AGC isn't running (which is unusual, but does + * happen with memory cameras). */ agcStatus_.exposureTime = 0.0s; agcStatus_.analogueGain = 0.0; + agcStatus_.digitalGain = 1.0; metadata.get("agc.status", agcStatus_); if (agcStatus_.exposureTime && agcStatus_.analogueGain) { @@ -417,6 +422,75 @@ void IpaBase::unmapBuffers(const std::vector &ids) } } +void IpaBase::prepareIspFillMetadata(const PrepareParams ¶ms, unsigned int ipaContext, + Span &embeddedBuffer, bool &hdrChange) +{ + RPiController::Metadata &rpiMetadata = rpiMetadata_[ipaContext]; + + rpiMetadata.clear(); + hdrChange = false; + + if (!isMemoryCamera_) { + /* + * Regular cameras need to get the device status from the sensor, find the + * appropriate embedded data buffers (if any), and so forth. + */ + fillDeviceStatus(params.sensorControls, ipaContext); + fillSyncParams(params, ipaContext); + + if (params.buffers.embedded) { + /* + * Pipeline handler has supplied us with an embedded data buffer, + * we must pass it to the CamHelper for parsing. + */ + auto it = buffers_.find(params.buffers.embedded); + ASSERT(it != buffers_.end()); + embeddedBuffer = it->second.planes()[0]; + } + + /* + * AGC wants to know the algorithm status from the time it actioned the + * sensor exposure/gain changes. So fetch it from the metadata list + * indexed by the IPA cookie returned, and put it in the current frame + * metadata. + * + * Note if the HDR mode has changed, as things like tonemaps may need updating. + */ + AgcStatus agcStatus; + RPiController::Metadata &delayedMetadata = rpiMetadata_[params.delayContext]; + if (!delayedMetadata.get("agc.status", agcStatus)) { + rpiMetadata.set("agc.delayed_status", agcStatus); + hdrChange = agcStatus.hdr.mode != hdrStatus_.mode; + hdrStatus_ = agcStatus.hdr; + } + } else { + /* + * Memory cameras make the device status from exposure/gain information passed + * in the request controls. Making a "fake" device status like this will cause + * the correct values to be used to set up all the tuning algorithms for this + * frame. + */ + DeviceStatus deviceStatus = {}; + + const auto exposureTime = params.requestControls.get(controls::ExposureTime); + if (exposureTime) + deviceStatus.exposureTime = Duration(*exposureTime * 1000); + else + deviceStatus.exposureTime = Duration(1ms); + + const auto analogueGain = params.requestControls.get(controls::AnalogueGain); + if (analogueGain) + deviceStatus.analogueGain = *analogueGain; + else + deviceStatus.analogueGain = 1.0; + + LOG(IPARPI, Debug) << "Exposure time " << deviceStatus.exposureTime + << " AG " << deviceStatus.analogueGain; + + rpiMetadata.set("device.status", deviceStatus); + } +} + void IpaBase::prepareIsp(const PrepareParams ¶ms) { applyControls(params.requestControls); @@ -430,37 +504,14 @@ void IpaBase::prepareIsp(const PrepareParams ¶ms) unsigned int ipaContext = params.ipaContext % rpiMetadata_.size(); RPiController::Metadata &rpiMetadata = rpiMetadata_[ipaContext]; Span embeddedBuffer; - - rpiMetadata.clear(); - fillDeviceStatus(params.sensorControls, ipaContext); - fillSyncParams(params, ipaContext); - - if (params.buffers.embedded) { - /* - * Pipeline handler has supplied us with an embedded data buffer, - * we must pass it to the CamHelper for parsing. - */ - auto it = buffers_.find(params.buffers.embedded); - ASSERT(it != buffers_.end()); - embeddedBuffer = it->second.planes()[0]; - } + bool hdrChange; /* - * AGC wants to know the algorithm status from the time it actioned the - * sensor exposure/gain changes. So fetch it from the metadata list - * indexed by the IPA cookie returned, and put it in the current frame - * metadata. - * - * Note if the HDR mode has changed, as things like tonemaps may need updating. + * This call fills in certain metadata items that subsequent calls to prepare + * will expect to find. Note that different things happen here for regular + * sensors and "memory cameras". */ - AgcStatus agcStatus; - bool hdrChange = false; - RPiController::Metadata &delayedMetadata = rpiMetadata_[params.delayContext]; - if (!delayedMetadata.get("agc.status", agcStatus)) { - rpiMetadata.set("agc.delayed_status", agcStatus); - hdrChange = agcStatus.hdr.mode != hdrStatus_.mode; - hdrStatus_ = agcStatus.hdr; - } + prepareIspFillMetadata(params, ipaContext, embeddedBuffer, hdrChange); /* * This may overwrite the DeviceStatus using values from the sensor @@ -650,6 +701,18 @@ void IpaBase::setMode(const IPACameraSensorInfo &sensorInfo) */ mode_.sensitivity = helper_->getModeSensitivity(mode_); + if (isMemoryCamera_) { + /* + * For memory cameras, the min/max gain/exposure doesn't make much + * sense (and our "sensor" won't have any V4L2 controls), but we do + * need to set the camera mode information that we have so far, so + * that the control algorithms will see it. + */ + helper_->setCameraMode(mode_); + + return; + } + const ControlInfo &gainCtrl = sensorCtrls_.at(V4L2_CID_ANALOGUE_GAIN); const ControlInfo &exposureTimeCtrl = sensorCtrls_.at(V4L2_CID_EXPOSURE); @@ -690,6 +753,10 @@ void IpaBase::setCameraTimeoutValue() bool IpaBase::validateSensorControls() { + /* Don't need any of these controls with memory cameras. */ + if (isMemoryCamera_) + return true; + static const uint32_t ctrls[] = { V4L2_CID_ANALOGUE_GAIN, V4L2_CID_EXPOSURE, @@ -955,6 +1022,11 @@ void IpaBase::applyControls(const ControlList &controls) case controls::EXPOSURE_TIME: { RPiController::AgcAlgorithm *agc = dynamic_cast( controller_.getAlgorithm("agc")); + + /* Don't print a "no AGC" warning if this is a memory camera. */ + if (!agc && isMemoryCamera_) + break; + if (!agc) { LOG(IPARPI, Warning) << "Could not set EXPOSURE_TIME - no AGC algorithm"; @@ -981,6 +1053,11 @@ void IpaBase::applyControls(const ControlList &controls) case controls::ANALOGUE_GAIN: { RPiController::AgcAlgorithm *agc = dynamic_cast( controller_.getAlgorithm("agc")); + + /* Don't print a "no AGC" warning if this is a memory camera. */ + if (!agc && isMemoryCamera_) + break; + if (!agc) { LOG(IPARPI, Warning) << "Could not set ANALOGUE_GAIN - no AGC algorithm"; diff --git a/src/ipa/rpi/common/ipa_base.h b/src/ipa/rpi/common/ipa_base.h index 0ebd80db..94efd9a9 100644 --- a/src/ipa/rpi/common/ipa_base.h +++ b/src/ipa/rpi/common/ipa_base.h @@ -45,6 +45,9 @@ public: void mapBuffers(const std::vector &buffers) override; void unmapBuffers(const std::vector &ids) override; + void prepareIspFillMetadata(const PrepareParams ¶ms, unsigned int ipaContext, + Span &embeddedBuffer, bool &hdrChange); + void prepareIsp(const PrepareParams ¶ms) override; void processStats(const ProcessParams ¶ms) override; @@ -143,6 +146,8 @@ private: utils::Duration manualPeriod; } flickerState_; + bool isMemoryCamera_; + bool cnnEnableInputTensor_; bool awbEnabled_; };