From patchwork Wed Aug 26 08:23:26 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?UsOzYmVydCBCb8W+aWs=?= X-Patchwork-Id: 28101 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4691DC333E for ; Wed, 26 Aug 2026 08:33:33 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id EAE246842C; Wed, 26 Aug 2026 10:33:29 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="P6tJmsBv"; dkim-atps=neutral Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E079668425 for ; Wed, 26 Aug 2026 10:23:38 +0200 (CEST) Received: by mail-ej1-x62b.google.com with SMTP id a640c23a62f3a-c1600d040e4so106221766b.1 for ; Wed, 26 Aug 2026 01:23:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787732618; x=1788337418; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=sR7eQIRTodmLB3PIU5BlGuGEzYYEqBDYfXYra7p9Wek=; b=P6tJmsBvL1FQCj9ISQediregl71oRQePGrdcZOr15uIUdLcIxpDF5WuKiVhzZWi3WV cIJdOWTcWuGsgs3w2PHuZuIkKNxNfEDBavoVId4H5XZJzZn98ErYLwdnx7ytKlTblNrC axILdwDtH7L8E0vgFAPBlIaHn/zY85AjQ8gt6AbrWG+nosRhRBk6msaXtQd4mgwCS/Hx 3QNIF0froIFbVy743aFLkTTzcZ26l4cV9YpvfVjoT6pYy4xejUb0jHHBBolrl5JAtwNA tyTFy4XgwijbItga0nE7vhFXkfvb2SnHC4TCO6aGCfx/0YwBhbEZsM7eHklIpZqtxO6A 4c1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787732618; x=1788337418; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=sR7eQIRTodmLB3PIU5BlGuGEzYYEqBDYfXYra7p9Wek=; b=YgC9uPa752UZhmaFXhxeCpY8FBR7yrUueY0cp7mInLpoHWmVhGV0KKJOHgyUwzzdKS r3rrB7XjT2kwttLkDRNQm2sZT6BTv2UZZwZgQQjtji3RJ5Z1BG4zt2hB2WD1S17jBd0t rc4jE9cvr3WNgxC2Go55IBgg3StbSd+taDnk5JrTU961U/aJ2PGDOJJTLDqjMIXqW0US 18j2ZmX9w8neEx2ZzEqN0GHQvtCB60A8H4DI4VNuLvDFXCBYB6Alfhhz6t+nR4yGsMSE uXY6f8JtyEuXC6eY5KjN6Dl7FdWp3MVDorNEJsT0/pUGBQ2XBM1Ke/6lE+sJN2Z6J3IS laOg== X-Gm-Message-State: AFuF++n4M5z9gJotrTsqKdIexQjFNRRv9p2UtEY/XYkpsdKOVp7143SE TyUIWsdXoQ6lL21GgdCwsRzzs/WTY2Ua/2ejmMk7Af2ozTaPuEb0+hm82p3S9JrneEI= X-Gm-Gg: AR+sD10Y6Ku3513FnbGK3GnyAReidiu/q15Ndj1wqLBZkaP4EfInMnkh7CQM2oSXt+J tqUZrkgWxNiLq9s7rjcW1tF1DUoIUrXpa4um1OIGp/6ZMGM+0mPHXQyxw+vW/C1HkTn+pqN+MEQ yqDdYLpiGpVH/7b+MAn3mWbjpL5Jj3xcEdpNbgFoufudSCFHm1gkHayNrHzdraSxs7fMDcCvpcY uwn4ruepJuY7HWIXtXVq8LJDx0Envb4HRcx2wQW0rXonQCIRmo6DmKHoIIDhZzcaa0+B6Nt13yj HaZtXLq1j0RDUlKNV1zrihZJncqev2dbFHdPyE+AM/zNcUEjRrzUA5W19bp1WeZ9+vswzfcR98s bn9yHBvVxbb6WEVoZmX0KEE4NQGZsZUI7t+NkWBlflyOBKsmzje78ZUTA+efL6XdMT/FOoabGFl pAH+t0l4SQOxkToCHTzD4vHfT6TrXPIdBxxJhMijCged+3c4/TL6rc3BGitQWAqaTF9wXF+cNad oURUuh/ga07UN9Wc7YPCz/wurB7+fL7REEI8gQvdAUTh6i3UA7Vg3BMtz1I694kUDb6OxVSQ0fm 1PglK9mPxXXevYI= X-Received: by 2002:a17:906:6a1d:b0:c15:c323:36e5 with SMTP id a640c23a62f3a-c24e2d2b08amr1414392366b.10.1787732618137; Wed, 26 Aug 2026 01:23:38 -0700 (PDT) Received: from robert-83cx (static-dsl-191.87-197-115.telecom.sk. [87.197.115.191]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-c250a5d69a7sm554222766b.8.2026.08.26.01.23.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Aug 2026 01:23:37 -0700 (PDT) From: Robert Bozik To: libcamera-devel@lists.libcamera.org Cc: Kieran Bingham , Milan Zamazal , =?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= , Hans de Goede , Robert Mader , Bryan O'Donoghue , Laurent Pinchart , Robert Bozik Subject: [PATCH 5/7] ipa: softisp: Converge faster and add digital gain to the AGC Date: Wed, 26 Aug 2026 10:23:26 +0200 Message-ID: <20260826082328.24176-6-robertbozik@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260826082328.24176-1-robertbozik@gmail.com> References: <20260826082328.24176-1-robertbozik@gmail.com> MIME-Version: 1.0 X-Mailman-Approved-At: Wed, 26 Aug 2026 10:33:21 +0200 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The AGC corrects the exposure by a small proportional step per statistics period, at most 15% and typically around 5%. Getting from a black start to a bright target then needs tens of statistics periods, and as each period spans several frames and the frames get long in low light, the convergence takes tens of seconds. Far from the target, jump by the ratio between the target and the measured mean sample value, bounded to a factor of two per step; the proportional correction takes over near the target so the convergence stays smooth. When the exposure and the analogue gain are both exhausted, apply a digital gain in the ISP by scaling the colour gains, after black level subtraction. It doesn't add information, so it is only used last and is bounded by the new maxDigitalGain tuning parameter (default 1.0, which disables it). The statistics are computed on the sensor data, so the histogram is scaled by the digital gain of the frame before the mean sample value is computed. The gain is reported in the DigitalGain metadata. Signed-off-by: Robert Bozik --- src/ipa/softisp/algorithms/agc.cpp | 94 ++++++++++++++++++++++++++---- src/ipa/softisp/algorithms/agc.h | 5 ++ src/ipa/softisp/ipa_context.h | 4 ++ 3 files changed, 91 insertions(+), 12 deletions(-) diff --git a/src/ipa/softisp/algorithms/agc.cpp b/src/ipa/softisp/algorithms/agc.cpp index ca403300..b6fc7203 100644 --- a/src/ipa/softisp/algorithms/agc.cpp +++ b/src/ipa/softisp/algorithms/agc.cpp @@ -66,12 +66,37 @@ static constexpr float kExpProportionalGain = 0.04; */ static constexpr float kExpMaxStep = 0.15; +/* + * Errors above this threshold are far from the target, and the small + * proportional steps would need tens of statistics periods to get there + * (a statistics period being several frames, and the frame possibly long + * in low light). For those, jump by the ratio between the target and the + * measured MSV instead, bounded by kExpMaxJump per step. The proportional + * correction takes over near the target, so the convergence stays smooth. + */ +static constexpr float kExpLargeError = 0.5; +static constexpr float kExpMaxJump = 2.0; + +/* + * Digital gain is applied by the ISP on top of the sensor exposure and + * analogue gain, only when those are exhausted. It doesn't add information, + * so the tuning file bounds it with maxDigitalGain (1.0 disables it). + */ +static constexpr double kDefaultMaxDigitalGain = 1.0; + Agc::Agc() { } -int Agc::init(IPAContext &context, [[maybe_unused]] const ValueNode &tuningData) +int Agc::init(IPAContext &context, const ValueNode &tuningData) { + maxDigitalGain_ = tuningData["maxDigitalGain"].get(kDefaultMaxDigitalGain); + if (maxDigitalGain_ < 1.0) { + LOG(IPASoftIspExposure, Warning) + << "maxDigitalGain " << maxDigitalGain_ << " below 1.0, ignored"; + maxDigitalGain_ = 1.0; + } + /* * Expose the frame duration limits the sensor can achieve in the * current mode. Whether the IPA can actually change the frame duration @@ -116,10 +141,24 @@ int Agc::configure(IPAContext &context, [[maybe_unused]] const IPAConfigInfo &co agc.maxFrameDuration = agc.minFrameDuration; } agc.vblank = cfg.vblankDef; + agc.dgain = 1.0; + context.configuration.agc.dgainMax = maxDigitalGain_; return 0; } +void Agc::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, DebayerParams *params) +{ + /* + * The digital gain scales the colour gains the ISP applies after black + * level subtraction. This runs after the AWB has set the gains, as the + * Agc algorithm is listed after Awb in the tuning file. + */ + frameContext.agc.digitalGain = context.activeState.agc.dgain; + params->gains *= frameContext.agc.digitalGain; +} + void Agc::queueRequest(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, const ControlList &controls) { @@ -201,6 +240,8 @@ void Agc::updateExposure(IPAContext &context, IPAFrameContext &frameContext, dou int32_t &exposure = frameContext.sensor.exposure; double &again = frameContext.sensor.gain; int32_t &vblank = frameContext.sensor.vblank; + double &dgain = frameContext.agc.digitalGain; + const double dgainMax = context.configuration.agc.dgainMax; int32_t vblankLo, vblankHi; vblankRange(context, frameContext, vblankLo, vblankHi); @@ -227,32 +268,47 @@ void Agc::updateExposure(IPAContext &context, IPAFrameContext &frameContext, dou } /* - * Compute a proportional correction factor. The sign of the error - * determines the direction: positive error means too dark (increase), - * negative means too bright (decrease). + * Compute the correction factor. The sign of the error determines the + * direction: positive error means too dark (increase), negative means + * too bright (decrease). Far from the target, jump by the measured + * ratio; near it, apply a small proportional step. */ - float step = std::clamp(static_cast(error) * kExpProportionalGain, - -kExpMaxStep, kExpMaxStep); - float factor = 1.0f + step; + float factor; + if (std::abs(error) > kExpLargeError) { + factor = std::clamp(static_cast(kExposureOptimal / std::max(exposureMSV, 0.1)), + 1.0f / kExpMaxJump, kExpMaxJump); + } else { + float step = std::clamp(static_cast(error) * kExpProportionalGain, + -kExpMaxStep, kExpMaxStep); + factor = 1.0f + step; + } if (factor > 1.0f) { /* * Scene too dark: increase exposure first (lengthening the - * frame when the limits allow it), then gain. + * frame when the limits allow it), then analogue gain, then + * digital gain. */ if (exposure < exposureMax) { int32_t next = static_cast(exposure * factor); exposure = std::max(next, exposure + 1); - } else { + } else if (again < context.configuration.agc.againMax) { double next = again * factor; if (next - again < context.configuration.agc.againMinStep) again += context.configuration.agc.againMinStep; else again = next; + } else { + dgain = std::min(dgain * factor, dgainMax); } } else { - /* Scene too bright: decrease gain first, then exposure. */ - if (again > context.configuration.agc.again10) { + /* + * Scene too bright: decrease digital gain first, then analogue + * gain, then exposure. + */ + if (dgain > 1.0) { + dgain = std::max(dgain * factor, 1.0); + } else if (again > context.configuration.agc.again10) { double next = again * factor; if (again - next < context.configuration.agc.againMinStep) again -= context.configuration.agc.againMinStep; @@ -272,14 +328,18 @@ void Agc::updateExposure(IPAContext &context, IPAFrameContext &frameContext, dou static_cast(context.configuration.agc.frameHeight), vblankLo, vblankHi); + dgain = std::clamp(dgain, 1.0, dgainMax); + context.activeState.agc.exposure = exposure; context.activeState.agc.again = again; context.activeState.agc.vblank = vblank; + context.activeState.agc.dgain = dgain; LOG(IPASoftIspExposure, Debug) << "exposureMSV " << exposureMSV << " error " << error << " factor " << factor << " exp " << exposure << " again " << again + << " dgain " << dgain << " vblank " << vblank << " (" << vblankLo << "-" << vblankHi << ")"; } @@ -293,6 +353,7 @@ void Agc::process(IPAContext &context, utils::Duration exposureTime = cfg.lineDuration * frameContext.sensor.exposure; metadata.set(controls::ExposureTime, exposureTime.get()); metadata.set(controls::AnalogueGain, frameContext.sensor.gain); + metadata.set(controls::DigitalGain, static_cast(frameContext.agc.digitalGain)); if (cfg.vblankSupported) { frameContext.agc.frameDuration = cfg.lineDuration * (cfg.frameHeight + frameContext.sensor.vblank); @@ -319,6 +380,7 @@ void Agc::process(IPAContext &context, frameContext.sensor.exposure = context.activeState.agc.exposure; frameContext.sensor.gain = context.activeState.agc.again; frameContext.sensor.vblank = context.activeState.agc.vblank; + frameContext.agc.digitalGain = context.activeState.agc.dgain; return; } @@ -344,8 +406,16 @@ void Agc::process(IPAContext &context, return; } + /* + * The statistics are computed on the sensor data, before the ISP + * applies the digital gain. Scale the histogram index by the digital + * gain of the frame so that the MSV reflects the output brightness. + */ + const double digitalGain = frameContext.agc.digitalGain; for (unsigned int i = 0; i < histogramSize; i++) { - unsigned int idx = (i - (i / yHistValsPerBinMod)) / yHistValsPerBin; + unsigned int scaled = std::min( + static_cast(i * digitalGain), histogramSize - 1); + unsigned int idx = (scaled - (scaled / yHistValsPerBinMod)) / yHistValsPerBin; exposureBins[idx] += histogram[blackLevelHistIdx + i]; } diff --git a/src/ipa/softisp/algorithms/agc.h b/src/ipa/softisp/algorithms/agc.h index 1aa77737..6e9ba728 100644 --- a/src/ipa/softisp/algorithms/agc.h +++ b/src/ipa/softisp/algorithms/agc.h @@ -24,6 +24,9 @@ public: void queueRequest(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const ControlList &controls) override; + void prepare(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + DebayerParams *params) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const SwIspStats *stats, @@ -34,6 +37,8 @@ private: void vblankRange(const IPAContext &context, const IPAFrameContext &frameContext, int32_t &vblankLo, int32_t &vblankHi) const; int32_t exposureMaxForVblank(const IPAContext &context, int32_t vblank) const; + + double maxDigitalGain_; }; } /* namespace ipa::softisp::algorithms */ diff --git a/src/ipa/softisp/ipa_context.h b/src/ipa/softisp/ipa_context.h index b5577768..de188f14 100644 --- a/src/ipa/softisp/ipa_context.h +++ b/src/ipa/softisp/ipa_context.h @@ -42,6 +42,8 @@ struct IPASessionConfiguration { /* Lines the sensor keeps between max exposure and frame length */ int32_t exposureMargin; uint32_t frameHeight; + /* Digital gain applied by the ISP after the sensor gain is exhausted */ + double dgainMax; } agc; struct { std::optional level; @@ -56,6 +58,7 @@ struct IPAActiveState { int32_t exposure; double again; int32_t vblank; + double dgain; bool valid; utils::Duration minFrameDuration; utils::Duration maxFrameDuration; @@ -91,6 +94,7 @@ struct IPAFrameContext : public FrameContext { utils::Duration minFrameDuration; utils::Duration maxFrameDuration; utils::Duration frameDuration; + double digitalGain; } agc; float gamma;