From patchwork Fri Nov 19 21:02:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 14663 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id EE17EBF415 for ; Fri, 19 Nov 2021 21:03:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8FC55603B5; Fri, 19 Nov 2021 22:03:09 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="IpvSD2sk"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9C3F460233 for ; Fri, 19 Nov 2021 22:03:05 +0100 (CET) Received: from pendragon.lan (62-78-145-57.bb.dnainternet.fi [62.78.145.57]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 449821C72 for ; Fri, 19 Nov 2021 22:03:05 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637355785; bh=rxvkaSWwq7rCGBvSitrsNFQ825869nmLjV8jJ8rxnFI=; h=From:To:Subject:Date:In-Reply-To:References:From; b=IpvSD2sk3rVRSVTnbZMRwblgYlyEpc2t7iemfQYMV6/1kuSbGg7u4syvhyPY3I22p zEuwUSL9H50Rlx0AFMjTZhnFtAZiT4vWbvl9mT4zgu3qAOFKqBgUoNcg1euP/CjUTw i0k8bW6PRabWMBtqBgSc6Kp3KhVf8yRMFJ2DVSSU= From: Laurent Pinchart To: libcamera-devel@lists.libcamera.org Date: Fri, 19 Nov 2021 23:02:35 +0200 Message-Id: <20211119210239.18540-2-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211119210239.18540-1-laurent.pinchart@ideasonboard.com> References: <20211119210239.18540-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 1/5] ipa: ipu3: agc: Drop kMaxLuminance constant X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The kMaxLuminance constant is badly named, it's not a maximum luminance, but the maximum integer value output by the AWB statistics engine for per-channel averages. The constant is used in a single place, hardcoding the value is actually more readable. Signed-off-by: Laurent Pinchart Reviewed-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/agc.cpp | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index bd02c474611c..43a39ffd57d6 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -61,9 +61,6 @@ static constexpr double kEvGainTarget = 0.5; /* Number of frames to wait before calculating stats on minimum exposure */ static constexpr uint32_t kNumStartupFrames = 10; -/* Maximum luminance used for brightness normalization */ -static constexpr uint32_t kMaxLuminance = 255; - /* * Normalized luma value target. * @@ -298,8 +295,7 @@ double Agc::computeInitialY(IPAFrameContext &frameContext, greenSum * frameContext.awb.gains.green * .587 + blueSum * frameContext.awb.gains.blue * .114; - /* Return the normalized relative luminance. */ - return Y_sum / (grid.height * grid.width) / kMaxLuminance; + return Y_sum / (grid.height * grid.width) / 255; } /** From patchwork Fri Nov 19 21:02:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 14664 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5FEC8BF415 for ; Fri, 19 Nov 2021 21:03:11 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 567206038C; Fri, 19 Nov 2021 22:03:10 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="aOc3vfpc"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2D91360233 for ; Fri, 19 Nov 2021 22:03:06 +0100 (CET) Received: from pendragon.lan (62-78-145-57.bb.dnainternet.fi [62.78.145.57]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id CDA1B1C19 for ; Fri, 19 Nov 2021 22:03:05 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637355786; bh=3Vl2pkE0XFr4qlES+kHAJcyXgVB8PRXyVCWVBMahkXw=; h=From:To:Subject:Date:In-Reply-To:References:From; b=aOc3vfpcc3gpytvQhNtcTQwRce3tEZK4NF62b8kNrGuvGXVF2AbYV0AAxglRvonvC by3fOtwDa2wKdbnb+/1Og4TKcsRJa5APmFSv4WwzZLrvd9Okv3bUux2qlplzVNC3Ah SjElu8km1314sIPVOhsPjBcnHBOhpUvbTR46iLJw= From: Laurent Pinchart To: libcamera-devel@lists.libcamera.org Date: Fri, 19 Nov 2021 23:02:36 +0200 Message-Id: <20211119210239.18540-3-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211119210239.18540-1-laurent.pinchart@ideasonboard.com> References: <20211119210239.18540-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 2/5] ipa: ipu3: agc: Standardize vocabulary on "relative luminance" X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The AGC computes the average relative luminance of the frame and calls the value "normalized luma", "brightness" or "initialY". The latter is the most accurate term, as the relative luminance is abbreviated Y, but the "initial" prefix isn't accurate. Standardize the vocabulary on "relative luminance" in code and comments, abbreviating it to Y when needed. While at it, rename variables to match the libcamera coding style. Signed-off-by: Laurent Pinchart Reviewed-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- Changes since v1: - Improve comment about relative luminance computation --- src/ipa/ipu3/algorithms/agc.cpp | 44 ++++++++++++++++----------------- src/ipa/ipu3/algorithms/agc.h | 8 +++--- 2 files changed, 26 insertions(+), 26 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 43a39ffd57d6..9cd2ded501ed 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -62,12 +62,12 @@ static constexpr double kEvGainTarget = 0.5; static constexpr uint32_t kNumStartupFrames = 10; /* - * Normalized luma value target. + * Relative luminance target. * * It's a number that's chosen so that, when the camera points at a grey * target, the resulting image brightness is considered right. */ -static constexpr double kNormalizedLumaTarget = 0.16; +static constexpr double kRelativeLuminanceTarget = 0.16; Agc::Agc() : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minShutterSpeed_(0s), @@ -250,12 +250,12 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) } /** - * \brief Estimate the average brightness of the frame + * \brief Estimate the relative luminance of the frame with a given gain * \param[in] frameContext The shared IPA frame context * \param[in] grid The grid used to store the statistics in the IPU3 * \param[in] stats The IPU3 statistics and ISP results * \param[in] currentYGain The gain calculated on the current brightness level - * \return The normalized luma + * \return The relative luminance * * Luma is the weighted sum of gamma-compressed R′G′B′ components of a color * video. The luma values are normalized as 0.0 to 1.0, with 1.0 being a @@ -263,12 +263,12 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) * luma here. * * More detailed information can be found in: - * https://en.wikipedia.org/wiki/Luma_(video) + * https://en.wikipedia.org/wiki/Relative_luminance */ -double Agc::computeInitialY(IPAFrameContext &frameContext, - const ipu3_uapi_grid_config &grid, - const ipu3_uapi_stats_3a *stats, - double currentYGain) +double Agc::estimateLuminance(IPAFrameContext &frameContext, + const ipu3_uapi_grid_config &grid, + const ipu3_uapi_stats_3a *stats, + double currentYGain) { double redSum = 0, greenSum = 0, blueSum = 0; @@ -288,14 +288,14 @@ double Agc::computeInitialY(IPAFrameContext &frameContext, } /* - * Estimate the sum of the brightness values, weighted with the gains - * applied on the channels in AWB as the Rec. 601 luma. + * Apply the AWB gains to approximate colours correctly, use the Rec. + * 601 formula to calculate the relative luminance, and normalize it. */ - double Y_sum = redSum * frameContext.awb.gains.red * .299 + - greenSum * frameContext.awb.gains.green * .587 + - blueSum * frameContext.awb.gains.blue * .114; + double ySum = redSum * frameContext.awb.gains.red * 0.299 + + greenSum * frameContext.awb.gains.green * 0.587 + + blueSum * frameContext.awb.gains.blue * 0.114; - return Y_sum / (grid.height * grid.width) / 255; + return ySum / (grid.height * grid.width) / 255; } /** @@ -311,22 +311,22 @@ void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) measureBrightness(stats, context.configuration.grid.bdsGrid); double currentYGain = 1.0; - double targetY = kNormalizedLumaTarget; + double yTarget = kRelativeLuminanceTarget; /* * Do this calculation a few times as brightness increase can be * non-linear when there are saturated regions. */ - for (int i = 0; i < 8; i++) { - double initialY = computeInitialY(context.frameContext, + for (unsigned int i = 0; i < 8; i++) { + double yValue = estimateLuminance(context.frameContext, context.configuration.grid.bdsGrid, stats, currentYGain); - double extra_gain = std::min(10.0, targetY / (initialY + .001)); + double extra_gain = std::min(10.0, yTarget / (yValue + .001)); currentYGain *= extra_gain; - LOG(IPU3Agc, Debug) << "Initial Y " << initialY - << " target " << targetY - << " gives gain " << currentYGain; + LOG(IPU3Agc, Debug) << "Y value: " << yValue + << ", Y target: " << yTarget + << ", gives gain " << currentYGain; if (extra_gain < 1.01) break; } diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 31c5a6e519d4..943c354a820e 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -35,10 +35,10 @@ private: const ipu3_uapi_grid_config &grid); void filterExposure(); void computeExposure(IPAFrameContext &frameContext, double currentYGain); - double computeInitialY(IPAFrameContext &frameContext, - const ipu3_uapi_grid_config &grid, - const ipu3_uapi_stats_3a *stats, - double currentYGain); + double estimateLuminance(IPAFrameContext &frameContext, + const ipu3_uapi_grid_config &grid, + const ipu3_uapi_stats_3a *stats, + double currentYGain); uint64_t frameCount_; From patchwork Fri Nov 19 21:02:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 14665 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0A73BBF415 for ; Fri, 19 Nov 2021 21:03:13 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B736460371; Fri, 19 Nov 2021 22:03:12 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="sYr+07cQ"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B3BFB6037A for ; Fri, 19 Nov 2021 22:03:06 +0100 (CET) Received: from pendragon.lan (62-78-145-57.bb.dnainternet.fi [62.78.145.57]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 3C7241C72 for ; Fri, 19 Nov 2021 22:03:06 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637355786; bh=8vyiuE7CMNjXJ1c/+aPG49uEPPJQ7rb6G2kJWawD4Hg=; h=From:To:Subject:Date:In-Reply-To:References:From; b=sYr+07cQzGzizawq72VGAQ8ZCyaU3jWs+Undwnnuq2uowEWgS7OkhXMveV6Qb4aZq 5Te5YTBnzDOu9hUkkhddjVwpXtlJpbXBM0HTnl+SXpNv3C3fJRVtIFDQRmvmJETX4C Kx/X1H4R6pBOPFHiSIE4+ZNqPL8t5P6EZgZRSrQs= From: Laurent Pinchart To: libcamera-devel@lists.libcamera.org Date: Fri, 19 Nov 2021 23:02:37 +0200 Message-Id: <20211119210239.18540-4-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211119210239.18540-1-laurent.pinchart@ideasonboard.com> References: <20211119210239.18540-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 3/5] ipa: ipu3: agc: Rename currentYGain X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The "current" prefix in the currentYGain variable name is confusing: - In Agc::estimateLuminance(), the variable contains the gain to be applied to the image, which is neither a "current" gain nor a "Y" gain. Rename it to "gain". - In Agc::computeExposure(), the variable contains the gain computed by the relative luminance method, so rename it to "yGain". While at it, rename variables to match the libcamera coding style. Signed-off-by: Laurent Pinchart Reviewed-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/agc.cpp | 32 ++++++++++++++++---------------- src/ipa/ipu3/algorithms/agc.h | 4 ++-- 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 9cd2ded501ed..2d196fd63c7e 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -173,9 +173,9 @@ void Agc::filterExposure() /** * \brief Estimate the new exposure and gain values * \param[inout] frameContext The shared IPA frame Context - * \param[in] currentYGain The gain calculated on the current brightness level + * \param[in] yGain The gain calculated based on the relative luminance target */ -void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) +void Agc::computeExposure(IPAFrameContext &frameContext, double yGain) { /* Get the effective exposure and gain applied on the sensor. */ uint32_t exposure = frameContext.sensor.exposure; @@ -189,8 +189,8 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) */ double evGain = kEvGainTarget * knumHistogramBins / iqMean_; - if (evGain < currentYGain) - evGain = currentYGain; + if (evGain < yGain) + evGain = yGain; /* Consider within 1% of the target as correctly exposed */ if (std::abs(evGain - 1.0) < 0.01) @@ -254,7 +254,7 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) * \param[in] frameContext The shared IPA frame context * \param[in] grid The grid used to store the statistics in the IPU3 * \param[in] stats The IPU3 statistics and ISP results - * \param[in] currentYGain The gain calculated on the current brightness level + * \param[in] gain The gain to apply to the frame * \return The relative luminance * * Luma is the weighted sum of gamma-compressed R′G′B′ components of a color @@ -268,7 +268,7 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double currentYGain) double Agc::estimateLuminance(IPAFrameContext &frameContext, const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, - double currentYGain) + double gain) { double redSum = 0, greenSum = 0, blueSum = 0; @@ -281,9 +281,9 @@ double Agc::estimateLuminance(IPAFrameContext &frameContext, &stats->awb_raw_buffer.meta_data[cellPosition] ); - redSum += cell->R_avg * currentYGain; - greenSum += (cell->Gr_avg + cell->Gb_avg) / 2 * currentYGain; - blueSum += cell->B_avg * currentYGain; + redSum += cell->R_avg * gain; + greenSum += (cell->Gr_avg + cell->Gb_avg) / 2 * gain; + blueSum += cell->B_avg * gain; } } @@ -310,7 +310,7 @@ void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) { measureBrightness(stats, context.configuration.grid.bdsGrid); - double currentYGain = 1.0; + double yGain = 1.0; double yTarget = kRelativeLuminanceTarget; /* @@ -320,18 +320,18 @@ void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) for (unsigned int i = 0; i < 8; i++) { double yValue = estimateLuminance(context.frameContext, context.configuration.grid.bdsGrid, - stats, currentYGain); - double extra_gain = std::min(10.0, yTarget / (yValue + .001)); + stats, yGain); + double extraGain = std::min(10.0, yTarget / (yValue + .001)); - currentYGain *= extra_gain; + yGain *= extraGain; LOG(IPU3Agc, Debug) << "Y value: " << yValue << ", Y target: " << yTarget - << ", gives gain " << currentYGain; - if (extra_gain < 1.01) + << ", gives gain " << yGain; + if (extraGain < 1.01) break; } - computeExposure(context.frameContext, currentYGain); + computeExposure(context.frameContext, yGain); frameCount_++; } diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 943c354a820e..0c868d6737f1 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -34,11 +34,11 @@ private: void measureBrightness(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid); void filterExposure(); - void computeExposure(IPAFrameContext &frameContext, double currentYGain); + void computeExposure(IPAFrameContext &frameContext, double yGain); double estimateLuminance(IPAFrameContext &frameContext, const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, - double currentYGain); + double gain); uint64_t frameCount_; From patchwork Fri Nov 19 21:02:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 14666 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 985EFC324F for ; Fri, 19 Nov 2021 21:03:13 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 363D960233; Fri, 19 Nov 2021 22:03:13 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="pCj5iZ1E"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3875160233 for ; Fri, 19 Nov 2021 22:03:07 +0100 (CET) Received: from pendragon.lan (62-78-145-57.bb.dnainternet.fi [62.78.145.57]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id CD17F1C19 for ; Fri, 19 Nov 2021 22:03:06 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637355787; bh=J/w+5W1Ntpsm6t1s3Dz4YCs6l0nmODQheND9T1MKP2I=; h=From:To:Subject:Date:In-Reply-To:References:From; b=pCj5iZ1EjtzTcGZhD7TOx8txbjp+Ojm2t/6IV/eAcgEeQQkZuu67gx+IsEE6KiuEh SH2wwQkLDbq26pXpInC+LxLx48pvjGnpFoN2vRNHRZO943LfVnxkY7f1uLaAFZf7qh Uh8EQSRvlHMSTncNos0Pg32DzUeB07JbKWcDypMI= From: Laurent Pinchart To: libcamera-devel@lists.libcamera.org Date: Fri, 19 Nov 2021 23:02:38 +0200 Message-Id: <20211119210239.18540-5-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211119210239.18540-1-laurent.pinchart@ideasonboard.com> References: <20211119210239.18540-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 4/5] ipa: ipu3: agc: Return the inter-quantile mean from measureBrightness() X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The inter-quantile mean is a value that is computed as part of the AGC run. It doesn't need to be stored in a member variable. Return it from measureBrightness(), which makes the flow of data easier to follow. Signed-off-by: Laurent Pinchart Reviewed-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- src/ipa/ipu3/algorithms/agc.cpp | 53 ++++++++++++++++++--------------- src/ipa/ipu3/algorithms/agc.h | 9 +++--- 2 files changed, 33 insertions(+), 29 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 2d196fd63c7e..c39e213f1dac 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -70,7 +70,7 @@ static constexpr uint32_t kNumStartupFrames = 10; static constexpr double kRelativeLuminanceTarget = 0.16; Agc::Agc() - : frameCount_(0), iqMean_(0.0), lineDuration_(0s), minShutterSpeed_(0s), + : frameCount_(0), lineDuration_(0s), minShutterSpeed_(0s), maxShutterSpeed_(0s), filteredExposure_(0s), currentExposure_(0s) { } @@ -108,9 +108,10 @@ int Agc::configure(IPAContext &context, const IPAConfigInfo &configInfo) * \brief Estimate the mean value of the top 2% of the histogram * \param[in] stats The statistics computed by the ImgU * \param[in] grid The grid used to store the statistics in the IPU3 + * \return The mean value of the top 2% of the histogram */ -void Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, - const ipu3_uapi_grid_config &grid) +double Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, + const ipu3_uapi_grid_config &grid) const { /* Initialise the histogram array */ uint32_t hist[knumHistogramBins] = { 0 }; @@ -135,8 +136,8 @@ void Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, } } - /* Estimate the quantile mean of the top 2% of the histogram */ - iqMean_ = Histogram(Span(hist)).interQuantileMean(0.98, 1.0); + /* Estimate the quantile mean of the top 2% of the histogram. */ + return Histogram(Span(hist)).interQuantileMean(0.98, 1.0); } /** @@ -174,28 +175,22 @@ void Agc::filterExposure() * \brief Estimate the new exposure and gain values * \param[inout] frameContext The shared IPA frame Context * \param[in] yGain The gain calculated based on the relative luminance target + * \param[in] iqMeanGain The gain calculated based on the relative luminance target */ -void Agc::computeExposure(IPAFrameContext &frameContext, double yGain) +void Agc::computeExposure(IPAFrameContext &frameContext, double yGain, + double iqMeanGain) { /* Get the effective exposure and gain applied on the sensor. */ uint32_t exposure = frameContext.sensor.exposure; double analogueGain = frameContext.sensor.gain; - /* - * Estimate the gain needed to have the proportion of pixels in a given - * desired range. iqMean_ returns the mean value of the top 2% of the - * cumulative histogram, and we want it to be as close as possible to a - * configured target. - */ - double evGain = kEvGainTarget * knumHistogramBins / iqMean_; - - if (evGain < yGain) - evGain = yGain; + /* Use the highest of the two gain estimates. */ + double evGain = std::max(yGain, iqMeanGain); /* Consider within 1% of the target as correctly exposed */ if (std::abs(evGain - 1.0) < 0.01) - LOG(IPU3Agc, Debug) << "We are well exposed (iqMean = " - << iqMean_ << ")"; + LOG(IPU3Agc, Debug) << "We are well exposed (evGain = " + << evGain << ")"; /* extracted from Rpi::Agc::computeTargetExposure */ @@ -308,15 +303,25 @@ double Agc::estimateLuminance(IPAFrameContext &frameContext, */ void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) { - measureBrightness(stats, context.configuration.grid.bdsGrid); + /* + * Estimate the gain needed to have the proportion of pixels in a given + * desired range. iqMean is the mean value of the top 2% of the + * cumulative histogram, and we want it to be as close as possible to a + * configured target. + */ + double iqMean = measureBrightness(stats, context.configuration.grid.bdsGrid); + double iqMeanGain = kEvGainTarget * knumHistogramBins / iqMean; + /* + * Estimate the gain needed to achieve a relative luminance target. To + * account for non-linearity caused by saturation, the value needs to be + * estimated in an iterative process, as multiplying by a gain will not + * increase the relative luminance by the same factor if some image + * regions are saturated. + */ double yGain = 1.0; double yTarget = kRelativeLuminanceTarget; - /* - * Do this calculation a few times as brightness increase can be - * non-linear when there are saturated regions. - */ for (unsigned int i = 0; i < 8; i++) { double yValue = estimateLuminance(context.frameContext, context.configuration.grid.bdsGrid, @@ -331,7 +336,7 @@ void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) break; } - computeExposure(context.frameContext, yGain); + computeExposure(context.frameContext, yGain, iqMeanGain); frameCount_++; } diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 0c868d6737f1..a04a81fb8eae 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -31,10 +31,11 @@ public: void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override; private: - void measureBrightness(const ipu3_uapi_stats_3a *stats, - const ipu3_uapi_grid_config &grid); + double measureBrightness(const ipu3_uapi_stats_3a *stats, + const ipu3_uapi_grid_config &grid) const; void filterExposure(); - void computeExposure(IPAFrameContext &frameContext, double yGain); + void computeExposure(IPAFrameContext &frameContext, double yGain, + double iqMeanGain); double estimateLuminance(IPAFrameContext &frameContext, const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, @@ -42,8 +43,6 @@ private: uint64_t frameCount_; - double iqMean_; - utils::Duration lineDuration_; utils::Duration minShutterSpeed_; utils::Duration maxShutterSpeed_; From patchwork Fri Nov 19 21:02:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 14667 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 540C9C3250 for ; Fri, 19 Nov 2021 21:03:14 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DC36F60398; Fri, 19 Nov 2021 22:03:13 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="NPGhxA4y"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9558A600B5 for ; Fri, 19 Nov 2021 22:03:07 +0100 (CET) Received: from pendragon.lan (62-78-145-57.bb.dnainternet.fi [62.78.145.57]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 3B99B1C72 for ; Fri, 19 Nov 2021 22:03:07 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637355787; bh=JPKLiwTHxFE2lUqAo+jBhpHlDvWBvy+KofV9Z+zhqQ0=; h=From:To:Subject:Date:In-Reply-To:References:From; b=NPGhxA4yj5qcf8KqQvsyhfl4l/wQJuguFQjv33WpOkWSTe/D8/qINy1VBNuBe83UT GzE4WTq5eBAonJ2fqxzqVn9u1mA0rQrq27r76BgObYJXM9WOTYipckX86dS+QElPgC 4oKjkvftKwx6OvXRHs1CSZlorP58yJW1fztKITLg= From: Laurent Pinchart To: libcamera-devel@lists.libcamera.org Date: Fri, 19 Nov 2021 23:02:39 +0200 Message-Id: <20211119210239.18540-6-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211119210239.18540-1-laurent.pinchart@ideasonboard.com> References: <20211119210239.18540-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 5/5] ipa: ipu3: agc: Saturate the averages when computing relative luminance X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The relative luminance is calculated using an iterative process to account for saturation in the sensor, as multiplying pixels by a gain doesn't increase the relative luminance by the same factor if some regions are saturated. Relative luminance estimation doesn't apply a saturation, which produces a value that doesn't match what the sensor will output, and defeats the point of the iterative process. Fix it. Fixes: f8f07f9468c6 ("ipa: ipu3: agc: Improve gain calculation") Signed-off-by: Laurent Pinchart Tested-by: Jean-Michel Hautbois Reviewed-by: Jean-Michel Hautbois Tested-by: Kieran Bingham Reviewed-by: Kieran Bingham --- Changes since v1: - Replace incorrect mention of analogue gain with gain - Fix typos --- src/ipa/ipu3/algorithms/agc.cpp | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index c39e213f1dac..582f0ae1b54b 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -252,10 +252,19 @@ void Agc::computeExposure(IPAFrameContext &frameContext, double yGain, * \param[in] gain The gain to apply to the frame * \return The relative luminance * - * Luma is the weighted sum of gamma-compressed R′G′B′ components of a color - * video. The luma values are normalized as 0.0 to 1.0, with 1.0 being a - * theoretical perfect reflector of 100% reference white. We use the Rec. 601 - * luma here. + * This function estimates the average relative luminance of the frame that + * would be output by the sensor if an additional \a gain was applied. + * + * The estimation is based on the AWB statistics for the current frame. Red, + * green and blue averages for all cells are first multiplied by the gain, and + * then saturated to approximate the sensor behaviour at high brightness + * values. The approximation is quite rough, as it doesn't take into account + * non-linearities when approaching saturation. + * + * The relative luminance (Y) is computed from the linear RGB components using + * the Rec. 601 formula. The values are normalized to the [0.0, 1.0] range, + * where 1.0 corresponds to a theoretical perfect reflector of 100% reference + * white. * * More detailed information can be found in: * https://en.wikipedia.org/wiki/Relative_luminance @@ -267,6 +276,7 @@ double Agc::estimateLuminance(IPAFrameContext &frameContext, { double redSum = 0, greenSum = 0, blueSum = 0; + /* Sum the per-channel averages, saturated to 255. */ for (unsigned int cellY = 0; cellY < grid.height; cellY++) { for (unsigned int cellX = 0; cellX < grid.width; cellX++) { uint32_t cellPosition = cellY * stride_ + cellX; @@ -275,10 +285,11 @@ double Agc::estimateLuminance(IPAFrameContext &frameContext, reinterpret_cast( &stats->awb_raw_buffer.meta_data[cellPosition] ); + const uint8_t G_avg = (cell->Gr_avg + cell->Gb_avg) / 2; - redSum += cell->R_avg * gain; - greenSum += (cell->Gr_avg + cell->Gb_avg) / 2 * gain; - blueSum += cell->B_avg * gain; + redSum += std::min(cell->R_avg * gain, 255.0); + greenSum += std::min(G_avg * gain, 255.0); + blueSum += std::min(cell->B_avg * gain, 255.0); } }