From patchwork Fri Sep 19 09:40:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24410 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id DD462C328C for ; Fri, 19 Sep 2025 09:41:00 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 499036B599; Fri, 19 Sep 2025 11:41:00 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="QAutSTn7"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1CDA669369 for ; Fri, 19 Sep 2025 11:40:57 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 85AD27E4; Fri, 19 Sep 2025 11:39:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274776; bh=oQcdMvtUb+EZcHrp/lYfdkzvy04kyfc66qUfF29BOKU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QAutSTn7ZAAssbzeditdOANRGG7o5Ecnf2ySofywU1PbFUOdJh67yZBNTgzoSXmYN FHk4qUbp8ZwKxnqz+OIx071swrGMZ8gBAd6g1ltzug+2ycSJWglAATk4S83bWc3dLm 3qlAZA5NLGXL0c+v+QG7SGqXfMFg5EUrkrspUGWM= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Kieran Bingham Subject: [PATCH v5 01/19] ipa: rkisp1: Add basic compression algorithm Date: Fri, 19 Sep 2025 11:40:16 +0200 Message-ID: <20250919094041.183031-2-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The i.MX8 M Plus has a compression curve inside the compand block. This curve is necessary to process HDR stitched data and is useful for other aspects like applying a digital gain to the incoming sensor data. Add a basic algorithm for the compression curve. This algorithm has a hardcoded input width of 20bit and output width of 12bit which matches the imx8mp pipeline. Only a static gain is supported in this version. Signed-off-by: Stefan Klug Reviewed-by: Kieran Bingham Reviewed-by: Paul Elder --- Changes in v5: - Removed unnecessary include - Added check for compress.supported in prepare() - Collected tag Changes in v4: - Moved activeState.compress.supported to configuration.compress.supported - Added documentation - Fixed typo in documentation - Replaced constant with RKISP1_CIF_ISP_COMPAND_NUM_POINTS Changes in v3: - Removed unused member - Fixed comment referencing copy-paste source - Ensure activeState.compress.supported stays false if unsupported --- src/ipa/rkisp1/algorithms/compress.cpp | 103 +++++++++++++++++++++++++ src/ipa/rkisp1/algorithms/compress.h | 30 +++++++ src/ipa/rkisp1/algorithms/meson.build | 1 + src/ipa/rkisp1/ipa_context.cpp | 19 +++++ src/ipa/rkisp1/ipa_context.h | 9 +++ 5 files changed, 162 insertions(+) create mode 100644 src/ipa/rkisp1/algorithms/compress.cpp create mode 100644 src/ipa/rkisp1/algorithms/compress.h diff --git a/src/ipa/rkisp1/algorithms/compress.cpp b/src/ipa/rkisp1/algorithms/compress.cpp new file mode 100644 index 000000000000..6445cd14bae3 --- /dev/null +++ b/src/ipa/rkisp1/algorithms/compress.cpp @@ -0,0 +1,103 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2025, Ideas On Board + * + * RkISP1 Compression curve + */ +#include "compress.h" + +#include + +#include +#include + +#include "linux/rkisp1-config.h" + +/** + * \file compress.h + */ + +namespace libcamera { + +namespace ipa::rkisp1::algorithms { + +/** + * \class Compress + * \brief RkISP1 Compress curve + * + * This algorithm implements support for the compression curve in the compand + * block available in the i.MX8 M Plus + * + * In its current version it only supports a static gain. This is useful for + * the agc algorithm to compensate for exposure/gain quantization effects. + * + * This algorithm doesn't have any configuration options. It needs to be + * configured per frame by other algorithms. + * + * Other algorithms can check configuration.compress.supported to see if + * compression is available. If it is available they can configure it per frame + * using frameContext.compress.enable and frameContext.compress.gain. + */ + +LOG_DEFINE_CATEGORY(RkISP1Compress) + +constexpr static int kRkISP1CompressInBits = 20; +constexpr static int kRkISP1CompressOutBits = 12; + +/** + * \copydoc libcamera::ipa::Algorithm::configure + */ +int Compress::configure(IPAContext &context, + [[maybe_unused]] const IPACameraSensorInfo &configInfo) +{ + if (context.configuration.paramFormat != V4L2_META_FMT_RK_ISP1_EXT_PARAMS || + !context.hw->compand) { + LOG(RkISP1Compress, Warning) + << "Compression is not supported by the hardware or kernel."; + return 0; + } + + context.configuration.compress.supported = true; + return 0; +} + +/** + * \copydoc libcamera::ipa::Algorithm::prepare + */ +void Compress::prepare([[maybe_unused]] IPAContext &context, + [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, + RkISP1Params *params) +{ + if (!context.configuration.compress.supported) + return; + + auto comp = params->block(); + comp.setEnabled(frameContext.compress.enable); + + if (!frameContext.compress.enable) + return; + + int xmax = (1 << kRkISP1CompressInBits); + int ymax = (1 << kRkISP1CompressOutBits); + int inLogStep = std::log2(xmax / RKISP1_CIF_ISP_COMPAND_NUM_POINTS); + + for (unsigned int i = 0; i < RKISP1_CIF_ISP_COMPAND_NUM_POINTS; i++) { + double x = (i + 1) * (1.0 / RKISP1_CIF_ISP_COMPAND_NUM_POINTS); + double y = x * frameContext.compress.gain; + + comp->px[i] = inLogStep; + comp->x[i] = std::min(x * xmax, xmax - 1); + comp->y[i] = std::min(y * ymax, ymax - 1); + } + + LOG(RkISP1Compress, Debug) << "Compression: " << kRkISP1CompressInBits + << " bits to " << kRkISP1CompressOutBits + << " bits gain: " << frameContext.compress.gain; +} + +REGISTER_IPA_ALGORITHM(Compress, "Compress") + +} /* namespace ipa::rkisp1::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/rkisp1/algorithms/compress.h b/src/ipa/rkisp1/algorithms/compress.h new file mode 100644 index 000000000000..87797b8ebcc5 --- /dev/null +++ b/src/ipa/rkisp1/algorithms/compress.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2025, Ideas On Board + * + * RkISP1 Compression curve + */ + +#pragma once + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::rkisp1::algorithms { + +class Compress : public Algorithm +{ +public: + Compress() = default; + ~Compress() = default; + + int configure(IPAContext &context, + const IPACameraSensorInfo &configInfo) override; + void prepare(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + RkISP1Params *params) override; +}; + +} /* namespace ipa::rkisp1::algorithms */ +} /* namespace libcamera */ diff --git a/src/ipa/rkisp1/algorithms/meson.build b/src/ipa/rkisp1/algorithms/meson.build index c66b0b70b82f..2e42a80cf99d 100644 --- a/src/ipa/rkisp1/algorithms/meson.build +++ b/src/ipa/rkisp1/algorithms/meson.build @@ -5,6 +5,7 @@ rkisp1_ipa_algorithms = files([ 'awb.cpp', 'blc.cpp', 'ccm.cpp', + 'compress.cpp', 'cproc.cpp', 'dpcc.cpp', 'dpf.cpp', diff --git a/src/ipa/rkisp1/ipa_context.cpp b/src/ipa/rkisp1/ipa_context.cpp index 6509610573c0..15cb0afe9fe8 100644 --- a/src/ipa/rkisp1/ipa_context.cpp +++ b/src/ipa/rkisp1/ipa_context.cpp @@ -66,6 +66,14 @@ namespace libcamera::ipa::rkisp1 { * operates in manual or automatic mode. */ +/** + * \var IPASessionConfiguration::compress + * \brief Compress parameters configuration of the IPA + * + * \var IPASessionConfiguration::agc.supported + * \brief true if compression is supported and the algorithm is loaded + */ + /** * \var IPASessionConfiguration::lsc * \brief Lens Shading Correction configuration of the IPA @@ -377,6 +385,17 @@ namespace libcamera::ipa::rkisp1 { * \brief Colour Correction Matrix */ +/** + * \var IPAFrameContext::compress + * \brief Compress parameters for this frame + * + * \struct IPAFrameContext::compress.enable + * \brief True if compression is enabled + * + * \var IPAFrameContext::compress.gain + * \brief The gain applied with the compression curve + */ + /** * \var IPAFrameContext::cproc * \brief Color Processing parameters for this frame diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index 7ccc7b501aff..a723c79b04d9 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -49,6 +49,10 @@ struct IPASessionConfiguration { bool enabled; } awb; + struct { + bool supported; + } compress; + struct { bool enabled; } lsc; @@ -158,6 +162,11 @@ struct IPAFrameContext : public FrameContext { bool update; } cproc; + struct { + bool enable; + double gain; + } compress; + struct { bool denoise; bool update; From patchwork Fri Sep 19 09:40:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24411 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9DBAEC3329 for ; Fri, 19 Sep 2025 09:41:03 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 7199B6B5A2; Fri, 19 Sep 2025 11:41:02 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="G9CO5gk9"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 820B469369 for ; Fri, 19 Sep 2025 11:40:59 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 55E64819; Fri, 19 Sep 2025 11:39:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274779; bh=nRScZh49NzUnocZocPpeDWVHZ+XH+kB4+onpDi6PLLA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=G9CO5gk9kL2cfkwjXISIW1osRUnUgQmhxSmsmkMhHsRJCt29v5ohKVZTSzzB+g+Jo 6iTIfzni5GahgQgJHjqQyeCzgQ/B9koy48rYYA1zg/5RgGJ/N/sup+VHeXdgd524rE WgB8n2EK9DSrxi8FCwZI08BEj6rPaGT/khNeFzMg= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Isaac Scott , Daniel Scally , Paul Elder Subject: [PATCH v5 02/19] tuning: rksip1: Add a static Compress entry Date: Fri, 19 Sep 2025 11:40:17 +0200 Message-ID: <20250919094041.183031-3-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a static Compress entry that gets added by default. Signed-off-by: Stefan Klug Reviewed-by: Isaac Scott Reviewed-by: Daniel Scally Reviewed-by: Paul Elder --- Changes in v4: - Collected tag Changes in v3: - Collected tags --- utils/tuning/rkisp1.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/utils/tuning/rkisp1.py b/utils/tuning/rkisp1.py index 207b717a029c..179d920c05df 100755 --- a/utils/tuning/rkisp1.py +++ b/utils/tuning/rkisp1.py @@ -27,6 +27,7 @@ awb = AWBRkISP1(debug=[lt.Debug.Plot]) blc = StaticModule('BlackLevelCorrection') ccm = CCMRkISP1(debug=[lt.Debug.Plot]) color_processing = StaticModule('ColorProcessing') +compress = StaticModule('Compress') filter = StaticModule('Filter') gamma_out = StaticModule('GammaOutCorrection', {'gamma': 2.2}) lsc = LSCRkISP1(debug=[lt.Debug.Plot], @@ -49,13 +50,14 @@ lsc = LSCRkISP1(debug=[lt.Debug.Plot], lux = LuxRkISP1(debug=[lt.Debug.Plot]) tuner = lt.Tuner('RkISP1') -tuner.add([agc, awb, blc, ccm, color_processing, filter, gamma_out, lsc, lux]) +tuner.add([agc, awb, blc, ccm, color_processing, filter, gamma_out, lsc, lux, compress]) tuner.set_input_parser(YamlParser()) tuner.set_output_formatter(YamlOutput()) # Bayesian AWB uses the lux value, so insert the lux algorithm before AWB. +# Compress is parameterized by others, so add it at the end. tuner.set_output_order([agc, lux, awb, blc, ccm, color_processing, - filter, gamma_out, lsc]) + filter, gamma_out, lsc, compress]) if __name__ == '__main__': sys.exit(tuner.run(sys.argv)) From patchwork Fri Sep 19 09:40:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24412 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id BDF5BC328C for ; Fri, 19 Sep 2025 09:41:06 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 266D96B5A5; Fri, 19 Sep 2025 11:41:06 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="PzVbY2+N"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 706B86B5A0 for ; Fri, 19 Sep 2025 11:41:02 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 3065F7E4; Fri, 19 Sep 2025 11:39:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274782; bh=utyQW5ybO12a+eoYtHAe559BLHxWYaUugkp+e1Mar/Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PzVbY2+NCq9CL4pRELgXQesDd5bGh1Je9asem0bbakQ+PcKxZmpg17JnLw+nQMdg3 BCRhgiafR4DfC0Zp3ujWYIH5bVDiHGdNWRIUj+tPIaGFsoja1MaoXpTw5WXfCnkCIY l0BaoX85KR2iAUmGHHHno0dZDdM3Xgk0AovIklP0= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Isaac Scott , Daniel Scally Subject: [PATCH v5 03/19] libipa: camera_sensor_helper: Add quantizeGain() function Date: Fri, 19 Sep 2025 11:40:18 +0200 Message-ID: <20250919094041.183031-4-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a small utility function that calculates the quantized gain that gets applied by a sensor when the gain code is set to gainCode(gain). This is needed by algorithms to calculate a digital correction gain that gets applied to mitigate the error introduce by quantization. Signed-off-by: Stefan Klug Reviewed-by: Isaac Scott Reviewed-by: Daniel Scally Reviewed-by: Paul Elder --- Changes in v3: - Remove virtual from quantizeGain() - Improved documentation - Collected tags --- src/ipa/libipa/camera_sensor_helper.cpp | 23 +++++++++++++++++++++++ src/ipa/libipa/camera_sensor_helper.h | 1 + 2 files changed, 24 insertions(+) diff --git a/src/ipa/libipa/camera_sensor_helper.cpp b/src/ipa/libipa/camera_sensor_helper.cpp index dcd69d9f2bbb..01f6470a196f 100644 --- a/src/ipa/libipa/camera_sensor_helper.cpp +++ b/src/ipa/libipa/camera_sensor_helper.cpp @@ -131,6 +131,29 @@ double CameraSensorHelper::gain(uint32_t gainCode) const } } +/** + * \brief Quantize the given gain value + * \param[in] _gain The real gain + * \param[out] quantizationGain The gain that is lost due to quantization + * + * This function returns the actual gain that is applied when the sensor's gain + * is set to gainCode(_gain). + * + * It shall be guaranteed that gainCode(_gain) == gainCode(quantizeGain(_gain)). + * + * If \a quantizationGain is provided it is populated with the gain that must be + * applied on top to correct for the losses due to quantization. + * + * \return The quantized real gain + */ +double CameraSensorHelper::quantizeGain(double _gain, double *quantizationGain) const +{ + double g = gain(gainCode(_gain)); + if (quantizationGain) + *quantizationGain = _gain / g; + return g; +} + /** * \struct CameraSensorHelper::AnalogueGainLinear * \brief Analogue gain constants for the linear gain model diff --git a/src/ipa/libipa/camera_sensor_helper.h b/src/ipa/libipa/camera_sensor_helper.h index a9300a64f1e7..bd3d0beec77f 100644 --- a/src/ipa/libipa/camera_sensor_helper.h +++ b/src/ipa/libipa/camera_sensor_helper.h @@ -29,6 +29,7 @@ public: std::optional blackLevel() const { return blackLevel_; } virtual uint32_t gainCode(double gain) const; virtual double gain(uint32_t gainCode) const; + double quantizeGain(double gain, double *quantizationGain) const; protected: struct AnalogueGainLinear { From patchwork Fri Sep 19 09:40:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24413 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1449BC328C for ; Fri, 19 Sep 2025 09:41:10 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8E4F96B5AA; Fri, 19 Sep 2025 11:41:09 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="j6ScMhps"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 78C206B59C for ; Fri, 19 Sep 2025 11:41:05 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 51272819; Fri, 19 Sep 2025 11:39:45 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274785; bh=xDVZ9ggp9ojc3L+LlFL01S76YA2IRaSQUNBafWCExyc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=j6ScMhpsEBOfiFRZGcl5kk7G86uNEoueABkeGF9Jk9SiwK7UcAxPy5YFOrYI9zAc/ 7kEGgV4BNxAczkKtDKBDNa+MbosKr8Y2665VzV89QJ118+vqNQN++dnxWTQXinEkzC oob3aOj7uXPRNykZeqxQoPWpDyNDJPRN+LQkFeyU= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Kieran Bingham , Daniel Scally Subject: [PATCH v5 04/19] libipa: exposure_mode_helper: Take exposure/gain quantization into account Date: Fri, 19 Sep 2025 11:40:19 +0200 Message-ID: <20250919094041.183031-5-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In ExposureModeHelper::splitExposure() the quantization of exposure time and gain is not taken into account. This can lead to visible flicker when the quantization steps are too big. As a preparation to fixing that, add a function to set the sensor line length and the current sensor mode helper and extend the clampXXX functions to return the quantization error. By default the exposure time quantization is assumed to be 1us and gain is assumed to not be quantized at all. Signed-off-by: Stefan Klug Reviewed-by: Kieran Bingham Reviewed-by: Daniel Scally Reviewed-by: Paul Elder --- Changes in v4: - Improved documentation Changes in v3: - Collected tags - Renamed lineLength to lineDuration - Remove the "no functional changes" sentence, as the returned exposureTime/gain are now quantized, which happened outside of this class before. So that is actually a functional change. --- src/ipa/libipa/exposure_mode_helper.cpp | 49 ++++++++++++++++++++----- src/ipa/libipa/exposure_mode_helper.h | 10 ++++- 2 files changed, 48 insertions(+), 11 deletions(-) diff --git a/src/ipa/libipa/exposure_mode_helper.cpp b/src/ipa/libipa/exposure_mode_helper.cpp index 0c1e99e31a47..b776a031b441 100644 --- a/src/ipa/libipa/exposure_mode_helper.cpp +++ b/src/ipa/libipa/exposure_mode_helper.cpp @@ -70,18 +70,37 @@ namespace ipa { * the runtime limits set through setLimits() instead. */ ExposureModeHelper::ExposureModeHelper(const Span> stages) + : lineDuration_(1us), minExposureTime_(0us), maxExposureTime_(0us), + minGain_(0), maxGain_(0), sensorHelper_(nullptr) { - minExposureTime_ = 0us; - maxExposureTime_ = 0us; - minGain_ = 0; - maxGain_ = 0; - for (const auto &[s, g] : stages) { exposureTimes_.push_back(s); gains_.push_back(g); } } +/** + * \brief Configure sensor details + * \param[in] lineDuration The current line length of the sensor + * \param[in] sensorHelper The sensor helper + * + * This function sets the line length and sensor helper. These are used in + * splitExposure() to take the quantization of the exposure and gain into + * account. + * + * When this has not been called, it is assumed that exposure is in micro second + * granularity and gain has no quantization at all. + * + * ExposureModeHelper keeps a pointer to the CameraSensorHelper, so the caller + * has to ensure that sensorHelper is valid until the next call to configure(). + */ +void ExposureModeHelper::configure(utils::Duration lineDuration, + const CameraSensorHelper *sensorHelper) +{ + lineDuration_ = lineDuration; + sensorHelper_ = sensorHelper; +} + /** * \brief Set the exposure time and gain limits * \param[in] minExposureTime The minimum exposure time supported @@ -108,14 +127,26 @@ void ExposureModeHelper::setLimits(utils::Duration minExposureTime, maxGain_ = maxGain; } -utils::Duration ExposureModeHelper::clampExposureTime(utils::Duration exposureTime) const +utils::Duration ExposureModeHelper::clampExposureTime(utils::Duration exposureTime, + double *quantizationGain) const { - return std::clamp(exposureTime, minExposureTime_, maxExposureTime_); + utils::Duration clamped; + utils::Duration exp; + + clamped = std::clamp(exposureTime, minExposureTime_, maxExposureTime_); + exp = static_cast(clamped / lineDuration_) * lineDuration_; + if (quantizationGain) + *quantizationGain = clamped / exp; + + return exp; } -double ExposureModeHelper::clampGain(double gain) const +double ExposureModeHelper::clampGain(double gain, double *quantizationGain) const { - return std::clamp(gain, minGain_, maxGain_); + double clamped = std::clamp(gain, minGain_, maxGain_); + if (!sensorHelper_) + return clamped; + return sensorHelper_->quantizeGain(clamped, quantizationGain); } /** diff --git a/src/ipa/libipa/exposure_mode_helper.h b/src/ipa/libipa/exposure_mode_helper.h index c5be1b6703a8..ac7e8da95c6c 100644 --- a/src/ipa/libipa/exposure_mode_helper.h +++ b/src/ipa/libipa/exposure_mode_helper.h @@ -14,6 +14,8 @@ #include #include +#include "camera_sensor_helper.h" + namespace libcamera { namespace ipa { @@ -24,6 +26,7 @@ public: ExposureModeHelper(const Span> stages); ~ExposureModeHelper() = default; + void configure(utils::Duration lineLength, const CameraSensorHelper *sensorHelper); void setLimits(utils::Duration minExposureTime, utils::Duration maxExposureTime, double minGain, double maxGain); @@ -36,16 +39,19 @@ public: double maxGain() const { return maxGain_; } private: - utils::Duration clampExposureTime(utils::Duration exposureTime) const; - double clampGain(double gain) const; + utils::Duration clampExposureTime(utils::Duration exposureTime, + double *quantizationGain = nullptr) const; + double clampGain(double gain, double *quantizationGain = nullptr) const; std::vector exposureTimes_; std::vector gains_; + utils::Duration lineDuration_; utils::Duration minExposureTime_; utils::Duration maxExposureTime_; double minGain_; double maxGain_; + const CameraSensorHelper *sensorHelper_; }; } /* namespace ipa */ From patchwork Fri Sep 19 09:40:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24414 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 47411C3329 for ; Fri, 19 Sep 2025 09:41:12 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BBDF16B5AC; Fri, 19 Sep 2025 11:41:11 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="jwyfyozz"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 742CA6B5A0 for ; Fri, 19 Sep 2025 11:41:08 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 45880819; Fri, 19 Sep 2025 11:39:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274788; bh=GMp4tviTO5X+GDxI2B6CSa8Wv63WvQ0SUMM2BTCrffE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jwyfyozzhMMtO7O+9GitBEHz5nwi6FaLct8GdbU9URXn4HYTvI/dGBhmUlXmjys+n YiRuUS4f0I5vUa4VYpUDQEBk8psRW/Qy/e2Czd3M+8bVaNT7DKMimTUg3WvDXXYr9U Ulqt3rU2+rXDXgqOwWlaIBzf7m+SSWDNJcYTHHlM= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Kieran Bingham , Daniel Scally , Paul Elder Subject: [PATCH v5 05/19] libipa: exposure_mode_helper: Remove double calculation of lastStageGain Date: Fri, 19 Sep 2025 11:40:20 +0200 Message-ID: <20250919094041.183031-6-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" lastStageGain gets recalculated unconditionally even though it is the stageGain of the last stage. Refactor for increased simplicity and efficiency. Signed-off-by: Stefan Klug Reviewed-by: Kieran Bingham Reviewed-by: Daniel Scally Reviewed-by: Paul Elder --- Changes in v4: - Collected tag Changes in v3: - Collected tags --- src/ipa/libipa/exposure_mode_helper.cpp | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/ipa/libipa/exposure_mode_helper.cpp b/src/ipa/libipa/exposure_mode_helper.cpp index b776a031b441..e806731047cf 100644 --- a/src/ipa/libipa/exposure_mode_helper.cpp +++ b/src/ipa/libipa/exposure_mode_helper.cpp @@ -198,10 +198,10 @@ ExposureModeHelper::splitExposure(utils::Duration exposure) const utils::Duration exposureTime; double stageGain = 1.0; + double lastStageGain = 1.0; double gain; for (unsigned int stage = 0; stage < gains_.size(); stage++) { - double lastStageGain = stage == 0 ? 1.0 : clampGain(gains_[stage - 1]); utils::Duration stageExposureTime = clampExposureTime(exposureTimes_[stage]); stageGain = clampGain(gains_[stage]); @@ -228,6 +228,8 @@ ExposureModeHelper::splitExposure(utils::Duration exposure) const return { exposureTime, gain, exposure / (exposureTime * gain) }; } + + lastStageGain = stageGain; } /* From patchwork Fri Sep 19 09:40:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24415 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id BBBD6C328C for ; Fri, 19 Sep 2025 09:41:15 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3B7AA6B5AA; Fri, 19 Sep 2025 11:41:15 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="hdma1xc5"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 729546B5A7 for ; Fri, 19 Sep 2025 11:41:11 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 10BD8842; Fri, 19 Sep 2025 11:39:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274791; bh=wuVocP0PxGUEMxd0Ahj37h1AcvbUWZUDWSbY5rD3MIA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hdma1xc575D0jBjuGHw712k7NuyYm/K6JsneWFuPyN+PDSNnWvTsovRcElaouQdI0 znynJm8prYhFb9s7tl2dpturAjNYvbevEGB4LUSUuFdMVc57zeXkNxhJDX9hB9fASN nhz3r1e6fH6dHl2feX9EgZirQtGR9p1e1/x28vhU= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Isaac Scott , Daniel Scally , Paul Elder Subject: [PATCH v5 06/19] libipa: exposure_mode_helper: Remove unnecessary clamp calls Date: Fri, 19 Sep 2025 11:40:21 +0200 Message-ID: <20250919094041.183031-7-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Except for the first iteration of the loop and in the case of an empty gains_ vector, the values were run through clamp two times which is unnecessary. Remove that by clamping the initial value. Signed-off-by: Stefan Klug Reviewed-by: Isaac Scott Reviewed-by: Daniel Scally Reviewed-by: Paul Elder --- Changes in v4: - Collected tag Changes in v3: - Collected tags --- src/ipa/libipa/exposure_mode_helper.cpp | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/ipa/libipa/exposure_mode_helper.cpp b/src/ipa/libipa/exposure_mode_helper.cpp index e806731047cf..f2de1d8e4229 100644 --- a/src/ipa/libipa/exposure_mode_helper.cpp +++ b/src/ipa/libipa/exposure_mode_helper.cpp @@ -197,8 +197,8 @@ ExposureModeHelper::splitExposure(utils::Duration exposure) const return { minExposureTime_, minGain_, exposure / (minExposureTime_ * minGain_) }; utils::Duration exposureTime; - double stageGain = 1.0; - double lastStageGain = 1.0; + double stageGain = clampGain(1.0); + double lastStageGain = stageGain; double gain; for (unsigned int stage = 0; stage < gains_.size(); stage++) { @@ -215,7 +215,7 @@ ExposureModeHelper::splitExposure(utils::Duration exposure) const /* Clamp the gain to lastStageGain and regulate exposureTime. */ if (stageExposureTime * lastStageGain >= exposure) { - exposureTime = clampExposureTime(exposure / clampGain(lastStageGain)); + exposureTime = clampExposureTime(exposure / lastStageGain); gain = clampGain(exposure / exposureTime); return { exposureTime, gain, exposure / (exposureTime * gain) }; @@ -223,7 +223,7 @@ ExposureModeHelper::splitExposure(utils::Duration exposure) const /* Clamp the exposureTime to stageExposureTime and regulate gain. */ if (stageExposureTime * stageGain >= exposure) { - exposureTime = clampExposureTime(stageExposureTime); + exposureTime = stageExposureTime; gain = clampGain(exposure / exposureTime); return { exposureTime, gain, exposure / (exposureTime * gain) }; @@ -239,7 +239,7 @@ ExposureModeHelper::splitExposure(utils::Duration exposure) const * stages to use then the default stageGain of 1.0 is used so that * exposure time is maxed before gain is touched at all. */ - exposureTime = clampExposureTime(exposure / clampGain(stageGain)); + exposureTime = clampExposureTime(exposure / stageGain); gain = clampGain(exposure / exposureTime); return { exposureTime, gain, exposure / (exposureTime * gain) }; From patchwork Fri Sep 19 09:40:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24416 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2081CC3329 for ; Fri, 19 Sep 2025 09:41:18 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9C18E6B5B2; Fri, 19 Sep 2025 11:41:17 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="d4zf0GPT"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0B38A6B5A5 for ; Fri, 19 Sep 2025 11:41:14 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id CBB7999F; Fri, 19 Sep 2025 11:39:53 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274793; bh=GT3SNiwf5b0a23Lu4SX6u1wUMPaGCvMLW+NqlVuAkrw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=d4zf0GPT874mgHhAo2jgA1PXoUxEEUDGkIRI81XGc3fP6LhRaZXN6YetvvFz+YfdA Hsl0Y0sty2iWrjz7aDUEnpBppNaWUc4R55VIKBtD5/RvokqqUkrfxBHuZR58VdhRsZ k6Is/wnQQkdsKIYJQ3WAxzzCW0ffCmpBD4nbchW0= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Daniel Scally , Kieran Bingham , Paul Elder Subject: [PATCH v5 07/19] libipa: agc_mean_luminance: Fix constraint logging Date: Fri, 19 Sep 2025 11:40:22 +0200 Message-ID: <20250919094041.183031-8-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The debug log statements in constraintClampGain() are after the assignment of gain. So they correctly log when the constraint applies, but the gain values logged are the same. Fix that. Fixes: 42e18c96bcb7 ("libipa: agc_mean_luminance: Add debug logging") Signed-off-by: Stefan Klug Reviewed-by: Daniel Scally Reviewed-by: Kieran Bingham Reviewed-by: Paul Elder --- Changes in v4: - Collected tags Changes in v3: - Added this patch --- src/ipa/libipa/agc_mean_luminance.cpp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp index ff96a381ffce..fce1a5064870 100644 --- a/src/ipa/libipa/agc_mean_luminance.cpp +++ b/src/ipa/libipa/agc_mean_luminance.cpp @@ -488,18 +488,18 @@ double AgcMeanLuminance::constraintClampGain(uint32_t constraintModeIndex, if (constraint.bound == AgcConstraint::Bound::Lower && newGain > gain) { - gain = newGain; LOG(AgcMeanLuminance, Debug) << "Apply lower bound: " << gain << " to " << newGain; + gain = newGain; } if (constraint.bound == AgcConstraint::Bound::Upper && newGain < gain) { - gain = newGain; LOG(AgcMeanLuminance, Debug) << "Apply upper bound: " << gain << " to " << newGain; + gain = newGain; } } From patchwork Fri Sep 19 09:40:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24417 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 772B6C328C for ; Fri, 19 Sep 2025 09:41:21 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C08716B5B2; Fri, 19 Sep 2025 11:41:20 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="oQQQBmaE"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 23A166B5A7 for ; Fri, 19 Sep 2025 11:41:17 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id EED6C819; Fri, 19 Sep 2025 11:39:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274797; bh=0vf3lFwYoPER+hvWa/iUoCJV/5mJeZC6GD89iKVaKnw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oQQQBmaEAK7lNmVbPGkvBIl7CXfVRofSkLXxYleAp5/bHL0TmK8wYxrqnR4DdGQSx jjTk9cJQM4ah7n4eeLAHmDUxrZrOXVX9wTBzMwFlhACSPXKXFtX9Oi/067ZS4TNWnz vt8M6hM98fEx+Pdr7wIAXkWuJrj8w6D9NWc0+iqg= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Daniel Scally , Kieran Bingham , Paul Elder Subject: [PATCH v5 08/19] libipa: agc_mean_luminance: Configure the exposure mode helpers Date: Fri, 19 Sep 2025 11:40:23 +0200 Message-ID: <20250919094041.183031-9-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a function to configure the exposure mode helpers with the line length and sensor helper to take quantization effects into account. Signed-off-by: Stefan Klug Reviewed-by: Daniel Scally Reviewed-by: Kieran Bingham Reviewed-by: Paul Elder --- Changes in v4: - Fixed wrong variable name in configure() documentation - Collected tags Changes in v3: - Renamed lineLength to lineDuration - Collected tag --- src/ipa/libipa/agc_mean_luminance.cpp | 15 +++++++++++++++ src/ipa/libipa/agc_mean_luminance.h | 1 + 2 files changed, 16 insertions(+) diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp index fce1a5064870..ea4709bf4e49 100644 --- a/src/ipa/libipa/agc_mean_luminance.cpp +++ b/src/ipa/libipa/agc_mean_luminance.cpp @@ -311,6 +311,21 @@ int AgcMeanLuminance::parseExposureModes(const YamlObject &tuningData) return 0; } +/** + * \brief Configure the exposure mode helpers + * \param[in] lineDuration The sensor line length + * \param[in] sensorHelper The sensor helper + * + * This function configures the exposure mode helpers so they can correctly + * take quantization effects into account. + */ +void AgcMeanLuminance::configure(utils::Duration lineDuration, + const CameraSensorHelper *sensorHelper) +{ + for (auto &[id, helper] : exposureModeHelpers_) + helper->configure(lineDuration, sensorHelper); +} + /** * \brief Parse tuning data for AeConstraintMode and AeExposureMode controls * \param[in] tuningData the YamlObject representing the tuning data diff --git a/src/ipa/libipa/agc_mean_luminance.h b/src/ipa/libipa/agc_mean_luminance.h index cad7ef845487..49985481ac51 100644 --- a/src/ipa/libipa/agc_mean_luminance.h +++ b/src/ipa/libipa/agc_mean_luminance.h @@ -42,6 +42,7 @@ public: double yTarget; }; + void configure(utils::Duration lineDuration, const CameraSensorHelper *sensorHelper); int parseTuningData(const YamlObject &tuningData); void setExposureCompensation(double gain) From patchwork Fri Sep 19 09:40:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24418 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1A164C3329 for ; Fri, 19 Sep 2025 09:41:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 62CF06B5BA; Fri, 19 Sep 2025 11:41:23 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="mfhti3IK"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 03E416B5A4 for ; Fri, 19 Sep 2025 11:41:20 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id CF3E799F; Fri, 19 Sep 2025 11:39:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274799; bh=Gex4cPUw8DH11uNgibUPAD59r/ghBk0tdid0R02R2WU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mfhti3IKxEw2KTMSOfSZejd4lXD2BYr9Qbjo01xjvW5HGIjS2JQCnZ3UOPnCGonIJ dBPSi0ewjH2YtyHtCcqh8Xq64/+gzT2ZtVAR1yzgvkBHneY/WCx7hTcQFeK6MxqYR7 00Km2xhqQrmrkQoKp/V9RA41m5u+k9QbXvMz56EQ= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Daniel Scally , Paul Elder Subject: [PATCH v5 09/19] libipa: exposure_mode_helper: Calculate quantization gain in splitExposure() Date: Fri, 19 Sep 2025 11:40:24 +0200 Message-ID: <20250919094041.183031-10-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Calculate the error introduced by quantization as "quantization gain" and return it separately from splitExposure(). It is not included in the digital gain, to not silently ignore the limits imposed by the AGC configuration. Signed-off-by: Stefan Klug Reviewed-by: Daniel Scally Reviewed-by: Paul Elder --- Changes in v4: - Collected tags Changes in v3: - Fixed mali-c55 build - Fixed return of splitExposure in case of gain & exposure time fixed - Calculate quantizationGain for exposure and gain separately and do not feed it into the next stage --- src/ipa/ipu3/algorithms/agc.cpp | 4 +- src/ipa/libipa/agc_mean_luminance.cpp | 7 ++-- src/ipa/libipa/agc_mean_luminance.h | 2 +- src/ipa/libipa/exposure_mode_helper.cpp | 51 +++++++++++++++++-------- src/ipa/libipa/exposure_mode_helper.h | 2 +- src/ipa/mali-c55/algorithms/agc.cpp | 4 +- src/ipa/rkisp1/algorithms/agc.cpp | 9 +++-- 7 files changed, 51 insertions(+), 28 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 39d0aebb0838..da045640d569 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -222,8 +222,8 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, utils::Duration effectiveExposureValue = exposureTime * analogueGain; utils::Duration newExposureTime; - double aGain, dGain; - std::tie(newExposureTime, aGain, dGain) = + double aGain, qGain, dGain; + std::tie(newExposureTime, aGain, qGain, dGain) = calculateNewEv(context.activeState.agc.constraintMode, context.activeState.agc.exposureMode, hist, effectiveExposureValue); diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp index ea4709bf4e49..41c7f72de8ed 100644 --- a/src/ipa/libipa/agc_mean_luminance.cpp +++ b/src/ipa/libipa/agc_mean_luminance.cpp @@ -566,11 +566,12 @@ utils::Duration AgcMeanLuminance::filterExposure(utils::Duration exposureValue) * * Calculate a new exposure value to try to obtain the target. The calculated * exposure value is filtered to prevent rapid changes from frame to frame, and - * divided into exposure time, analogue and digital gain. + * divided into exposure time, analogue, quantization and digital gain. * - * \return Tuple of exposure time, analogue gain, and digital gain + * \return Tuple of exposure time, analogue gain, quantization gain and digital + * gain */ -std::tuple +std::tuple AgcMeanLuminance::calculateNewEv(uint32_t constraintModeIndex, uint32_t exposureModeIndex, const Histogram &yHist, diff --git a/src/ipa/libipa/agc_mean_luminance.h b/src/ipa/libipa/agc_mean_luminance.h index 49985481ac51..fbb526f6ae8e 100644 --- a/src/ipa/libipa/agc_mean_luminance.h +++ b/src/ipa/libipa/agc_mean_luminance.h @@ -68,7 +68,7 @@ public: return controls_; } - std::tuple + std::tuple calculateNewEv(uint32_t constraintModeIndex, uint32_t exposureModeIndex, const Histogram &yHist, utils::Duration effectiveExposureValue); diff --git a/src/ipa/libipa/exposure_mode_helper.cpp b/src/ipa/libipa/exposure_mode_helper.cpp index f2de1d8e4229..29e316d9d091 100644 --- a/src/ipa/libipa/exposure_mode_helper.cpp +++ b/src/ipa/libipa/exposure_mode_helper.cpp @@ -178,14 +178,24 @@ double ExposureModeHelper::clampGain(double gain, double *quantizationGain) cons * required exposure, the helper falls-back to simply maximising the exposure * time first, followed by analogue gain, followed by digital gain. * - * \return Tuple of exposure time, analogue gain, and digital gain + * During the calculations the gain missed due to quantization is recorded and + * returned as quantization gain. The quantization gain is not included in the + * digital gain. So to exactly apply the given exposure, both quantization gain + * and digital gain must be applied. + * + * \return Tuple of exposure time, analogue gain, quantization gain and digital + * gain */ -std::tuple +std::tuple ExposureModeHelper::splitExposure(utils::Duration exposure) const { ASSERT(maxExposureTime_); ASSERT(maxGain_); + utils::Duration exposureTime; + double gain; + double quantGain; + double quantGain2; bool gainFixed = minGain_ == maxGain_; bool exposureTimeFixed = minExposureTime_ == maxExposureTime_; @@ -193,16 +203,21 @@ ExposureModeHelper::splitExposure(utils::Duration exposure) const * There's no point entering the loop if we cannot change either gain * nor exposure time anyway. */ - if (exposureTimeFixed && gainFixed) - return { minExposureTime_, minGain_, exposure / (minExposureTime_ * minGain_) }; + if (exposureTimeFixed && gainFixed) { + exposureTime = clampExposureTime(minExposureTime_, &quantGain); + gain = clampGain(minGain_, &quantGain2); + quantGain *= quantGain2; + + return { exposureTime, gain, quantGain, + exposure / (exposureTime * gain * quantGain) }; + } - utils::Duration exposureTime; double stageGain = clampGain(1.0); double lastStageGain = stageGain; - double gain; for (unsigned int stage = 0; stage < gains_.size(); stage++) { - utils::Duration stageExposureTime = clampExposureTime(exposureTimes_[stage]); + utils::Duration stageExposureTime = clampExposureTime(exposureTimes_[stage], + &quantGain); stageGain = clampGain(gains_[stage]); /* @@ -215,18 +230,22 @@ ExposureModeHelper::splitExposure(utils::Duration exposure) const /* Clamp the gain to lastStageGain and regulate exposureTime. */ if (stageExposureTime * lastStageGain >= exposure) { - exposureTime = clampExposureTime(exposure / lastStageGain); - gain = clampGain(exposure / exposureTime); + exposureTime = clampExposureTime(exposure / lastStageGain, &quantGain); + gain = clampGain(exposure / exposureTime, &quantGain2); + quantGain *= quantGain2; - return { exposureTime, gain, exposure / (exposureTime * gain) }; + return { exposureTime, gain, quantGain, + exposure / (exposureTime * gain * quantGain) }; } /* Clamp the exposureTime to stageExposureTime and regulate gain. */ if (stageExposureTime * stageGain >= exposure) { exposureTime = stageExposureTime; - gain = clampGain(exposure / exposureTime); + gain = clampGain(exposure / exposureTime, &quantGain2); + quantGain *= quantGain2; - return { exposureTime, gain, exposure / (exposureTime * gain) }; + return { exposureTime, gain, quantGain, + exposure / (exposureTime * gain * quantGain) }; } lastStageGain = stageGain; @@ -239,10 +258,12 @@ ExposureModeHelper::splitExposure(utils::Duration exposure) const * stages to use then the default stageGain of 1.0 is used so that * exposure time is maxed before gain is touched at all. */ - exposureTime = clampExposureTime(exposure / stageGain); - gain = clampGain(exposure / exposureTime); + exposureTime = clampExposureTime(exposure / stageGain, &quantGain); + gain = clampGain(exposure / exposureTime, &quantGain2); + quantGain *= quantGain2; - return { exposureTime, gain, exposure / (exposureTime * gain) }; + return { exposureTime, gain, quantGain, + exposure / (exposureTime * gain * quantGain) }; } /** diff --git a/src/ipa/libipa/exposure_mode_helper.h b/src/ipa/libipa/exposure_mode_helper.h index ac7e8da95c6c..968192ddc5af 100644 --- a/src/ipa/libipa/exposure_mode_helper.h +++ b/src/ipa/libipa/exposure_mode_helper.h @@ -30,7 +30,7 @@ public: void setLimits(utils::Duration minExposureTime, utils::Duration maxExposureTime, double minGain, double maxGain); - std::tuple + std::tuple splitExposure(utils::Duration exposure) const; utils::Duration minExposureTime() const { return minExposureTime_; } diff --git a/src/ipa/mali-c55/algorithms/agc.cpp b/src/ipa/mali-c55/algorithms/agc.cpp index 15963994b2d6..88f8664c9823 100644 --- a/src/ipa/mali-c55/algorithms/agc.cpp +++ b/src/ipa/mali-c55/algorithms/agc.cpp @@ -381,8 +381,8 @@ void Agc::process(IPAContext &context, utils::Duration effectiveExposureValue = currentShutter * totalGain; utils::Duration shutterTime; - double aGain, dGain; - std::tie(shutterTime, aGain, dGain) = + double aGain, qGain, dGain; + std::tie(shutterTime, aGain, qGain, dGain) = calculateNewEv(activeState.agc.constraintMode, activeState.agc.exposureMode, statistics_.yHist, effectiveExposureValue); diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 35440b67e999..0a29326841fb 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -567,15 +567,16 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, setExposureCompensation(pow(2.0, frameContext.agc.exposureValue)); utils::Duration newExposureTime; - double aGain, dGain; - std::tie(newExposureTime, aGain, dGain) = + double aGain, qGain, dGain; + std::tie(newExposureTime, aGain, qGain, dGain) = calculateNewEv(frameContext.agc.constraintMode, frameContext.agc.exposureMode, hist, effectiveExposureValue); LOG(RkISP1Agc, Debug) - << "Divided up exposure time, analogue gain and digital gain are " - << newExposureTime << ", " << aGain << " and " << dGain; + << "Divided up exposure time, analogue gain, quantization gain" + << " and digital gain are " << newExposureTime << ", " << aGain + << ", " << qGain << " and " << dGain; IPAActiveState &activeState = context.activeState; /* Update the estimated exposure and gain. */ From patchwork Fri Sep 19 09:40:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24419 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 03B63C328C for ; Fri, 19 Sep 2025 09:41:25 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 685856B5B9; Fri, 19 Sep 2025 11:41:25 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="m8HgF/r0"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 962856B5A9 for ; Fri, 19 Sep 2025 11:41:22 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 6598E842; Fri, 19 Sep 2025 11:40:02 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274802; bh=8NrnZ1Svug0IEpVFOG2l9atAlmoOVvrNZc39ea8krp4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=m8HgF/r02W8YnKsziYOK9UkKBJAEd9mdL7su4BKnu6PRuKx/AHQgvT/L5+xJ62x55 p7aXM6bVuQI4LT3JAqsPNQ/fAb/HB4DvJh+tUeoUpIS2bPKlT6Nw03toS/MxJI0wQz 1lmsfaVLpx85TkUPaxfIggRGfyrgbhqGA4YYYORU= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Kieran Bingham , Daniel Scally , Paul Elder Subject: [PATCH v5 10/19] ipa: rkisp1: agc: Add correction for exposure quantization Date: Fri, 19 Sep 2025 11:40:25 +0200 Message-ID: <20250919094041.183031-11-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" There are several occasions where quantization can lead to visible effects. In WDR mode it can happen that exposure times get set to very low values (Sometimes 2-3 lines). This intentionally introduced underexposure is corrected by the GWDR module. As exposure time is quantized by lines, the smallest possible change in exposure time now results in a quite visible change in perceived brightness. On some sensors the possible gain steps are also quite large leading to visible jumps if e.g. if the exposure time is fixed. Mitigate that by applying a global gain to account for the error introduced by the exposure quantization. ToDo: This needs perfect frame synchronous control of the sensor to work properly which is not guaranteed in all cases. It still improves the behavior with the current regulation and can easily be skipped, be removing the compress algorithm from the tuning file. Signed-off-by: Stefan Klug Reviewed-by: Kieran Bingham Reviewed-by: Daniel Scally Reviewed-by: Paul Elder --- Changes in v4: - Include quantization gain in effective exposure value - Correctly initialize quantizationGain in frameContext - Collected tags Changes in v3: - Collected tags --- src/ipa/rkisp1/algorithms/agc.cpp | 33 ++++++++++++++++++++++++++++--- src/ipa/rkisp1/ipa_context.h | 2 ++ 2 files changed, 32 insertions(+), 3 deletions(-) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 0a29326841fb..fe2f66cf6ee4 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -176,6 +176,7 @@ int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) context.activeState.agc.automatic.gain = context.configuration.sensor.minAnalogueGain; context.activeState.agc.automatic.exposure = 10ms / context.configuration.sensor.lineDuration; + context.activeState.agc.automatic.quantizationGain = 1.0; context.activeState.agc.manual.gain = context.activeState.agc.automatic.gain; context.activeState.agc.manual.exposure = context.activeState.agc.automatic.exposure; context.activeState.agc.autoExposureEnabled = !context.configuration.raw; @@ -199,6 +200,9 @@ int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) context.configuration.agc.measureWindow.h_size = configInfo.outputSize.width; context.configuration.agc.measureWindow.v_size = configInfo.outputSize.height; + AgcMeanLuminance::configure(context.configuration.sensor.lineDuration, + context.camHelper.get()); + setLimits(context.configuration.sensor.minExposureTime, context.configuration.sensor.maxExposureTime, context.configuration.sensor.minAnalogueGain, @@ -283,6 +287,10 @@ void Agc::queueRequest(IPAContext &context, if (!frameContext.agc.autoGainEnabled) frameContext.agc.gain = agc.manual.gain; + if (!frameContext.agc.autoExposureEnabled && + !frameContext.agc.autoGainEnabled) + frameContext.agc.quantizationGain = 1.0; + const auto &meteringMode = controls.get(controls::AeMeteringMode); if (meteringMode) { frameContext.agc.updateMetering = agc.meteringMode != *meteringMode; @@ -336,12 +344,17 @@ void Agc::prepare(IPAContext &context, const uint32_t frame, { uint32_t activeAutoExposure = context.activeState.agc.automatic.exposure; double activeAutoGain = context.activeState.agc.automatic.gain; + double activeAutoQGain = context.activeState.agc.automatic.quantizationGain; /* Populate exposure and gain in auto mode */ - if (frameContext.agc.autoExposureEnabled) + if (frameContext.agc.autoExposureEnabled) { frameContext.agc.exposure = activeAutoExposure; - if (frameContext.agc.autoGainEnabled) + frameContext.agc.quantizationGain = activeAutoQGain; + } + if (frameContext.agc.autoGainEnabled) { frameContext.agc.gain = activeAutoGain; + frameContext.agc.quantizationGain = activeAutoQGain; + } /* * Populate manual exposure and gain from the active auto values when @@ -354,6 +367,12 @@ void Agc::prepare(IPAContext &context, const uint32_t frame, if (!frameContext.agc.autoGainEnabled && frameContext.agc.autoGainModeChange) { context.activeState.agc.manual.gain = activeAutoGain; frameContext.agc.gain = activeAutoGain; + frameContext.agc.quantizationGain = activeAutoQGain; + } + + if (context.configuration.compress.supported) { + frameContext.compress.enable = true; + frameContext.compress.gain = frameContext.agc.quantizationGain; } if (frame > 0 && !frameContext.agc.updateMetering) @@ -564,6 +583,14 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, double analogueGain = frameContext.sensor.gain; utils::Duration effectiveExposureValue = exposureTime * analogueGain; + /* + * Include the quantization gain if it was applied. Do not use + * compress.gain because it will include gains that shall not be + * reported to the user when HDR is implemented. + */ + if (frameContext.compress.enable) + effectiveExposureValue *= frameContext.agc.quantizationGain; + setExposureCompensation(pow(2.0, frameContext.agc.exposureValue)); utils::Duration newExposureTime; @@ -582,7 +609,7 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, /* Update the estimated exposure and gain. */ activeState.agc.automatic.exposure = newExposureTime / lineDuration; activeState.agc.automatic.gain = aGain; - + activeState.agc.automatic.quantizationGain = qGain; /* * Expand the target frame duration so that we do not run faster than * the minimum frame duration when we have short exposures. diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index a723c79b04d9..35d25d555e65 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -81,6 +81,7 @@ struct IPAActiveState { struct { uint32_t exposure; double gain; + double quantizationGain; } automatic; bool autoExposureEnabled; @@ -135,6 +136,7 @@ struct IPAFrameContext : public FrameContext { uint32_t exposure; double gain; double exposureValue; + double quantizationGain; uint32_t vblank; bool autoExposureEnabled; bool autoGainEnabled; From patchwork Fri Sep 19 09:40:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24420 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0C474C328C for ; Fri, 19 Sep 2025 09:41:30 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A97606B5C4; Fri, 19 Sep 2025 11:41:29 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="TnuiEtUL"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8E6AF6B5BF for ; Fri, 19 Sep 2025 11:41:25 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 2052E99F; Fri, 19 Sep 2025 11:40:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274805; bh=ZEdXfHFpaIB4O0iiGoYtfIMWXOEazI7FHupCfh9Hz/c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TnuiEtUL6PQZ8YQDV0ji+LtoCzaZSmXqu8hcqCWMkfGnqEcHl4UJUB1HYtTLYa3sl HqBO0q0mYkWmJKIbjK1vNibTL0OXLRG2myIS/LW/hZQOEQA7w/0JdDfMkps6ygQEwR 0GJrxN8lwOVqX8ogqtEcPl+2c8kGeuTYwpjTGXwU= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Kieran Bingham , Paul Elder , Daniel Scally Subject: [PATCH v5 11/19] pipeline: rkisp1: Add error log when parameter queuing fails Date: Fri, 19 Sep 2025 11:40:26 +0200 Message-ID: <20250919094041.183031-12-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When the extensible parameters queued to the kernel contain an unknown block type it fails with -EINVAL. This should not happen as user land is supposed to check for the supported parameter types. But it took a while to figure out where things went wrong. Add a error statement when queuing of the parameter buffer fails for whatever reason. Signed-off-by: Stefan Klug Reviewed-by: Kieran Bingham Reviewed-by: Paul Elder Reviewed-by: Daniel Scally --- Changes in v3: - Collected tag Changelog from former series: Changes in v4: - Improved commit message Changes in v3: - Collected tags - Removed hint regarding unsupported parameter types as this will be handled using the now upstreamed RKISP1_CID_SUPPORTED_PARAMS_BLOCKS. Changes in v2: - Also print the error code in case of failure --- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index cfcbb3b2590a..a91a443aeaf7 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -422,7 +422,14 @@ void RkISP1CameraData::paramsComputed(unsigned int frame, unsigned int bytesused return; info->paramBuffer->_d()->metadata().planes()[0].bytesused = bytesused; - pipe->param_->queueBuffer(info->paramBuffer); + + int ret = pipe->param_->queueBuffer(info->paramBuffer); + if (ret < 0) { + LOG(RkISP1, Error) << "Failed to queue parameter buffer: " + << strerror(-ret); + return; + } + pipe->stat_->queueBuffer(info->statBuffer); if (info->mainPathBuffer) From patchwork Fri Sep 19 09:40:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24421 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 14033C3329 for ; Fri, 19 Sep 2025 09:41:32 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 655D969369; Fri, 19 Sep 2025 11:41:32 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="fsltjixJ"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9F9BC6B5BD for ; Fri, 19 Sep 2025 11:41:28 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 77E3B842; Fri, 19 Sep 2025 11:40:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274808; bh=KntSfjma75Nzy9LoNDqRskT+yp8r2D+kpUL1rNFia7A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fsltjixJEygo7ZysauHxjqNCurqIhL2l/t53Fb2c9Iif5q4SPrQoXrD/GfKAxXtnZ zEm4f7o2gIbNkrgFDQV229NEwW3cVZY85E6I5m9tVSvOTKEr/u/KP9O5SGDJAMVfbk aE9ckLP9OpUUpBpwDl68caIg08A+yBq4S2otWVZQ= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Kieran Bingham , Paul Elder Subject: [PATCH v5 12/19] include: linux: Partially update linux headers from v6.16-rc1-310-gd968e50b5c26 Date: Fri, 19 Sep 2025 11:40:27 +0200 Message-ID: <20250919094041.183031-13-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Update rkisp1-config.h and v4l2-controls.h from the next branch of https://gitlab.freedesktop.org/linux-media/media-committers.git to include the WDR related updates. The rest was left as is to minimize the risk of issues due to last minute changes in the upstream process. Signed-off-by: Stefan Klug Reviewed-by: Kieran Bingham Acked-by: Paul Elder --- Changes in v4: - Collected tags Changes in v3: - Limit changes to the bare minimum required for rkisp1 Changes in v2: - Updated headers from linux-media next branch --- include/linux/rkisp1-config.h | 108 +++++++++++++++++++++++++++++++++- include/linux/v4l2-controls.h | 6 ++ 2 files changed, 112 insertions(+), 2 deletions(-) diff --git a/include/linux/rkisp1-config.h b/include/linux/rkisp1-config.h index edbc6cb65d1c..d323bfa72d8e 100644 --- a/include/linux/rkisp1-config.h +++ b/include/linux/rkisp1-config.h @@ -169,6 +169,13 @@ */ #define RKISP1_CIF_ISP_COMPAND_NUM_POINTS 64 +/* + * Wide Dynamic Range + */ +#define RKISP1_CIF_ISP_WDR_CURVE_NUM_INTERV 32 +#define RKISP1_CIF_ISP_WDR_CURVE_NUM_COEFF (RKISP1_CIF_ISP_WDR_CURVE_NUM_INTERV + 1) +#define RKISP1_CIF_ISP_WDR_CURVE_NUM_DY_REGS 4 + /* * Measurement types */ @@ -889,6 +896,72 @@ struct rkisp1_cif_isp_compand_curve_config { __u32 y[RKISP1_CIF_ISP_COMPAND_NUM_POINTS]; }; +/** + * struct rkisp1_cif_isp_wdr_tone_curve - Tone mapping curve definition for WDR. + * + * @dY: the dYn increments for horizontal (input) axis of the tone curve. + * each 3-bit dY value represents an increment of 2**(value+3). + * dY[0] bits 0:2 is increment dY1, bit 3 unused + * dY[0] bits 4:6 is increment dY2, bit 7 unused + * ... + * dY[0] bits 28:30 is increment dY8, bit 31 unused + * ... and so on till dY[3] bits 28:30 is increment dY32, bit 31 unused. + * @ym: the Ym values for the vertical (output) axis of the tone curve. + * each value is 13 bit. + */ +struct rkisp1_cif_isp_wdr_tone_curve { + __u32 dY[RKISP1_CIF_ISP_WDR_CURVE_NUM_DY_REGS]; + __u16 ym[RKISP1_CIF_ISP_WDR_CURVE_NUM_COEFF]; +}; + +/** + * struct rkisp1_cif_isp_wdr_iref_config - Illumination reference config for WDR. + * + * Use illumination reference value as described below, instead of only the + * luminance (Y) value for tone mapping and gain calculations: + * IRef = (rgb_factor * RGBMax_tr + (8 - rgb_factor) * Y)/8 + * + * @rgb_factor: defines how much influence the RGBmax approach has in + * comparison to Y (valid values are 0..8). + * @use_y9_8: use Y*9/8 for maximum value calculation along with the + * default of R, G, B for noise reduction. + * @use_rgb7_8: decrease RGBMax by 7/8 for noise reduction. + * @disable_transient: disable transient calculation between Y and RGBY_max. + */ +struct rkisp1_cif_isp_wdr_iref_config { + __u8 rgb_factor; + __u8 use_y9_8; + __u8 use_rgb7_8; + __u8 disable_transient; +}; + +/** + * struct rkisp1_cif_isp_wdr_config - Configuration for wide dynamic range. + * + * @tone_curve: tone mapping curve. + * @iref_config: illumination reference configuration. (when use_iref is true) + * @rgb_offset: RGB offset value for RGB operation mode. (12 bits) + * @luma_offset: luminance offset value for RGB operation mode. (12 bits) + * @dmin_thresh: lower threshold for deltaMin value. (12 bits) + * @dmin_strength: strength factor for deltaMin. (valid range is 0x00..0x10) + * @use_rgb_colorspace: use RGB instead of luminance/chrominance colorspace. + * @bypass_chroma_mapping: disable chrominance mapping (only valid if + * use_rgb_colorspace = 0) + * @use_iref: use illumination reference instead of Y for tone mapping + * and gain calculations. + */ +struct rkisp1_cif_isp_wdr_config { + struct rkisp1_cif_isp_wdr_tone_curve tone_curve; + struct rkisp1_cif_isp_wdr_iref_config iref_config; + __u16 rgb_offset; + __u16 luma_offset; + __u16 dmin_thresh; + __u8 dmin_strength; + __u8 use_rgb_colorspace; + __u8 bypass_chroma_mapping; + __u8 use_iref; +}; + /*---------- PART2: Measurement Statistics ------------*/ /** @@ -1059,6 +1132,7 @@ struct rkisp1_stat_buffer { * @RKISP1_EXT_PARAMS_BLOCK_TYPE_COMPAND_BLS: BLS in the compand block * @RKISP1_EXT_PARAMS_BLOCK_TYPE_COMPAND_EXPAND: Companding expand curve * @RKISP1_EXT_PARAMS_BLOCK_TYPE_COMPAND_COMPRESS: Companding compress curve + * @RKISP1_EXT_PARAMS_BLOCK_TYPE_WDR: Wide dynamic range */ enum rkisp1_ext_params_block_type { RKISP1_EXT_PARAMS_BLOCK_TYPE_BLS, @@ -1081,11 +1155,15 @@ enum rkisp1_ext_params_block_type { RKISP1_EXT_PARAMS_BLOCK_TYPE_COMPAND_BLS, RKISP1_EXT_PARAMS_BLOCK_TYPE_COMPAND_EXPAND, RKISP1_EXT_PARAMS_BLOCK_TYPE_COMPAND_COMPRESS, + RKISP1_EXT_PARAMS_BLOCK_TYPE_WDR, }; #define RKISP1_EXT_PARAMS_FL_BLOCK_DISABLE (1U << 0) #define RKISP1_EXT_PARAMS_FL_BLOCK_ENABLE (1U << 1) +/* A bitmask of parameters blocks supported on the current hardware. */ +#define RKISP1_CID_SUPPORTED_PARAMS_BLOCKS (V4L2_CID_USER_RKISP1_BASE + 0x01) + /** * struct rkisp1_ext_params_block_header - RkISP1 extensible parameters block * header @@ -1460,6 +1538,23 @@ struct rkisp1_ext_params_compand_curve_config { struct rkisp1_cif_isp_compand_curve_config config; } __attribute__((aligned(8))); +/** + * struct rkisp1_ext_params_wdr_config - RkISP1 extensible params + * Wide dynamic range config + * + * RkISP1 extensible parameters WDR block. + * Identified by :c:type:`RKISP1_EXT_PARAMS_BLOCK_TYPE_WDR` + * + * @header: The RkISP1 extensible parameters header, see + * :c:type:`rkisp1_ext_params_block_header` + * @config: WDR configuration, see + * :c:type:`rkisp1_cif_isp_wdr_config` + */ +struct rkisp1_ext_params_wdr_config { + struct rkisp1_ext_params_block_header header; + struct rkisp1_cif_isp_wdr_config config; +} __attribute__((aligned(8))); + /* * The rkisp1_ext_params_compand_curve_config structure is counted twice as it * is used for both the COMPAND_EXPAND and COMPAND_COMPRESS block types. @@ -1484,7 +1579,8 @@ struct rkisp1_ext_params_compand_curve_config { sizeof(struct rkisp1_ext_params_afc_config) +\ sizeof(struct rkisp1_ext_params_compand_bls_config) +\ sizeof(struct rkisp1_ext_params_compand_curve_config) +\ - sizeof(struct rkisp1_ext_params_compand_curve_config)) + sizeof(struct rkisp1_ext_params_compand_curve_config) +\ + sizeof(struct rkisp1_ext_params_wdr_config)) /** * enum rksip1_ext_param_buffer_version - RkISP1 extensible parameters version @@ -1520,6 +1616,14 @@ enum rksip1_ext_param_buffer_version { * V4L2 control. If such control is not available, userspace should assume only * RKISP1_EXT_PARAM_BUFFER_V1 is supported by the driver. * + * The read-only V4L2 control ``RKISP1_CID_SUPPORTED_PARAMS_BLOCKS`` can be used + * to query the blocks supported by the device. It contains a bitmask where each + * bit represents the availability of the corresponding entry from the + * :c:type:`rkisp1_ext_params_block_type` enum. The current and default values + * of the control represents the blocks supported by the device instance, while + * the maximum value represents the blocks supported by the kernel driver, + * independently of the device instance. + * * For each ISP block that userspace wants to configure, a block-specific * structure is appended to the @data buffer, one after the other without gaps * in between nor overlaps. Userspace shall populate the @data_size field with @@ -1528,7 +1632,7 @@ enum rksip1_ext_param_buffer_version { * The expected memory layout of the parameters buffer is:: * * +-------------------- struct rkisp1_ext_params_cfg -------------------+ - * | version = RKISP_EXT_PARAMS_BUFFER_V1; | + * | version = RKISP1_EXT_PARAM_BUFFER_V1; | * | data_size = sizeof(struct rkisp1_ext_params_bls_config) | * | + sizeof(struct rkisp1_ext_params_dpcc_config); | * | +------------------------- data ---------------------------------+ | diff --git a/include/linux/v4l2-controls.h b/include/linux/v4l2-controls.h index 882a81805783..4cfae0414894 100644 --- a/include/linux/v4l2-controls.h +++ b/include/linux/v4l2-controls.h @@ -217,6 +217,12 @@ enum v4l2_colorfx { */ #define V4L2_CID_USER_THP7312_BASE (V4L2_CID_USER_BASE + 0x11c0) +/* + * The base for Rockchip ISP1 driver controls. + * We reserve 16 controls for this driver. + */ +#define V4L2_CID_USER_RKISP1_BASE (V4L2_CID_USER_BASE + 0x1220) + /* MPEG-class control IDs */ /* The MPEG controls are applicable to all codec controls * and the 'MPEG' part of the define is historical */ From patchwork Fri Sep 19 09:40:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24422 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id DC578C328C for ; Fri, 19 Sep 2025 09:41:36 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 72BBF6B5C2; Fri, 19 Sep 2025 11:41:36 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="NMOmdyKi"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 930C26B5BA for ; Fri, 19 Sep 2025 11:41:31 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 6765FD3E; Fri, 19 Sep 2025 11:40:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274811; bh=/pVOmLIZnIp1gu90GepU9SqG6OJvbmpojHTW1RnqeNs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NMOmdyKi+1e/l/xGkZorhz4dM+wfrIOPLYfXWNeVXRu05nx5t6So92rRgyCn8Enwy i5eMm3GczXqBIp967OZnE+BzBVgm+aqBvn8QQ2soTE4PJ5pANpsxH5wMB3a+Tp/o7D gquZ0od8tQFKzis2uikqOEDWW8M7roYpojPlHqRw= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Paul Elder , Kieran Bingham Subject: [PATCH v5 13/19] ipa: rkisp1: Switch histogram to RGB combined mode Date: Fri, 19 Sep 2025 11:40:28 +0200 Message-ID: <20250919094041.183031-14-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The Y mode of the histogram gets captured at the ISP output, before the output formatter. This has the side effect that the first and the last bins are empty in case of limited YUV range. Another side effect is that gamma and GWDR processing is included in the histogram which makes algorithm development very difficult. In RGB mode the histogram is taken after xtalk (CCM) and is therefore independent of gamma and WDR. The limited range issue also does not apply. In the ISP reference it is however stated that "it is not possible to calculate a luminance or grayscale histogram from an RGB histogram since the position information is lost during its generation". During testing the RGB histogram provided good data and better algorithmic stability at a possible (but not measured) inaccuracy. Another option would be to pass the color space information into the IPA and strip the histogram accordingly. For ease of implementation switch to the RGB mode. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder Reviewed-by: Kieran Bingham --- Changes in v4: - Tiny fixes in the block comment (no doxygen & whitespace) - Collected tags Changes in v3: - Added block comment inside the code --- src/ipa/rkisp1/algorithms/agc.cpp | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index fe2f66cf6ee4..174d810edc65 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -397,7 +397,24 @@ void Agc::prepare(IPAContext &context, const uint32_t frame, hstConfig.setEnabled(true); hstConfig->meas_window = context.configuration.agc.measureWindow; - hstConfig->mode = RKISP1_CIF_ISP_HISTOGRAM_MODE_Y_HISTOGRAM; + /* + * The Y mode of the histogram gets captured at the ISP output, before + * the output formatter. This has the side effect that the first and + * the last bins are empty in case of limited YUV range. Another side + * effect is that gamma and GWDR processing is included in the histogram + * which makes algorithm development very difficult. In RGB mode the + * histogram is taken after xtalk (CCM) and is therefore independent of + * gamma and WDR. The limited range issue also does not apply. In the + * ISP reference it is however stated that "it is not possible to + * calculate a luminance or grayscale histogram from an RGB histogram + * since the position information is lost during its generation". + * + * During testing the RGB histogram provided good data and better + * algorithmic stability at a possible (but not measured) inaccuracy. + * + * \todo For a proper fix support for HIST64 is needed. + */ + hstConfig->mode = RKISP1_CIF_ISP_HISTOGRAM_MODE_RGB_COMBINED; Span weights{ hstConfig->hist_weight, From patchwork Fri Sep 19 09:40:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24423 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D22CAC328C for ; Fri, 19 Sep 2025 09:41:40 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 16FA769369; Fri, 19 Sep 2025 11:41:40 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="P7+IFXD/"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 104146B5B9 for ; Fri, 19 Sep 2025 11:41:35 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id B0299EFF; Fri, 19 Sep 2025 11:40:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274815; bh=bmSsIku7p9XKWO8vu2+aB9z0rvtjj+9wtxaELrf2Sl8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=P7+IFXD/6ux4tKQY19wDhBMq9s3DQNrzBSNaR30M2/O5Gn5ZqTpqH5pklBdgtUWoD aiKyejvqaFAhDYooAR5+mU2UbLuh1d3kH/YCFeWOvVJiqS9L+v91P4mWudCX4b1NTX Ik1BFtQVMW+BqBSP6wIP86r/RmYZ7mmD9klLvBNA= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Kieran Bingham , Daniel Scally , Paul Elder Subject: [PATCH v5 14/19] pipeline: rkisp1: Query kernel for available params blocks Date: Fri, 19 Sep 2025 11:40:29 +0200 Message-ID: <20250919094041.183031-15-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Query the params device for RKISP1_CID_SUPPORTED_PARAMS_BLOCKS and inject the information into the IPA hardware context for use by the algorithms. To be able to modify the hardware configuration at runtime, replace the pointer with an instance and create a copy of the static hardware specific data. Signed-off-by: Stefan Klug Reviewed-by: Kieran Bingham Reviewed-by: Daniel Scally Reviewed-by: Paul Elder --- Changes in v4: - Keep the static data const and create a copy of the configuration to be able to modify it. Changes in v3: - Removed unnecessary log statement - Collected tags - Added case for getControls() failing internally --- include/libcamera/ipa/rkisp1.mojom | 2 +- src/ipa/rkisp1/algorithms/agc.cpp | 10 ++++---- src/ipa/rkisp1/algorithms/blc.cpp | 4 ++-- src/ipa/rkisp1/algorithms/compress.cpp | 2 +- src/ipa/rkisp1/algorithms/goc.cpp | 4 ++-- src/ipa/rkisp1/algorithms/lux.cpp | 2 +- src/ipa/rkisp1/ipa_context.h | 5 ++-- src/ipa/rkisp1/rkisp1.cpp | 12 +++++++--- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 29 ++++++++++++++++++++---- 9 files changed, 49 insertions(+), 21 deletions(-) diff --git a/include/libcamera/ipa/rkisp1.mojom b/include/libcamera/ipa/rkisp1.mojom index 043ad27ea199..068e898848c4 100644 --- a/include/libcamera/ipa/rkisp1.mojom +++ b/include/libcamera/ipa/rkisp1.mojom @@ -16,7 +16,7 @@ struct IPAConfigInfo { interface IPARkISP1Interface { init(libcamera.IPASettings settings, - uint32 hwRevision, + uint32 hwRevision, uint32 supportedBlocks, libcamera.IPACameraSensorInfo sensorInfo, libcamera.ControlInfoMap sensorControls) => (int32 ret, libcamera.ControlInfoMap ipaControls); diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 174d810edc65..d0337fd0027f 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -56,7 +56,7 @@ int Agc::parseMeteringModes(IPAContext &context, const YamlObject &tuningData) std::vector weights = value.getList().value_or(std::vector{}); - if (weights.size() != context.hw->numHistogramWeights) { + if (weights.size() != context.hw.numHistogramWeights) { LOG(RkISP1Agc, Warning) << "Failed to read metering mode'" << key << "'"; continue; @@ -68,7 +68,7 @@ int Agc::parseMeteringModes(IPAContext &context, const YamlObject &tuningData) if (meteringModes_.empty()) { LOG(RkISP1Agc, Warning) << "No metering modes read from tuning file; defaulting to matrix"; - std::vector weights(context.hw->numHistogramWeights, 1); + std::vector weights(context.hw.numHistogramWeights, 1); meteringModes_[controls::MeteringMatrix] = weights; } @@ -418,7 +418,7 @@ void Agc::prepare(IPAContext &context, const uint32_t frame, Span weights{ hstConfig->hist_weight, - context.hw->numHistogramWeights + context.hw.numHistogramWeights }; std::vector &modeWeights = meteringModes_.at(frameContext.agc.meteringMode); std::copy(modeWeights.begin(), modeWeights.end(), weights.begin()); @@ -556,9 +556,9 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, const rkisp1_cif_isp_stat *params = &stats->params; /* The lower 4 bits are fractional and meant to be discarded. */ - Histogram hist({ params->hist.hist_bins, context.hw->numHistogramBins }, + Histogram hist({ params->hist.hist_bins, context.hw.numHistogramBins }, [](uint32_t x) { return x >> 4; }); - expMeans_ = { params->ae.exp_mean, context.hw->numAeCells }; + expMeans_ = { params->ae.exp_mean, context.hw.numAeCells }; std::vector &modeWeights = meteringModes_.at(frameContext.agc.meteringMode); weights_ = { modeWeights.data(), modeWeights.size() }; diff --git a/src/ipa/rkisp1/algorithms/blc.cpp b/src/ipa/rkisp1/algorithms/blc.cpp index 98cb7145e164..32fc44ffff92 100644 --- a/src/ipa/rkisp1/algorithms/blc.cpp +++ b/src/ipa/rkisp1/algorithms/blc.cpp @@ -114,7 +114,7 @@ int BlackLevelCorrection::configure(IPAContext &context, * of the extensible parameters format. */ supported_ = context.configuration.paramFormat == V4L2_META_FMT_RK_ISP1_EXT_PARAMS || - !context.hw->compand; + !context.hw.compand; if (!supported_) LOG(RkISP1Blc, Warning) @@ -140,7 +140,7 @@ void BlackLevelCorrection::prepare(IPAContext &context, if (!supported_) return; - if (context.hw->compand) { + if (context.hw.compand) { auto config = params->block(); config.setEnabled(true); diff --git a/src/ipa/rkisp1/algorithms/compress.cpp b/src/ipa/rkisp1/algorithms/compress.cpp index 6445cd14bae3..2afa449608b7 100644 --- a/src/ipa/rkisp1/algorithms/compress.cpp +++ b/src/ipa/rkisp1/algorithms/compress.cpp @@ -51,7 +51,7 @@ int Compress::configure(IPAContext &context, [[maybe_unused]] const IPACameraSensorInfo &configInfo) { if (context.configuration.paramFormat != V4L2_META_FMT_RK_ISP1_EXT_PARAMS || - !context.hw->compand) { + !context.hw.compand) { LOG(RkISP1Compress, Warning) << "Compression is not supported by the hardware or kernel."; return 0; diff --git a/src/ipa/rkisp1/algorithms/goc.cpp b/src/ipa/rkisp1/algorithms/goc.cpp index a9493678dba7..a0e7030fe5db 100644 --- a/src/ipa/rkisp1/algorithms/goc.cpp +++ b/src/ipa/rkisp1/algorithms/goc.cpp @@ -50,7 +50,7 @@ const float kDefaultGamma = 2.2f; */ int GammaOutCorrection::init(IPAContext &context, const YamlObject &tuningData) { - if (context.hw->numGammaOutSamples != + if (context.hw.numGammaOutSamples != RKISP1_CIF_ISP_GAMMA_OUT_MAX_SAMPLES_V10) { LOG(RkISP1Gamma, Error) << "Gamma is not implemented for RkISP1 V12"; @@ -101,7 +101,7 @@ void GammaOutCorrection::prepare(IPAContext &context, IPAFrameContext &frameContext, RkISP1Params *params) { - ASSERT(context.hw->numGammaOutSamples == + ASSERT(context.hw.numGammaOutSamples == RKISP1_CIF_ISP_GAMMA_OUT_MAX_SAMPLES_V10); if (!frameContext.goc.update) diff --git a/src/ipa/rkisp1/algorithms/lux.cpp b/src/ipa/rkisp1/algorithms/lux.cpp index dd05f18d5e94..e8da69810008 100644 --- a/src/ipa/rkisp1/algorithms/lux.cpp +++ b/src/ipa/rkisp1/algorithms/lux.cpp @@ -61,7 +61,7 @@ void Lux::process(IPAContext &context, /* \todo Deduplicate the histogram calculation from AGC */ const rkisp1_cif_isp_stat *params = &stats->params; - Histogram yHist({ params->hist.hist_bins, context.hw->numHistogramBins }, + Histogram yHist({ params->hist.hist_bins, context.hw.numHistogramBins }, [](uint32_t x) { return x >> 4; }); double lux = lux_.estimateLux(exposureTime, gain, 1.0, yHist); diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index 35d25d555e65..67427c01ac68 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -36,6 +36,7 @@ struct IPAHwSettings { unsigned int numHistogramBins; unsigned int numHistogramWeights; unsigned int numGammaOutSamples; + uint32_t supportedBlocks; bool compand; }; @@ -201,11 +202,11 @@ struct IPAFrameContext : public FrameContext { struct IPAContext { IPAContext(unsigned int frameContextSize) - : hw(nullptr), frameContexts(frameContextSize) + : frameContexts(frameContextSize) { } - const IPAHwSettings *hw; + IPAHwSettings hw; IPACameraSensorInfo sensorInfo; IPASessionConfiguration configuration; IPAActiveState activeState; diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index cf66d5553dcd..fa22bfc34904 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -52,6 +52,7 @@ public: IPARkISP1(); int init(const IPASettings &settings, unsigned int hwRevision, + uint32_t supportedBlocks, const IPACameraSensorInfo &sensorInfo, const ControlInfoMap &sensorControls, ControlInfoMap *ipaControls) override; @@ -94,6 +95,7 @@ const IPAHwSettings ipaHwSettingsV10{ RKISP1_CIF_ISP_HIST_BIN_N_MAX_V10, RKISP1_CIF_ISP_HISTOGRAM_WEIGHT_GRIDS_SIZE_V10, RKISP1_CIF_ISP_GAMMA_OUT_MAX_SAMPLES_V10, + 0, false, }; @@ -102,6 +104,7 @@ const IPAHwSettings ipaHwSettingsIMX8MP{ RKISP1_CIF_ISP_HIST_BIN_N_MAX_V10, RKISP1_CIF_ISP_HISTOGRAM_WEIGHT_GRIDS_SIZE_V10, RKISP1_CIF_ISP_GAMMA_OUT_MAX_SAMPLES_V10, + 0, true, }; @@ -110,6 +113,7 @@ const IPAHwSettings ipaHwSettingsV12{ RKISP1_CIF_ISP_HIST_BIN_N_MAX_V12, RKISP1_CIF_ISP_HISTOGRAM_WEIGHT_GRIDS_SIZE_V12, RKISP1_CIF_ISP_GAMMA_OUT_MAX_SAMPLES_V12, + 0, false, }; @@ -132,6 +136,7 @@ std::string IPARkISP1::logPrefix() const } int IPARkISP1::init(const IPASettings &settings, unsigned int hwRevision, + uint32_t supportedBlocks, const IPACameraSensorInfo &sensorInfo, const ControlInfoMap &sensorControls, ControlInfoMap *ipaControls) @@ -139,13 +144,13 @@ int IPARkISP1::init(const IPASettings &settings, unsigned int hwRevision, /* \todo Add support for other revisions */ switch (hwRevision) { case RKISP1_V10: - context_.hw = &ipaHwSettingsV10; + context_.hw = ipaHwSettingsV10; break; case RKISP1_V_IMX8MP: - context_.hw = &ipaHwSettingsIMX8MP; + context_.hw = ipaHwSettingsIMX8MP; break; case RKISP1_V12: - context_.hw = &ipaHwSettingsV12; + context_.hw = ipaHwSettingsV12; break; default: LOG(IPARkISP1, Error) @@ -153,6 +158,7 @@ int IPARkISP1::init(const IPASettings &settings, unsigned int hwRevision, << " is currently not supported"; return -ENODEV; } + context_.hw.supportedBlocks = supportedBlocks; LOG(IPARkISP1, Debug) << "Hardware revision is " << hwRevision; diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index a91a443aeaf7..ecd13831539f 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -100,7 +100,7 @@ public: PipelineHandlerRkISP1 *pipe(); const PipelineHandlerRkISP1 *pipe() const; - int loadIPA(unsigned int hwRevision); + int loadIPA(unsigned int hwRevision, uint32_t supportedBlocks); Stream mainPathStream_; Stream selfPathStream_; @@ -383,7 +383,7 @@ const PipelineHandlerRkISP1 *RkISP1CameraData::pipe() const return static_cast(Camera::Private::pipe()); } -int RkISP1CameraData::loadIPA(unsigned int hwRevision) +int RkISP1CameraData::loadIPA(unsigned int hwRevision, uint32_t supportedBlocks) { ipa_ = IPAManager::createIPA(pipe(), 1, 1); if (!ipa_) @@ -405,7 +405,8 @@ int RkISP1CameraData::loadIPA(unsigned int hwRevision) } ret = ipa_->init({ ipaTuningFile, sensor_->model() }, hwRevision, - sensorInfo, sensor_->controls(), &ipaControls_); + supportedBlocks, sensorInfo, sensor_->controls(), + &ipaControls_); if (ret < 0) { LOG(RkISP1, Error) << "IPA initialization failure"; return ret; @@ -1313,6 +1314,12 @@ int PipelineHandlerRkISP1::updateControls(RkISP1CameraData *data) return 0; } +/* + * By default we assume all the blocks that were included in the first + * extensible parameters series are available. That is the lower 20bits. + */ +const uint32_t kDefaultExtParamsBlocks = 0xfffff; + int PipelineHandlerRkISP1::createCamera(MediaEntity *sensor) { int ret; @@ -1350,7 +1357,21 @@ int PipelineHandlerRkISP1::createCamera(MediaEntity *sensor) isp_->frameStart.connect(data->delayedCtrls_.get(), &DelayedControls::applyControls); - ret = data->loadIPA(media_->hwRevision()); + uint32_t supportedBlocks = kDefaultExtParamsBlocks; + + auto &controls = param_->controls(); + if (controls.find(RKISP1_CID_SUPPORTED_PARAMS_BLOCKS) != controls.end()) { + auto list = param_->getControls({ { RKISP1_CID_SUPPORTED_PARAMS_BLOCKS } }); + if (!list.empty()) + supportedBlocks = static_cast( + list.get(RKISP1_CID_SUPPORTED_PARAMS_BLOCKS) + .get()); + } else { + LOG(RkISP1, Error) + << "Failed to query supported params blocks. Falling back to defaults."; + } + + ret = data->loadIPA(media_->hwRevision(), supportedBlocks); if (ret) return ret; From patchwork Fri Sep 19 09:40:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24424 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7EB32C3329 for ; Fri, 19 Sep 2025 09:41:43 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id EBD186B5E0; Fri, 19 Sep 2025 11:41:41 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="VOvjQDwZ"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id EBF026B5C8 for ; Fri, 19 Sep 2025 11:41:37 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 9BE43819; Fri, 19 Sep 2025 11:40:17 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274817; bh=NiZvUMrY2M4TQbAJAj4uMOM4cpMQWjAcK/qcd89IooM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VOvjQDwZafn2H1wyhgt7ZLD9FaJKGEuZ0NTWYlO00B/QiW/DqT8NyOUZ00XR3LxKL nvkNlMLkJM0o8fNWNeOqlV4Zf4irI/0hVJqbCb8+Ks8EkYjVCoXPCDQBT57V0Du5kI GgW2h3zPchx5Xt15Vnh0PukWi/0ndlYSKlSgbCcQ= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Daniel Scally , Paul ELder Subject: [PATCH v5 15/19] libipa: agc_mean_luminance: Introduce effectiveYTarget() accessor Date: Fri, 19 Sep 2025 11:40:30 +0200 Message-ID: <20250919094041.183031-16-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The upcoming WDR algorithm needs to know the effective y target (Which includes the current ExposureValue). Add an accessor for that. Signed-off-by: Stefan Klug Reviewed-by: Daniel Scally Reviewed-by: Paul ELder --- Changes in v4: - Fixed typos in documentation - Collected tags Changes in v3: - Added this patch --- src/ipa/libipa/agc_mean_luminance.cpp | 16 ++++++++++++++-- src/ipa/libipa/agc_mean_luminance.h | 2 ++ 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp index 41c7f72de8ed..45eba97516f3 100644 --- a/src/ipa/libipa/agc_mean_luminance.cpp +++ b/src/ipa/libipa/agc_mean_luminance.cpp @@ -458,8 +458,7 @@ void AgcMeanLuminance::setLimits(utils::Duration minExposureTime, */ double AgcMeanLuminance::estimateInitialGain() const { - double yTarget = std::min(relativeLuminanceTarget_ * exposureCompensation_, - kMaxRelativeLuminanceTarget); + double yTarget = effectiveYTarget(); double yGain = 1.0; /* @@ -521,6 +520,19 @@ double AgcMeanLuminance::constraintClampGain(uint32_t constraintModeIndex, return gain; } +/** + * \brief Get the currently effective y target + * + * This function returns the current y target including exposure compensation. + * + * \return The y target value + */ +double AgcMeanLuminance::effectiveYTarget() const +{ + return std::min(relativeLuminanceTarget_ * exposureCompensation_, + kMaxRelativeLuminanceTarget); +} + /** * \brief Apply a filter on the exposure value to limit the speed of changes * \param[in] exposureValue The target exposure from the AGC algorithm diff --git a/src/ipa/libipa/agc_mean_luminance.h b/src/ipa/libipa/agc_mean_luminance.h index fbb526f6ae8e..950b7b893754 100644 --- a/src/ipa/libipa/agc_mean_luminance.h +++ b/src/ipa/libipa/agc_mean_luminance.h @@ -72,6 +72,8 @@ public: calculateNewEv(uint32_t constraintModeIndex, uint32_t exposureModeIndex, const Histogram &yHist, utils::Duration effectiveExposureValue); + double effectiveYTarget() const; + void resetFrameCount() { frameCount_ = 0; From patchwork Fri Sep 19 09:40:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24425 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 957D4C3331 for ; Fri, 19 Sep 2025 09:41:45 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 675586B5C8; Fri, 19 Sep 2025 11:41:44 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="vRx0c5d0"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2561262C38 for ; Fri, 19 Sep 2025 11:41:41 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id E3B87C67; Fri, 19 Sep 2025 11:40:20 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274821; bh=8QESrQ2P6bkZ2N/UKPNrA4RaLN0gtxHRQyM5r/MAJM4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vRx0c5d06lrgaRXrSpBcOqfbUhCELlB54/wyQnd/SbFStDjr/pZH1nMPk+JOwYfnC YdecA9pstMq+zdDasiLoaMznuaA5Kb+upj3SSwLk1hNKX5jH2N8O++bR//2Ced34d8 Y2LIdwaMCNEWnbp50Nd3pc9JQnEsthZ1C0DQ7cM8= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Paul Elder , Kieran Bingham Subject: [PATCH v5 16/19] libipa: agc_mean_luminance: Add support for additional constraints Date: Fri, 19 Sep 2025 11:40:31 +0200 Message-ID: <20250919094041.183031-17-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add support for additional constraints added at runtime. This is for example useful for WDR use cases where you want to add an upper constraint to limit the amount of saturated pixels. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder Reviewed-by: Kieran Bingham --- Changes in v5: - Collected tag Changes in v4: - Added missing docs for the constraints parameter - Collected tag Changes in v3: - Added this patch --- src/ipa/ipu3/algorithms/agc.cpp | 2 +- src/ipa/libipa/agc_mean_luminance.cpp | 17 +++++++++++++---- src/ipa/libipa/agc_mean_luminance.h | 3 ++- src/ipa/mali-c55/algorithms/agc.cpp | 3 ++- src/ipa/rkisp1/algorithms/agc.cpp | 4 ++-- 5 files changed, 20 insertions(+), 9 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index da045640d569..b0d89541da85 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -117,7 +117,7 @@ int Agc::configure(IPAContext &context, /* \todo Run this again when FrameDurationLimits is passed in */ setLimits(minExposureTime_, maxExposureTime_, minAnalogueGain_, - maxAnalogueGain_); + maxAnalogueGain_, {}); resetFrameCount(); return 0; diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp index 45eba97516f3..64f36bd75dd2 100644 --- a/src/ipa/libipa/agc_mean_luminance.cpp +++ b/src/ipa/libipa/agc_mean_luminance.cpp @@ -7,6 +7,7 @@ #include "agc_mean_luminance.h" +#include #include #include @@ -408,16 +409,20 @@ int AgcMeanLuminance::parseTuningData(const YamlObject &tuningData) * \param[in] maxExposureTime Maximum ewposure time to allow * \param[in] minGain Minimum gain to allow * \param[in] maxGain Maximum gain to allow + * \param[in] constraints Additional constraints to apply * * This function calls \ref ExposureModeHelper::setLimits() for each * ExposureModeHelper that has been created for this class. */ void AgcMeanLuminance::setLimits(utils::Duration minExposureTime, utils::Duration maxExposureTime, - double minGain, double maxGain) + double minGain, double maxGain, + std::vector constraints) { for (auto &[id, helper] : exposureModeHelpers_) helper->setLimits(minExposureTime, maxExposureTime, minGain, maxGain); + + additionalConstraints_ = std::move(constraints); } /** @@ -495,8 +500,7 @@ double AgcMeanLuminance::constraintClampGain(uint32_t constraintModeIndex, const Histogram &hist, double gain) { - std::vector &constraints = constraintModes_[constraintModeIndex]; - for (const AgcConstraint &constraint : constraints) { + auto applyConstraint = [&gain, &hist](const AgcConstraint &constraint) { double newGain = constraint.yTarget * hist.bins() / hist.interQuantileMean(constraint.qLo, constraint.qHi); @@ -515,7 +519,12 @@ double AgcMeanLuminance::constraintClampGain(uint32_t constraintModeIndex, << newGain; gain = newGain; } - } + }; + + std::vector &constraints = constraintModes_[constraintModeIndex]; + std::for_each(constraints.begin(), constraints.end(), applyConstraint); + + std::for_each(additionalConstraints_.begin(), additionalConstraints_.end(), applyConstraint); return gain; } diff --git a/src/ipa/libipa/agc_mean_luminance.h b/src/ipa/libipa/agc_mean_luminance.h index 950b7b893754..d7ec548e3e58 100644 --- a/src/ipa/libipa/agc_mean_luminance.h +++ b/src/ipa/libipa/agc_mean_luminance.h @@ -51,7 +51,7 @@ public: } void setLimits(utils::Duration minExposureTime, utils::Duration maxExposureTime, - double minGain, double maxGain); + double minGain, double maxGain, std::vector constraints); std::map> constraintModes() { @@ -97,6 +97,7 @@ private: utils::Duration filteredExposure_; double relativeLuminanceTarget_; + std::vector additionalConstraints_; std::map> constraintModes_; std::map> exposureModeHelpers_; ControlInfoMap::Map controls_; diff --git a/src/ipa/mali-c55/algorithms/agc.cpp b/src/ipa/mali-c55/algorithms/agc.cpp index 88f8664c9823..f60fddac3f04 100644 --- a/src/ipa/mali-c55/algorithms/agc.cpp +++ b/src/ipa/mali-c55/algorithms/agc.cpp @@ -173,7 +173,8 @@ int Agc::configure(IPAContext &context, setLimits(context.configuration.agc.minShutterSpeed, context.configuration.agc.maxShutterSpeed, context.configuration.agc.minAnalogueGain, - context.configuration.agc.maxAnalogueGain); + context.configuration.agc.maxAnalogueGain, + {}); resetFrameCount(); diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index d0337fd0027f..d1b6bb7196f4 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -206,7 +206,7 @@ int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) setLimits(context.configuration.sensor.minExposureTime, context.configuration.sensor.maxExposureTime, context.configuration.sensor.minAnalogueGain, - context.configuration.sensor.maxAnalogueGain); + context.configuration.sensor.maxAnalogueGain, {}); resetFrameCount(); @@ -590,7 +590,7 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, maxAnalogueGain = frameContext.agc.gain; } - setLimits(minExposureTime, maxExposureTime, minAnalogueGain, maxAnalogueGain); + setLimits(minExposureTime, maxExposureTime, minAnalogueGain, maxAnalogueGain, {}); /* * The Agc algorithm needs to know the effective exposure value that was From patchwork Fri Sep 19 09:40:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24426 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 93192C328C for ; Fri, 19 Sep 2025 09:41:49 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E6F836B5F1; Fri, 19 Sep 2025 11:41:48 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="R2pG337H"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DDA0E6B59A for ; Fri, 19 Sep 2025 11:41:43 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id AF555842; Fri, 19 Sep 2025 11:40:23 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274823; bh=Hf8pcymcjtAho++wmGoIxgQhaksSfOG8IgrUvraKMos=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=R2pG337HyQSLGZflAmzw2gKv+UbQ+WVCD4ArMWoYwOBgu5YSfQImzJKgdG5fxUtkz 8jsOMsE0Nz0WSAHJ81a6FqcF8YQwedqGUhP9fuYwdqmyt8ab7EqI0SumxGpiW7DjTT aT+NFGs2dK/T3e0nrfM6rK+m7CgHRvJauqIOmZCU= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Daniel Scally , Paul Elder Subject: [PATCH v5 17/19] rkisp1: agc: Agc add yTarget to frame context Date: Fri, 19 Sep 2025 11:40:32 +0200 Message-ID: <20250919094041.183031-18-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The upcoming WDR algorithm needs access to the effective yTarget value for its calculations. Add it to the frame context. Signed-off-by: Stefan Klug Reviewed-by: Daniel Scally Reviewed-by: Paul Elder --- Changes in v4: - Collected tags --- src/ipa/rkisp1/algorithms/agc.cpp | 5 +++++ src/ipa/rkisp1/ipa_context.h | 2 ++ 2 files changed, 7 insertions(+) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index d1b6bb7196f4..046a1ac9caa2 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -208,6 +208,8 @@ int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) context.configuration.sensor.minAnalogueGain, context.configuration.sensor.maxAnalogueGain, {}); + context.activeState.agc.automatic.yTarget = effectiveYTarget(); + resetFrameCount(); return 0; @@ -375,6 +377,8 @@ void Agc::prepare(IPAContext &context, const uint32_t frame, frameContext.compress.gain = frameContext.agc.quantizationGain; } + frameContext.agc.yTarget = context.activeState.agc.automatic.yTarget; + if (frame > 0 && !frameContext.agc.updateMetering) return; @@ -627,6 +631,7 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, activeState.agc.automatic.exposure = newExposureTime / lineDuration; activeState.agc.automatic.gain = aGain; activeState.agc.automatic.quantizationGain = qGain; + activeState.agc.automatic.yTarget = effectiveYTarget(); /* * Expand the target frame duration so that we do not run faster than * the minimum frame duration when we have short exposures. diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index 67427c01ac68..113b90428008 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -83,6 +83,7 @@ struct IPAActiveState { uint32_t exposure; double gain; double quantizationGain; + double yTarget; } automatic; bool autoExposureEnabled; @@ -139,6 +140,7 @@ struct IPAFrameContext : public FrameContext { double exposureValue; double quantizationGain; uint32_t vblank; + double yTarget; bool autoExposureEnabled; bool autoGainEnabled; controls::AeConstraintModeEnum constraintMode; From patchwork Fri Sep 19 09:40:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24427 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 722F5C3329 for ; Fri, 19 Sep 2025 09:41:51 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AB4F76B5F8; Fri, 19 Sep 2025 11:41:50 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="mewOF5EO"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DCF4F6B5E3 for ; Fri, 19 Sep 2025 11:41:46 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 91475D3E; Fri, 19 Sep 2025 11:40:26 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274826; bh=Fnt7c+vv1pKsVyayOa3W3inpEUFox78T7Zi+5vVUDys=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mewOF5EO2BRBGGSTYMoDbwYPLBp5wUNmjif75rA8mFFMFqalmev/tX3AEtaQ5ztzw vknjH2CFEKXV6f9DEOtYtaxfVmvUU5gXEr5HKreNGhrwIlaQPyRHTg11SBOnaacxlf 9QHeUgA7RBzBe9klmN2ZiYfR3qK+NE4/oKeA65cE= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Paul Elder , Kieran Bingham Subject: [PATCH v5 18/19] ipa: rkisp1: Add WDR algorithm Date: Fri, 19 Sep 2025 11:40:33 +0200 Message-ID: <20250919094041.183031-19-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a WDR algorithm to do global tone mapping. Global tone mapping is used to increase the perceived dynamic range of an image. The typical effect is that in areas that are normally overexposed, additional structure becomes visible. The overall idea is that the algorithm applies an exposure value correction to underexpose the image to the point where only a small number of saturated pixels is left. This artificial underexposure is then mitigated by applying a tone mapping curve. This algorithm implements 4 tone mapping strategies: - Linear - Power - Exponential - Histogram equalization Signed-off-by: Stefan Klug Reviewed-by: Paul Elder Reviewed-by: Kieran Bingham --- Changes in v5: - Moved Wdr controls from draft to core - Fixed a few style issue in comments - Corrected the copyright date in header - Collected tag Changes in v4: - Properly initialize activeState.wdr.gain - Fixed some typos and improved wording Changes in v3: - Removed the need for a separate regulation loop, by not relying on an added ExposureValue but by applying a constraint to AGC regulation and deducing the required WDR gain from the histogram. This makes the structure easier and hopefully less prone to oscillations. - Dropped debug metadata as this needs more discussions and is not required for the algorithm to work. - Dropped minExposureValue as it is not used anymore. - Added a damping on the gain applied by the WDR curve - Moved strength into activeState so that it is reset on configure() Changes in v2: - Fixed default value for min bright pixels - Added check for supported params type - Reset PID controller - Various fixes from Pauls review --- src/ipa/rkisp1/algorithms/agc.cpp | 7 +- src/ipa/rkisp1/algorithms/meson.build | 1 + src/ipa/rkisp1/algorithms/wdr.cpp | 493 ++++++++++++++++++++++++++ src/ipa/rkisp1/algorithms/wdr.h | 58 +++ src/ipa/rkisp1/ipa_context.h | 14 + src/ipa/rkisp1/params.cpp | 1 + src/ipa/rkisp1/params.h | 2 + src/libcamera/control_ids_core.yaml | 62 ++++ 8 files changed, 637 insertions(+), 1 deletion(-) create mode 100644 src/ipa/rkisp1/algorithms/wdr.cpp create mode 100644 src/ipa/rkisp1/algorithms/wdr.h diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 046a1ac9caa2..f5a3c917cb69 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -594,7 +594,12 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, maxAnalogueGain = frameContext.agc.gain; } - setLimits(minExposureTime, maxExposureTime, minAnalogueGain, maxAnalogueGain, {}); + std::vector additionalConstraints; + if (context.activeState.wdr.mode != controls::WdrOff) + additionalConstraints.push_back(context.activeState.wdr.constraint); + + setLimits(minExposureTime, maxExposureTime, minAnalogueGain, maxAnalogueGain, + std::move(additionalConstraints)); /* * The Agc algorithm needs to know the effective exposure value that was diff --git a/src/ipa/rkisp1/algorithms/meson.build b/src/ipa/rkisp1/algorithms/meson.build index 2e42a80cf99d..d329dbfb432d 100644 --- a/src/ipa/rkisp1/algorithms/meson.build +++ b/src/ipa/rkisp1/algorithms/meson.build @@ -14,4 +14,5 @@ rkisp1_ipa_algorithms = files([ 'gsl.cpp', 'lsc.cpp', 'lux.cpp', + 'wdr.cpp', ]) diff --git a/src/ipa/rkisp1/algorithms/wdr.cpp b/src/ipa/rkisp1/algorithms/wdr.cpp new file mode 100644 index 000000000000..45144913dcd8 --- /dev/null +++ b/src/ipa/rkisp1/algorithms/wdr.cpp @@ -0,0 +1,493 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2025, Ideas On Board + * + * RkISP1 Wide Dynamic Range control + */ + +#include "wdr.h" + +#include +#include + +#include "libcamera/internal/yaml_parser.h" + +#include +#include +#include + +#include "linux/rkisp1-config.h" + +/** + * \file wdr.h + */ + +namespace libcamera { + +namespace ipa::rkisp1::algorithms { + +/** + * \class WideDynamicRange + * \brief RkISP1 Wide Dynamic Range algorithm + * + * This algorithm implements automatic global tone mapping for the RkISP1. + * Global tone mapping is done by the GWDR hardware block and applies + * a global tone mapping curve to the image to increase the perceived dynamic + * range. Imagine an indoor scene with bright outside visible through the + * windows. With normal exposure settings, the windows will be completely + * saturated and no structure (sky/clouds) will be visible because the AEGC has + * to increase overall exposure to reach a certain level of mean brightness. In + * WDR mode, the algorithm will artifically reduce the exposure time so that the + * texture and colours become visible in the formerly saturated areas. Then the + * global tone mapping curve is applied to mitigate the loss of brightness. + * + * Calculating that tone mapping curve is the most difficult part. This + * algorithm implements four tone mapping strategies: + * - Linear: The tone mapping curve is a combination of two linear functions + * with one kneepoint + * - Power: The tone mapping curve follows a power function + * - Exponential: The tone mapping curve follows an exponential function + * - HistogramEqualization: The tone mapping curve tries to equalize the + * histogram + * + * The overall strategy is the same in all cases: Add a constraint to the AEGC + * regulation so that the number of nearly saturated pixels goes below a given + * threshold (default 2%). This threshold can either be specified in the tuning + * file or set via the WdrMaxBrightPixels control. + * + * The global tone mapping curve is then calculated so that it accounts for the + * reduction of brightness due to the exposure constraint. We'll call this the + * WDR-gain. As the result of tone mapping is very difficult to quantize and is + * by definition a lossy process there is not a single "correct" solution on how + * this curve should look like. + * + * The approach taken here is based on a simple linear model. Consider a pixel + * that was originally 50% grey. It will have its exposure pushed down by the + * WDR's initial exposure compensation. This value then needs to be pushed back + * up by the tone mapping curve so that it is 50% grey again. This point serves + * as our kneepoint. To get to this kneepoint, this pixel and all darker pixels + * (to the left of the kneepoint on the tone mapping curve) will simply have the + * exposure compensation undone by WDR-gain. This cancels out the + * original exposure compensation, which was 1/WDR-gain. The remaining + * brigher pixels (to the right of the kneepoint on the tone mapping curve) will + * be compressed. The WdrStrength control adjusts the gain of the left part of + * the tone mapping curve. + * + * In the Power and Exponential modes, the curves are calculated so that they + * pass through that kneepoint. + * + * The histogram equalization mode tries to equalize the histogram of the + * image and acts independently of the calculated exposure value. + * + * \code{.unparsed} + * algorithms: + * - WideDynamicRange: + * ExposureConstraint: + * MaxBrightPixels: 0.02 + * yTarget: 0.95 + * \endcode + */ + +LOG_DEFINE_CATEGORY(RkISP1Wdr) + +static constexpr unsigned int kTonecurveXIntervals = RKISP1_CIF_ISP_WDR_CURVE_NUM_INTERV; + +/* + * Increasing interval sizes. The intervals are crafted so that they sum + * up to 4096. This results in better fitting curves than the constant intervals + * (all entries are 4) + */ +static constexpr std::array kLoglikeIntervals = { + { 0, 0, 0, 0, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 6, 6 } +}; + +WideDynamicRange::WideDynamicRange() +{ +} + +/** + * \copydoc libcamera::ipa::Algorithm::init + */ +int WideDynamicRange::init([[maybe_unused]] IPAContext &context, + [[maybe_unused]] const YamlObject &tuningData) +{ + if (!(context.hw.supportedBlocks & 1 << RKISP1_EXT_PARAMS_BLOCK_TYPE_WDR)) { + LOG(RkISP1Wdr, Error) + << "Wide Dynamic Range not supported by the hardware or kernel."; + return -ENOTSUP; + } + + toneCurveIntervalValues_ = kLoglikeIntervals; + + /* Calculate a list of normed x values */ + toneCurveX_[0] = 0.0; + int lastValue = 0; + for (unsigned int i = 1; i < toneCurveX_.size(); i++) { + lastValue += std::pow(2, toneCurveIntervalValues_[i - 1] + 3); + lastValue = std::min(lastValue, 4096); + toneCurveX_[i] = lastValue / 4096.0; + } + + exposureConstraintMaxBrightPixels_ = 0.02; + exposureConstraintY_ = 0.95; + + const auto &constraint = tuningData["ExposureConstraint"]; + if (!constraint.isDictionary()) { + LOG(RkISP1Wdr, Warning) + << "ExposureConstraint not found in tuning data." + "Using default values MaxBrightPixels: " + << exposureConstraintMaxBrightPixels_ + << " yTarget: " << exposureConstraintY_; + } else { + exposureConstraintMaxBrightPixels_ = + constraint["MaxBrightPixels"] + .get() + .value_or(exposureConstraintMaxBrightPixels_); + exposureConstraintY_ = + constraint["yTarget"] + .get() + .value_or(exposureConstraintY_); + } + + context.ctrlMap[&controls::WdrMode] = + ControlInfo(controls::WdrModeValues, controls::WdrOff); + context.ctrlMap[&controls::WdrStrength] = + ControlInfo(0.0f, 2.0f, 1.0f); + context.ctrlMap[&controls::WdrMaxBrightPixels] = + ControlInfo(0.0f, 1.0f, static_cast(exposureConstraintMaxBrightPixels_)); + + applyCompensationLinear(1.0, 0.0); + + return 0; +} + +/** + * \copydoc libcamera::ipa::Algorithm::configure + */ +int WideDynamicRange::configure(IPAContext &context, + [[maybe_unused]] const IPACameraSensorInfo &configInfo) +{ + context.activeState.wdr.mode = controls::WdrOff; + context.activeState.wdr.gain = 1.0; + context.activeState.wdr.strength = 1.0; + auto &constraint = context.activeState.wdr.constraint; + constraint.bound = AgcMeanLuminance::AgcConstraint::Bound::Upper; + constraint.qHi = 1.0; + constraint.qLo = 1.0 - exposureConstraintMaxBrightPixels_; + constraint.yTarget = exposureConstraintY_; + return 0; +} + +void WideDynamicRange::applyHistogramEqualization(double strength) +{ + if (hist_.empty()) + return; + + /* + * Apply a factor on strength, so that it roughly matches the optical + * impression that is produced by the other algorithms. The goal is that + * the user can switch algorithms for different looks but similar + * "strength". + */ + strength *= 0.65; + + /* + * In a fully equalized histogram, all bins have the same value. Try + * to equalize the histogram by applying a gain or damping depending on + * the distance of the actual bin value from that norm. + */ + std::vector gains; + gains.resize(hist_.size()); + double sum = 0; + double norm = 1.0 / (gains.size()); + for (unsigned i = 0; i < hist_.size(); i++) { + double diff = 1.0 + strength * (hist_[i] - norm) / norm; + gains[i] = diff; + sum += diff; + } + + /* Never amplify the last entry. */ + gains.back() = std::max(gains.back(), 1.0); + + double scale = gains.size() / sum; + for (auto &v : gains) + v *= scale; + + Pwl pwl; + double step = 1.0 / gains.size(); + double lastX = 0; + double lastY = 0; + + pwl.append(lastX, lastY); + for (unsigned int i = 0; i < gains.size() - 1; i++) { + lastY += gains[i] * step; + lastX += step; + pwl.append(lastX, lastY); + } + pwl.append(1.0, 1.0); + + for (unsigned int i = 0; i < toneCurveX_.size(); i++) + toneCurveY_[i] = pwl.eval(toneCurveX_[i]); +} + +Vector WideDynamicRange::kneePoint(double gain, double strength) +{ + gain = std::pow(gain, strength); + double y = 0.5; + double x = y / gain; + + return { { x, y } }; +} + +void WideDynamicRange::applyCompensationLinear(double gain, double strength) +{ + auto kp = kneePoint(gain, strength); + double g1 = kp.y() / kp.x(); + double g2 = (kp.y() - 1) / (kp.x() - 1); + + for (unsigned int i = 0; i < toneCurveX_.size(); i++) { + double x = toneCurveX_[i]; + double y; + if (x <= kp.x()) { + y = g1 * x; + } else { + y = g2 * x + 1 - g2; + } + toneCurveY_[i] = y; + } +} + +void WideDynamicRange::applyCompensationPower(double gain, double strength) +{ + double e = 1.0; + if (strength > 1e-6) { + auto kp = kneePoint(gain, strength); + /* Calculate an exponent to go through the knee point. */ + e = log(kp.y()) / log(kp.x()); + } + + /* + * The power function tends to be extremely steep at the beginning. This + * leads to noise and image artifacts in the dark areas. To mitigate + * that, we add a short linear section at the beginning of the curve. + * The connection between linear and power is the point where the linear + * section reaches the y level yLin. The power curve is then scaled so + * that it starts at the connection point with the steepness it would + * have at y=yLin but still goes through 1,1 + */ + double yLin = 0.1; + /* x position of the connection point */ + double xb = yLin / gain; + /* x offset for the scaled power function */ + double q = xb - std::exp(std::log(yLin) / e); + + for (unsigned int i = 0; i < toneCurveX_.size(); i++) { + double x = toneCurveX_[i]; + if (x < xb) { + toneCurveY_[i] = x * gain; + } else { + x = (x - q) / (1 - q); + toneCurveY_[i] = std::pow(x, e); + } + } +} + +void WideDynamicRange::applyCompensationExponential(double gain, double strength) +{ + double k = 0.1; + auto kp = kneePoint(gain, strength); + double kx = kp.x(); + double ky = kp.y(); + + if (kx > ky) { + LOG(RkISP1Wdr, Warning) << "Invalid knee point: " << kp; + kx = ky; + } + + /* + * The exponential curve is based on the function proposed by Glozman + * et al. in + * S. Glozman, T. Kats, and O. Yadid-Pecht, "Exponent Operator Based + * Tone Mapping Algorithm for Color Wide Dynamic Range Images," 2011. + * + * That function uses a k factor as parameter for the WDR compression + * curve: + * k=0: maximum compression + * k=infinity: linear curve + * + * To calculate a k factor that results in a curve that passes through + * the kneepoint, the equation needs to be solved for k after inserting + * the kneepoint. This can be formulated as search for a zero point. + * Unfortunately there is no closed solution for that transformation. + * Using newton's method to approximate the value is numerically + * unstable. + * + * Luckily the function only crosses the x axis once and for the set of + * possible kneepoints, a negative and a positive point can be guessed. + * The approximation is then implemented using bisection. + */ + if (std::abs(kx - ky) < 0.001) { + k = 1e8; + } else { + double kl = 0.0001; + double kh = 1000; + + auto fk = [=](double v) { + return std::exp(-kx / v) - + ky * std::exp(-1.0 / v) + ky - 1.0; + }; + + ASSERT(fk(kl) < 0); + ASSERT(fk(kh) > 0); + + k = kh / 10.0; + while (fk(k) > 0) { + kh = k; + k /= 10.0; + } + + do { + k = (kl + kh) / 2; + if (fk(k) < 0) + kl = k; + else + kh = k; + } while (std::abs(kh - kl) > 1e-3); + } + + double a = 1.0 / (1.0 - std::exp(-1.0 / k)); + for (unsigned int i = 0; i < toneCurveX_.size(); i++) + toneCurveY_[i] = a * (1.0 - std::exp(-toneCurveX_[i] / k)); +} + +/** + * \copydoc libcamera::ipa::Algorithm::queueRequest + */ +void WideDynamicRange::queueRequest([[maybe_unused]] IPAContext &context, + [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, + const ControlList &controls) +{ + auto &activeState = context.activeState; + + const auto &mode = controls.get(controls::WdrMode); + if (mode) + activeState.wdr.mode = static_cast(*mode); + + const auto &brightPixels = controls.get(controls::WdrMaxBrightPixels); + if (brightPixels) + activeState.wdr.constraint.qLo = 1.0 - *brightPixels; + + const auto &strength = controls.get(controls::WdrStrength); + if (strength) + activeState.wdr.strength = *strength; + + frameContext.wdr.mode = activeState.wdr.mode; + frameContext.wdr.strength = activeState.wdr.strength; +} + +/** + * \copydoc libcamera::ipa::Algorithm::prepare + */ +void WideDynamicRange::prepare(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, + RkISP1Params *params) +{ + if (!params) { + LOG(RkISP1Wdr, Warning) << "Params is null"; + return; + } + + auto mode = frameContext.wdr.mode; + + auto config = params->block(); + config.setEnabled(mode != controls::WdrOff); + + /* Calculate how much EV we need to compensate with the WDR curve. */ + double gain = context.activeState.wdr.gain; + frameContext.wdr.gain = gain; + + if (mode == controls::WdrOff) { + applyCompensationLinear(1.0, 0.0); + } else if (mode == controls::WdrLinear) { + applyCompensationLinear(gain, frameContext.wdr.strength); + } else if (mode == controls::WdrPower) { + applyCompensationPower(gain, frameContext.wdr.strength); + } else if (mode == controls::WdrExponential) { + applyCompensationExponential(gain, frameContext.wdr.strength); + } else if (mode == controls::WdrHistogramEqualization) { + applyHistogramEqualization(frameContext.wdr.strength); + } + + /* Reset value */ + config->dmin_strength = 0x10; + config->dmin_thresh = 0; + + for (unsigned int i = 0; i < kTonecurveXIntervals; i++) { + int v = toneCurveIntervalValues_[i]; + config->tone_curve.dY[i / 8] |= (v & 0x07) << ((i % 8) * 4); + } + + /* + * Fix the curve to adhere to the hardware constraints. Don't apply a + * constraint on the first element, which is most likely zero anyways. + */ + int lastY = toneCurveY_[0] * 4096.0; + for (unsigned int i = 0; i < toneCurveX_.size(); i++) { + int diff = static_cast(toneCurveY_[i] * 4096.0) - lastY; + diff = std::clamp(diff, -2048, 2048); + lastY = lastY + diff; + config->tone_curve.ym[i] = lastY; + } +} + +void WideDynamicRange::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, + const rkisp1_stat_buffer *stats, + ControlList &metadata) +{ + if (!stats || !(stats->meas_type & RKISP1_CIF_ISP_STAT_HIST)) { + LOG(RkISP1Wdr, Warning) << "No histogram data in statistics"; + return; + } + + const rkisp1_cif_isp_stat *params = &stats->params; + auto mode = frameContext.wdr.mode; + + metadata.set(controls::WdrMode, mode); + + Histogram cumHist({ params->hist.hist_bins, context.hw.numHistogramBins }, + [](uint32_t x) { return x >> 4; }); + + /* Calculate the gain needed to reach the requested yTarget. */ + double value = cumHist.interQuantileMean(0, 1.0) / cumHist.bins(); + double gain = context.activeState.agc.automatic.yTarget / value; + gain = std::max(gain, 1.0); + + double speed = 0.2; + gain = gain * speed + context.activeState.wdr.gain * (1.0 - speed); + + context.activeState.wdr.gain = gain; + + std::vector hist; + double sum = 0; + for (unsigned i = 0; i < context.hw.numHistogramBins; i++) { + double v = params->hist.hist_bins[i] >> 4; + hist.push_back(v); + sum += v; + } + + /* Scale so that the entries sum up to 1. */ + double scale = 1.0 / sum; + for (auto &v : hist) + v *= scale; + hist_.swap(hist); +} + +REGISTER_IPA_ALGORITHM(WideDynamicRange, "WideDynamicRange") + +} /* namespace ipa::rkisp1::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/rkisp1/algorithms/wdr.h b/src/ipa/rkisp1/algorithms/wdr.h new file mode 100644 index 000000000000..46f7cdeea69d --- /dev/null +++ b/src/ipa/rkisp1/algorithms/wdr.h @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2025, Ideas On Board + * + * RkISP1 Wide Dynamic Range control + */ + +#pragma once + +#include + +#include "linux/rkisp1-config.h" + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::rkisp1::algorithms { + +class WideDynamicRange : public Algorithm +{ +public: + WideDynamicRange(); + ~WideDynamicRange() = default; + + int init(IPAContext &context, const YamlObject &tuningData) override; + int configure(IPAContext &context, const IPACameraSensorInfo &configInfo) override; + + void queueRequest(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const ControlList &controls) override; + void prepare(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + RkISP1Params *params) override; + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const rkisp1_stat_buffer *stats, + ControlList &metadata) override; + +private: + Vector kneePoint(double gain, double strength); + void applyCompensationLinear(double gain, double strength); + void applyCompensationPower(double gain, double strength); + void applyCompensationExponential(double gain, double strength); + void applyHistogramEqualization(double strength); + + double exposureConstraintMaxBrightPixels_; + double exposureConstraintY_; + + std::vector hist_; + + std::array toneCurveIntervalValues_; + std::array toneCurveX_; + std::array toneCurveY_; +}; + +} /* namespace ipa::rkisp1::algorithms */ +} /* namespace libcamera */ diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index 113b90428008..f85a130d9c23 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -26,6 +26,7 @@ #include #include +#include "libipa/agc_mean_luminance.h" namespace libcamera { @@ -131,6 +132,13 @@ struct IPAActiveState { struct { double gamma; } goc; + + struct { + controls::WdrModeEnum mode; + AgcMeanLuminance::AgcConstraint constraint; + double gain; + double strength; + } wdr; }; struct IPAFrameContext : public FrameContext { @@ -200,6 +208,12 @@ struct IPAFrameContext : public FrameContext { struct { double lux; } lux; + + struct { + controls::WdrModeEnum mode; + double strength; + double gain; + } wdr; }; struct IPAContext { diff --git a/src/ipa/rkisp1/params.cpp b/src/ipa/rkisp1/params.cpp index 4c0b051ce65d..5edb36c91b87 100644 --- a/src/ipa/rkisp1/params.cpp +++ b/src/ipa/rkisp1/params.cpp @@ -74,6 +74,7 @@ const std::map kBlockTypeInfo = { RKISP1_BLOCK_TYPE_ENTRY_EXT(CompandBls, COMPAND_BLS, compand_bls), RKISP1_BLOCK_TYPE_ENTRY_EXT(CompandExpand, COMPAND_EXPAND, compand_curve), RKISP1_BLOCK_TYPE_ENTRY_EXT(CompandCompress, COMPAND_COMPRESS, compand_curve), + RKISP1_BLOCK_TYPE_ENTRY_EXT(Wdr, WDR, wdr), }; } /* namespace */ diff --git a/src/ipa/rkisp1/params.h b/src/ipa/rkisp1/params.h index 40450e34497a..2e60528d102e 100644 --- a/src/ipa/rkisp1/params.h +++ b/src/ipa/rkisp1/params.h @@ -40,6 +40,7 @@ enum class BlockType { CompandBls, CompandExpand, CompandCompress, + Wdr, }; namespace details { @@ -74,6 +75,7 @@ RKISP1_DEFINE_BLOCK_TYPE(Afc, afc) RKISP1_DEFINE_BLOCK_TYPE(CompandBls, compand_bls) RKISP1_DEFINE_BLOCK_TYPE(CompandExpand, compand_curve) RKISP1_DEFINE_BLOCK_TYPE(CompandCompress, compand_curve) +RKISP1_DEFINE_BLOCK_TYPE(Wdr, wdr) } /* namespace details */ diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml index eec4b4f937ee..f781865859ac 100644 --- a/src/libcamera/control_ids_core.yaml +++ b/src/libcamera/control_ids_core.yaml @@ -1283,5 +1283,67 @@ controls: \sa SensorTimestamp The FrameWallClock control can only be returned in metadata. + - WdrMode: + type: int32_t + direction: inout + description: | + Set the WDR mode. + + The WDR mode is used to select the algorithm used for global tone + mapping. It will automatically reduce the exposure time of the sensor + so that there are only a small number of saturated pixels in the image. + The algorithm then compensates for the loss of brightness by applying a + global tone mapping curve to the image. + enum: + - name: WdrOff + value: 0 + description: Wdr is disabled. + - name: WdrLinear + value: 1 + description: + Apply a linear global tone mapping curve. + + A curve with two linear sections is applied. This produces good + results at the expense of a slightly artificial look. + - name: WdrPower + value: 2 + description: | + Apply a power global tone mapping curve. + + This curve has high gain values on the dark areas of an image and + high compression values on the bright area. It therefore tends to + produce noticeable noise artifacts. + - name: WdrExponential + value: 3 + description: | + Apply an exponential global tone mapping curve. + + This curve has lower gain values in dark areas compared to the power + curve but produces a more natural look compared to the linear curve. + It is therefore the best choice for most scenes. + - name: WdrHistogramEqualization + value: 4 + description: | + Apply histogram equalization. + + This curve preserves most of the information of the image at the + expense of a very artificial look. It is therefore best suited for + technical analysis. + - WdrStrength: + type: float + direction: in + description: | + Specify the strength of the wdr algorithm. The exact meaning of this + value is specific to the algorithm in use. Usually a value of 0 means no + global tone mapping is applied. A values of 1 is the default value and + the correct value for most scenes. A value above 1 increases the global + tone mapping effect and can lead to unrealistic image effects. + - WdrMaxBrightPixels: + type: float + direction: in + description: | + Percentage of allowed (nearly) saturated pixels. The WDR algorithm + reduces the WdrExposureValue until the amount of pixels that are close + to saturation is lower than this value. ... From patchwork Fri Sep 19 09:40:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 24428 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5AFF0C3331 for ; Fri, 19 Sep 2025 09:41:54 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E3A706B5F3; Fri, 19 Sep 2025 11:41:53 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="LfQZL2QF"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8665B6B5F3 for ; Fri, 19 Sep 2025 11:41:49 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:4d54:eab8:98ca:163b]) by perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 5998EEFE; Fri, 19 Sep 2025 11:40:29 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1758274829; bh=g12oOSHrLQEB28SlVnbv2bihdBCrBjTQBNEkNZ/tE3k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LfQZL2QF7NY5v1OvDjU8JLkHDItIG+aLPvkGqDHN3wcUvMa4fy+TUPuGlEG0/c8b7 +QZzunRsv6Uag06lYdhEVeAIDSAT4bU9POfw1h/NZwW8mrUFgdv1EY25n4FcL+gkdW kgZuvC4TsgII1kqKdbqC8Y7lcNulCITygYZ4mQss= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Paul Elder , Kieran Bingham Subject: [PATCH v5 19/19] tuning: rksip1: Add a static WideDynamicRange entry Date: Fri, 19 Sep 2025 11:40:34 +0200 Message-ID: <20250919094041.183031-20-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250919094041.183031-1-stefan.klug@ideasonboard.com> References: <20250919094041.183031-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a static WideDynamicRange entry that gets added by default. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder Reviewed-by: Kieran Bingham --- Changes in v3: - Collected tag - Removed MinExposureValue as it is not needed by the algorithm anymore --- utils/tuning/rkisp1.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/utils/tuning/rkisp1.py b/utils/tuning/rkisp1.py index 179d920c05df..0526449558fe 100755 --- a/utils/tuning/rkisp1.py +++ b/utils/tuning/rkisp1.py @@ -48,16 +48,18 @@ lsc = LSCRkISP1(debug=[lt.Debug.Plot], # values. This can also be a custom function. smoothing_function=lt.smoothing.MedianBlur(3),) lux = LuxRkISP1(debug=[lt.Debug.Plot]) +wdr = StaticModule('WideDynamicRange', + {'ExposureConstraint': {'MaxBrightPixels': 0.02, 'yTarget': 0.95}}) tuner = lt.Tuner('RkISP1') -tuner.add([agc, awb, blc, ccm, color_processing, filter, gamma_out, lsc, lux, compress]) +tuner.add([agc, awb, blc, ccm, color_processing, filter, gamma_out, lsc, lux, wdr, compress]) tuner.set_input_parser(YamlParser()) tuner.set_output_formatter(YamlOutput()) # Bayesian AWB uses the lux value, so insert the lux algorithm before AWB. # Compress is parameterized by others, so add it at the end. tuner.set_output_order([agc, lux, awb, blc, ccm, color_processing, - filter, gamma_out, lsc, compress]) + filter, gamma_out, lsc, wdr, compress]) if __name__ == '__main__': sys.exit(tuner.run(sys.argv))