From patchwork Thu Jan 23 14:07:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Scally X-Patchwork-Id: 22635 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 59164BDE6B for ; Thu, 23 Jan 2025 14:07:52 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 799C468564; Thu, 23 Jan 2025 15:07:49 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="pwV60fLh"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E047A61879 for ; Thu, 23 Jan 2025 15:07:45 +0100 (CET) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 11B15D1F; Thu, 23 Jan 2025 15:06:42 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1737641202; bh=+1sktMH3NyNzi9oYk3qIY3hescA5XHgRgeofWToMEyI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pwV60fLh/toelyfFEB271B4UuvB4WRRqXEE3w10tghVuevxtFpxYnZbIRQA2J4FdU tgwJQeVVMssMgL9Nt/7DuhWvTy/enk4HqA1eMpLOFkU5e4H8YO6U0UYSceQ/r41diM fzMMLQeU406BrO51HOmsnctNWb8Ed4EPoDjctwpw= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally , Laurent Pinchart , Paul Elder Subject: [PATCH v2 1/3] ipa: ipu3: Add skeleton Agc::queueRequest() function Date: Thu, 23 Jan 2025 14:07:25 +0000 Message-Id: <20250123140727.458567-2-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250123140727.458567-1-dan.scally@ideasonboard.com> References: <20250123140727.458567-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The IPU3 IPA's AGC algorithm currently does not implement a queueRequest() function. We will shortly need to use it, but to avoid lots of noise in the future commit that adds a function call in to that function add it as a skeleton here. Signed-off-by: Daniel Scally Reviewed-by: Laurent Pinchart Reviewed-by: Paul Elder --- Changes in v2: - None src/ipa/ipu3/algorithms/agc.cpp | 10 ++++++++++ src/ipa/ipu3/algorithms/agc.h | 3 +++ 2 files changed, 13 insertions(+) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 39d0aebb..383b046c 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -123,6 +123,16 @@ int Agc::configure(IPAContext &context, return 0; } +/** + * \copydoc libcamera::ipa::Algorithm::queueRequest + */ +void Agc::queueRequest([[maybe_unused]] typename Module::Context &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] typename Module::FrameContext &frameContext, + [[maybe_unused]] const ControlList &controls) +{ +} + Histogram Agc::parseStatistics(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid) { diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 890c271b..c82b7acb 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -32,6 +32,9 @@ public: int init(IPAContext &context, const YamlObject &tuningData) override; int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; + void queueRequest(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const ControlList &controls) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const ipu3_uapi_stats_3a *stats, From patchwork Thu Jan 23 14:07:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Scally X-Patchwork-Id: 22636 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 62AA9C3226 for ; Thu, 23 Jan 2025 14:07:54 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8D6046855C; Thu, 23 Jan 2025 15:07:51 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ceGjsVlZ"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0564E61878 for ; Thu, 23 Jan 2025 15:07:46 +0100 (CET) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 697AD1081; Thu, 23 Jan 2025 15:06:42 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1737641202; bh=TOMu82SphVk7MFOmJTgjlewOu9jdsrMrgssQKtJznRM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ceGjsVlZnURtY0jnM50PotDi5sPWzwLKR2k689xHU2lIvZL17OT91p+z7s3f56iC+ CylibrHvXl54HQf6J3mUh3qpCVPT4up/JeBFfNbZp9IReTP+7HrkuJ+g6TxT9vHm0z kczBmY1ZItB+ZSdItkSLUDcli3pZyZFz7DME4ROA= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally Subject: [PATCH v2 2/3] ipa: libipa: Adjust for flicker in ExposureModeHelper Date: Thu, 23 Jan 2025 14:07:26 +0000 Message-Id: <20250123140727.458567-3-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250123140727.458567-1-dan.scally@ideasonboard.com> References: <20250123140727.458567-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Update the ExposureModeHelper class to compensate for flickering light sources in the ExposureModeHelper::splitExposure() function. The adjustment simply caps exposure time at a multiple of the given flicker period and compensates for any loss in the effective exposure value by increasing analogue and then digital gain. Initially in the one call-site for this function, a std::nullopt is passed to make this a no-op. Signed-off-by: Daniel Scally --- Changes in v2: - Added a function to perform all the exposure time functionality in a single place so we're not repeating ourselves - Called that function in all the return sites rather than just one so the flicker mitigation takes effect using exposure from the stages list - Switched the flickerPeriod input to a std::optional - Clamped the calculated exposure time to guarantee it can't go beneath the configured minima src/ipa/libipa/agc_mean_luminance.cpp | 3 +- src/ipa/libipa/exposure_mode_helper.cpp | 37 ++++++++++++++++++++++--- src/ipa/libipa/exposure_mode_helper.h | 6 +++- 3 files changed, 40 insertions(+), 6 deletions(-) diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp index 02555a44..273ec4e5 100644 --- a/src/ipa/libipa/agc_mean_luminance.cpp +++ b/src/ipa/libipa/agc_mean_luminance.cpp @@ -8,6 +8,7 @@ #include "agc_mean_luminance.h" #include +#include #include #include @@ -560,7 +561,7 @@ AgcMeanLuminance::calculateNewEv(uint32_t constraintModeIndex, newExposureValue = filterExposure(newExposureValue); frameCount_++; - return exposureModeHelper->splitExposure(newExposureValue); + return exposureModeHelper->splitExposure(newExposureValue, std::nullopt); } /** diff --git a/src/ipa/libipa/exposure_mode_helper.cpp b/src/ipa/libipa/exposure_mode_helper.cpp index f235316d..4e1ba943 100644 --- a/src/ipa/libipa/exposure_mode_helper.cpp +++ b/src/ipa/libipa/exposure_mode_helper.cpp @@ -7,6 +7,7 @@ #include "exposure_mode_helper.h" #include +#include #include @@ -118,9 +119,31 @@ double ExposureModeHelper::clampGain(double gain) const return std::clamp(gain, minGain_, maxGain_); } +utils::Duration +ExposureModeHelper::calculateExposureTime(utils::Duration exposure, double stageGain, + std::optional flickerPeriod) const +{ + utils::Duration exposureTime; + + exposureTime = clampExposureTime(exposure / stageGain); + + /* + * If we haven't been given a flicker period to adjust for or if it's + * longer than the exposure time that we need to set then there's not + * much we can do to compensate. + */ + if (!flickerPeriod.has_value() || flickerPeriod.value() >= exposureTime) + return exposureTime; + + unsigned int flickerPeriods = exposureTime / flickerPeriod.value(); + + return clampExposureTime(flickerPeriods * flickerPeriod.value()); +} + /** * \brief Split exposure into exposure time and gain * \param[in] exposure Exposure value + * \param[in] flickerPeriod The period of a flickering light source * * This function divides a given exposure into exposure time, analogue and * digital gain by iterating through stages of exposure time and gain limits. @@ -147,10 +170,15 @@ double ExposureModeHelper::clampGain(double gain) const * required exposure, the helper falls-back to simply maximising the exposure * time first, followed by analogue gain, followed by digital gain. * + * Once the exposure time has been determined from the modes, an adjustment is + * made to compensate for a flickering light source by fixing the exposure time + * to an exact multiple of the flicker period. Any effective exposure value that + * is lost is added back via analogue and digital gain. + * * \return Tuple of exposure time, analogue gain, and digital gain */ std::tuple -ExposureModeHelper::splitExposure(utils::Duration exposure) const +ExposureModeHelper::splitExposure(utils::Duration exposure, std::optional flickerPeriod) const { ASSERT(maxExposureTime_); ASSERT(maxGain_); @@ -183,14 +211,14 @@ ExposureModeHelper::splitExposure(utils::Duration exposure) const */ if (stageExposureTime * lastStageGain >= exposure) { - exposureTime = clampExposureTime(exposure / clampGain(lastStageGain)); + exposureTime = calculateExposureTime(exposure, clampGain(lastStageGain), flickerPeriod); gain = clampGain(exposure / exposureTime); return { exposureTime, gain, exposure / (exposureTime * gain) }; } if (stageExposureTime * stageGain >= exposure) { - exposureTime = clampExposureTime(exposure / clampGain(stageGain)); + exposureTime = calculateExposureTime(exposure, clampGain(stageGain), flickerPeriod); gain = clampGain(exposure / exposureTime); return { exposureTime, gain, exposure / (exposureTime * gain) }; @@ -204,7 +232,8 @@ ExposureModeHelper::splitExposure(utils::Duration exposure) const * stages to use then the default stageGain of 1.0 is used so that * exposure time is maxed before gain is touched at all. */ - exposureTime = clampExposureTime(exposure / clampGain(stageGain)); + exposureTime = calculateExposureTime(exposure, clampGain(stageGain), flickerPeriod); + gain = clampGain(exposure / exposureTime); return { exposureTime, gain, exposure / (exposureTime * gain) }; diff --git a/src/ipa/libipa/exposure_mode_helper.h b/src/ipa/libipa/exposure_mode_helper.h index c5be1b67..a1d8c6bf 100644 --- a/src/ipa/libipa/exposure_mode_helper.h +++ b/src/ipa/libipa/exposure_mode_helper.h @@ -7,6 +7,7 @@ #pragma once +#include #include #include #include @@ -28,7 +29,7 @@ public: double minGain, double maxGain); std::tuple - splitExposure(utils::Duration exposure) const; + splitExposure(utils::Duration exposure, std::optional flickerPeriod) const; utils::Duration minExposureTime() const { return minExposureTime_; } utils::Duration maxExposureTime() const { return maxExposureTime_; } @@ -38,6 +39,9 @@ public: private: utils::Duration clampExposureTime(utils::Duration exposureTime) const; double clampGain(double gain) const; + utils::Duration + calculateExposureTime(utils::Duration exposureTime, double stageGain, + std::optional flickerPeriod) const; std::vector exposureTimes_; std::vector gains_; From patchwork Thu Jan 23 14:07:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Scally X-Patchwork-Id: 22637 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A37BABDE6B for ; Thu, 23 Jan 2025 14:07:55 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 418B568566; Thu, 23 Jan 2025 15:07:53 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="maqFTaZR"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 58EB36855B for ; Thu, 23 Jan 2025 15:07:46 +0100 (CET) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id B0E7A89A; Thu, 23 Jan 2025 15:06:42 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1737641202; bh=aRdq7cLKNsCqtN3GGXJJYxx7IUhDmvcDMHJ3PDLWDCs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=maqFTaZRO4hHWsW9mFLoHjtcUPjbldhVj1EGJWQybjIIYuPdNEJTYdLqcyAIX8wgx 79TRpifEuytFwCEpdrjBJjGLWiyPxPXFmLybjx5vnybBUKXdrZ5LyYcWh1Cqombcks D3nwrMI3LLujT8gcoIKgVwqnGhXXALjTnZx/+qhE= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally , Laurent Pinchart , Paul Elder Subject: [PATCH v2 3/3] ipa: libipa: Add flicker controls to AgcMeanLuminance Date: Thu, 23 Jan 2025 14:07:27 +0000 Message-Id: <20250123140727.458567-4-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250123140727.458567-1-dan.scally@ideasonboard.com> References: <20250123140727.458567-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add controls for AeFlickerMode and AeFlickerPeriod to the AgcMeanLuminance class. The controls passed to an algorithm's queueRequest() are forwarded to AgcMeanLuminance through a new function, and the values then passed to the ExposureModeHelper to be used in calculating the new exposure time. Take the opportunity to call the new parseControls() function in each of the derived class' queueRequest() function. Signed-off-by: Daniel Scally Reviewed-by: Laurent Pinchart Reviewed-by: Paul Elder --- Changes in v2: - Declared the controls using a span rather than min and max values - Used a std::optional to pass to the ExposureModeHelper. I didn't use it in the IPAContext, because it can't really be modelled as a single variable option there. src/ipa/ipu3/algorithms/agc.cpp | 3 +- src/ipa/libipa/agc_mean_luminance.cpp | 47 ++++++++++++++++++++++++++- src/ipa/libipa/agc_mean_luminance.h | 5 +++ src/ipa/mali-c55/algorithms/agc.cpp | 2 ++ src/ipa/rkisp1/algorithms/agc.cpp | 2 ++ 5 files changed, 57 insertions(+), 2 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 383b046c..b993aaa7 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -129,8 +129,9 @@ int Agc::configure(IPAContext &context, void Agc::queueRequest([[maybe_unused]] typename Module::Context &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] typename Module::FrameContext &frameContext, - [[maybe_unused]] const ControlList &controls) + const ControlList &controls) { + parseControls(controls); } Histogram Agc::parseStatistics(const ipu3_uapi_stats_3a *stats, diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp index 273ec4e5..65285bf1 100644 --- a/src/ipa/libipa/agc_mean_luminance.cpp +++ b/src/ipa/libipa/agc_mean_luminance.cpp @@ -137,6 +137,10 @@ static constexpr double kDefaultRelativeLuminanceTarget = 0.16; AgcMeanLuminance::AgcMeanLuminance() : frameCount_(0), filteredExposure_(0s), relativeLuminanceTarget_(0) { + controls_[&controls::AeFlickerMode] = ControlInfo({ { ControlValue(controls::FlickerOff), + ControlValue(controls::FlickerManual) } }, + ControlValue(controls::FlickerOff)); + controls_[&controls::AeFlickerPeriod] = ControlInfo(100, 1000000); } AgcMeanLuminance::~AgcMeanLuminance() = default; @@ -480,6 +484,39 @@ double AgcMeanLuminance::constraintClampGain(uint32_t constraintModeIndex, return gain; } +/** + * \brief Parse the controls passed to an algorithm for the ones we need + * \param[in] controls the ControlList passed to an algorithm by the IPA + * + * This function must be called by a derived class in its queueRequest() + * function so that we can extract the controls needed by this base class. + */ +void AgcMeanLuminance::parseControls(const ControlList &controls) +{ + const auto &flickerMode = controls.get(controls::AeFlickerMode); + if (flickerMode) { + switch (*flickerMode) { + case controls::AeFlickerModeEnum::FlickerOff: + case controls::AeFlickerModeEnum::FlickerManual: + flickerMode_ = static_cast(*flickerMode); + break; + default: + LOG(AgcMeanLuminance, Error) + << "Flicker mode " << *flickerMode << " is not supported"; + break; + } + } + + const auto &flickerPeriod = controls.get(controls::AeFlickerPeriod); + if (flickerPeriod) { + /* + * If at some future point we support automatic flicker + * mitigation then this will need revision. + */ + flickerPeriod_ = *flickerPeriod * 1.0us; + } +} + /** * \brief Apply a filter on the exposure value to limit the speed of changes * \param[in] exposureValue The target exposure from the AGC algorithm @@ -561,7 +598,15 @@ AgcMeanLuminance::calculateNewEv(uint32_t constraintModeIndex, newExposureValue = filterExposure(newExposureValue); frameCount_++; - return exposureModeHelper->splitExposure(newExposureValue, std::nullopt); + + std::optional flickerPeriod; + + if (flickerMode_ == controls::AeFlickerModeEnum::FlickerManual) + flickerPeriod = flickerPeriod_; + else + flickerPeriod = std::nullopt; + + return exposureModeHelper->splitExposure(newExposureValue, flickerPeriod); } /** diff --git a/src/ipa/libipa/agc_mean_luminance.h b/src/ipa/libipa/agc_mean_luminance.h index c41391cb..b9b36687 100644 --- a/src/ipa/libipa/agc_mean_luminance.h +++ b/src/ipa/libipa/agc_mean_luminance.h @@ -14,6 +14,7 @@ #include +#include #include #include "libcamera/internal/yaml_parser.h" @@ -71,6 +72,8 @@ public: frameCount_ = 0; } + void parseControls(const ControlList &controls); + private: virtual double estimateLuminance(const double gain) const = 0; @@ -87,6 +90,8 @@ private: uint64_t frameCount_; utils::Duration filteredExposure_; double relativeLuminanceTarget_; + utils::Duration flickerPeriod_; + controls::AeFlickerModeEnum flickerMode_; std::map> constraintModes_; std::map> exposureModeHelpers_; diff --git a/src/ipa/mali-c55/algorithms/agc.cpp b/src/ipa/mali-c55/algorithms/agc.cpp index 70667db3..6a80c44f 100644 --- a/src/ipa/mali-c55/algorithms/agc.cpp +++ b/src/ipa/mali-c55/algorithms/agc.cpp @@ -238,6 +238,8 @@ void Agc::queueRequest(IPAContext &context, const uint32_t frame, << "Digital gain set to " << agc.manual.ispGain << " on request sequence " << frame; } + + parseControls(controls); } size_t Agc::fillGainParamBlock(IPAContext &context, IPAFrameContext &frameContext, diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 1680669c..fd2eda96 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -315,6 +315,8 @@ void Agc::queueRequest(IPAContext &context, agc.maxFrameDuration = maxFrameDuration; } frameContext.agc.maxFrameDuration = agc.maxFrameDuration; + + parseControls(controls); } /**