From patchwork Fri Jan 17 14:34:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 22581 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2703DBD7D8 for ; Fri, 17 Jan 2025 14:34:28 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D224C6854A; Fri, 17 Jan 2025 15:34:24 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="MBO5TDl6"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7F1EF6851B for ; Fri, 17 Jan 2025 15:34:21 +0100 (CET) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 08969874; Fri, 17 Jan 2025 15:33:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1737124402; bh=c98Vnrgel42oLF875b2YKlI3NVdnbzSpmKC0kcTjipc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MBO5TDl6/+aLZsgKs5v0/q12QxpCwN5CctiC4+iXnwcp8j0r2sDm/bO8pPEJe0yqI m7RGlpurL7hZLRp8YA5Q2ErJ9Ns+4K2LJzU6k+9nZfVBZcUWvSTa6lcAnaCaLC5vDa /CZUhdvWUhys/rUep3z6aaFgDIfafUTJwlJwdhMc= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally Subject: [PATCH 1/3] ipa: ipu3: Add skeleton Agc::queueRequest() function Date: Fri, 17 Jan 2025 14:34:08 +0000 Message-Id: <20250117143410.20363-2-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250117143410.20363-1-dan.scally@ideasonboard.com> References: <20250117143410.20363-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The IPU3 IPA's AGC algorithm currently does not implement a queueRequest() function. We will shortly need to use it, but to avoid lots of noise in the future commit that adds a function call in to that function add it as a skeleton here. Signed-off-by: Daniel Scally Reviewed-by: Laurent Pinchart Reviewed-by: Paul Elder --- src/ipa/ipu3/algorithms/agc.cpp | 10 ++++++++++ src/ipa/ipu3/algorithms/agc.h | 3 +++ 2 files changed, 13 insertions(+) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 39d0aebb..383b046c 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -123,6 +123,16 @@ int Agc::configure(IPAContext &context, return 0; } +/** + * \copydoc libcamera::ipa::Algorithm::queueRequest + */ +void Agc::queueRequest([[maybe_unused]] typename Module::Context &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] typename Module::FrameContext &frameContext, + [[maybe_unused]] const ControlList &controls) +{ +} + Histogram Agc::parseStatistics(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid) { diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 890c271b..c82b7acb 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -32,6 +32,9 @@ public: int init(IPAContext &context, const YamlObject &tuningData) override; int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; + void queueRequest(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const ControlList &controls) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const ipu3_uapi_stats_3a *stats, From patchwork Fri Jan 17 14:34:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 22582 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D021CBD7D8 for ; Fri, 17 Jan 2025 14:34:29 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A9EC068549; Fri, 17 Jan 2025 15:34:26 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="DBDaRhQN"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 951456851D for ; Fri, 17 Jan 2025 15:34:21 +0100 (CET) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 49C27CE6; Fri, 17 Jan 2025 15:33:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1737124402; bh=fEvddwlwSY6laB9o9fibIRkBZw8Bd0TVC8qqw3R32mk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DBDaRhQNIc3yqFghK1x1Pl35nVmz3arsuRSZvlfXSGtPq3Bs0lDKGcSsUQe3dncTe uR5P8y42gszZ4XffYTrfnONnJR3K4QF8xGkSf69dWtw3eShbaPy3KPXUlnfAhQAV6W c9DpLemKKaQ9dhw8X8Nv+EK3QXx9ldCE06iux9mM= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally Subject: [PATCH 2/3] ipa: libipa: Adjust for flicker in ExposureModeHelper Date: Fri, 17 Jan 2025 14:34:09 +0000 Message-Id: <20250117143410.20363-3-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250117143410.20363-1-dan.scally@ideasonboard.com> References: <20250117143410.20363-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Update the ExposureModeHelper class to compensate for flickering light sources in the ::splitExposure() function. The adjustment simply caps exposure time at a multiple of the given flicker period and compensates for any loss in the effective exposure value by increasing analogue and then digital gain. Initially in the one call-site for this function, a flicker period of 0us is passed, making this a no-op. Signed-off-by: Daniel Scally --- src/ipa/libipa/agc_mean_luminance.cpp | 2 +- src/ipa/libipa/exposure_mode_helper.cpp | 20 +++++++++++++++++++- src/ipa/libipa/exposure_mode_helper.h | 2 +- 3 files changed, 21 insertions(+), 3 deletions(-) diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp index 02555a44..b5e6afe3 100644 --- a/src/ipa/libipa/agc_mean_luminance.cpp +++ b/src/ipa/libipa/agc_mean_luminance.cpp @@ -560,7 +560,7 @@ AgcMeanLuminance::calculateNewEv(uint32_t constraintModeIndex, newExposureValue = filterExposure(newExposureValue); frameCount_++; - return exposureModeHelper->splitExposure(newExposureValue); + return exposureModeHelper->splitExposure(newExposureValue, 0us); } /** diff --git a/src/ipa/libipa/exposure_mode_helper.cpp b/src/ipa/libipa/exposure_mode_helper.cpp index f235316d..038aa33c 100644 --- a/src/ipa/libipa/exposure_mode_helper.cpp +++ b/src/ipa/libipa/exposure_mode_helper.cpp @@ -121,6 +121,7 @@ double ExposureModeHelper::clampGain(double gain) const /** * \brief Split exposure into exposure time and gain * \param[in] exposure Exposure value + * \param[in] flickerPeriod The period of a flickering light source * * This function divides a given exposure into exposure time, analogue and * digital gain by iterating through stages of exposure time and gain limits. @@ -147,10 +148,15 @@ double ExposureModeHelper::clampGain(double gain) const * required exposure, the helper falls-back to simply maximising the exposure * time first, followed by analogue gain, followed by digital gain. * + * Once the exposure time has been determined from the modes, an adjustment is + * made to compensate for a flickering light source by fixing the exposure time + * to an exact multiple of the flicker period. Any effective exposure value that + * is lost is added back via analogue and digital gain. + * * \return Tuple of exposure time, analogue gain, and digital gain */ std::tuple -ExposureModeHelper::splitExposure(utils::Duration exposure) const +ExposureModeHelper::splitExposure(utils::Duration exposure, utils::Duration flickerPeriod) const { ASSERT(maxExposureTime_); ASSERT(maxGain_); @@ -205,6 +211,18 @@ ExposureModeHelper::splitExposure(utils::Duration exposure) const * exposure time is maxed before gain is touched at all. */ exposureTime = clampExposureTime(exposure / clampGain(stageGain)); + + /* + * Finally, if we have been given a flicker period we need to reduce the + * exposure time to be a multiple of that period (if possible). The + * effect of this should be to hide the flicker. + */ + if (flickerPeriod > 0us && flickerPeriod < exposureTime) { + unsigned int flickerPeriods = exposureTime / flickerPeriod; + if (flickerPeriods) + exposureTime = flickerPeriods * flickerPeriod; + } + gain = clampGain(exposure / exposureTime); return { exposureTime, gain, exposure / (exposureTime * gain) }; diff --git a/src/ipa/libipa/exposure_mode_helper.h b/src/ipa/libipa/exposure_mode_helper.h index c5be1b67..a5fcc366 100644 --- a/src/ipa/libipa/exposure_mode_helper.h +++ b/src/ipa/libipa/exposure_mode_helper.h @@ -28,7 +28,7 @@ public: double minGain, double maxGain); std::tuple - splitExposure(utils::Duration exposure) const; + splitExposure(utils::Duration exposure, utils::Duration flickerPeriod) const; utils::Duration minExposureTime() const { return minExposureTime_; } utils::Duration maxExposureTime() const { return maxExposureTime_; } From patchwork Fri Jan 17 14:34:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 22583 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B4917C3309 for ; Fri, 17 Jan 2025 14:34:30 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8548A68545; Fri, 17 Jan 2025 15:34:27 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="CLkyYvgd"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id CAAE668545 for ; Fri, 17 Jan 2025 15:34:21 +0100 (CET) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 8970E6DE; Fri, 17 Jan 2025 15:33:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1737124402; bh=wplAYg7WxuBGs06vdPeCIFMd5551GtMlf8JOjJZ3KsM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CLkyYvgd7VnSiyn8dvRDXH/+uWEl7hODDqGnJBdiXwMYlfD+kuKmdi8VT2RLkd/c3 wC4t9AAG4li4U5ANjhZCVWR4BTe1VV577ZZvPsV1cYAD40an5/RMNYyvYfvs0ze2i5 jJyee6FbbrONuZLdhQJOIr2moFF5MOiTQM33yK3c= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally Subject: [PATCH 3/3] ipa: libipa: Add flicker controls to AgcMeanLuminance Date: Fri, 17 Jan 2025 14:34:10 +0000 Message-Id: <20250117143410.20363-4-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250117143410.20363-1-dan.scally@ideasonboard.com> References: <20250117143410.20363-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add controls for AeFlickerMode and AeFlickerPeriod to the AgcMeanLuminance class. The controls passed to an algorithm's queueRequest() are forwarded to AgcMeanLuminance through a new function, and the values then passed to the ExposureModeHelper to be used in calculating the new exposure time. Take the opportunity to call the new parseControls() function in each of the derived class' queueRequest() function. Signed-off-by: Daniel Scally Reviewed-by: Laurent Pinchart --- src/ipa/ipu3/algorithms/agc.cpp | 3 +- src/ipa/libipa/agc_mean_luminance.cpp | 47 ++++++++++++++++++++++++++- src/ipa/libipa/agc_mean_luminance.h | 5 +++ src/ipa/mali-c55/algorithms/agc.cpp | 2 ++ src/ipa/rkisp1/algorithms/agc.cpp | 2 ++ 5 files changed, 57 insertions(+), 2 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 383b046c..b993aaa7 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -129,8 +129,9 @@ int Agc::configure(IPAContext &context, void Agc::queueRequest([[maybe_unused]] typename Module::Context &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] typename Module::FrameContext &frameContext, - [[maybe_unused]] const ControlList &controls) + const ControlList &controls) { + parseControls(controls); } Histogram Agc::parseStatistics(const ipu3_uapi_stats_3a *stats, diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp index b5e6afe3..4acf7b55 100644 --- a/src/ipa/libipa/agc_mean_luminance.cpp +++ b/src/ipa/libipa/agc_mean_luminance.cpp @@ -136,6 +136,10 @@ static constexpr double kDefaultRelativeLuminanceTarget = 0.16; AgcMeanLuminance::AgcMeanLuminance() : frameCount_(0), filteredExposure_(0s), relativeLuminanceTarget_(0) { + controls_[&controls::AeFlickerMode] = ControlInfo(static_cast(controls::FlickerOff), + static_cast(controls::FlickerManual), + static_cast(controls::FlickerOff)); + controls_[&controls::AeFlickerPeriod] = ControlInfo(100, 1000000); } AgcMeanLuminance::~AgcMeanLuminance() = default; @@ -479,6 +483,39 @@ double AgcMeanLuminance::constraintClampGain(uint32_t constraintModeIndex, return gain; } +/** + * \brief Parse the controls passed to an algorithm for the ones we need + * \param[in] controls the ControlList passed to an algorithm by the IPA + * + * This function must be called by a derived class in its queueRequest() + * function so that we can extract the controls needed by this base class. + */ +void AgcMeanLuminance::parseControls(const ControlList &controls) +{ + const auto &flickerMode = controls.get(controls::AeFlickerMode); + if (flickerMode) { + switch (*flickerMode) { + case controls::AeFlickerModeEnum::FlickerOff: + case controls::AeFlickerModeEnum::FlickerManual: + flickerMode_ = static_cast(*flickerMode); + break; + default: + LOG(AgcMeanLuminance, Error) + << "Flicker mode " << *flickerMode << " is not supported"; + break; + } + } + + const auto &flickerPeriod = controls.get(controls::AeFlickerPeriod); + if (flickerPeriod) { + /* + * If at some future point we support automatic flicker + * mitigation then this will need revision. + */ + flickerPeriod_ = *flickerPeriod * 1.0us; + } +} + /** * \brief Apply a filter on the exposure value to limit the speed of changes * \param[in] exposureValue The target exposure from the AGC algorithm @@ -560,7 +597,15 @@ AgcMeanLuminance::calculateNewEv(uint32_t constraintModeIndex, newExposureValue = filterExposure(newExposureValue); frameCount_++; - return exposureModeHelper->splitExposure(newExposureValue, 0us); + + utils::Duration flickerPeriod; + + if (flickerMode_ == controls::AeFlickerModeEnum::FlickerManual) + flickerPeriod = flickerPeriod_; + else + flickerPeriod = 0us; + + return exposureModeHelper->splitExposure(newExposureValue, flickerPeriod); } /** diff --git a/src/ipa/libipa/agc_mean_luminance.h b/src/ipa/libipa/agc_mean_luminance.h index c41391cb..b9b36687 100644 --- a/src/ipa/libipa/agc_mean_luminance.h +++ b/src/ipa/libipa/agc_mean_luminance.h @@ -14,6 +14,7 @@ #include +#include #include #include "libcamera/internal/yaml_parser.h" @@ -71,6 +72,8 @@ public: frameCount_ = 0; } + void parseControls(const ControlList &controls); + private: virtual double estimateLuminance(const double gain) const = 0; @@ -87,6 +90,8 @@ private: uint64_t frameCount_; utils::Duration filteredExposure_; double relativeLuminanceTarget_; + utils::Duration flickerPeriod_; + controls::AeFlickerModeEnum flickerMode_; std::map> constraintModes_; std::map> exposureModeHelpers_; diff --git a/src/ipa/mali-c55/algorithms/agc.cpp b/src/ipa/mali-c55/algorithms/agc.cpp index 70667db3..6a80c44f 100644 --- a/src/ipa/mali-c55/algorithms/agc.cpp +++ b/src/ipa/mali-c55/algorithms/agc.cpp @@ -238,6 +238,8 @@ void Agc::queueRequest(IPAContext &context, const uint32_t frame, << "Digital gain set to " << agc.manual.ispGain << " on request sequence " << frame; } + + parseControls(controls); } size_t Agc::fillGainParamBlock(IPAContext &context, IPAFrameContext &frameContext, diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 40e5a8f4..7b1763a2 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -275,6 +275,8 @@ void Agc::queueRequest(IPAContext &context, agc.maxFrameDuration = maxFrameDuration; } frameContext.agc.maxFrameDuration = agc.maxFrameDuration; + + parseControls(controls); } /**