From patchwork Wed Apr 24 12:49:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19937 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id BBBF9C3200 for ; Wed, 24 Apr 2024 12:49:42 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 50A7663408; Wed, 24 Apr 2024 14:49:39 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="Jc47R/hL"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E6D82633EC for ; Wed, 24 Apr 2024 14:49:34 +0200 (CEST) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C001E7E9; Wed, 24 Apr 2024 14:48:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1713962922; bh=gWcdBPI+pLxGTihk5wyi1Tu4BUUfoChnbIvfitaZsQ4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Jc47R/hL11+68DhFAgbhGpJAjnVw+HnQRt5oNk4HYvpmYGCAac16hTXnkwSgJ776J YYee1gnPDvmDx6nM25rqxYwOhrW41dwSbOL9FdhUJiqbgd4v4iL5jhBZ6N7V8nhNiu 18zZnnV2BiuTAsvqqOZXm8twMvYUMZmOADqC9h4U= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally , Stefan Klug , Paul Elder Subject: [PATCH v3 1/8] ipa: libipa: Allow creation of empty Histogram Date: Wed, 24 Apr 2024 13:49:10 +0100 Message-Id: <20240424124917.1250837-2-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240424124917.1250837-1-dan.scally@ideasonboard.com> References: <20240424124917.1250837-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" For convenience's sake allow the creation of empty Histograms so they can be embedded within other Classes and filled out with data at some later point in time. Reviewed-by: Stefan Klug Reviewed-by: Paul Elder Signed-off-by: Daniel Scally --- Changes in v3: - None Changes in v2: - Added documentation src/ipa/libipa/histogram.cpp | 9 +++++++++ src/ipa/libipa/histogram.h | 1 + 2 files changed, 10 insertions(+) diff --git a/src/ipa/libipa/histogram.cpp b/src/ipa/libipa/histogram.cpp index 6b5cde8e..c1aac59b 100644 --- a/src/ipa/libipa/histogram.cpp +++ b/src/ipa/libipa/histogram.cpp @@ -28,6 +28,15 @@ namespace ipa { * specified bin. It can be used to find quantiles and averages between quantiles. */ +/** + * \fn Histogram::Histogram() + * \brief Construct an empty Histogram + * + * This empty constructor exists largely to allow Histograms to be embedded in + * other classes which may be created before the contents of the Histogram are + * known. + */ + /** * \brief Create a cumulative histogram * \param[in] data A pre-sorted histogram to be passed diff --git a/src/ipa/libipa/histogram.h b/src/ipa/libipa/histogram.h index 05bb4b80..54bb2a19 100644 --- a/src/ipa/libipa/histogram.h +++ b/src/ipa/libipa/histogram.h @@ -22,6 +22,7 @@ namespace ipa { class Histogram { public: + Histogram() { cumulative_.push_back(0); } Histogram(Span data); size_t bins() const { return cumulative_.size() - 1; } uint64_t total() const { return cumulative_[cumulative_.size() - 1]; } From patchwork Wed Apr 24 12:49:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19938 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A4A63C3200 for ; Wed, 24 Apr 2024 12:49:45 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 6F2B8633F3; Wed, 24 Apr 2024 14:49:40 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ZOKI7ebG"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1AF0D633F2 for ; Wed, 24 Apr 2024 14:49:35 +0200 (CEST) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 1589BEBB; Wed, 24 Apr 2024 14:48:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1713962923; bh=3tT2IBEdggplsveVwPjuks1RjOi34fXDITbQTVtNvPI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZOKI7ebGKogdQWFBIhPuND29l6fAVyCibEkrf+hclOZPxbge4GK9gomwZetd3djFL VtsOKC6LSjJPl8A0OiZ2WE8OCrANhGuAzYlZDzArlCfjcfzTd43TQQPHm0COyf0w0g Acv+YoCkpkMJQajla2Oin2ksCe8qBCfn82iXa8kw= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally , Stefan Klug , Jacopo Mondi , Paul Elder Subject: [PATCH v3 2/8] libcamera: controls: Generate enum value-name maps Date: Wed, 24 Apr 2024 13:49:11 +0100 Message-Id: <20240424124917.1250837-3-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240424124917.1250837-1-dan.scally@ideasonboard.com> References: <20240424124917.1250837-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Generate maps for each control enum which associate strings that represent the enum values with the values themselves. This change will allow us to refer to enumerated control values using the string. For example if we want to pass variables to an algorithm for use when a control has a particular value we can embed within tuning files a dictionary that uses the control values as keys. Reviewed-by: Stefan Klug Reviewed-by: Jacopo Mondi Reviewed-by: Paul Elder Signed-off-by: Daniel Scally --- Changes in v3: - None Changes in v2: - Expanded commit message include/libcamera/control_ids.h.in | 2 ++ include/libcamera/property_ids.h.in | 2 ++ utils/gen-controls.py | 19 +++++++++++++++++++ 3 files changed, 23 insertions(+) diff --git a/include/libcamera/control_ids.h.in b/include/libcamera/control_ids.h.in index d53b1cf7..58dd48e1 100644 --- a/include/libcamera/control_ids.h.in +++ b/include/libcamera/control_ids.h.in @@ -10,7 +10,9 @@ #pragma once #include +#include #include +#include #include diff --git a/include/libcamera/property_ids.h.in b/include/libcamera/property_ids.h.in index 43372c71..f51ba028 100644 --- a/include/libcamera/property_ids.h.in +++ b/include/libcamera/property_ids.h.in @@ -9,7 +9,9 @@ #pragma once +#include #include +#include #include diff --git a/utils/gen-controls.py b/utils/gen-controls.py index 6cd5e362..4fe1e705 100755 --- a/utils/gen-controls.py +++ b/utils/gen-controls.py @@ -140,6 +140,12 @@ ${description} */''') enum_values_start = string.Template('''extern const std::array ${name}Values = {''') enum_values_values = string.Template('''\tstatic_cast(${name}),''') + name_value_map_doc = string.Template('''/** + * \\var ${name}NameValueMap + * \\brief Map of all $name supported value names (in std::string format) to value + */''') + name_value_map_start = string.Template('''extern const std::map ${name}NameValueMap = {''') + name_value_values = string.Template('''\t{ "${name}", ${name} },''') ctrls_doc = {} ctrls_def = {} @@ -183,6 +189,7 @@ ${description} values_info = { 'name': info['name'], + 'type': ctrl.type, 'size': num_entries, } target_doc.append(enum_values_doc.substitute(values_info)) @@ -194,6 +201,15 @@ ${description} target_def.append(enum_values_values.substitute(value_info)) target_def.append("};") + target_doc.append(name_value_map_doc.substitute(values_info)) + target_def.append(name_value_map_start.substitute(values_info)) + for enum in ctrl.enum_values: + value_info = { + 'name': enum.name + } + target_def.append(name_value_values.substitute(value_info)) + target_def.append("};") + target_doc.append(doc_template.substitute(info)) target_def.append(def_template.substitute(info)) @@ -231,6 +247,7 @@ def generate_h(controls, mode, ranges): enum_template_start = string.Template('''enum ${name}Enum {''') enum_value_template = string.Template('''\t${name} = ${value},''') enum_values_template = string.Template('''extern const std::array ${name}Values;''') + name_value_map_template = string.Template('''extern const std::map ${name}NameValueMap;''') template = string.Template('''extern const Control<${type}> ${name};''') ctrls = {} @@ -273,9 +290,11 @@ def generate_h(controls, mode, ranges): values_info = { 'name': info['name'], + 'type': ctrl.type, 'size': num_entries, } target_ctrls.append(enum_values_template.substitute(values_info)) + target_ctrls.append(name_value_map_template.substitute(values_info)) target_ctrls.append(template.substitute(info)) id_value[vendor] += 1 From patchwork Wed Apr 24 12:49:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19939 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B48D3C3200 for ; Wed, 24 Apr 2024 12:49:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 30623633F5; Wed, 24 Apr 2024 14:49:42 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="bEss/eTH"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4DAD0633F3 for ; Wed, 24 Apr 2024 14:49:35 +0200 (CEST) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 665DB6B3; Wed, 24 Apr 2024 14:48:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1713962923; bh=48DzKXfp55ONd4XPFVwZVzEQhWn3vtK3oQyCy+JqLQo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bEss/eTHkCzUczTZx+qMdblv5Rr1nn4ylxtMfsvVHL0JhAhhgyx5ell9ogFLP62yy E/ubP+Hg2Wwgr6unUnSpV8IX7jNccEeXcSl2M3zZjefihVyEMsMqAVlYeMpLGuuAPQ 5jiZAgQbcqO8xm4kuG7zBN0O/gLX8IpWsXyIoLQU= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Paul Elder , Daniel Scally Subject: [PATCH v3 3/8] ipa: libipa: Add ExposureModeHelper Date: Wed, 24 Apr 2024 13:49:12 +0100 Message-Id: <20240424124917.1250837-4-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240424124917.1250837-1-dan.scally@ideasonboard.com> References: <20240424124917.1250837-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Paul Elder Add a helper for managing exposure modes and splitting exposure times into shutter and gain values. Reviewed-by: Paul Elder Signed-off-by: Paul Elder Signed-off-by: Daniel Scally Reviewed-by: Jacopo Mondi --- Changes in v3: - Referred to "shutter time" instead of "shutter" - Removed ::init() and swapped its functionality to the constructor - Replaced the parameter to the constructor (ex-::init()) with a Span instead of a vector. - Lots of documentation updates and function renaming Changes in v2: - Expanded the documentation - Dropped the overloads for fixed shutter / gain - the same functionality is instead done by setting min and max shutter and gain to the same value - Changed ::init() to consume a vector of pairs instead of two separate vectors - Reworked splitExposure() src/ipa/libipa/exposure_mode_helper.cpp | 246 ++++++++++++++++++++++++ src/ipa/libipa/exposure_mode_helper.h | 53 +++++ src/ipa/libipa/meson.build | 2 + 3 files changed, 301 insertions(+) create mode 100644 src/ipa/libipa/exposure_mode_helper.cpp create mode 100644 src/ipa/libipa/exposure_mode_helper.h diff --git a/src/ipa/libipa/exposure_mode_helper.cpp b/src/ipa/libipa/exposure_mode_helper.cpp new file mode 100644 index 00000000..47a300f3 --- /dev/null +++ b/src/ipa/libipa/exposure_mode_helper.cpp @@ -0,0 +1,246 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Paul Elder + * + * exposure_mode_helper.cpp - Helper class that performs computations relating to exposure + */ +#include "exposure_mode_helper.h" + +#include + +#include + +/** + * \file exposure_mode_helper.h + * \brief Helper class that performs computations relating to exposure + * + * AEGC algorithms have a need to split exposure between shutter time, analogue + * and digital gain. Multiple implementations do so based on paired stages of + * shutter time and gain limits; provide a helper to avoid duplicating the code. + */ + +namespace libcamera { + +using namespace std::literals::chrono_literals; + +LOG_DEFINE_CATEGORY(ExposureModeHelper) + +namespace ipa { + +/** + * \class ExposureModeHelper + * \brief Class for splitting exposure into shutter time and total gain + * + * The ExposureModeHelper class provides a standard interface through which an + * AEGC algorithm can divide exposure between shutter time and gain. It is + * configured with a set of shutter time and gain pairs and works by initially + * fixing gain at 1.0 and increasing shutter time up to the shutter time value + * from the first pair in the set in an attempt to meet the required exposure + * value. + * + * If the required exposure is not achievable by the first shutter time value + * alone it ramps gain up to the value from the first pair in the set. If the + * required exposure is still not met it then allows shutter time to ramp up to + * the shutter time value from the second pair in the set, and continues in this + * vein until either the required exposure time is met, or else the hardware's + * shutter time or gain limits are reached. + * + * This method allows users to strike a balance between a well-exposed image and + * an acceptable frame-rate, as opposed to simply maximising shutter time + * followed by gain. The same helpers can be used to perform the latter + * operation if needed by passing an empty set of pairs to the initialisation + * function. + * + * The gain values may exceed a camera sensor's analogue gain limits if either + * it or the IPA is also capable of digital gain. The configure() function must + * be called with the hardware's limits to inform the helper of those + * constraints. Any gain that is needed will be applied as analogue gain first + * until the hardware's limit is reached, following which digital gain will be + * used. + */ + +/** + * \brief Construct an ExposureModeHelper instance + * \param[in] stages The vector of paired shutter time and gain limits + * + * The input stages are shutter time and _total_ gain pairs; the gain + * encompasses both analogue and digital gain. + * + * The vector of stages may be empty. In that case, the helper will simply use + * the runtime limits set through setShutterGainLimits() instead. + */ +ExposureModeHelper::ExposureModeHelper(const Span> stages) +{ + minShutter_ = 0us; + maxShutter_ = 0us; + minGain_ = 0; + maxGain_ = 0; + + for (const auto &[s, g] : stages) { + shutters_.push_back(s); + gains_.push_back(g); + } +} + +/** + * \brief Set the shutter time and gain limits + * \param[in] minShutter The minimum shutter time supported + * \param[in] maxShutter The maximum shutter time supported + * \param[in] minGain The minimum analogue gain supported + * \param[in] maxGain The maximum analogue gain supported + * + * This function configures the shutter time and analogue gain limits that need + * to be adhered to as the helper divides up exposure. Note that this function + * *must* be called whenever those limits change and before splitExposure() is + * used. + * + * If the algorithm using the helpers needs to indicate that either shutter time + * or analogue gain or both should be fixed it can do so by setting both the + * minima and maxima to the same value. + */ +void ExposureModeHelper::setLimits(utils::Duration minShutter, + utils::Duration maxShutter, + double minGain, double maxGain) +{ + minShutter_ = minShutter; + maxShutter_ = maxShutter; + minGain_ = minGain; + maxGain_ = maxGain; +} + +utils::Duration ExposureModeHelper::clampShutter(utils::Duration shutter) const +{ + return std::clamp(shutter, minShutter_, maxShutter_); +} + +double ExposureModeHelper::clampGain(double gain) const +{ + return std::clamp(gain, minGain_, maxGain_); +} + +/** + * \brief Split exposure time into shutter time and gain + * \param[in] exposure Exposure time + * + * This function divides a given exposure time into shutter time, analogue and + * digital gain by iterating through stages of shutter time and gain limits. At + * each stage the current stage's shutter time limit is multiplied by the + * previous stage's gain limit (or 1.0 initially) to see if the combination of + * the two can meet the required exposure time. If they cannot then the current + * stage's shutter time limit is multiplied by the same stage's gain limit to + * see if that combination can meet the required exposure time. If they cannot + * then the function moves to consider the next stage. + * + * When a combination of shutter time and gain _stage_ limits are found that are + * sufficient to meet the required exposure time, the function attempts to + * reduce shutter time as much as possible whilst fixing gain and still meeting + * the exposure time. If a _runtime_ limit prevents shutter time from being + * lowered enough to meet the exposure time with gain fixed at the stage limit, + * gain is also lowered to compensate. + * + * Once the shutter time and gain values are ascertained, gain is assigned as + * analogue gain as much as possible, with digital gain only in use if the + * maximum analogue gain runtime limit is unable to accomodate the exposure + * value. + * + * If no combination of shutter time and gain limits is found that meets the + * required exposure time, the helper falls-back to simply maximising the + * shutter time first, followed by analogue gain, followed by digital gain. + * + * \return Tuple of shutter time, analogue gain, and digital gain + */ +std::tuple +ExposureModeHelper::splitExposure(utils::Duration exposure) const +{ + ASSERT(maxShutter_); + ASSERT(maxGain_); + + bool gainFixed = minGain_ == maxGain_; + bool shutterFixed = minShutter_ == maxShutter_; + + /* + * There's no point entering the loop if we cannot change either gain + * nor shutter anyway. + */ + if (shutterFixed && gainFixed) + return { minShutter_, minGain_, exposure / (minShutter_ * minGain_) }; + + utils::Duration shutter; + double stageGain; + double gain; + + for (unsigned int stage = 0; stage < gains_.size(); stage++) { + double lastStageGain = stage == 0 ? 1.0 : clampGain(gains_[stage - 1]); + utils::Duration stageShutter = clampShutter(shutters_[stage]); + stageGain = clampGain(gains_[stage]); + + /* + * We perform the clamping on both shutter and gain in case the + * helper has had limits set that prevent those values being + * lowered beyond a certain minimum...this can happen at runtime + * for various reasons and so would not be known when the stage + * limits are initialised. + */ + + if (stageShutter * lastStageGain >= exposure) { + shutter = clampShutter(exposure / clampGain(lastStageGain)); + gain = clampGain(exposure / shutter); + + return { shutter, gain, exposure / (shutter * gain) }; + } + + if (stageShutter * stageGain >= exposure) { + shutter = clampShutter(exposure / clampGain(stageGain)); + gain = clampGain(exposure / shutter); + + return { shutter, gain, exposure / (shutter * gain) }; + } + } + + /* + * From here on all we can do is max out the shutter time, followed by + * the analogue gain. If we still haven't achieved the target we send + * the rest of the exposure time to digital gain. If we were given no + * stages to use then set stageGain to 1.0 so that shutter time is maxed + * before gain touched at all. + */ + if (gains_.empty()) + stageGain = 1.0; + + shutter = clampShutter(exposure / clampGain(stageGain)); + gain = clampGain(exposure / shutter); + + return { shutter, gain, exposure / (shutter * gain) }; +} + +/** + * \fn ExposureModeHelper::minShutter() + * \brief Retrieve the configured minimum shutter time limit set through + * setShutterGainLimits() + * \return The minShutter_ value + */ + +/** + * \fn ExposureModeHelper::maxShutter() + * \brief Retrieve the configured maximum shutter time set through + * setShutterGainLimits() + * \return The maxShutter_ value + */ + +/** + * \fn ExposureModeHelper::minGain() + * \brief Retrieve the configured minimum gain set through + * setShutterGainLimits() + * \return The minGain_ value + */ + +/** + * \fn ExposureModeHelper::maxGain() + * \brief Retrieve the configured maximum gain set through + * setShutterGainLimits() + * \return The maxGain_ value + */ + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/exposure_mode_helper.h b/src/ipa/libipa/exposure_mode_helper.h new file mode 100644 index 00000000..0ffc164e --- /dev/null +++ b/src/ipa/libipa/exposure_mode_helper.h @@ -0,0 +1,53 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Paul Elder + * + * exposure_mode_helper.h - Helper class that performs computations relating to exposure + */ + +#pragma once + +#include +#include +#include + +#include +#include + +namespace libcamera { + +namespace ipa { + +class ExposureModeHelper +{ +public: + ExposureModeHelper(const Span> stages); + ~ExposureModeHelper() = default; + + void setLimits(utils::Duration minShutter, utils::Duration maxShutter, + double minGain, double maxGain); + + std::tuple + splitExposure(utils::Duration exposure) const; + + utils::Duration minShutter() const { return minShutter_; } + utils::Duration maxShutter() const { return maxShutter_; } + double minGain() const { return minGain_; } + double maxGain() const { return maxGain_; } + +private: + utils::Duration clampShutter(utils::Duration shutter) const; + double clampGain(double gain) const; + + std::vector shutters_; + std::vector gains_; + + utils::Duration minShutter_; + utils::Duration maxShutter_; + double minGain_; + double maxGain_; +}; + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index 016b8e0e..37fbd177 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -3,6 +3,7 @@ libipa_headers = files([ 'algorithm.h', 'camera_sensor_helper.h', + 'exposure_mode_helper.h', 'fc_queue.h', 'histogram.h', 'module.h', @@ -11,6 +12,7 @@ libipa_headers = files([ libipa_sources = files([ 'algorithm.cpp', 'camera_sensor_helper.cpp', + 'exposure_mode_helper.cpp', 'fc_queue.cpp', 'histogram.cpp', 'module.cpp', From patchwork Wed Apr 24 12:49:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19940 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8B837C328D for ; Wed, 24 Apr 2024 12:49:49 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 101FC633FC; Wed, 24 Apr 2024 14:49:43 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="XB8WjWM4"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A5E43633F4 for ; Wed, 24 Apr 2024 14:49:35 +0200 (CEST) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id ABB3C1153; Wed, 24 Apr 2024 14:48:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1713962923; bh=sghFNvYNCAtX9ZTm4FVDToPM49OgDbCowMH/WZJ01Z8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XB8WjWM4b5Otw0/SPOGktS7IhxvfzwG3ypNppEAzSXQjD16g+hAX2gTgE73WLC3eB lkeA7TCzxu6hY92rDd+6KewwW9vwcgtYnBpIRcVVaooM00Tb6MFuMX+eqip98scwm8 UUnWg1TEQQy2fKFeQnk0w/OF9VtJDDdMDVpmMj0U= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally , Stefan Klug , Paul Elder Subject: [PATCH v3 4/8] ipa: libipa: Add AgcMeanLuminance base class Date: Wed, 24 Apr 2024 13:49:13 +0100 Message-Id: <20240424124917.1250837-5-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240424124917.1250837-1-dan.scally@ideasonboard.com> References: <20240424124917.1250837-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The Agc algorithms for the RkIsp1 and IPU3 IPAs do the same thing in very large part; following the Rpi IPA's algorithm in spirit with a few tunable values in that IPA being hardcoded in the libipa ones. Add a new base class for AgcMeanLuminance which implements the same algorithm and additionally parses yaml tuning files to inform an IPA module's Agc algorithm about valid constraint and exposure modes and their associated bounds. Reviewed-by: Stefan Klug Reviewed-by: Paul Elder Signed-off-by: Daniel Scally Reviewed-by: Jacopo Mondi --- Changes in v3: - Finished renaming the class in all references to it (I hope...) - Updated documentation to try to be a bit clearer about how to use the base class - Made functions const/private where possible Changes in v2: - Renamed the class and files - Expanded the documentation - Added parseTuningData() so derived classes can call a single function to cover all the parsing in ::init(). src/ipa/libipa/agc_mean_luminance.cpp | 590 ++++++++++++++++++++++++++ src/ipa/libipa/agc_mean_luminance.h | 90 ++++ src/ipa/libipa/meson.build | 2 + 3 files changed, 682 insertions(+) create mode 100644 src/ipa/libipa/agc_mean_luminance.cpp create mode 100644 src/ipa/libipa/agc_mean_luminance.h diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp new file mode 100644 index 00000000..7305db71 --- /dev/null +++ b/src/ipa/libipa/agc_mean_luminance.cpp @@ -0,0 +1,590 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Ideas on Board Oy + * + * agc_mean_luminance.cpp - Base class for mean luminance AGC algorithms + */ + +#include "agc_mean_luminance.h" + +#include + +#include +#include + +#include "exposure_mode_helper.h" + +using namespace libcamera::controls; + +/** + * \file agc_mean_luminance.h + * \brief Base class implementing mean luminance AEGC + */ + +namespace libcamera { + +using namespace std::literals::chrono_literals; + +LOG_DEFINE_CATEGORY(AgcMeanLuminance) + +namespace ipa { + +/* + * Number of frames for which to run the algorithm at full speed, before slowing + * down to prevent large and jarring changes in exposure from frame to frame. + */ +static constexpr uint32_t kNumStartupFrames = 10; + +/* + * Default relative luminance target + * + * This value should be chosen so that when the camera points at a grey target, + * the resulting image brightness looks "right". Custom values can be passed + * as the relativeLuminanceTarget value in sensor tuning files. + */ +static constexpr double kDefaultRelativeLuminanceTarget = 0.16; + +/** + * \struct AgcMeanLuminance::AgcConstraint + * \brief The boundaries and target for an AeConstraintMode constraint + * + * This structure describes an AeConstraintMode constraint for the purposes of + * this algorithm. These constraints are expressed as a pair of quantile + * boundaries for a histogram, along with a luminance target and a bounds-type. + * The algorithm uses the constraints by ensuring that the defined portion of a + * luminance histogram (I.E. lying between the two quantiles) is above or below + * the given luminance value. + */ + +/** + * \enum AgcMeanLuminance::AgcConstraint::Bound + * \brief Specify whether the constraint defines a lower or upper bound + * \var AgcMeanLuminance::AgcConstraint::lower + * \brief The constraint defines a lower bound + * \var AgcMeanLuminance::AgcConstraint::upper + * \brief The constraint defines an upper bound + */ + +/** + * \var AgcMeanLuminance::AgcConstraint::bound + * \brief The type of constraint bound + */ + +/** + * \var AgcMeanLuminance::AgcConstraint::qLo + * \brief The lower quantile to use for the constraint + */ + +/** + * \var AgcMeanLuminance::AgcConstraint::qHi + * \brief The upper quantile to use for the constraint + */ + +/** + * \var AgcMeanLuminance::AgcConstraint::yTarget + * \brief The luminance target for the constraint + */ + +/** + * \class AgcMeanLuminance + * \brief A mean-based auto-exposure algorithm + * + * This algorithm calculates a shutter time, analogue and digital gain such that + * the normalised mean luminance value of an image is driven towards a target, + * which itself is discovered from tuning data. The algorithm is a two-stage + * process. + * + * In the first stage, an initial gain value is derived by iteratively comparing + * the gain-adjusted mean luminance across an entire image against a target, and + * selecting a value which pushes it as closely as possible towards the target. + * + * In the second stage we calculate the gain required to drive the average of a + * section of a histogram to a target value, where the target and the boundaries + * of the section of the histogram used in the calculation are taken from the + * values defined for the currently configured AeConstraintMode within the + * tuning data. This class provides a helper function to parse those tuning data + * to discover the constraints, and so requires a specific format for those + * data which is described in \ref parseTuningData(). The gain from the first + * stage is then clamped to the gain from this stage. + * + * The final gain is used to adjust the effective exposure value of the image, + * and that new exposure value is divided into shutter time, analogue gain and + * digital gain according to the selected AeExposureMode. This class uses the + * \ref ExposureModeHelper class to assist in that division, and expects the + * data needed to initialise that class to be present in tuning data in a + * format described in \ref parseTuningData(). + * + * In order to be able to derive an AGC implementation from this class, an IPA + * needs to be able to do the following: + * + * 1. Provide a luminance estimation across an entire image. + * 2. Provide a luminance Histogram for the image to use in calculating + * constraint compliance. The precision of the Histogram that is available + * will determine the supportable precision of the constraints. + * + * Users should derive their Agc class from this class as well as their IPA's + * Algorithm class, and provide an overriding estimateLuminance() function for + * this class to use. They must call parseTuningData() in init(), and must also + * call setLimits() and resetFrameCounter() in configure(). They may then use + * calculateNewEv() in process(). If the limits passed to setLimits() change for + * any reason (for example, in response to a FrameDurationLimit control being + * passed in queueRequest()) then setLimits() must be called again with the new + * values. + */ + +AgcMeanLuminance::AgcMeanLuminance() + : frameCount_(0), filteredExposure_(0s), relativeLuminanceTarget_(0) +{ +} + +/** + * \brief Parse the relative luminance target from the tuning data + * \param[in] tuningData The YamlObject holding the algorithm's tuning data + */ +void AgcMeanLuminance::parseRelativeLuminanceTarget(const YamlObject &tuningData) +{ + relativeLuminanceTarget_ = + tuningData["relativeLuminanceTarget"].get(kDefaultRelativeLuminanceTarget); +} + +/** + * \brief Parse an AeConstraintMode constraint from tuning data + * \param[in] modeDict the YamlObject holding the constraint data + * \param[in] id The constraint ID from AeConstraintModeEnum + */ +void AgcMeanLuminance::parseConstraint(const YamlObject &modeDict, int32_t id) +{ + for (const auto &[boundName, content] : modeDict.asDict()) { + if (boundName != "upper" && boundName != "lower") { + LOG(AgcMeanLuminance, Warning) + << "Ignoring unknown constraint bound '" << boundName << "'"; + continue; + } + + unsigned int idx = static_cast(boundName == "upper"); + AgcConstraint::Bound bound = static_cast(idx); + double qLo = content["qLo"].get().value_or(0.98); + double qHi = content["qHi"].get().value_or(1.0); + double yTarget = + content["yTarget"].getList().value_or(std::vector{ 0.5 }).at(0); + + AgcConstraint constraint = { bound, qLo, qHi, yTarget }; + + if (!constraintModes_.count(id)) + constraintModes_[id] = {}; + + if (idx) + constraintModes_[id].push_back(constraint); + else + constraintModes_[id].insert(constraintModes_[id].begin(), constraint); + } +} + +int AgcMeanLuminance::parseConstraintModes(const YamlObject &tuningData) +{ + std::vector availableConstraintModes; + + const YamlObject &yamlConstraintModes = tuningData[controls::AeConstraintMode.name()]; + if (yamlConstraintModes.isDictionary()) { + for (const auto &[modeName, modeDict] : yamlConstraintModes.asDict()) { + if (AeConstraintModeNameValueMap.find(modeName) == + AeConstraintModeNameValueMap.end()) { + LOG(AgcMeanLuminance, Warning) + << "Skipping unknown constraint mode '" << modeName << "'"; + continue; + } + + if (!modeDict.isDictionary()) { + LOG(AgcMeanLuminance, Error) + << "Invalid constraint mode '" << modeName << "'"; + return -EINVAL; + } + + parseConstraint(modeDict, + AeConstraintModeNameValueMap.at(modeName)); + availableConstraintModes.push_back( + AeConstraintModeNameValueMap.at(modeName)); + } + } + + /* + * If the tuning data file contains no constraints then we use the + * default constraint that the IPU3/RkISP1 Agc algorithms were adhering + * to anyway before centralisation; this constraint forces the top 2% of + * the histogram to be at least 0.5. + */ + if (constraintModes_.empty()) { + AgcConstraint constraint = { + AgcConstraint::Bound::lower, + 0.98, + 1.0, + 0.5 + }; + + constraintModes_[controls::ConstraintNormal].insert( + constraintModes_[controls::ConstraintNormal].begin(), + constraint); + availableConstraintModes.push_back( + AeConstraintModeNameValueMap.at("ConstraintNormal")); + } + + controls_[&controls::AeConstraintMode] = ControlInfo(availableConstraintModes); + + return 0; +} + +int AgcMeanLuminance::parseExposureModes(const YamlObject &tuningData) +{ + std::vector availableExposureModes; + + const YamlObject &yamlExposureModes = tuningData[controls::AeExposureMode.name()]; + if (yamlExposureModes.isDictionary()) { + for (const auto &[modeName, modeValues] : yamlExposureModes.asDict()) { + if (AeExposureModeNameValueMap.find(modeName) == + AeExposureModeNameValueMap.end()) { + LOG(AgcMeanLuminance, Warning) + << "Skipping unknown exposure mode '" << modeName << "'"; + continue; + } + + if (!modeValues.isDictionary()) { + LOG(AgcMeanLuminance, Error) + << "Invalid exposure mode '" << modeName << "'"; + return -EINVAL; + } + + std::vector shutters = + modeValues["shutter"].getList().value_or(std::vector{}); + std::vector gains = + modeValues["gain"].getList().value_or(std::vector{}); + + if (shutters.size() != gains.size()) { + LOG(AgcMeanLuminance, Error) + << "Shutter and gain array sizes unequal"; + return -EINVAL; + } + + if (shutters.empty()) { + LOG(AgcMeanLuminance, Error) + << "Shutter and gain arrays are empty"; + return -EINVAL; + } + + std::vector> stages; + for (unsigned int i = 0; i < shutters.size(); i++) { + stages.push_back({ + std::chrono::microseconds(shutters[i]), + gains[i] + }); + } + + std::shared_ptr helper = + std::make_shared(stages); + + exposureModeHelpers_[AeExposureModeNameValueMap.at(modeName)] = helper; + availableExposureModes.push_back(AeExposureModeNameValueMap.at(modeName)); + } + } + + /* + * If we don't have any exposure modes in the tuning data we create an + * ExposureModeHelper using an empty vector of stages. This will result + * in the ExposureModeHelper simply driving the shutter as high as + * possible before touching gain. + */ + if (availableExposureModes.empty()) { + int32_t exposureModeId = AeExposureModeNameValueMap.at("ExposureNormal"); + std::vector> stages = { }; + + std::shared_ptr helper = + std::make_shared(stages); + + exposureModeHelpers_[exposureModeId] = helper; + availableExposureModes.push_back(exposureModeId); + } + + controls_[&controls::AeExposureMode] = ControlInfo(availableExposureModes); + + return 0; +} + +/** + * \brief Parse tuning data for AeConstraintMode and AeExposureMode controls + * \param[in] tuningData the YamlObject representing the tuning data + * + * This function parses tuning data to build the list of allowed values for the + * AeConstraintMode and AeExposureMode controls. Those tuning data must provide + * the data in a specific format; the Agc algorithm's tuning data should contain + * a dictionary called AeConstraintMode containing per-mode setting dictionaries + * with the key being a value from \ref controls::AeConstraintModeNameValueMap. + * Each mode dict may contain either a "lower" or "upper" key or both, for + * example: + * + * \code{.unparsed} + * algorithms: + * - Agc: + * AeConstraintMode: + * ConstraintNormal: + * lower: + * qLo: 0.98 + * qHi: 1.0 + * yTarget: 0.5 + * ConstraintHighlight: + * lower: + * qLo: 0.98 + * qHi: 1.0 + * yTarget: 0.5 + * upper: + * qLo: 0.98 + * qHi: 1.0 + * yTarget: 0.8 + * + * \endcode + * + * For the AeExposureMode control the data should contain a dictionary called + * AeExposureMode containing per-mode setting dictionaries with the key being a + * value from \ref controls::AeExposureModeNameValueMap. Each mode dict should + * contain an array of shutter times with the key "shutter" and an array of gain + * values with the key "gain", in this format: + * + * \code{.unparsed} + * algorithms: + * - Agc: + * AeExposureMode: + * ExposureNormal: + * shutter: [ 100, 10000, 30000, 60000, 120000 ] + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ] + * ExposureShort: + * shutter: [ 100, 10000, 30000, 60000, 120000 ] + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ] + * + * \endcode + * + * \return 0 on success or a negative error code + */ +int AgcMeanLuminance::parseTuningData(const YamlObject &tuningData) +{ + int ret; + + parseRelativeLuminanceTarget(tuningData); + + ret = parseConstraintModes(tuningData); + if (ret) + return ret; + + ret = parseExposureModes(tuningData); + if (ret) + return ret; + + return 0; +} + +/** + * \brief Set the ExposureModeHelper limits for this class + * \param[in] minShutter Minimum shutter time to allow + * \param[in] maxShutter Maximum shutter time to allow + * \param[in] minGain Minimum gain to allow + * \param[in] maxGain Maximum gain to allow + * + * This function calls \ref ExposureModeHelper::setLimits() for each + * ExposureModeHelper that has been created for this class. + */ +void AgcMeanLuminance::setLimits(utils::Duration minShutter, + utils::Duration maxShutter, + double minGain, double maxGain) +{ + for (auto &[id, helper] : exposureModeHelpers_) + helper->setLimits(minShutter, maxShutter, minGain, maxGain); +} + +/** + * \fn AgcMeanLuminance::constraintModes() + * \brief Get the constraint modes that have been parsed from tuning data + */ + +/** + * \fn AgcMeanLuminance::exposureModeHelpers() + * \brief Get the ExposureModeHelpers that have been parsed from tuning data + */ + +/** + * \fn AgcMeanLuminance::controls() + * \brief Get the controls that have been generated after parsing tuning data + */ + +/** + * \fn AgcMeanLuminance::estimateLuminance(const double gain) + * \brief Estimate the luminance of an image, adjusted by a given gain + * \param[in] gain The gain with which to adjust the luminance estimate + * + * This function estimates the average relative luminance of the frame that + * would be output by the sensor if an additional \a gain was applied. It is a + * pure virtual function because estimation of luminance is a hardware-specific + * operation, which depends wholly on the format of the stats that are delivered + * to libcamera from the ISP. Derived classes must override this function with + * one that calculates the normalised mean luminance value across the entire + * image. + * + * \return The normalised relative luminance of the image + */ + +/** + * \brief Estimate the initial gain needed to achieve a relative luminance + * target + * \return The calculated initial gain + */ +double AgcMeanLuminance::estimateInitialGain() const +{ + double yTarget = relativeLuminanceTarget_; + double yGain = 1.0; + + /* + * To account for non-linearity caused by saturation, the value needs to + * be estimated in an iterative process, as multiplying by a gain will + * not increase the relative luminance by the same factor if some image + * regions are saturated. + */ + for (unsigned int i = 0; i < 8; i++) { + double yValue = estimateLuminance(yGain); + double extra_gain = std::min(10.0, yTarget / (yValue + .001)); + + yGain *= extra_gain; + LOG(AgcMeanLuminance, Debug) << "Y value: " << yValue + << ", Y target: " << yTarget + << ", gives gain " << yGain; + + if (utils::abs_diff(extra_gain, 1.0) < 0.01) + break; + } + + return yGain; +} + +/** + * \brief Clamp gain within the bounds of a defined constraint + * \param[in] constraintModeIndex The index of the constraint to adhere to + * \param[in] hist A histogram over which to calculate inter-quantile means + * \param[in] gain The gain to clamp + * + * \return The gain clamped within the constraint bounds + */ +double AgcMeanLuminance::constraintClampGain(uint32_t constraintModeIndex, + const Histogram &hist, + double gain) +{ + std::vector &constraints = constraintModes_[constraintModeIndex]; + for (const AgcConstraint &constraint : constraints) { + double newGain = constraint.yTarget * hist.bins() / + hist.interQuantileMean(constraint.qLo, constraint.qHi); + + if (constraint.bound == AgcConstraint::Bound::lower && + newGain > gain) + gain = newGain; + + if (constraint.bound == AgcConstraint::Bound::upper && + newGain < gain) + gain = newGain; + } + + return gain; +} + +/** + * \brief Apply a filter on the exposure value to limit the speed of changes + * \param[in] exposureValue The target exposure from the AGC algorithm + * + * The speed of the filter is adaptive, and will produce the target quicker + * during startup, or when the target exposure is within 20% of the most recent + * filter output. + * + * \return The filtered exposure + */ +utils::Duration AgcMeanLuminance::filterExposure(utils::Duration exposureValue) +{ + double speed = 0.2; + + /* Adapt instantly if we are in startup phase. */ + if (frameCount_ < kNumStartupFrames) + speed = 1.0; + + /* + * If we are close to the desired result, go faster to avoid making + * multiple micro-adjustments. + * \todo Make this customisable? + */ + if (filteredExposure_ < 1.2 * exposureValue && + filteredExposure_ > 0.8 * exposureValue) + speed = sqrt(speed); + + filteredExposure_ = speed * exposureValue + + filteredExposure_ * (1.0 - speed); + + return filteredExposure_; +} + +/** + * \brief Calculate the new exposure value + * \param[in] constraintModeIndex The index of the current constraint mode + * \param[in] exposureModeIndex The index of the current exposure mode + * \param[in] yHist A Histogram from the ISP statistics to use in constraining + * the calculated gain + * \param[in] effectiveExposureValue The EV applied to the frame from which the + * statistics in use derive + * + * Calculate a new exposure value to try to obtain the target. The calculated + * exposure value is filtered to prevent rapid changes from frame to frame, and + * divided into shutter time, analogue and digital gain. + * + * \return Tuple of shutter time, analogue gain, and digital gain + */ +std::tuple +AgcMeanLuminance::calculateNewEv(uint32_t constraintModeIndex, + uint32_t exposureModeIndex, + const Histogram &yHist, + utils::Duration effectiveExposureValue) +{ + /* + * The pipeline handler should validate that we have received an allowed + * value for AeExposureMode. + */ + std::shared_ptr exposureModeHelper = + exposureModeHelpers_.at(exposureModeIndex); + + double gain = estimateInitialGain(); + gain = constraintClampGain(constraintModeIndex, yHist, gain); + + /* + * We don't check whether we're already close to the target, because + * even if the effective exposure value is the same as the last frame's + * we could have switched to an exposure mode that would require a new + * pass through the splitExposure() function. + */ + + utils::Duration newExposureValue = effectiveExposureValue * gain; + utils::Duration maxTotalExposure = exposureModeHelper->maxShutter() + * exposureModeHelper->maxGain(); + newExposureValue = std::min(newExposureValue, maxTotalExposure); + + /* + * We filter the exposure value to make sure changes are not too jarring + * from frame to frame. + */ + newExposureValue = filterExposure(newExposureValue); + + frameCount_++; + return exposureModeHelper->splitExposure(newExposureValue); +} + +/** + * \fn AgcMeanLuminance::resetFrameCount() + * \brief Reset the frame counter + * + * This function resets the internal frame counter, which exists to help the + * algorithm decide whether it should respond instantly or not. The expectation + * is for derived classes to call this function before each camera start call, + * either in configure() or queueRequest() if the frame number is zero. + */ + +}; /* namespace ipa */ + +}; /* namespace libcamera */ diff --git a/src/ipa/libipa/agc_mean_luminance.h b/src/ipa/libipa/agc_mean_luminance.h new file mode 100644 index 00000000..9f91e08f --- /dev/null +++ b/src/ipa/libipa/agc_mean_luminance.h @@ -0,0 +1,90 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Ideas on Board Oy + * + agc_mean_luminance.h - Base class for mean luminance AGC algorithms + */ + +#pragma once + +#include +#include + +#include + +#include "libcamera/internal/yaml_parser.h" + +#include "exposure_mode_helper.h" +#include "histogram.h" + +namespace libcamera { + +namespace ipa { + +class AgcMeanLuminance +{ +public: + AgcMeanLuminance(); + virtual ~AgcMeanLuminance() = default; + + struct AgcConstraint { + enum class Bound { + lower = 0, + upper = 1 + }; + Bound bound; + double qLo; + double qHi; + double yTarget; + }; + + int parseTuningData(const YamlObject &tuningData); + + void setLimits(utils::Duration minShutter, utils::Duration maxShutter, + double minGain, double maxGain); + + std::map> constraintModes() + { + return constraintModes_; + } + + std::map> exposureModeHelpers() + { + return exposureModeHelpers_; + } + + ControlInfoMap::Map controls() + { + return controls_; + } + + std::tuple + calculateNewEv(uint32_t constraintModeIndex, uint32_t exposureModeIndex, + const Histogram &yHist, utils::Duration effectiveExposureValue); + void resetFrameCount() { frameCount_ = 0; } + +private: + virtual double estimateLuminance(const double gain) const = 0; + + void parseRelativeLuminanceTarget(const YamlObject &tuningData); + void parseConstraint(const YamlObject &modeDict, int32_t id); + int parseConstraintModes(const YamlObject &tuningData); + int parseExposureModes(const YamlObject &tuningData); + double estimateInitialGain() const; + double constraintClampGain(uint32_t constraintModeIndex, + const Histogram &hist, + double gain); + utils::Duration filterExposure(utils::Duration exposureValue); + + uint64_t frameCount_; + utils::Duration filteredExposure_; + double relativeLuminanceTarget_; + + std::map> constraintModes_; + std::map> exposureModeHelpers_; + ControlInfoMap::Map controls_; +}; + +}; /* namespace ipa */ + +}; /* namespace libcamera */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index 37fbd177..7ce885da 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -1,6 +1,7 @@ # SPDX-License-Identifier: CC0-1.0 libipa_headers = files([ + 'agc_mean_luminance.h', 'algorithm.h', 'camera_sensor_helper.h', 'exposure_mode_helper.h', @@ -10,6 +11,7 @@ libipa_headers = files([ ]) libipa_sources = files([ + 'agc_mean_luminance.cpp', 'algorithm.cpp', 'camera_sensor_helper.cpp', 'exposure_mode_helper.cpp', From patchwork Wed Apr 24 12:49:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19941 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id EFDF6C3200 for ; Wed, 24 Apr 2024 12:49:50 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id EC21C63408; Wed, 24 Apr 2024 14:49:44 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="Z43GZN9L"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E6390633F5 for ; Wed, 24 Apr 2024 14:49:35 +0200 (CEST) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 0981A11D6; Wed, 24 Apr 2024 14:48:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1713962924; bh=D9agxx/JGb646rcc252NJmykVoWiDlOIUNuOJdHkgSY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Z43GZN9Lhu3q1ikvd1fkh+wYJ+fWY6wi5c5VS+vRLSOiHIa9yh2CNaRr/JiQIW9RG 6D75h7UE19kTfV+px569nzqJ2N+/B6miHybbIx5qyj3k9Vl+sWjCihExCOiCravrAU mLbKUIiu0gWBHk54hDCGcnFI26BSMjNAb3nvVkLM= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally , Stefan Klug Subject: [PATCH v3 5/8] ipa: ipu3: Derive ipu3::algorithms::Agc from AgcMeanLuminance Date: Wed, 24 Apr 2024 13:49:14 +0100 Message-Id: <20240424124917.1250837-6-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240424124917.1250837-1-dan.scally@ideasonboard.com> References: <20240424124917.1250837-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In preparation for switching to a derivation of AgcMeanLuminance, add a function to parse and store the statistics for easy retrieval in an overriding estimateLuminance() function. Now that we have a MeanLuminanceAgc class that centralises our AEGC algorithm, derive the IPU3's Agc class from it and plumb in the necessary framework to enable it to be used. For simplicity's sake this commit switches the algorithm to use the derived class, but does not remove the bespoke functions at this time. Reviewed-by: Stefan Klug Signed-off-by: Daniel Scally --- Changes in v3: - Minor style fixes Changes in v2: - Squashed the patch adding parseStatistics() - Used the first available constraint and exposure modes rather than assuming the "Normal" mode was available - Used a vector of RGB tuples in estimateLuminance() rather than 3 vectors - Stored a copy of the bdsGrid and AWB gains rather than a pointer to the context for use in estimateLuminance() - Remembered to report the controls out to IPAIPU3::updateControls() src/ipa/ipu3/algorithms/agc.cpp | 119 ++++++++++++++++++++++++++++++-- src/ipa/ipu3/algorithms/agc.h | 14 +++- src/ipa/ipu3/ipa_context.cpp | 3 + src/ipa/ipu3/ipa_context.h | 5 ++ src/ipa/ipu3/ipu3.cpp | 3 +- 5 files changed, 135 insertions(+), 9 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 606a237a..b1ba27fe 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -71,11 +71,33 @@ static constexpr uint32_t kNumStartupFrames = 10; static constexpr double kRelativeLuminanceTarget = 0.16; Agc::Agc() - : frameCount_(0), minShutterSpeed_(0s), - maxShutterSpeed_(0s), filteredExposure_(0s) + : minShutterSpeed_(0s), maxShutterSpeed_(0s) { } +/** + * \brief Initialise the AGC algorithm from tuning files + * \param[in] context The shared IPA context + * \param[in] tuningData The YamlObject containing Agc tuning data + * + * This function calls the base class' tuningData parsers to discover which + * control values are supported. + * + * \return 0 on success or errors from the base class + */ +int Agc::init(IPAContext &context, const YamlObject &tuningData) +{ + int ret; + + ret = parseTuningData(tuningData); + if (ret) + return ret; + + context.ctrlMap.merge(controls()); + + return 0; +} + /** * \brief Configure the AGC given a configInfo * \param[in] context The shared IPA context @@ -90,6 +112,7 @@ int Agc::configure(IPAContext &context, IPAActiveState &activeState = context.activeState; stride_ = configuration.grid.stride; + bdsGrid_ = configuration.grid.bdsGrid; minShutterSpeed_ = configuration.agc.minShutterSpeed; maxShutterSpeed_ = std::min(configuration.agc.maxShutterSpeed, @@ -103,6 +126,15 @@ int Agc::configure(IPAContext &context, activeState.agc.exposure = 10ms / configuration.sensor.lineDuration; frameCount_ = 0; + + context.activeState.agc.constraintMode = constraintModes().begin()->first; + context.activeState.agc.exposureMode = exposureModeHelpers().begin()->first; + + /* \todo Run this again when FrameDurationLimits is passed in */ + setLimits(minShutterSpeed_, maxShutterSpeed_, minAnalogueGain_, + maxAnalogueGain_); + resetFrameCount(); + return 0; } @@ -142,6 +174,39 @@ double Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, return Histogram(Span(hist)).interQuantileMean(0.98, 1.0); } +Histogram Agc::parseStatistics(const ipu3_uapi_stats_3a *stats, + const ipu3_uapi_grid_config &grid) +{ + uint32_t hist[knumHistogramBins] = { 0 }; + + rgbTriples_.clear(); + + for (unsigned int cellY = 0; cellY < grid.height; cellY++) { + for (unsigned int cellX = 0; cellX < grid.width; cellX++) { + uint32_t cellPosition = cellY * stride_ + cellX; + + const ipu3_uapi_awb_set_item *cell = + reinterpret_cast( + &stats->awb_raw_buffer.meta_data[cellPosition]); + + rgbTriples_.push_back({ + cell->R_avg, + (cell->Gr_avg + cell->Gb_avg) / 2, + cell->B_avg + }); + + /* + * Store the average green value to estimate the + * brightness. Even the overexposed pixels are + * taken into account. + */ + hist[(cell->Gr_avg + cell->Gb_avg) / 2]++; + } + } + + return Histogram(Span(hist)); +} + /** * \brief Apply a filter on the exposure value to limit the speed of changes * \param[in] exposureValue The target exposure from the AGC algorithm @@ -247,11 +312,6 @@ void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, LOG(IPU3Agc, Debug) << "Divided up shutter and gain are " << shutterTime << " and " << stepGain; - - IPAActiveState &activeState = context.activeState; - /* Update the estimated exposure and gain. */ - activeState.agc.exposure = shutterTime / configuration.sensor.lineDuration; - activeState.agc.gain = stepGain; } /** @@ -314,6 +374,23 @@ double Agc::estimateLuminance(IPAActiveState &activeState, return ySum / (grid.height * grid.width) / 255; } +double Agc::estimateLuminance(double gain) const +{ + double redSum = 0, greenSum = 0, blueSum = 0; + + for (unsigned int i = 0; i < rgbTriples_.size(); i++) { + redSum += std::min(std::get<0>(rgbTriples_[i]) * gain, 255.0); + greenSum += std::min(std::get<1>(rgbTriples_[i]) * gain, 255.0); + blueSum += std::min(std::get<2>(rgbTriples_[i]) * gain, 255.0); + } + + double ySum = redSum * rGain_ * 0.299 + + greenSum * gGain_ * 0.587 + + blueSum * bGain_ * 0.114; + + return ySum / (bdsGrid_.height * bdsGrid_.width) / 255; +} + /** * \brief Process IPU3 statistics, and run AGC operations * \param[in] context The shared IPA context @@ -366,8 +443,36 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, computeExposure(context, frameContext, yGain, iqMeanGain); frameCount_++; + Histogram hist = parseStatistics(stats, context.configuration.grid.bdsGrid); + rGain_ = context.activeState.awb.gains.red; + gGain_ = context.activeState.awb.gains.blue; + bGain_ = context.activeState.awb.gains.green; + + /* + * The Agc algorithm needs to know the effective exposure value that was + * applied to the sensor when the statistics were collected. + */ utils::Duration exposureTime = context.configuration.sensor.lineDuration * frameContext.sensor.exposure; + double analogueGain = frameContext.sensor.gain; + utils::Duration effectiveExposureValue = exposureTime * analogueGain; + + utils::Duration shutterTime; + double aGain, dGain; + std::tie(shutterTime, aGain, dGain) = + calculateNewEv(context.activeState.agc.constraintMode, + context.activeState.agc.exposureMode, hist, + effectiveExposureValue); + + LOG(IPU3Agc, Debug) + << "Divided up shutter, analogue gain and digital gain are " + << shutterTime << ", " << aGain << " and " << dGain; + + IPAActiveState &activeState = context.activeState; + /* Update the estimated exposure and gain. */ + activeState.agc.exposure = shutterTime / context.configuration.sensor.lineDuration; + activeState.agc.gain = aGain; + metadata.set(controls::AnalogueGain, frameContext.sensor.gain); metadata.set(controls::ExposureTime, exposureTime.get()); diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 9d6e3ff1..1d6855a9 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -13,6 +13,9 @@ #include +#include "libipa/agc_mean_luminance.h" +#include "libipa/histogram.h" + #include "algorithm.h" namespace libcamera { @@ -21,12 +24,13 @@ struct IPACameraSensorInfo; namespace ipa::ipu3::algorithms { -class Agc : public Algorithm +class Agc : public Algorithm, public AgcMeanLuminance { public: Agc(); ~Agc() = default; + int init(IPAContext &context, const YamlObject &tuningData) override; int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, @@ -43,6 +47,9 @@ private: const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, double gain); + double estimateLuminance(double gain) const override; + Histogram parseStatistics(const ipu3_uapi_stats_3a *stats, + const ipu3_uapi_grid_config &grid); uint64_t frameCount_; @@ -55,6 +62,11 @@ private: utils::Duration filteredExposure_; uint32_t stride_; + double rGain_; + double gGain_; + double bGain_; + ipu3_uapi_grid_config bdsGrid_; + std::vector> rgbTriples_; }; } /* namespace ipa::ipu3::algorithms */ diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index 959f314f..c4fb5642 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -47,6 +47,9 @@ namespace libcamera::ipa::ipu3 { * * \var IPAContext::activeState * \brief The current state of IPA algorithms + * + * \var IPAContext::ctrlMap + * \brief A ControlInfoMap::Map of controls populated by the algorithms */ /** diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index e9a3863b..a92cb6ce 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -12,6 +12,7 @@ #include +#include #include #include @@ -55,6 +56,8 @@ struct IPAActiveState { struct { uint32_t exposure; double gain; + uint32_t constraintMode; + uint32_t exposureMode; } agc; struct { @@ -85,6 +88,8 @@ struct IPAContext { IPAActiveState activeState; FCQueue frameContexts; + + ControlInfoMap::Map ctrlMap; }; } /* namespace ipa::ipu3 */ diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 08ee6eb3..4809eb60 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -189,7 +189,7 @@ private: }; IPAIPU3::IPAIPU3() - : context_({ {}, {}, { kMaxFrameContexts } }) + : context_({ {}, {}, { kMaxFrameContexts }, {} }) { } @@ -287,6 +287,7 @@ void IPAIPU3::updateControls(const IPACameraSensorInfo &sensorInfo, frameDurations[1], frameDurations[2]); + controls.merge(context_.ctrlMap); *ipaControls = ControlInfoMap(std::move(controls), controls::controls); } From patchwork Wed Apr 24 12:49:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19942 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D5FBDC32C9 for ; Wed, 24 Apr 2024 12:49:51 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3463A633FF; Wed, 24 Apr 2024 14:49:46 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="M+aSs/Bt"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3C60961B43 for ; Wed, 24 Apr 2024 14:49:36 +0200 (CEST) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 4CA207E9; Wed, 24 Apr 2024 14:48:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1713962924; bh=45RoIet6CPkmK0RjX+MSq0X7weRQrlvHZ/mQo3D24yQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=M+aSs/BtgtfdyvkNoZu6Huv+xiofUq/DVVuGDHjTtQzI4vzl9C1GMN834wBQO4XTO uKxO5dOUFHdX6r76UuRpVVAfdlamSoczh6UOu6VG+kHapoeZZbEImIVFbDhHsFWYBC lIZkX+xT9PjbExRf5QWf03EI4P0k8u/41QPiRyi8= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally , Jacopo Mondi , Stefan Klug Subject: [PATCH v3 6/8] ipa: ipu3: Remove bespoke AGC functions from IPU3 Date: Wed, 24 Apr 2024 13:49:15 +0100 Message-Id: <20240424124917.1250837-7-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240424124917.1250837-1-dan.scally@ideasonboard.com> References: <20240424124917.1250837-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that the IPU3's Agc is derived from MeanLuminanceAgc we can delete all the unecessary bespoke functions. Reviewed-by: Jacopo Mondi Reviewed-by: Stefan Klug Signed-off-by: Daniel Scally --- Changes in v3: - \return not @return Changes in v2: - Kept the documentation for estimateLuminance() src/ipa/ipu3/algorithms/agc.cpp | 227 +------------------------------- src/ipa/ipu3/algorithms/agc.h | 13 -- 2 files changed, 3 insertions(+), 237 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index b1ba27fe..e1bca07f 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -125,8 +125,6 @@ int Agc::configure(IPAContext &context, activeState.agc.gain = minAnalogueGain_; activeState.agc.exposure = 10ms / configuration.sensor.lineDuration; - frameCount_ = 0; - context.activeState.agc.constraintMode = constraintModes().begin()->first; context.activeState.agc.exposureMode = exposureModeHelpers().begin()->first; @@ -138,42 +136,6 @@ int Agc::configure(IPAContext &context, return 0; } -/** - * \brief Estimate the mean value of the top 2% of the histogram - * \param[in] stats The statistics computed by the ImgU - * \param[in] grid The grid used to store the statistics in the IPU3 - * \return The mean value of the top 2% of the histogram - */ -double Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, - const ipu3_uapi_grid_config &grid) const -{ - /* Initialise the histogram array */ - uint32_t hist[knumHistogramBins] = { 0 }; - - for (unsigned int cellY = 0; cellY < grid.height; cellY++) { - for (unsigned int cellX = 0; cellX < grid.width; cellX++) { - uint32_t cellPosition = cellY * stride_ + cellX; - - const ipu3_uapi_awb_set_item *cell = - reinterpret_cast( - &stats->awb_raw_buffer.meta_data[cellPosition] - ); - - uint8_t gr = cell->Gr_avg; - uint8_t gb = cell->Gb_avg; - /* - * Store the average green value to estimate the - * brightness. Even the overexposed pixels are - * taken into account. - */ - hist[(gr + gb) / 2]++; - } - } - - /* Estimate the quantile mean of the top 2% of the histogram. */ - return Histogram(Span(hist)).interQuantileMean(0.98, 1.0); -} - Histogram Agc::parseStatistics(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid) { @@ -207,123 +169,9 @@ Histogram Agc::parseStatistics(const ipu3_uapi_stats_3a *stats, return Histogram(Span(hist)); } -/** - * \brief Apply a filter on the exposure value to limit the speed of changes - * \param[in] exposureValue The target exposure from the AGC algorithm - * - * The speed of the filter is adaptive, and will produce the target quicker - * during startup, or when the target exposure is within 20% of the most recent - * filter output. - * - * \return The filtered exposure - */ -utils::Duration Agc::filterExposure(utils::Duration exposureValue) -{ - double speed = 0.2; - - /* Adapt instantly if we are in startup phase. */ - if (frameCount_ < kNumStartupFrames) - speed = 1.0; - - /* - * If we are close to the desired result, go faster to avoid making - * multiple micro-adjustments. - * \todo Make this customisable? - */ - if (filteredExposure_ < 1.2 * exposureValue && - filteredExposure_ > 0.8 * exposureValue) - speed = sqrt(speed); - - filteredExposure_ = speed * exposureValue + - filteredExposure_ * (1.0 - speed); - - LOG(IPU3Agc, Debug) << "After filtering, exposure " << filteredExposure_; - - return filteredExposure_; -} - -/** - * \brief Estimate the new exposure and gain values - * \param[inout] frameContext The shared IPA frame Context - * \param[in] yGain The gain calculated based on the relative luminance target - * \param[in] iqMeanGain The gain calculated based on the relative luminance target - */ -void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, - double yGain, double iqMeanGain) -{ - const IPASessionConfiguration &configuration = context.configuration; - /* Get the effective exposure and gain applied on the sensor. */ - uint32_t exposure = frameContext.sensor.exposure; - double analogueGain = frameContext.sensor.gain; - - /* Use the highest of the two gain estimates. */ - double evGain = std::max(yGain, iqMeanGain); - - /* Consider within 1% of the target as correctly exposed */ - if (utils::abs_diff(evGain, 1.0) < 0.01) - LOG(IPU3Agc, Debug) << "We are well exposed (evGain = " - << evGain << ")"; - - /* extracted from Rpi::Agc::computeTargetExposure */ - - /* Calculate the shutter time in seconds */ - utils::Duration currentShutter = exposure * configuration.sensor.lineDuration; - - /* - * Update the exposure value for the next computation using the values - * of exposure and gain really used by the sensor. - */ - utils::Duration effectiveExposureValue = currentShutter * analogueGain; - - LOG(IPU3Agc, Debug) << "Actual total exposure " << currentShutter * analogueGain - << " Shutter speed " << currentShutter - << " Gain " << analogueGain - << " Needed ev gain " << evGain; - - /* - * Calculate the current exposure value for the scene as the latest - * exposure value applied multiplied by the new estimated gain. - */ - utils::Duration exposureValue = effectiveExposureValue * evGain; - - /* Clamp the exposure value to the min and max authorized */ - utils::Duration maxTotalExposure = maxShutterSpeed_ * maxAnalogueGain_; - exposureValue = std::min(exposureValue, maxTotalExposure); - LOG(IPU3Agc, Debug) << "Target total exposure " << exposureValue - << ", maximum is " << maxTotalExposure; - - /* - * Filter the exposure. - * \todo estimate if we need to desaturate - */ - exposureValue = filterExposure(exposureValue); - - /* - * Divide the exposure value as new exposure and gain values. - * - * Push the shutter time up to the maximum first, and only then - * increase the gain. - */ - utils::Duration shutterTime = - std::clamp(exposureValue / minAnalogueGain_, - minShutterSpeed_, maxShutterSpeed_); - double stepGain = std::clamp(exposureValue / shutterTime, - minAnalogueGain_, maxAnalogueGain_); - LOG(IPU3Agc, Debug) << "Divided up shutter and gain are " - << shutterTime << " and " - << stepGain; -} - /** * \brief Estimate the relative luminance of the frame with a given gain - * \param[in] frameContext The shared IPA frame context - * \param[in] grid The grid used to store the statistics in the IPU3 - * \param[in] stats The IPU3 statistics and ISP results - * \param[in] gain The gain to apply to the frame - * \return The relative luminance - * - * This function estimates the average relative luminance of the frame that - * would be output by the sensor if an additional \a gain was applied. + * \param[in] gain The gain to apply in estimating luminance * * The estimation is based on the AWB statistics for the current frame. Red, * green and blue averages for all cells are first multiplied by the gain, and @@ -338,42 +186,9 @@ void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, * * More detailed information can be found in: * https://en.wikipedia.org/wiki/Relative_luminance + * + * \return The relative luminance of the frame */ -double Agc::estimateLuminance(IPAActiveState &activeState, - const ipu3_uapi_grid_config &grid, - const ipu3_uapi_stats_3a *stats, - double gain) -{ - double redSum = 0, greenSum = 0, blueSum = 0; - - /* Sum the per-channel averages, saturated to 255. */ - for (unsigned int cellY = 0; cellY < grid.height; cellY++) { - for (unsigned int cellX = 0; cellX < grid.width; cellX++) { - uint32_t cellPosition = cellY * stride_ + cellX; - - const ipu3_uapi_awb_set_item *cell = - reinterpret_cast( - &stats->awb_raw_buffer.meta_data[cellPosition] - ); - const uint8_t G_avg = (cell->Gr_avg + cell->Gb_avg) / 2; - - redSum += std::min(cell->R_avg * gain, 255.0); - greenSum += std::min(G_avg * gain, 255.0); - blueSum += std::min(cell->B_avg * gain, 255.0); - } - } - - /* - * Apply the AWB gains to approximate colours correctly, use the Rec. - * 601 formula to calculate the relative luminance, and normalize it. - */ - double ySum = redSum * activeState.awb.gains.red * 0.299 - + greenSum * activeState.awb.gains.green * 0.587 - + blueSum * activeState.awb.gains.blue * 0.114; - - return ySum / (grid.height * grid.width) / 255; -} - double Agc::estimateLuminance(double gain) const { double redSum = 0, greenSum = 0, blueSum = 0; @@ -407,42 +222,6 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, const ipu3_uapi_stats_3a *stats, ControlList &metadata) { - /* - * Estimate the gain needed to have the proportion of pixels in a given - * desired range. iqMean is the mean value of the top 2% of the - * cumulative histogram, and we want it to be as close as possible to a - * configured target. - */ - double iqMean = measureBrightness(stats, context.configuration.grid.bdsGrid); - double iqMeanGain = kEvGainTarget * knumHistogramBins / iqMean; - - /* - * Estimate the gain needed to achieve a relative luminance target. To - * account for non-linearity caused by saturation, the value needs to be - * estimated in an iterative process, as multiplying by a gain will not - * increase the relative luminance by the same factor if some image - * regions are saturated. - */ - double yGain = 1.0; - double yTarget = kRelativeLuminanceTarget; - - for (unsigned int i = 0; i < 8; i++) { - double yValue = estimateLuminance(context.activeState, - context.configuration.grid.bdsGrid, - stats, yGain); - double extraGain = std::min(10.0, yTarget / (yValue + .001)); - - yGain *= extraGain; - LOG(IPU3Agc, Debug) << "Y value: " << yValue - << ", Y target: " << yTarget - << ", gives gain " << yGain; - if (extraGain < 1.01) - break; - } - - computeExposure(context, frameContext, yGain, iqMeanGain); - frameCount_++; - Histogram hist = parseStatistics(stats, context.configuration.grid.bdsGrid); rGain_ = context.activeState.awb.gains.red; gGain_ = context.activeState.awb.gains.blue; diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 1d6855a9..762cc072 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -38,29 +38,16 @@ public: ControlList &metadata) override; private: - double measureBrightness(const ipu3_uapi_stats_3a *stats, - const ipu3_uapi_grid_config &grid) const; - utils::Duration filterExposure(utils::Duration currentExposure); - void computeExposure(IPAContext &context, IPAFrameContext &frameContext, - double yGain, double iqMeanGain); - double estimateLuminance(IPAActiveState &activeState, - const ipu3_uapi_grid_config &grid, - const ipu3_uapi_stats_3a *stats, - double gain); double estimateLuminance(double gain) const override; Histogram parseStatistics(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid); - uint64_t frameCount_; - utils::Duration minShutterSpeed_; utils::Duration maxShutterSpeed_; double minAnalogueGain_; double maxAnalogueGain_; - utils::Duration filteredExposure_; - uint32_t stride_; double rGain_; double gGain_; From patchwork Wed Apr 24 12:49:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19943 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 95CA8C328D for ; Wed, 24 Apr 2024 12:49:52 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 56AF76340E; Wed, 24 Apr 2024 14:49:47 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="sg/C6RFU"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 86BE8633F7 for ; Wed, 24 Apr 2024 14:49:36 +0200 (CEST) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 9360911D6; Wed, 24 Apr 2024 14:48:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1713962924; bh=38kAgtdyIhphVmWeOOq5kSoIZbxpxz9/iMfoQBUaj/E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sg/C6RFUELqf8ZewdVZovrrKbzzW2DwRczLI1z0MnYSDYGMfjGYVKQBKbPZjqCwd0 HfF2hV6kqhMB3qd1abs7VRfGMTcYb4qk3VGxQ21MeKBmBpMrfidajTDilo28lMvVhf 34UbacZFTJBA59pnJZPzvU2TkuZZaOONBQRXENMw= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally , Stefan Klug , Paul Elder Subject: [PATCH v3 7/8] ipa: rkisp1: Derive rkisp1::algorithms::Agc from AgcMeanLuminance Date: Wed, 24 Apr 2024 13:49:16 +0100 Message-Id: <20240424124917.1250837-8-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240424124917.1250837-1-dan.scally@ideasonboard.com> References: <20240424124917.1250837-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that we have a AgcMeanLuminance class that centralises our AEGC algorithm, derive the RkISP1's Agc class from it and plumb in the necessary framework to enable it to be used. For simplicities sake this commit switches the algorithm to use the derived class, but does not remove the bespoke functions at this time. Reviewed-by: Stefan Klug Reviewed-by: Paul Elder Signed-off-by: Daniel Scally --- Changes in v3: - Minor style fixes Changes in v2: - Squashed the patch adding parseStatistics() - Used the first available constraint and exposure modes rather than assuming the "Normal" mode was available - Stopped storing a Histogram in the class members when it's only needed during ::process() - Remembered to report the controls out to IPARkISP1::updateControls() src/ipa/rkisp1/algorithms/agc.cpp | 81 +++++++++++++++++++++++++++++-- src/ipa/rkisp1/algorithms/agc.h | 9 +++- src/ipa/rkisp1/ipa_context.h | 5 ++ src/ipa/rkisp1/rkisp1.cpp | 3 +- 4 files changed, 91 insertions(+), 7 deletions(-) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 47a6f7b2..f8d1b9a3 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -64,6 +64,29 @@ Agc::Agc() supportsRaw_ = true; } +/** + * \brief Initialise the AGC algorithm from tuning files + * \param[in] context The shared IPA context + * \param[in] tuningData The YamlObject containing Agc tuning data + * + * This function calls the base class' tuningData parsers to discover which + * control values are supported. + * + * \return 0 on success or errors from the base class + */ +int Agc::init(IPAContext &context, const YamlObject &tuningData) +{ + int ret; + + ret = parseTuningData(tuningData); + if (ret) + return ret; + + context.ctrlMap.merge(controls()); + + return 0; +} + /** * \brief Configure the AGC given a configInfo * \param[in] context The shared IPA context @@ -81,6 +104,9 @@ int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) context.activeState.agc.manual.exposure = context.activeState.agc.automatic.exposure; context.activeState.agc.autoEnabled = !context.configuration.raw; + context.activeState.agc.constraintMode = constraintModes().begin()->first; + context.activeState.agc.exposureMode = exposureModeHelpers().begin()->first; + /* * Define the measurement window for AGC as a centered rectangle * covering 3/4 of the image width and height. @@ -95,6 +121,15 @@ int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) * frame index */ frameCount_ = 0; + + /* \todo Run this again when FrameDurationLimits is passed in */ + setLimits(context.configuration.sensor.minShutterSpeed, + context.configuration.sensor.maxShutterSpeed, + context.configuration.sensor.minAnalogueGain, + context.configuration.sensor.maxAnalogueGain); + + resetFrameCount(); + return 0; } @@ -234,7 +269,6 @@ void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, double yGain, double iqMeanGain) { IPASessionConfiguration &configuration = context.configuration; - IPAActiveState &activeState = context.activeState; /* Get the effective exposure and gain applied on the sensor. */ uint32_t exposure = frameContext.sensor.exposure; @@ -300,10 +334,6 @@ void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, LOG(RkISP1Agc, Debug) << "Divided up shutter and gain are " << shutterTime << " and " << stepGain; - - /* Update the estimated exposure and gain. */ - activeState.agc.automatic.exposure = shutterTime / configuration.sensor.lineDuration; - activeState.agc.automatic.gain = stepGain; } /** @@ -373,6 +403,19 @@ void Agc::fillMetadata(IPAContext &context, IPAFrameContext &frameContext, metadata.set(controls::FrameDuration, frameDuration.get()); } +double Agc::estimateLuminance(double gain) const +{ + double ySum = 0.0; + + /* Sum the averages, saturated to 255. */ + for (uint8_t expMean : expMeans_) + ySum += std::min(expMean * gain, 255.0); + + /* \todo Weight with the AWB gains */ + + return ySum / expMeans_.size() / 255; +} + /** * \brief Process RkISP1 statistics, and run AGC operations * \param[in] context The shared IPA context @@ -438,7 +481,35 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, computeExposure(context, frameContext, yGain, iqMeanGain); frameCount_++; + expMeans_ = { params->ae.exp_mean, context.hw->numAeCells }; + + /* + * The Agc algorithm needs to know the effective exposure value that was + * applied to the sensor when the statistics were collected. + */ + utils::Duration exposureTime = context.configuration.sensor.lineDuration + * frameContext.sensor.exposure; + double analogueGain = frameContext.sensor.gain; + utils::Duration effectiveExposureValue = exposureTime * analogueGain; + + utils::Duration shutterTime; + double aGain, dGain; + std::tie(shutterTime, aGain, dGain) = + calculateNewEv(context.activeState.agc.constraintMode, + context.activeState.agc.exposureMode, + Histogram(hist), effectiveExposureValue); + + LOG(RkISP1Agc, Debug) + << "Divided up shutter, analogue gain and digital gain are " + << shutterTime << ", " << aGain << " and " << dGain; + + IPAActiveState &activeState = context.activeState; + /* Update the estimated exposure and gain. */ + activeState.agc.automatic.exposure = shutterTime / context.configuration.sensor.lineDuration; + activeState.agc.automatic.gain = aGain; + fillMetadata(context, frameContext, metadata); + expMeans_ = {}; } REGISTER_IPA_ALGORITHM(Agc, "Agc") diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h index fb82a33f..34504459 100644 --- a/src/ipa/rkisp1/algorithms/agc.h +++ b/src/ipa/rkisp1/algorithms/agc.h @@ -14,18 +14,22 @@ #include +#include "libipa/agc_mean_luminance.h" +#include "libipa/histogram.h" + #include "algorithm.h" namespace libcamera { namespace ipa::rkisp1::algorithms { -class Agc : public Algorithm +class Agc : public Algorithm, public AgcMeanLuminance { public: Agc(); ~Agc() = default; + int init(IPAContext &context, const YamlObject &tuningData) override; int configure(IPAContext &context, const IPACameraSensorInfo &configInfo) override; void queueRequest(IPAContext &context, const uint32_t frame, @@ -47,10 +51,13 @@ private: double measureBrightness(Span hist) const; void fillMetadata(IPAContext &context, IPAFrameContext &frameContext, ControlList &metadata); + double estimateLuminance(double gain) const override; uint64_t frameCount_; utils::Duration filteredExposure_; + + Span expMeans_; }; } /* namespace ipa::rkisp1::algorithms */ diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index 10d8f38c..256b75eb 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -12,6 +12,7 @@ #include +#include #include #include @@ -67,6 +68,8 @@ struct IPAActiveState { } automatic; bool autoEnabled; + uint32_t constraintMode; + uint32_t exposureMode; } agc; struct { @@ -151,6 +154,8 @@ struct IPAContext { IPAActiveState activeState; FCQueue frameContexts; + + ControlInfoMap::Map ctrlMap; }; } /* namespace ipa::rkisp1 */ diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 9dc5f53c..d8610095 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -119,7 +119,7 @@ const ControlInfoMap::Map rkisp1Controls{ } /* namespace */ IPARkISP1::IPARkISP1() - : context_({ {}, {}, {}, { kMaxFrameContexts } }) + : context_({ {}, {}, {}, { kMaxFrameContexts }, {} }) { } @@ -427,6 +427,7 @@ void IPARkISP1::updateControls(const IPACameraSensorInfo &sensorInfo, frameDurations[1], frameDurations[2]); + ctrlMap.merge(context_.ctrlMap); *ipaControls = ControlInfoMap(std::move(ctrlMap), controls::controls); } From patchwork Wed Apr 24 12:49:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19944 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4B696C32CA for ; Wed, 24 Apr 2024 12:49:53 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8F78B63413; Wed, 24 Apr 2024 14:49:48 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="tPcoVlUq"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E954A633EC for ; Wed, 24 Apr 2024 14:49:36 +0200 (CEST) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id DF2EB7E9; Wed, 24 Apr 2024 14:48:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1713962925; bh=icsVYqpzWKMdxWoJx49k/I/aMqjnFH3qlC4wOOXU8Bc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tPcoVlUqs/Y2oWZK0nnsKmEkhyNyWR3zTL+tdgkCChBm7b7Jj8UTfUZwaKOLx+SU6 EP1sIlN6NBVEaY9z8cgQ90018uzRlKiYmk8+l/DrEj93XaWp3bCbU04i0Y1X1LgqlO uueOGvz4IH+IUo9NW165bw12/BcIB596FXBjMGyg= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally , Stefan Klug , Paul Elder , Jacopo Mondi Subject: [PATCH v3 8/8] ipa: rkisp1: Remove bespoke Agc functions Date: Wed, 24 Apr 2024 13:49:17 +0100 Message-Id: <20240424124917.1250837-9-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240424124917.1250837-1-dan.scally@ideasonboard.com> References: <20240424124917.1250837-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that the rkisp1 Agc algorithm is a derivation of MeanLuminanceAgc we can remove the bespoke functions from the IPA's class. Reviewed-by: Stefan Klug Reviewed-by: Paul Elder Reviewed-by: Jacopo Mondi Signed-off-by: Daniel Scally --- Changes in v3: - None Changes in v2: - Kept the documentation for estimateLuminance() src/ipa/rkisp1/algorithms/agc.cpp | 231 ++---------------------------- src/ipa/rkisp1/algorithms/agc.h | 9 -- 2 files changed, 14 insertions(+), 226 deletions(-) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index f8d1b9a3..3e98496b 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -36,30 +36,7 @@ namespace ipa::rkisp1::algorithms { LOG_DEFINE_CATEGORY(RkISP1Agc) -/* Minimum limit for analogue gain value */ -static constexpr double kMinAnalogueGain = 1.0; - -/* \todo Honour the FrameDurationLimits control instead of hardcoding a limit */ -static constexpr utils::Duration kMaxShutterSpeed = 60ms; - -/* Number of frames to wait before calculating stats on minimum exposure */ -static constexpr uint32_t kNumStartupFrames = 10; - -/* Target value to reach for the top 2% of the histogram */ -static constexpr double kEvGainTarget = 0.5; - -/* - * Relative luminance target. - * - * It's a number that's chosen so that, when the camera points at a grey - * target, the resulting image brightness is considered right. - * - * \todo Why is the value different between IPU3 and RkISP1 ? - */ -static constexpr double kRelativeLuminanceTarget = 0.4; - Agc::Agc() - : frameCount_(0), filteredExposure_(0s) { supportsRaw_ = true; } @@ -116,12 +93,6 @@ int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) context.configuration.agc.measureWindow.h_size = 3 * configInfo.outputSize.width / 4; context.configuration.agc.measureWindow.v_size = 3 * configInfo.outputSize.height / 4; - /* - * \todo Use the upcoming per-frame context API that will provide a - * frame index - */ - frameCount_ = 0; - /* \todo Run this again when FrameDurationLimits is passed in */ setLimits(context.configuration.sensor.minShutterSpeed, context.configuration.sensor.maxShutterSpeed, @@ -223,122 +194,24 @@ void Agc::prepare(IPAContext &context, const uint32_t frame, params->module_en_update |= RKISP1_CIF_ISP_MODULE_HST; } -/** - * \brief Apply a filter on the exposure value to limit the speed of changes - * \param[in] exposureValue The target exposure from the AGC algorithm - * - * The speed of the filter is adaptive, and will produce the target quicker - * during startup, or when the target exposure is within 20% of the most recent - * filter output. - * - * \return The filtered exposure - */ -utils::Duration Agc::filterExposure(utils::Duration exposureValue) -{ - double speed = 0.2; - - /* Adapt instantly if we are in startup phase. */ - if (frameCount_ < kNumStartupFrames) - speed = 1.0; - - /* - * If we are close to the desired result, go faster to avoid making - * multiple micro-adjustments. - * \todo Make this customisable? - */ - if (filteredExposure_ < 1.2 * exposureValue && - filteredExposure_ > 0.8 * exposureValue) - speed = sqrt(speed); - - filteredExposure_ = speed * exposureValue + - filteredExposure_ * (1.0 - speed); - - LOG(RkISP1Agc, Debug) << "After filtering, exposure " << filteredExposure_; - - return filteredExposure_; -} - -/** - * \brief Estimate the new exposure and gain values - * \param[inout] context The shared IPA Context - * \param[in] frameContext The FrameContext for this frame - * \param[in] yGain The gain calculated on the current brightness level - * \param[in] iqMeanGain The gain calculated based on the relative luminance target - */ -void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, - double yGain, double iqMeanGain) +void Agc::fillMetadata(IPAContext &context, IPAFrameContext &frameContext, + ControlList &metadata) { - IPASessionConfiguration &configuration = context.configuration; - - /* Get the effective exposure and gain applied on the sensor. */ - uint32_t exposure = frameContext.sensor.exposure; - double analogueGain = frameContext.sensor.gain; - - /* Use the highest of the two gain estimates. */ - double evGain = std::max(yGain, iqMeanGain); - - utils::Duration minShutterSpeed = configuration.sensor.minShutterSpeed; - utils::Duration maxShutterSpeed = std::min(configuration.sensor.maxShutterSpeed, - kMaxShutterSpeed); - - double minAnalogueGain = std::max(configuration.sensor.minAnalogueGain, - kMinAnalogueGain); - double maxAnalogueGain = configuration.sensor.maxAnalogueGain; - - /* Consider within 1% of the target as correctly exposed. */ - if (utils::abs_diff(evGain, 1.0) < 0.01) - return; - - /* extracted from Rpi::Agc::computeTargetExposure. */ - - /* Calculate the shutter time in seconds. */ - utils::Duration currentShutter = exposure * configuration.sensor.lineDuration; - - /* - * Update the exposure value for the next computation using the values - * of exposure and gain really used by the sensor. - */ - utils::Duration effectiveExposureValue = currentShutter * analogueGain; - - LOG(RkISP1Agc, Debug) << "Actual total exposure " << currentShutter * analogueGain - << " Shutter speed " << currentShutter - << " Gain " << analogueGain - << " Needed ev gain " << evGain; - - /* - * Calculate the current exposure value for the scene as the latest - * exposure value applied multiplied by the new estimated gain. - */ - utils::Duration exposureValue = effectiveExposureValue * evGain; - - /* Clamp the exposure value to the min and max authorized. */ - utils::Duration maxTotalExposure = maxShutterSpeed * maxAnalogueGain; - exposureValue = std::min(exposureValue, maxTotalExposure); - LOG(RkISP1Agc, Debug) << "Target total exposure " << exposureValue - << ", maximum is " << maxTotalExposure; - - /* - * Divide the exposure value as new exposure and gain values. - * \todo estimate if we need to desaturate - */ - exposureValue = filterExposure(exposureValue); + utils::Duration exposureTime = context.configuration.sensor.lineDuration + * frameContext.sensor.exposure; + metadata.set(controls::AnalogueGain, frameContext.sensor.gain); + metadata.set(controls::ExposureTime, exposureTime.get()); - /* - * Push the shutter time up to the maximum first, and only then - * increase the gain. - */ - utils::Duration shutterTime = std::clamp(exposureValue / minAnalogueGain, - minShutterSpeed, maxShutterSpeed); - double stepGain = std::clamp(exposureValue / shutterTime, - minAnalogueGain, maxAnalogueGain); - LOG(RkISP1Agc, Debug) << "Divided up shutter and gain are " - << shutterTime << " and " - << stepGain; + /* \todo Use VBlank value calculated from each frame exposure. */ + uint32_t vTotal = context.configuration.sensor.size.height + + context.configuration.sensor.defVBlank; + utils::Duration frameDuration = context.configuration.sensor.lineDuration + * vTotal; + metadata.set(controls::FrameDuration, frameDuration.get()); } /** * \brief Estimate the relative luminance of the frame with a given gain - * \param[in] expMeans The mean luminance values, from the RkISP1 statistics * \param[in] gain The gain to apply to the frame * * This function estimates the average relative luminance of the frame that @@ -352,8 +225,6 @@ void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, * YUV doesn't take into account the fact that the R, G and B components * contribute differently to the relative luminance. * - * \todo Have a dedicated YUV algorithm ? - * * The values are normalized to the [0.0, 1.0] range, where 1.0 corresponds to a * theoretical perfect reflector of 100% reference white. * @@ -362,47 +233,6 @@ void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, * * \return The relative luminance */ -double Agc::estimateLuminance(Span expMeans, double gain) -{ - double ySum = 0.0; - - /* Sum the averages, saturated to 255. */ - for (uint8_t expMean : expMeans) - ySum += std::min(expMean * gain, 255.0); - - /* \todo Weight with the AWB gains */ - - return ySum / expMeans.size() / 255; -} - -/** - * \brief Estimate the mean value of the top 2% of the histogram - * \param[in] hist The histogram statistics computed by the RkISP1 - * \return The mean value of the top 2% of the histogram - */ -double Agc::measureBrightness(Span hist) const -{ - Histogram histogram{ hist }; - /* Estimate the quantile mean of the top 2% of the histogram. */ - return histogram.interQuantileMean(0.98, 1.0); -} - -void Agc::fillMetadata(IPAContext &context, IPAFrameContext &frameContext, - ControlList &metadata) -{ - utils::Duration exposureTime = context.configuration.sensor.lineDuration - * frameContext.sensor.exposure; - metadata.set(controls::AnalogueGain, frameContext.sensor.gain); - metadata.set(controls::ExposureTime, exposureTime.get()); - - /* \todo Use VBlank value calculated from each frame exposure. */ - uint32_t vTotal = context.configuration.sensor.size.height - + context.configuration.sensor.defVBlank; - utils::Duration frameDuration = context.configuration.sensor.lineDuration - * vTotal; - metadata.set(controls::FrameDuration, frameDuration.get()); -} - double Agc::estimateLuminance(double gain) const { double ySum = 0.0; @@ -447,40 +277,7 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, const rkisp1_cif_isp_stat *params = &stats->params; ASSERT(stats->meas_type & RKISP1_CIF_ISP_STAT_AUTOEXP); - Span ae{ params->ae.exp_mean, context.hw->numAeCells }; - Span hist{ - params->hist.hist_bins, - context.hw->numHistogramBins - }; - - double iqMean = measureBrightness(hist); - double iqMeanGain = kEvGainTarget * hist.size() / iqMean; - - /* - * Estimate the gain needed to achieve a relative luminance target. To - * account for non-linearity caused by saturation, the value needs to be - * estimated in an iterative process, as multiplying by a gain will not - * increase the relative luminance by the same factor if some image - * regions are saturated. - */ - double yGain = 1.0; - double yTarget = kRelativeLuminanceTarget; - - for (unsigned int i = 0; i < 8; i++) { - double yValue = estimateLuminance(ae, yGain); - double extra_gain = std::min(10.0, yTarget / (yValue + .001)); - - yGain *= extra_gain; - LOG(RkISP1Agc, Debug) << "Y value: " << yValue - << ", Y target: " << yTarget - << ", gives gain " << yGain; - if (extra_gain < 1.01) - break; - } - - computeExposure(context, frameContext, yGain, iqMeanGain); - frameCount_++; - + Histogram hist({ params->hist.hist_bins, context.hw->numHistogramBins }); expMeans_ = { params->ae.exp_mean, context.hw->numAeCells }; /* @@ -497,7 +294,7 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, std::tie(shutterTime, aGain, dGain) = calculateNewEv(context.activeState.agc.constraintMode, context.activeState.agc.exposureMode, - Histogram(hist), effectiveExposureValue); + hist, effectiveExposureValue); LOG(RkISP1Agc, Debug) << "Divided up shutter, analogue gain and digital gain are " diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h index 34504459..f2f5b59d 100644 --- a/src/ipa/rkisp1/algorithms/agc.h +++ b/src/ipa/rkisp1/algorithms/agc.h @@ -44,19 +44,10 @@ public: ControlList &metadata) override; private: - void computeExposure(IPAContext &Context, IPAFrameContext &frameContext, - double yGain, double iqMeanGain); - utils::Duration filterExposure(utils::Duration exposureValue); - double estimateLuminance(Span expMeans, double gain); - double measureBrightness(Span hist) const; void fillMetadata(IPAContext &context, IPAFrameContext &frameContext, ControlList &metadata); double estimateLuminance(double gain) const override; - uint64_t frameCount_; - - utils::Duration filteredExposure_; - Span expMeans_; };