From patchwork Wed Apr 17 13:15:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19893 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 40174C3213 for ; Wed, 17 Apr 2024 13:16:02 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 38E9763360; Wed, 17 Apr 2024 15:15:58 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="lMyNwe96"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B5ED86333E for ; Wed, 17 Apr 2024 15:15:55 +0200 (CEST) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 668F1B7E; Wed, 17 Apr 2024 15:15:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1713359708; bh=L42UjfQnNuQ9DdPr4ZojsxRqIJJjIkTcGJWScPeL+AM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lMyNwe9668/iX2mpChxNDQmjYIOCp0P1M5P6PVtl8Skm7C7GkvyO7KQSPga8mmSQ8 QT5TqdOSK9SMOrcJbge4rZBMj9fbhtZExm3AyDmEYqiL9bnluebj1yNvh5/HPYeHoT IGhxWasgoG/W3pqb7Oi6Izf9V7HkyDoLLKuL8Jm4= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally , Stefan Klug , Paul Elder Subject: [PATCH v2 1/8] ipa: libipa: Allow creation of empty Histogram Date: Wed, 17 Apr 2024 14:15:29 +0100 Message-Id: <20240417131536.484129-2-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240417131536.484129-1-dan.scally@ideasonboard.com> References: <20240417131536.484129-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" For convenience's sake allow the creation of empty Histograms so they can be embedded within other Classes and filled out with data at some later point in time. Reviewed-by: Stefan Klug Reviewed-by: Paul Elder Signed-off-by: Daniel Scally --- Changes in v2: - Added documentation src/ipa/libipa/histogram.cpp | 9 +++++++++ src/ipa/libipa/histogram.h | 1 + 2 files changed, 10 insertions(+) diff --git a/src/ipa/libipa/histogram.cpp b/src/ipa/libipa/histogram.cpp index 6b5cde8e..c1aac59b 100644 --- a/src/ipa/libipa/histogram.cpp +++ b/src/ipa/libipa/histogram.cpp @@ -28,6 +28,15 @@ namespace ipa { * specified bin. It can be used to find quantiles and averages between quantiles. */ +/** + * \fn Histogram::Histogram() + * \brief Construct an empty Histogram + * + * This empty constructor exists largely to allow Histograms to be embedded in + * other classes which may be created before the contents of the Histogram are + * known. + */ + /** * \brief Create a cumulative histogram * \param[in] data A pre-sorted histogram to be passed diff --git a/src/ipa/libipa/histogram.h b/src/ipa/libipa/histogram.h index 05bb4b80..54bb2a19 100644 --- a/src/ipa/libipa/histogram.h +++ b/src/ipa/libipa/histogram.h @@ -22,6 +22,7 @@ namespace ipa { class Histogram { public: + Histogram() { cumulative_.push_back(0); } Histogram(Span data); size_t bins() const { return cumulative_.size() - 1; } uint64_t total() const { return cumulative_[cumulative_.size() - 1]; } From patchwork Wed Apr 17 13:15:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19894 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7CC45C3285 for ; Wed, 17 Apr 2024 13:16:03 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4CF5663368; Wed, 17 Apr 2024 15:15:59 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="gUHEmfXl"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DDFCE63340 for ; Wed, 17 Apr 2024 15:15:55 +0200 (CEST) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id B91831815; Wed, 17 Apr 2024 15:15:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1713359709; bh=GSyGmtldUbDzvBvOzFAnN3wzCNoqvZXPp6r+i5KS25M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gUHEmfXlKuCNfMtLJ4pyCniR8Ukuc6AALo5uhjjv0DWUOey83T+hFhy6ve7lujtMy 3fAgKNWDaI+DUh/AvEWGTybDSLoc33d5xbSo7FsDcV10lUv1q5PdGPqlP2qKOe8DAx qR+Un1c13Tkr8/0dr4hpxrjtuRYxylepojqXCP8w= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally , Stefan Klug , Jacopo Mondi , Paul Elder Subject: [PATCH v2 2/8] libcamera: controls: Generate enum value-name maps Date: Wed, 17 Apr 2024 14:15:30 +0100 Message-Id: <20240417131536.484129-3-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240417131536.484129-1-dan.scally@ideasonboard.com> References: <20240417131536.484129-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Generate maps for each control enum which associate strings that represent the enum values with the values themselves. This change will allow us to refer to enumerated control values using the string. For example if we want to pass variables to an algorithm for use when a control has a particular value we can embed within tuning files a dictionary that uses the control values as keys. Reviewed-by: Stefan Klug Reviewed-by: Jacopo Mondi Reviewed-by: Paul Elder Signed-off-by: Daniel Scally --- Changes in v2: - Expanded commit message include/libcamera/control_ids.h.in | 2 ++ include/libcamera/property_ids.h.in | 2 ++ utils/gen-controls.py | 19 +++++++++++++++++++ 3 files changed, 23 insertions(+) diff --git a/include/libcamera/control_ids.h.in b/include/libcamera/control_ids.h.in index d53b1cf7..58dd48e1 100644 --- a/include/libcamera/control_ids.h.in +++ b/include/libcamera/control_ids.h.in @@ -10,7 +10,9 @@ #pragma once #include +#include #include +#include #include diff --git a/include/libcamera/property_ids.h.in b/include/libcamera/property_ids.h.in index 43372c71..f51ba028 100644 --- a/include/libcamera/property_ids.h.in +++ b/include/libcamera/property_ids.h.in @@ -9,7 +9,9 @@ #pragma once +#include #include +#include #include diff --git a/utils/gen-controls.py b/utils/gen-controls.py index 6cd5e362..4fe1e705 100755 --- a/utils/gen-controls.py +++ b/utils/gen-controls.py @@ -140,6 +140,12 @@ ${description} */''') enum_values_start = string.Template('''extern const std::array ${name}Values = {''') enum_values_values = string.Template('''\tstatic_cast(${name}),''') + name_value_map_doc = string.Template('''/** + * \\var ${name}NameValueMap + * \\brief Map of all $name supported value names (in std::string format) to value + */''') + name_value_map_start = string.Template('''extern const std::map ${name}NameValueMap = {''') + name_value_values = string.Template('''\t{ "${name}", ${name} },''') ctrls_doc = {} ctrls_def = {} @@ -183,6 +189,7 @@ ${description} values_info = { 'name': info['name'], + 'type': ctrl.type, 'size': num_entries, } target_doc.append(enum_values_doc.substitute(values_info)) @@ -194,6 +201,15 @@ ${description} target_def.append(enum_values_values.substitute(value_info)) target_def.append("};") + target_doc.append(name_value_map_doc.substitute(values_info)) + target_def.append(name_value_map_start.substitute(values_info)) + for enum in ctrl.enum_values: + value_info = { + 'name': enum.name + } + target_def.append(name_value_values.substitute(value_info)) + target_def.append("};") + target_doc.append(doc_template.substitute(info)) target_def.append(def_template.substitute(info)) @@ -231,6 +247,7 @@ def generate_h(controls, mode, ranges): enum_template_start = string.Template('''enum ${name}Enum {''') enum_value_template = string.Template('''\t${name} = ${value},''') enum_values_template = string.Template('''extern const std::array ${name}Values;''') + name_value_map_template = string.Template('''extern const std::map ${name}NameValueMap;''') template = string.Template('''extern const Control<${type}> ${name};''') ctrls = {} @@ -273,9 +290,11 @@ def generate_h(controls, mode, ranges): values_info = { 'name': info['name'], + 'type': ctrl.type, 'size': num_entries, } target_ctrls.append(enum_values_template.substitute(values_info)) + target_ctrls.append(name_value_map_template.substitute(values_info)) target_ctrls.append(template.substitute(info)) id_value[vendor] += 1 From patchwork Wed Apr 17 13:15:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19895 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7478EC3213 for ; Wed, 17 Apr 2024 13:16:04 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 80C166336C; Wed, 17 Apr 2024 15:16:01 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="bU0TrXU0"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2967961B54 for ; Wed, 17 Apr 2024 15:15:56 +0200 (CEST) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 2097A63F; Wed, 17 Apr 2024 15:15:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1713359709; bh=aXniyL4nNy5kE3pchIQ3LFPkBE/O75lGYCEQDj60lqA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bU0TrXU0QX6BHVkRCwFuG2RrCqIGWN4c2ugxlwb0aKtat70G3OsKTxWBA11TiqZ7b IMmx4Di4rWIPJc96sQlT618hz1kkEnbH8+ZN4RY3wlPelW4HNwsLoXJvjuC2UKSsI/ M9UHC6Kw5knPM4VxIozecQ+gboQPwJc7jwpLQw88= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Paul Elder , Daniel Scally Subject: [PATCH v2 3/8] ipa: libipa: Add ExposureModeHelper Date: Wed, 17 Apr 2024 14:15:31 +0100 Message-Id: <20240417131536.484129-4-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240417131536.484129-1-dan.scally@ideasonboard.com> References: <20240417131536.484129-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Paul Elder Add a helper for managing exposure modes and splitting exposure times into shutter and gain values. Signed-off-by: Paul Elder Signed-off-by: Daniel Scally Reviewed-by: Paul Elder --- Changes in v2: - Expanded the documentation - Dropped the overloads for fixed shutter / gain - the same functionality is instead done by setting min and max shutter and gain to the same value - Changed ::init() to consume a vector of pairs instead of two separate vectors - Reworked splitExposure() src/ipa/libipa/exposure_mode_helper.cpp | 257 ++++++++++++++++++++++++ src/ipa/libipa/exposure_mode_helper.h | 53 +++++ src/ipa/libipa/meson.build | 2 + 3 files changed, 312 insertions(+) create mode 100644 src/ipa/libipa/exposure_mode_helper.cpp create mode 100644 src/ipa/libipa/exposure_mode_helper.h diff --git a/src/ipa/libipa/exposure_mode_helper.cpp b/src/ipa/libipa/exposure_mode_helper.cpp new file mode 100644 index 00000000..6463de18 --- /dev/null +++ b/src/ipa/libipa/exposure_mode_helper.cpp @@ -0,0 +1,257 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Paul Elder + * + * exposure_mode_helper.cpp - Helper class that performs computations relating to exposure + */ +#include "exposure_mode_helper.h" + +#include + +#include + +/** + * \file exposure_mode_helper.h + * \brief Helper class that performs computations relating to exposure + * + * AEGC algorithms have a need to split exposure between shutter and gain. + * Multiple implementations do so based on paired sets of shutter and gain + * limits; provide a helper to avoid duplicating the code. + */ + +namespace libcamera { + +LOG_DEFINE_CATEGORY(ExposureModeHelper) + +namespace ipa { + +/** + * \class ExposureModeHelper + * \brief Class for splitting exposure into shutter and gain + * + * The ExposureModeHelper class provides a standard interface through which an + * AEGC algorithm can divide exposure between shutter and gain. It is configured + * with a set of shutter and gain pairs and works by initially fixing gain at + * 1.0 and increasing shutter up to the shutter value from the first pair in the + * set in an attempt to meet the required exposure value. + * + * If the required exposure is not achievable by the first shutter value alone + * it ramps gain up to the value from the first pair in the set. If the required + * exposure is still not met it then allows shutter to ramp up to the shutter + * value from the second pair in the set, and continues in this vein until + * either the required exposure value is met, or else the hardware's shutter or + * gain limits are reached. + * + * This method allows users to strike a balance between a well-exposed image and + * an acceptable frame-rate, as opposed to simply maximising shutter followed by + * gain. The same helpers can be used to perform the latter operation if needed + * by passing an empty set of pairs to the initialisation function. + * + * The gain values may exceed a camera sensor's analogue gain limits if either + * it or the IPA is also capable of digital gain. The configure() function must + * be called with the hardware's limits to inform the helper of those + * constraints. Any gain that is needed will be applied as analogue gain first + * until the hardware's limit is reached, following which digital gain will be + * used. + */ + +/** + * \brief Construct an ExposureModeHelper instance + */ +ExposureModeHelper::ExposureModeHelper() + : minShutter_(0), maxShutter_(0), minGain_(0), maxGain_(0) +{ +} + +/** + * \brief Initialize an ExposureModeHelper instance + * \param[in] stages The vector of paired shutter time and gain limits + * + * This function is expected to be called a single time once the algorithm that + * is using these helpers has built the necessary list of shutter and gain pairs + * to use (archetypically by parsing them from a tuning file during the + * algorithm's .init() call). + * + * The input steps are shutter and _total_ gain pairs; the gain encompasses both + * analogue and digital gain. + * + * The vector of stages may be empty. In that case, the helper will simply use + * the runtime limits set through setShutterGainLimits() instead. + */ +void ExposureModeHelper::init(const std::vector> stages) +{ + /* We only need to check shutters_, as gains_ is filled alongside it */ + if (!shutters_.empty()) { + LOG(ExposureModeHelper, Warning) + << "Initialization attempted multiple times"; + return; + } + + for (auto stage : stages) { + shutters_.push_back(stage.first); + gains_.push_back(stage.second); + } +} + +/** + * \brief Set the shutter and gain limits + * \param[in] minShutter The minimum shutter time supported + * \param[in] maxShutter The maximum shutter time supported + * \param[in] minGain The minimum analogue gain supported + * \param[in] maxGain The maximum analogue gain supported + * + * This function configures the shutter and analogue gain limits that need to be + * adhered to as the helper divides up exposure. Note that these function *must* + * be called whenever those limits change and before splitExposure() is used. + * + * If the algorithm using the helpers needs to indicate that either shutter, + * analogue gain or both should be fixed it can do so by setting both the minima + * and maxima to the same value. + */ +void ExposureModeHelper::setShutterGainLimits(utils::Duration minShutter, + utils::Duration maxShutter, + double minGain, + double maxGain) +{ + minShutter_ = minShutter; + maxShutter_ = maxShutter; + minGain_ = minGain; + maxGain_ = maxGain; +} + +utils::Duration ExposureModeHelper::clampShutter(utils::Duration shutter) const +{ + return std::clamp(shutter, minShutter_, maxShutter_); +} + +double ExposureModeHelper::clampGain(double gain) const +{ + return std::clamp(gain, minGain_, maxGain_); +} + +/** + * \brief Split exposure time into shutter and gain + * \param[in] exposure Exposure time + * + * This function divides a given exposure value into shutter time, analogue and + * digital gain by iterating through sets of shutter and gain limits. At each + * stage the current stage's shutter limit is multiplied by the previous stage's + * gain limit (or 1.0 initially) to see if the combination of the two can meet + * the required exposure value. If they cannot then splitExpothe current stage's shutter + * limit is multiplied by the same stage's gain limit to see if that combination + * can meet the required exposure value. If they cannot then the function moves + * to consider the next stage. + * + * When a combination of shutter and gain _stage_ limits are found that are + * sufficient to meet the required exposure value, the function attempts to + * reduce shutter time as much as possible whilst fixing gain and still meeting + * the exposure value. If a _runtime_ limit prevents shutter time from being + * lowered enough to meet the exposure value with gain fixed at the stage limit, + * gain is also lowered to compensate. + * + * Once the shutter time and gain values are ascertained, gain is assigned as + * analogue gain as much as possible, with digital gain only in use if the + * maximum analogue gain runtime limit is unable to accomodate the exposure + * value. + * + * If no combination of shutter and gain limits is found that meets the required + * exposure value, the helper falls-back to simply maximising the shutter time + * first, followed by analogue gain, followed by digital gain. + * + * @return Tuple of shutter time, analogue gain, and digital gain + */ +std::tuple +ExposureModeHelper::splitExposure(utils::Duration exposure) const +{ + ASSERT(maxShutter_); + ASSERT(maxGain_); + + bool gainFixed = minGain_ == maxGain_; + bool shutterFixed = minShutter_ == maxShutter_; + + /* + * There's no point entering the loop if we cannot change either gain + * nor shutter anyway. + */ + if (shutterFixed && gainFixed) + return { minShutter_, minGain_, exposure / (minShutter_ * minGain_) }; + + utils::Duration shutter; + double stageGain; + double gain; + + for (unsigned int stage = 0; stage < gains_.size(); stage++) { + double lastStageGain = stage == 0 ? 1.0 : clampGain(gains_[stage - 1]); + utils::Duration stageShutter = clampShutter(shutters_[stage]); + stageGain = clampGain(gains_[stage]); + + /* + * We perform the clamping on both shutter and gain in case the + * helper has had limits set that prevent those values being + * lowered beyond a certain minimum...this can happen at runtime + * for various reasons and so would not be known when the stage + * limits are initialised. + */ + + if (stageShutter * lastStageGain >= exposure) { + shutter = clampShutter(exposure / clampGain(lastStageGain)); + gain = clampGain(exposure / shutter); + + return { shutter, gain, exposure / (shutter * gain) }; + } + + if (stageShutter * stageGain >= exposure) { + shutter = clampShutter(exposure / clampGain(stageGain)); + gain = clampGain(exposure / shutter); + + return { shutter, gain, exposure / (shutter * gain) }; + } + } + + /* + * From here on all we can do is max out the shutter, followed by the + * analogue gain. If we still haven't achieved the target we send the + * rest of the exposure time to digital gain. If we were given no stages + * to use then set stageGain to 1.0 so that shutter is maxed before gain + * touched at all. + */ + if (gains_.empty()) + stageGain = 1.0; + + shutter = clampShutter(exposure / clampGain(stageGain)); + gain = clampGain(exposure / shutter); + + return { shutter, gain, exposure / (shutter * gain) }; +} + +/** + * \fn ExposureModeHelper::minShutter() + * \brief Retrieve the configured minimum shutter time limit set through + * setShutterGainLimits() + * \return The minShutter_ value + */ + +/** + * \fn ExposureModeHelper::maxShutter() + * \brief Retrieve the configured maximum shutter time set through + * setShutterGainLimits() + * \return The maxShutter_ value + */ + +/** + * \fn ExposureModeHelper::minGain() + * \brief Retrieve the configured minimum gain set through + * setShutterGainLimits() + * \return The minGain_ value + */ + +/** + * \fn ExposureModeHelper::maxGain() + * \brief Retrieve the configured maximum gain set through + * setShutterGainLimits() + * \return The maxGain_ value + */ + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/exposure_mode_helper.h b/src/ipa/libipa/exposure_mode_helper.h new file mode 100644 index 00000000..1f672135 --- /dev/null +++ b/src/ipa/libipa/exposure_mode_helper.h @@ -0,0 +1,53 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Paul Elder + * + * exposure_mode_helper.h - Helper class that performs computations relating to exposure + */ + +#pragma once + +#include +#include + +#include + +namespace libcamera { + +namespace ipa { + +class ExposureModeHelper +{ +public: + ExposureModeHelper(); + ~ExposureModeHelper() = default; + + void init(const std::vector> stages); + void setShutterGainLimits(utils::Duration minShutter, + utils::Duration maxShutter, + double minGain, double maxGain); + + std::tuple + splitExposure(utils::Duration exposure) const; + + utils::Duration minShutter() const { return minShutter_; } + utils::Duration maxShutter() const { return maxShutter_; } + double minGain() const { return minGain_; } + double maxGain() const { return maxGain_; } + +private: + utils::Duration clampShutter(utils::Duration shutter) const; + double clampGain(double gain) const; + + std::vector shutters_; + std::vector gains_; + + utils::Duration minShutter_; + utils::Duration maxShutter_; + double minGain_; + double maxGain_; +}; + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index 016b8e0e..37fbd177 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -3,6 +3,7 @@ libipa_headers = files([ 'algorithm.h', 'camera_sensor_helper.h', + 'exposure_mode_helper.h', 'fc_queue.h', 'histogram.h', 'module.h', @@ -11,6 +12,7 @@ libipa_headers = files([ libipa_sources = files([ 'algorithm.cpp', 'camera_sensor_helper.cpp', + 'exposure_mode_helper.cpp', 'fc_queue.cpp', 'histogram.cpp', 'module.cpp', From patchwork Wed Apr 17 13:15:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19896 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7FE01C32C9 for ; Wed, 17 Apr 2024 13:16:05 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 7BA7163367; Wed, 17 Apr 2024 15:16:04 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="U+gjS6+p"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7007963354 for ; Wed, 17 Apr 2024 15:15:56 +0200 (CEST) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 6CE65182C; Wed, 17 Apr 2024 15:15:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1713359709; bh=8aGp9gL9UtnoQOkZdBhB/faAUlsA0QECsMBeXo3Hk2M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=U+gjS6+p4/XK5eCJU+FhJ8viMNb56F6Ze8UUR3y/hWbCkVYrqz1Gyx1tqu7gIN36V f38fMGc/WeCWS9EmreMoK8dEgwORLVEGVLVnau1n9FZS2DtYVQ50vB0QqW0QgHTGXe uRNU5H4FaZ5Pfy2GZ9TTY/z1qIJomFeyi6Fz/njk= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally Subject: [PATCH v2 4/8] ipa: libipa: Add MeanLuminanceAgc base class Date: Wed, 17 Apr 2024 14:15:32 +0100 Message-Id: <20240417131536.484129-5-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240417131536.484129-1-dan.scally@ideasonboard.com> References: <20240417131536.484129-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The Agc algorithms for the RkIsp1 and IPU3 IPAs do the same thing in very large part; following the Rpi IPA's algorithm in spirit with a few tunable values in that IPA being hardcoded in the libipa ones. Add a new base class for MeanLuminanceAgc which implements the same algorithm and additionally parses yaml tuning files to inform an IPA module's Agc algorithm about valid constraint and exposure modes and their associated bounds. Signed-off-by: Daniel Scally Reviewed-by: Stefan Klug Reviewed-by: Paul Elder --- Changes in v2: - Renamed the class and files - Expanded the documentation - Added parseTuningData() so derived classes can call a single function to cover all the parsing in ::init(). src/ipa/libipa/agc_mean_luminance.cpp | 581 ++++++++++++++++++++++++++ src/ipa/libipa/agc_mean_luminance.h | 91 ++++ src/ipa/libipa/meson.build | 2 + 3 files changed, 674 insertions(+) create mode 100644 src/ipa/libipa/agc_mean_luminance.cpp create mode 100644 src/ipa/libipa/agc_mean_luminance.h diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp new file mode 100644 index 00000000..02e223cf --- /dev/null +++ b/src/ipa/libipa/agc_mean_luminance.cpp @@ -0,0 +1,581 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Ideas on Board Oy + * + * agc_mean_luminance.cpp - Base class for mean luminance AGC algorithms + */ + +#include "agc_mean_luminance.h" + +#include + +#include +#include + +#include "exposure_mode_helper.h" + +using namespace libcamera::controls; + +/** + * \file agc_mean_luminance.h + * \brief Base class implementing mean luminance AEGC + */ + +namespace libcamera { + +using namespace std::literals::chrono_literals; + +LOG_DEFINE_CATEGORY(AgcMeanLuminance) + +namespace ipa { + +/* + * Number of frames for which to run the algorithm at full speed, before slowing + * down to prevent large and jarring changes in exposure from frame to frame. + */ +static constexpr uint32_t kNumStartupFrames = 10; + +/* + * Default relative luminance target + * + * This value should be chosen so that when the camera points at a grey target, + * the resulting image brightness looks "right". Custom values can be passed + * as the relativeLuminanceTarget value in sensor tuning files. + */ +static constexpr double kDefaultRelativeLuminanceTarget = 0.16; + +/** + * \struct AgcMeanLuminance::AgcConstraint + * \brief The boundaries and target for an AeConstraintMode constraint + * + * This structure describes an AeConstraintMode constraint for the purposes of + * this algorithm. The algorithm will apply the constraints by calculating the + * Histogram's inter-quantile mean between the given quantiles and ensure that + * the resulting value is the right side of the given target (as defined by the + * boundary and luminance target). + */ + +/** + * \enum AgcMeanLuminance::AgcConstraint::Bound + * \brief Specify whether the constraint defines a lower or upper bound + * \var AgcMeanLuminance::AgcConstraint::lower + * \brief The constraint defines a lower bound + * \var AgcMeanLuminance::AgcConstraint::upper + * \brief The constraint defines an upper bound + */ + +/** + * \var AgcMeanLuminance::AgcConstraint::bound + * \brief The type of constraint bound + */ + +/** + * \var AgcMeanLuminance::AgcConstraint::qLo + * \brief The lower quantile to use for the constraint + */ + +/** + * \var AgcMeanLuminance::AgcConstraint::qHi + * \brief The upper quantile to use for the constraint + */ + +/** + * \var AgcMeanLuminance::AgcConstraint::yTarget + * \brief The luminance target for the constraint + */ + +/** + * \class AgcMeanLuminance + * \brief A mean-based auto-exposure algorithm + * + * This algorithm calculates a shutter time, analogue and digital gain such that + * the normalised mean luminance value of an image is driven towards a target, + * which itself is discovered from tuning data. The algorithm is a two-stage + * process. + * + * In the first stage, an initial gain value is derived by iteratively comparing + * the gain-adjusted mean luminance across an entire image against a target, and + * selecting a value which pushes it as closely as possible towards the target. + * + * In the second stage we calculate the gain required to drive the average of a + * section of a histogram to a target value, where the target and the boundaries + * of the section of the histogram used in the calculation are taken from the + * values defined for the currently configured AeConstraintMode within the + * tuning data. This class provides a helper function to parse those tuning data + * to discover the constraints, and so requires a specific format for those + * data which is described in \ref parseTuningData(). The gain from the first + * stage is then clamped to the gain from this stage. + * + * The final gain is used to adjust the effective exposure value of the image, + * and that new exposure value is divided into shutter time, analogue gain and + * digital gain according to the selected AeExposureMode. This class expects to + * use the \ref ExposureModeHelper class to assist in that division, and expects + * the data needed to initialise that class to be present in tuning data in a + * format described in \ref parseTuningData(). + * + * In order to be able to derive an AGC implementation from this class, an IPA + * needs to be able to do the following: + * + * 1. Provide a luminance estimation across an entire image. + * 2. Provide a luminance Histogram for the image to use in calculating + * constraint compliance. The precision of the Histogram that is available + * will determine the supportable precision of the constraints. + */ + +AgcMeanLuminance::AgcMeanLuminance() + : frameCount_(0), filteredExposure_(0s), relativeLuminanceTarget_(0) +{ +} + +/** + * \brief Parse the relative luminance target from the tuning data + * \param[in] tuningData The YamlObject holding the algorithm's tuning data + */ +void AgcMeanLuminance::parseRelativeLuminanceTarget(const YamlObject &tuningData) +{ + relativeLuminanceTarget_ = + tuningData["relativeLuminanceTarget"].get(kDefaultRelativeLuminanceTarget); +} + +/** + * \brief Parse an AeConstraintMode constraint from tuning data + * \param[in] modeDict the YamlObject holding the constraint data + * \param[in] id The constraint ID from AeConstraintModeEnum + */ +void AgcMeanLuminance::parseConstraint(const YamlObject &modeDict, int32_t id) +{ + for (const auto &[boundName, content] : modeDict.asDict()) { + if (boundName != "upper" && boundName != "lower") { + LOG(AgcMeanLuminance, Warning) + << "Ignoring unknown constraint bound '" << boundName << "'"; + continue; + } + + unsigned int idx = static_cast(boundName == "upper"); + AgcConstraint::Bound bound = static_cast(idx); + double qLo = content["qLo"].get().value_or(0.98); + double qHi = content["qHi"].get().value_or(1.0); + double yTarget = + content["yTarget"].getList().value_or(std::vector{ 0.5 }).at(0); + + AgcConstraint constraint = { bound, qLo, qHi, yTarget }; + + if (!constraintModes_.count(id)) + constraintModes_[id] = {}; + + if (idx) + constraintModes_[id].push_back(constraint); + else + constraintModes_[id].insert(constraintModes_[id].begin(), constraint); + } +} + +int AgcMeanLuminance::parseConstraintModes(const YamlObject &tuningData) +{ + std::vector availableConstraintModes; + + const YamlObject &yamlConstraintModes = tuningData[controls::AeConstraintMode.name()]; + if (yamlConstraintModes.isDictionary()) { + for (const auto &[modeName, modeDict] : yamlConstraintModes.asDict()) { + if (AeConstraintModeNameValueMap.find(modeName) == + AeConstraintModeNameValueMap.end()) { + LOG(AgcMeanLuminance, Warning) + << "Skipping unknown constraint mode '" << modeName << "'"; + continue; + } + + if (!modeDict.isDictionary()) { + LOG(AgcMeanLuminance, Error) + << "Invalid constraint mode '" << modeName << "'"; + return -EINVAL; + } + + parseConstraint(modeDict, + AeConstraintModeNameValueMap.at(modeName)); + availableConstraintModes.push_back( + AeConstraintModeNameValueMap.at(modeName)); + } + } + + /* + * If the tuning data file contains no constraints then we use the + * default constraint that the various Agc algorithms were adhering to + * anyway before centralisation. + */ + if (constraintModes_.empty()) { + AgcConstraint constraint = { + AgcConstraint::Bound::lower, + 0.98, + 1.0, + 0.5 + }; + + constraintModes_[controls::ConstraintNormal].insert( + constraintModes_[controls::ConstraintNormal].begin(), + constraint); + availableConstraintModes.push_back( + AeConstraintModeNameValueMap.at("ConstraintNormal")); + } + + controls_[&controls::AeConstraintMode] = ControlInfo(availableConstraintModes); + + return 0; +} + +int AgcMeanLuminance::parseExposureModes(const YamlObject &tuningData) +{ + std::vector availableExposureModes; + + const YamlObject &yamlExposureModes = tuningData[controls::AeExposureMode.name()]; + if (yamlExposureModes.isDictionary()) { + for (const auto &[modeName, modeValues] : yamlExposureModes.asDict()) { + if (AeExposureModeNameValueMap.find(modeName) == + AeExposureModeNameValueMap.end()) { + LOG(AgcMeanLuminance, Warning) + << "Skipping unknown exposure mode '" << modeName << "'"; + continue; + } + + if (!modeValues.isDictionary()) { + LOG(AgcMeanLuminance, Error) + << "Invalid exposure mode '" << modeName << "'"; + return -EINVAL; + } + + std::vector shutters = + modeValues["shutter"].getList().value_or(std::vector{}); + std::vector gains = + modeValues["gain"].getList().value_or(std::vector{}); + + if (shutters.size() != gains.size()) { + LOG(AgcMeanLuminance, Error) + << "Shutter and gain array sizes unequal"; + return -EINVAL; + } + + if (shutters.empty()) { + LOG(AgcMeanLuminance, Error) + << "Shutter and gain arrays are empty"; + return -EINVAL; + } + + std::vector> stages; + for (unsigned int i = 0; i < shutters.size(); i++) { + stages.push_back({ + std::chrono::microseconds(shutters[i]), + gains[i] + }); + } + + std::shared_ptr helper = + std::make_shared(); + helper->init(stages); + + exposureModeHelpers_[AeExposureModeNameValueMap.at(modeName)] = helper; + availableExposureModes.push_back(AeExposureModeNameValueMap.at(modeName)); + } + } + + /* + * If we don't have any exposure modes in the tuning data we create an + * ExposureModeHelper using an empty vector of stages. This will result + * in the ExposureModeHelper simply driving the shutter as high as + * possible before touching gain. + */ + if (availableExposureModes.empty()) { + int32_t exposureModeId = AeExposureModeNameValueMap.at("ExposureNormal"); + std::vector> stages = { }; + + std::shared_ptr helper = + std::make_shared(); + helper->init(stages); + + exposureModeHelpers_[exposureModeId] = helper; + availableExposureModes.push_back(exposureModeId); + } + + controls_[&controls::AeExposureMode] = ControlInfo(availableExposureModes); + + return 0; +} + +/** + * \brief Parse tuning data for AeConstraintMode and AeExposureMode controls + * \param[in] tuningData the YamlObject representing the tuning data + * + * This function parses tuning data to build the list of allowed values for the + * AeConstraintMode and AeExposureMode controls. Those tuning data must provide + * the data in a specific format; the Agc algorithm's tuning data should contain + * a dictionary called AeConstraintMode containing per-mode setting dictionaries + * with the key being a value from \ref controls::AeConstraintModeNameValueMap. + * Each mode dict may contain either a "lower" or "upper" key or both, for + * example: + * + * \code{.unparsed} + * algorithms: + * - Agc: + * AeConstraintMode: + * ConstraintNormal: + * lower: + * qLo: 0.98 + * qHi: 1.0 + * yTarget: 0.5 + * ConstraintHighlight: + * lower: + * qLo: 0.98 + * qHi: 1.0 + * yTarget: 0.5 + * upper: + * qLo: 0.98 + * qHi: 1.0 + * yTarget: 0.8 + * + * \endcode + * + * For the AeExposureMode control the data should contain a dictionary called + * AeExposureMode containing per-mode setting dictionaries with the key being a + * value from \ref controls::AeExposureModeNameValueMap. Each mode dict should + * contain an array of shutter times with the key "shutter" and an array of gain + * values with the key "gain", in this format: + * + * \code{.unparsed} + * algorithms: + * - Agc: + * AeExposureMode: + * ExposureNormal: + * shutter: [ 100, 10000, 30000, 60000, 120000 ] + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ] + * ExposureShort: + * shutter: [ 100, 10000, 30000, 60000, 120000 ] + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ] + * + * \endcode + * + * @return 0 on success or a negative error code + */ +int AgcMeanLuminance::parseTuningData(const YamlObject &tuningData) +{ + int ret; + + parseRelativeLuminanceTarget(tuningData); + + ret = parseConstraintModes(tuningData); + if (ret) + return ret; + + ret = parseExposureModes(tuningData); + if (ret) + return ret; + + return 0; +} + +/** + * \brief configure the ExposureModeHelpers for this class + * \param[in] minShutter Minimum shutter time to allow + * \param[in] maxShutter Maximum shutter time to allow + * \param[in] minGain Minimum gain to allow + * \param[in] maxGain Maximum gain to allow + * + * This function calls \ref ExposureModeHelper::setShutterGainLimits() for each + * ExposureModeHelper that has been created for this class. + */ +void AgcMeanLuminance::configureExposureModeHelpers(utils::Duration minShutter, + utils::Duration maxShutter, + double minGain, + double maxGain) +{ + for (auto &[id, helper] : exposureModeHelpers_) + helper->setShutterGainLimits(minShutter, maxShutter, minGain, maxGain); +} + +/** + * \fn AgcMeanLuminance::constraintModes() + * \brief Get the constraint modes that have been parsed from tuning data + */ + +/** + * \fn AgcMeanLuminance::exposureModeHelpers() + * \brief Get the ExposureModeHelpers that have been parsed from tuning data + */ + +/** + * \fn AgcMeanLuminance::controls() + * \brief Get the controls that have been generated after parsing tuning data + */ + +/** + * \fn AgcMeanLuminance::estimateLuminance(const double gain) + * \brief Estimate the luminance of an image, adjusted by a given gain + * \param[in] gain The gain with which to adjust the luminance estimate + * + * This function estimates the average relative luminance of the frame that + * would be output by the sensor if an additional \a gain was applied. It is a + * pure virtual function because estimation of luminance is a hardware-specific + * operation, which depends wholly on the format of the stats that are delivered + * to libcamera from the ISP. Derived classes must implement an overriding + * function that calculates the normalised mean luminance value across the + * entire image. + * + * \return The normalised relative luminance of the image + */ + +/** + * \brief Estimate the initial gain needed to achieve a relative luminance + * target + * + * To account for non-linearity caused by saturation, the value needs to be + * estimated in an iterative process, as multiplying by a gain will not increase + * the relative luminance by the same factor if some image regions are saturated + * + * \return The calculated initial gain + */ +double AgcMeanLuminance::estimateInitialGain() +{ + double yTarget = relativeLuminanceTarget_; + double yGain = 1.0; + + for (unsigned int i = 0; i < 8; i++) { + double yValue = estimateLuminance(yGain); + double extra_gain = std::min(10.0, yTarget / (yValue + .001)); + + yGain *= extra_gain; + LOG(AgcMeanLuminance, Debug) << "Y value: " << yValue + << ", Y target: " << yTarget + << ", gives gain " << yGain; + + if (utils::abs_diff(extra_gain, 1.0) < 0.01) + break; + } + + return yGain; +} + +/** + * \brief Clamp gain within the bounds of a defined constraint + * \param[in] constraintModeIndex The index of the constraint to adhere to + * \param[in] hist A histogram over which to calculate inter-quantile means + * \param[in] gain The gain to clamp + * + * \return The gain clamped within the constraint bounds + */ +double AgcMeanLuminance::constraintClampGain(uint32_t constraintModeIndex, + const Histogram &hist, + double gain) +{ + std::vector &constraints = constraintModes_[constraintModeIndex]; + for (const AgcConstraint &constraint : constraints) { + double newGain = constraint.yTarget * hist.bins() / + hist.interQuantileMean(constraint.qLo, constraint.qHi); + + if (constraint.bound == AgcConstraint::Bound::lower && + newGain > gain) + gain = newGain; + + if (constraint.bound == AgcConstraint::Bound::upper && + newGain < gain) + gain = newGain; + } + + return gain; +} + +/** + * \brief Apply a filter on the exposure value to limit the speed of changes + * \param[in] exposureValue The target exposure from the AGC algorithm + * + * The speed of the filter is adaptive, and will produce the target quicker + * during startup, or when the target exposure is within 20% of the most recent + * filter output. + * + * \return The filtered exposure + */ +utils::Duration AgcMeanLuminance::filterExposure(utils::Duration exposureValue) +{ + double speed = 0.2; + + /* Adapt instantly if we are in startup phase. */ + if (frameCount_ < kNumStartupFrames) + speed = 1.0; + + /* + * If we are close to the desired result, go faster to avoid making + * multiple micro-adjustments. + * \todo Make this customisable? + */ + if (filteredExposure_ < 1.2 * exposureValue && + filteredExposure_ > 0.8 * exposureValue) + speed = sqrt(speed); + + filteredExposure_ = speed * exposureValue + + filteredExposure_ * (1.0 - speed); + + return filteredExposure_; +} + +/** + * \brief Calculate the new exposure value + * \param[in] constraintModeIndex The index of the current constraint mode + * \param[in] exposureModeIndex The index of the current exposure mode + * \param[in] yHist A Histogram from the ISP statistics to use in constraining + * the calculated gain + * \param[in] effectiveExposureValue The EV applied to the frame from which the + * statistics in use derive + * + * Calculate a new exposure value to try to obtain the target. The calculated + * exposure value is filtered to prevent rapid changes from frame to frame, and + * divided into shutter time, analogue and digital gain. + * + * \return Tuple of shutter time, analogue gain, and digital gain + */ +std::tuple +AgcMeanLuminance::calculateNewEv(uint32_t constraintModeIndex, + uint32_t exposureModeIndex, + const Histogram &yHist, + utils::Duration effectiveExposureValue) +{ + /* + * The pipeline handler should validate that we have received an allowed + * value for AeExposureMode. + */ + std::shared_ptr exposureModeHelper = + exposureModeHelpers_.at(exposureModeIndex); + + double gain = estimateInitialGain(); + gain = constraintClampGain(constraintModeIndex, yHist, gain); + + /* + * We don't check whether we're already close to the target, because + * even if the effective exposure value is the same as the last frame's + * we could have switched to an exposure mode that would require a new + * pass through the splitExposure() function. + */ + + utils::Duration newExposureValue = effectiveExposureValue * gain; + utils::Duration maxTotalExposure = exposureModeHelper->maxShutter() + * exposureModeHelper->maxGain(); + newExposureValue = std::min(newExposureValue, maxTotalExposure); + + /* + * We filter the exposure value to make sure changes are not too jarring + * from frame to frame. + */ + newExposureValue = filterExposure(newExposureValue); + + frameCount_++; + return exposureModeHelper->splitExposure(newExposureValue); +} + +/** + * \fn AgcMeanLuminance::resetFrameCount() + * \brief Reset the frame counter + * + * This function resets the internal frame counter, which exists to help the + * algorithm decide whether it should respond instantly or not. The expectation + * is for derived classes to call this function before each camera start call, + * either in configure() or queueRequest() if the frame number is zero. + */ + +}; /* namespace ipa */ + +}; /* namespace libcamera */ diff --git a/src/ipa/libipa/agc_mean_luminance.h b/src/ipa/libipa/agc_mean_luminance.h new file mode 100644 index 00000000..e48dc498 --- /dev/null +++ b/src/ipa/libipa/agc_mean_luminance.h @@ -0,0 +1,91 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Ideas on Board Oy + * + agc_mean_luminance.h - Base class for mean luminance AGC algorithms + */ + +#pragma once + +#include +#include + +#include + +#include "libcamera/internal/yaml_parser.h" + +#include "exposure_mode_helper.h" +#include "histogram.h" + +namespace libcamera { + +namespace ipa { + +class AgcMeanLuminance +{ +public: + AgcMeanLuminance(); + virtual ~AgcMeanLuminance() = default; + + struct AgcConstraint { + enum class Bound { + lower = 0, + upper = 1 + }; + Bound bound; + double qLo; + double qHi; + double yTarget; + }; + + int parseTuningData(const YamlObject &tuningData); + + void configureExposureModeHelpers(utils::Duration minShutter, + utils::Duration maxShutter, + double minGain, + double maxGain); + + std::map> constraintModes() + { + return constraintModes_; + } + + std::map> exposureModeHelpers() + { + return exposureModeHelpers_; + } + + ControlInfoMap::Map controls() + { + return controls_; + } + + double estimateInitialGain(); + double constraintClampGain(uint32_t constraintModeIndex, + const Histogram &hist, + double gain); + utils::Duration filterExposure(utils::Duration exposureValue); + std::tuple + calculateNewEv(uint32_t constraintModeIndex, uint32_t exposureModeIndex, + const Histogram &yHist, utils::Duration effectiveExposureValue); + void resetFrameCount() { frameCount_ = 0; } +private: + virtual double estimateLuminance(const double gain) = 0; + + void parseRelativeLuminanceTarget(const YamlObject &tuningData); + void parseConstraint(const YamlObject &modeDict, int32_t id); + int parseConstraintModes(const YamlObject &tuningData); + int parseExposureModes(const YamlObject &tuningData); + + uint64_t frameCount_; + utils::Duration filteredExposure_; + double relativeLuminanceTarget_; + + std::map> constraintModes_; + std::map> exposureModeHelpers_; + ControlInfoMap::Map controls_; +}; + +}; /* namespace ipa */ + +}; /* namespace libcamera */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index 37fbd177..7ce885da 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -1,6 +1,7 @@ # SPDX-License-Identifier: CC0-1.0 libipa_headers = files([ + 'agc_mean_luminance.h', 'algorithm.h', 'camera_sensor_helper.h', 'exposure_mode_helper.h', @@ -10,6 +11,7 @@ libipa_headers = files([ ]) libipa_sources = files([ + 'agc_mean_luminance.cpp', 'algorithm.cpp', 'camera_sensor_helper.cpp', 'exposure_mode_helper.cpp', From patchwork Wed Apr 17 13:15:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19897 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5EC78C3285 for ; Wed, 17 Apr 2024 13:16:07 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D1FB663363; Wed, 17 Apr 2024 15:16:06 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="pKltxEXx"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id BB43763352 for ; Wed, 17 Apr 2024 15:15:56 +0200 (CEST) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id AE04F1890; Wed, 17 Apr 2024 15:15:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1713359709; bh=liyCiuQOxZxyopNk4w2ter7m2T5dPbgxWyZRTEnGqu4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pKltxEXx9TNiCdj6TXdcsBOObgvQLL0J1yJEy8OWM1SxGQd/7OGNR8UHFJNd1+5Ri AzFsxQeSSM34WBg74fJ311iPByl3WTWOnn0YNaRylD9kA1zNN+PYiTxJN2fVGRoYdR 0G/MMawGQwdqv1rDDN8TRsreJfihCfMivRyEOiw4= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally Subject: [PATCH v2 5/8] ipa: ipu3: Derive ipu3::algorithms::Agc from MeanLuminanceAgc Date: Wed, 17 Apr 2024 14:15:33 +0100 Message-Id: <20240417131536.484129-6-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240417131536.484129-1-dan.scally@ideasonboard.com> References: <20240417131536.484129-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In preparation for switching to a derivation of MeanLuminanceAgc, add a function to parse and store the statistics for easy retrieval in an overriding estimateLuminance() function. Now that we have a MeanLuminanceAgc class that centralises our AEGC algorithm, derive the IPU3's Agc class from it and plumb in the necessary framework to enable it to be used. For simplicity's sake this commit switches the algorithm to use the derived class, but does not remove the bespoke functions at this time. Signed-off-by: Daniel Scally Reviewed-by: Stefan Klug --- Changes in v2: - Squashed the patch adding parseStatistics() - Used the first available constraint and exposure modes rather than assuming the "Normal" mode was available - Used a vector of RGB tuples in estimateLuminance() rather than 3 vectors - Stored a copy of the bdsGrid and AWB gains rather than a pointer to the context for use in estimateLuminance() - Remembered to report the controls out to IPAIPU3::updateControls() src/ipa/ipu3/algorithms/agc.cpp | 120 ++++++++++++++++++++++++++++++-- src/ipa/ipu3/algorithms/agc.h | 14 +++- src/ipa/ipu3/ipa_context.cpp | 3 + src/ipa/ipu3/ipa_context.h | 5 ++ src/ipa/ipu3/ipu3.cpp | 3 +- 5 files changed, 136 insertions(+), 9 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 606a237a..3b9761bd 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -71,11 +71,33 @@ static constexpr uint32_t kNumStartupFrames = 10; static constexpr double kRelativeLuminanceTarget = 0.16; Agc::Agc() - : frameCount_(0), minShutterSpeed_(0s), - maxShutterSpeed_(0s), filteredExposure_(0s) + : minShutterSpeed_(0s), maxShutterSpeed_(0s) { } +/** + * \brief Initialise the AGC algorithm from tuning files + * \param[in] context The shared IPA context + * \param[in] tuningData The YamlObject containing Agc tuning data + * + * This function calls the base class' tuningData parsers to discover which + * control values are supported. + * + * \return 0 on success or errors from the base class + */ +int Agc::init(IPAContext &context, const YamlObject &tuningData) +{ + int ret; + + ret = parseTuningData(tuningData); + if (ret) + return ret; + + context.ctrlMap.merge(controls()); + + return 0; +} + /** * \brief Configure the AGC given a configInfo * \param[in] context The shared IPA context @@ -90,6 +112,7 @@ int Agc::configure(IPAContext &context, IPAActiveState &activeState = context.activeState; stride_ = configuration.grid.stride; + bdsGrid_ = configuration.grid.bdsGrid; minShutterSpeed_ = configuration.agc.minShutterSpeed; maxShutterSpeed_ = std::min(configuration.agc.maxShutterSpeed, @@ -103,6 +126,16 @@ int Agc::configure(IPAContext &context, activeState.agc.exposure = 10ms / configuration.sensor.lineDuration; frameCount_ = 0; + + context.activeState.agc.constraintMode = constraintModes().begin()->first; + context.activeState.agc.exposureMode = exposureModeHelpers().begin()->first; + + /* \todo Run this again when FrameDurationLimits is passed in */ + configureExposureModeHelpers(minShutterSpeed_, maxShutterSpeed_, + minAnalogueGain_, maxAnalogueGain_); + + resetFrameCount(); + return 0; } @@ -142,6 +175,39 @@ double Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, return Histogram(Span(hist)).interQuantileMean(0.98, 1.0); } +Histogram Agc::parseStatistics(const ipu3_uapi_stats_3a *stats, + const ipu3_uapi_grid_config &grid) +{ + uint32_t hist[knumHistogramBins] = { 0 }; + + rgbTriples_.clear(); + + for (unsigned int cellY = 0; cellY < grid.height; cellY++) { + for (unsigned int cellX = 0; cellX < grid.width; cellX++) { + uint32_t cellPosition = cellY * stride_ + cellX; + + const ipu3_uapi_awb_set_item *cell = + reinterpret_cast( + &stats->awb_raw_buffer.meta_data[cellPosition]); + + rgbTriples_.push_back({ + cell->R_avg, + (cell->Gr_avg + cell->Gb_avg) / 2, + cell->B_avg + }); + + /* + * Store the average green value to estimate the + * brightness. Even the overexposed pixels are + * taken into account. + */ + hist[(cell->Gr_avg + cell->Gb_avg) / 2]++; + } + } + + return Histogram(Span(hist)); +} + /** * \brief Apply a filter on the exposure value to limit the speed of changes * \param[in] exposureValue The target exposure from the AGC algorithm @@ -247,11 +313,6 @@ void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, LOG(IPU3Agc, Debug) << "Divided up shutter and gain are " << shutterTime << " and " << stepGain; - - IPAActiveState &activeState = context.activeState; - /* Update the estimated exposure and gain. */ - activeState.agc.exposure = shutterTime / configuration.sensor.lineDuration; - activeState.agc.gain = stepGain; } /** @@ -314,6 +375,23 @@ double Agc::estimateLuminance(IPAActiveState &activeState, return ySum / (grid.height * grid.width) / 255; } +double Agc::estimateLuminance(double gain) +{ + double redSum = 0, greenSum = 0, blueSum = 0; + + for (unsigned int i = 0; i < rgbTriples_.size(); i++) { + redSum += std::min(std::get<0>(rgbTriples_[i]) * gain, 255.0); + greenSum += std::min(std::get<1>(rgbTriples_[i]) * gain, 255.0); + blueSum += std::min(std::get<2>(rgbTriples_[i]) * gain, 255.0); + } + + double ySum = redSum * rGain_ * 0.299 + + greenSum * gGain_ * 0.587 + + blueSum * bGain_ * 0.114; + + return ySum / (bdsGrid_.height * bdsGrid_.width) / 255; +} + /** * \brief Process IPU3 statistics, and run AGC operations * \param[in] context The shared IPA context @@ -366,8 +444,36 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, computeExposure(context, frameContext, yGain, iqMeanGain); frameCount_++; + Histogram hist = parseStatistics(stats, context.configuration.grid.bdsGrid); + rGain_ = context.activeState.awb.gains.red; + gGain_ = context.activeState.awb.gains.blue; + bGain_ = context.activeState.awb.gains.green; + + /* + * The Agc algorithm needs to know the effective exposure value that was + * applied to the sensor when the statistics were collected. + */ utils::Duration exposureTime = context.configuration.sensor.lineDuration * frameContext.sensor.exposure; + double analogueGain = frameContext.sensor.gain; + utils::Duration effectiveExposureValue = exposureTime * analogueGain; + + utils::Duration shutterTime; + double aGain, dGain; + std::tie(shutterTime, aGain, dGain) = + calculateNewEv(context.activeState.agc.constraintMode, + context.activeState.agc.exposureMode, hist, + effectiveExposureValue); + + LOG(IPU3Agc, Debug) + << "Divided up shutter, analogue gain and digital gain are " + << shutterTime << ", " << aGain << " and " << dGain; + + IPAActiveState &activeState = context.activeState; + /* Update the estimated exposure and gain. */ + activeState.agc.exposure = shutterTime / context.configuration.sensor.lineDuration; + activeState.agc.gain = aGain; + metadata.set(controls::AnalogueGain, frameContext.sensor.gain); metadata.set(controls::ExposureTime, exposureTime.get()); diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 9d6e3ff1..40f32188 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -13,6 +13,9 @@ #include +#include "libipa/agc_mean_luminance.h" +#include "libipa/histogram.h" + #include "algorithm.h" namespace libcamera { @@ -21,12 +24,13 @@ struct IPACameraSensorInfo; namespace ipa::ipu3::algorithms { -class Agc : public Algorithm +class Agc : public Algorithm, public AgcMeanLuminance { public: Agc(); ~Agc() = default; + int init(IPAContext &context, const YamlObject &tuningData) override; int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, @@ -43,6 +47,9 @@ private: const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, double gain); + double estimateLuminance(double gain) override; + Histogram parseStatistics(const ipu3_uapi_stats_3a *stats, + const ipu3_uapi_grid_config &grid); uint64_t frameCount_; @@ -55,6 +62,11 @@ private: utils::Duration filteredExposure_; uint32_t stride_; + double rGain_; + double gGain_; + double bGain_; + ipu3_uapi_grid_config bdsGrid_; + std::vector> rgbTriples_; }; } /* namespace ipa::ipu3::algorithms */ diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index 959f314f..c4fb5642 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -47,6 +47,9 @@ namespace libcamera::ipa::ipu3 { * * \var IPAContext::activeState * \brief The current state of IPA algorithms + * + * \var IPAContext::ctrlMap + * \brief A ControlInfoMap::Map of controls populated by the algorithms */ /** diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index e9a3863b..a92cb6ce 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -12,6 +12,7 @@ #include +#include #include #include @@ -55,6 +56,8 @@ struct IPAActiveState { struct { uint32_t exposure; double gain; + uint32_t constraintMode; + uint32_t exposureMode; } agc; struct { @@ -85,6 +88,8 @@ struct IPAContext { IPAActiveState activeState; FCQueue frameContexts; + + ControlInfoMap::Map ctrlMap; }; } /* namespace ipa::ipu3 */ diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 08ee6eb3..4809eb60 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -189,7 +189,7 @@ private: }; IPAIPU3::IPAIPU3() - : context_({ {}, {}, { kMaxFrameContexts } }) + : context_({ {}, {}, { kMaxFrameContexts }, {} }) { } @@ -287,6 +287,7 @@ void IPAIPU3::updateControls(const IPACameraSensorInfo &sensorInfo, frameDurations[1], frameDurations[2]); + controls.merge(context_.ctrlMap); *ipaControls = ControlInfoMap(std::move(controls), controls::controls); } From patchwork Wed Apr 17 13:15:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19898 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 6618DC3213 for ; Wed, 17 Apr 2024 13:16:08 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id CA0566336B; Wed, 17 Apr 2024 15:16:07 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="fWqa1lcN"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 23AC663363 for ; Wed, 17 Apr 2024 15:15:57 +0200 (CEST) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 076FDB7E; Wed, 17 Apr 2024 15:15:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1713359710; bh=/nN1mJOMfkfUaxWWtqn6DOazVJlnICfCsX4JO8QfPjk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fWqa1lcN/4L0igAiqEvHAWIue1JPet4SNB55ZmqJmqLMXO3Nfhh5b3PGnOchBSptI 1ZrYjenq09iEd9JV0RV/fiA4o6NJCK6moAsJtHa2WzkFOZb1hdq2W/O+F63zxI0Hsu BQUwVTqj2RC34KE0oSq2boUxthQSkqHVSf1BSa+E= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally , Stefan Klug Subject: [PATCH v2 6/8] ipa: ipu3: Remove bespoke AGC functions from IPU3 Date: Wed, 17 Apr 2024 14:15:34 +0100 Message-Id: <20240417131536.484129-7-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240417131536.484129-1-dan.scally@ideasonboard.com> References: <20240417131536.484129-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that the IPU3's Agc is derived from MeanLuminanceAgc we can delete all the unecessary bespoke functions. Reviewed-by: Stefan Klug Signed-off-by: Daniel Scally Reviewed-by: Jacopo Mondi --- Changes in v2: - Kept the documentation for estimateLuminance() src/ipa/ipu3/algorithms/agc.cpp | 227 +------------------------------- src/ipa/ipu3/algorithms/agc.h | 13 -- 2 files changed, 3 insertions(+), 237 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 3b9761bd..46fc3b33 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -125,8 +125,6 @@ int Agc::configure(IPAContext &context, activeState.agc.gain = minAnalogueGain_; activeState.agc.exposure = 10ms / configuration.sensor.lineDuration; - frameCount_ = 0; - context.activeState.agc.constraintMode = constraintModes().begin()->first; context.activeState.agc.exposureMode = exposureModeHelpers().begin()->first; @@ -139,42 +137,6 @@ int Agc::configure(IPAContext &context, return 0; } -/** - * \brief Estimate the mean value of the top 2% of the histogram - * \param[in] stats The statistics computed by the ImgU - * \param[in] grid The grid used to store the statistics in the IPU3 - * \return The mean value of the top 2% of the histogram - */ -double Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, - const ipu3_uapi_grid_config &grid) const -{ - /* Initialise the histogram array */ - uint32_t hist[knumHistogramBins] = { 0 }; - - for (unsigned int cellY = 0; cellY < grid.height; cellY++) { - for (unsigned int cellX = 0; cellX < grid.width; cellX++) { - uint32_t cellPosition = cellY * stride_ + cellX; - - const ipu3_uapi_awb_set_item *cell = - reinterpret_cast( - &stats->awb_raw_buffer.meta_data[cellPosition] - ); - - uint8_t gr = cell->Gr_avg; - uint8_t gb = cell->Gb_avg; - /* - * Store the average green value to estimate the - * brightness. Even the overexposed pixels are - * taken into account. - */ - hist[(gr + gb) / 2]++; - } - } - - /* Estimate the quantile mean of the top 2% of the histogram. */ - return Histogram(Span(hist)).interQuantileMean(0.98, 1.0); -} - Histogram Agc::parseStatistics(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid) { @@ -208,123 +170,9 @@ Histogram Agc::parseStatistics(const ipu3_uapi_stats_3a *stats, return Histogram(Span(hist)); } -/** - * \brief Apply a filter on the exposure value to limit the speed of changes - * \param[in] exposureValue The target exposure from the AGC algorithm - * - * The speed of the filter is adaptive, and will produce the target quicker - * during startup, or when the target exposure is within 20% of the most recent - * filter output. - * - * \return The filtered exposure - */ -utils::Duration Agc::filterExposure(utils::Duration exposureValue) -{ - double speed = 0.2; - - /* Adapt instantly if we are in startup phase. */ - if (frameCount_ < kNumStartupFrames) - speed = 1.0; - - /* - * If we are close to the desired result, go faster to avoid making - * multiple micro-adjustments. - * \todo Make this customisable? - */ - if (filteredExposure_ < 1.2 * exposureValue && - filteredExposure_ > 0.8 * exposureValue) - speed = sqrt(speed); - - filteredExposure_ = speed * exposureValue + - filteredExposure_ * (1.0 - speed); - - LOG(IPU3Agc, Debug) << "After filtering, exposure " << filteredExposure_; - - return filteredExposure_; -} - -/** - * \brief Estimate the new exposure and gain values - * \param[inout] frameContext The shared IPA frame Context - * \param[in] yGain The gain calculated based on the relative luminance target - * \param[in] iqMeanGain The gain calculated based on the relative luminance target - */ -void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, - double yGain, double iqMeanGain) -{ - const IPASessionConfiguration &configuration = context.configuration; - /* Get the effective exposure and gain applied on the sensor. */ - uint32_t exposure = frameContext.sensor.exposure; - double analogueGain = frameContext.sensor.gain; - - /* Use the highest of the two gain estimates. */ - double evGain = std::max(yGain, iqMeanGain); - - /* Consider within 1% of the target as correctly exposed */ - if (utils::abs_diff(evGain, 1.0) < 0.01) - LOG(IPU3Agc, Debug) << "We are well exposed (evGain = " - << evGain << ")"; - - /* extracted from Rpi::Agc::computeTargetExposure */ - - /* Calculate the shutter time in seconds */ - utils::Duration currentShutter = exposure * configuration.sensor.lineDuration; - - /* - * Update the exposure value for the next computation using the values - * of exposure and gain really used by the sensor. - */ - utils::Duration effectiveExposureValue = currentShutter * analogueGain; - - LOG(IPU3Agc, Debug) << "Actual total exposure " << currentShutter * analogueGain - << " Shutter speed " << currentShutter - << " Gain " << analogueGain - << " Needed ev gain " << evGain; - - /* - * Calculate the current exposure value for the scene as the latest - * exposure value applied multiplied by the new estimated gain. - */ - utils::Duration exposureValue = effectiveExposureValue * evGain; - - /* Clamp the exposure value to the min and max authorized */ - utils::Duration maxTotalExposure = maxShutterSpeed_ * maxAnalogueGain_; - exposureValue = std::min(exposureValue, maxTotalExposure); - LOG(IPU3Agc, Debug) << "Target total exposure " << exposureValue - << ", maximum is " << maxTotalExposure; - - /* - * Filter the exposure. - * \todo estimate if we need to desaturate - */ - exposureValue = filterExposure(exposureValue); - - /* - * Divide the exposure value as new exposure and gain values. - * - * Push the shutter time up to the maximum first, and only then - * increase the gain. - */ - utils::Duration shutterTime = - std::clamp(exposureValue / minAnalogueGain_, - minShutterSpeed_, maxShutterSpeed_); - double stepGain = std::clamp(exposureValue / shutterTime, - minAnalogueGain_, maxAnalogueGain_); - LOG(IPU3Agc, Debug) << "Divided up shutter and gain are " - << shutterTime << " and " - << stepGain; -} - /** * \brief Estimate the relative luminance of the frame with a given gain - * \param[in] frameContext The shared IPA frame context - * \param[in] grid The grid used to store the statistics in the IPU3 - * \param[in] stats The IPU3 statistics and ISP results - * \param[in] gain The gain to apply to the frame - * \return The relative luminance - * - * This function estimates the average relative luminance of the frame that - * would be output by the sensor if an additional \a gain was applied. + * \param[in] gain The gain to apply in estimating luminance * * The estimation is based on the AWB statistics for the current frame. Red, * green and blue averages for all cells are first multiplied by the gain, and @@ -339,42 +187,9 @@ void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, * * More detailed information can be found in: * https://en.wikipedia.org/wiki/Relative_luminance + * + * @return The relative luminance of the frame */ -double Agc::estimateLuminance(IPAActiveState &activeState, - const ipu3_uapi_grid_config &grid, - const ipu3_uapi_stats_3a *stats, - double gain) -{ - double redSum = 0, greenSum = 0, blueSum = 0; - - /* Sum the per-channel averages, saturated to 255. */ - for (unsigned int cellY = 0; cellY < grid.height; cellY++) { - for (unsigned int cellX = 0; cellX < grid.width; cellX++) { - uint32_t cellPosition = cellY * stride_ + cellX; - - const ipu3_uapi_awb_set_item *cell = - reinterpret_cast( - &stats->awb_raw_buffer.meta_data[cellPosition] - ); - const uint8_t G_avg = (cell->Gr_avg + cell->Gb_avg) / 2; - - redSum += std::min(cell->R_avg * gain, 255.0); - greenSum += std::min(G_avg * gain, 255.0); - blueSum += std::min(cell->B_avg * gain, 255.0); - } - } - - /* - * Apply the AWB gains to approximate colours correctly, use the Rec. - * 601 formula to calculate the relative luminance, and normalize it. - */ - double ySum = redSum * activeState.awb.gains.red * 0.299 - + greenSum * activeState.awb.gains.green * 0.587 - + blueSum * activeState.awb.gains.blue * 0.114; - - return ySum / (grid.height * grid.width) / 255; -} - double Agc::estimateLuminance(double gain) { double redSum = 0, greenSum = 0, blueSum = 0; @@ -408,42 +223,6 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, const ipu3_uapi_stats_3a *stats, ControlList &metadata) { - /* - * Estimate the gain needed to have the proportion of pixels in a given - * desired range. iqMean is the mean value of the top 2% of the - * cumulative histogram, and we want it to be as close as possible to a - * configured target. - */ - double iqMean = measureBrightness(stats, context.configuration.grid.bdsGrid); - double iqMeanGain = kEvGainTarget * knumHistogramBins / iqMean; - - /* - * Estimate the gain needed to achieve a relative luminance target. To - * account for non-linearity caused by saturation, the value needs to be - * estimated in an iterative process, as multiplying by a gain will not - * increase the relative luminance by the same factor if some image - * regions are saturated. - */ - double yGain = 1.0; - double yTarget = kRelativeLuminanceTarget; - - for (unsigned int i = 0; i < 8; i++) { - double yValue = estimateLuminance(context.activeState, - context.configuration.grid.bdsGrid, - stats, yGain); - double extraGain = std::min(10.0, yTarget / (yValue + .001)); - - yGain *= extraGain; - LOG(IPU3Agc, Debug) << "Y value: " << yValue - << ", Y target: " << yTarget - << ", gives gain " << yGain; - if (extraGain < 1.01) - break; - } - - computeExposure(context, frameContext, yGain, iqMeanGain); - frameCount_++; - Histogram hist = parseStatistics(stats, context.configuration.grid.bdsGrid); rGain_ = context.activeState.awb.gains.red; gGain_ = context.activeState.awb.gains.blue; diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 40f32188..945d1846 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -38,29 +38,16 @@ public: ControlList &metadata) override; private: - double measureBrightness(const ipu3_uapi_stats_3a *stats, - const ipu3_uapi_grid_config &grid) const; - utils::Duration filterExposure(utils::Duration currentExposure); - void computeExposure(IPAContext &context, IPAFrameContext &frameContext, - double yGain, double iqMeanGain); - double estimateLuminance(IPAActiveState &activeState, - const ipu3_uapi_grid_config &grid, - const ipu3_uapi_stats_3a *stats, - double gain); double estimateLuminance(double gain) override; Histogram parseStatistics(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid); - uint64_t frameCount_; - utils::Duration minShutterSpeed_; utils::Duration maxShutterSpeed_; double minAnalogueGain_; double maxAnalogueGain_; - utils::Duration filteredExposure_; - uint32_t stride_; double rGain_; double gGain_; From patchwork Wed Apr 17 13:15:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19899 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 63136C32CA for ; Wed, 17 Apr 2024 13:16:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B5B5563368; Wed, 17 Apr 2024 15:16:08 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="GmEfe7Oq"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5FB3F63365 for ; Wed, 17 Apr 2024 15:15:57 +0200 (CEST) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 4E8841815; Wed, 17 Apr 2024 15:15:10 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1713359710; bh=sgYmjPNwvsNEFKyq4BgXci7sH83fHuf1IlXouiX4HA8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GmEfe7OqT9eqR84OH1zgH+Oc/rS+onWMIlNxUF70SSHa3mL5ED3ueLIUg6q5CYJRG c+MYSh4HLoGc0ecyoJYW2hcID7yGQ+QrHo5WnTMyfiR08CR5EW4+mWnrOhPAl+25d5 Lmtn3zDo6EZo3cqxlfnetcTv0ktniDWEvjpdfjOw= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally Subject: [PATCH v2 7/8] ipa: rkisp1: Derive rkisp1::algorithms::Agc from MeanLuminanceAgc Date: Wed, 17 Apr 2024 14:15:35 +0100 Message-Id: <20240417131536.484129-8-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240417131536.484129-1-dan.scally@ideasonboard.com> References: <20240417131536.484129-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that we have a MeanLuminanceAgc class that centralises our AEGC algorithm, derive the RkISP1's Agc class from it and plumb in the necessary framework to enable it to be used. For simplicities sake this commit switches the algorithm to use the derived class, but does not remove the bespoke functions at this time. Signed-off-by: Daniel Scally Reviewed-by: Stefan Klug Reviewed-by: Paul Elder --- Changes in v2: - Squashed the patch adding parseStatistics() - Used the first available constraint and exposure modes rather than assuming the "Normal" mode was available - Stopped storing a Histogram in the class members when it's only needed during ::process() - Remembered to report the controls out to IPARkISP1::updateControls() src/ipa/rkisp1/algorithms/agc.cpp | 82 +++++++++++++++++++++++++++++-- src/ipa/rkisp1/algorithms/agc.h | 9 +++- src/ipa/rkisp1/ipa_context.h | 5 ++ src/ipa/rkisp1/rkisp1.cpp | 3 +- 4 files changed, 92 insertions(+), 7 deletions(-) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 47a6f7b2..0d66fcca 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -64,6 +64,30 @@ Agc::Agc() supportsRaw_ = true; } +/** + * \brief Initialise the AGC algorithm from tuning files + * + * \param[in] context The shared IPA context + * \param[in] tuningData The YamlObject containing Agc tuning data + * + * This function calls the base class' tuningData parsers to discover which + * control values are supported. + * + * \return 0 on success or errors from the base class + */ +int Agc::init(IPAContext &context, const YamlObject &tuningData) +{ + int ret; + + ret = parseTuningData(tuningData); + if (ret) + return ret; + + context.ctrlMap.merge(controls()); + + return 0; +} + /** * \brief Configure the AGC given a configInfo * \param[in] context The shared IPA context @@ -81,6 +105,9 @@ int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) context.activeState.agc.manual.exposure = context.activeState.agc.automatic.exposure; context.activeState.agc.autoEnabled = !context.configuration.raw; + context.activeState.agc.constraintMode = constraintModes().begin()->first; + context.activeState.agc.exposureMode = exposureModeHelpers().begin()->first; + /* * Define the measurement window for AGC as a centered rectangle * covering 3/4 of the image width and height. @@ -95,6 +122,15 @@ int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) * frame index */ frameCount_ = 0; + + /* \todo Run this again when FrameDurationLimits is passed in */ + configureExposureModeHelpers(context.configuration.sensor.minShutterSpeed, + context.configuration.sensor.maxShutterSpeed, + context.configuration.sensor.minAnalogueGain, + context.configuration.sensor.maxAnalogueGain); + + resetFrameCount(); + return 0; } @@ -234,7 +270,6 @@ void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, double yGain, double iqMeanGain) { IPASessionConfiguration &configuration = context.configuration; - IPAActiveState &activeState = context.activeState; /* Get the effective exposure and gain applied on the sensor. */ uint32_t exposure = frameContext.sensor.exposure; @@ -300,10 +335,6 @@ void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, LOG(RkISP1Agc, Debug) << "Divided up shutter and gain are " << shutterTime << " and " << stepGain; - - /* Update the estimated exposure and gain. */ - activeState.agc.automatic.exposure = shutterTime / configuration.sensor.lineDuration; - activeState.agc.automatic.gain = stepGain; } /** @@ -373,6 +404,19 @@ void Agc::fillMetadata(IPAContext &context, IPAFrameContext &frameContext, metadata.set(controls::FrameDuration, frameDuration.get()); } +double Agc::estimateLuminance(double gain) +{ + double ySum = 0.0; + + /* Sum the averages, saturated to 255. */ + for (uint8_t expMean : expMeans_) + ySum += std::min(expMean * gain, 255.0); + + /* \todo Weight with the AWB gains */ + + return ySum / expMeans_.size() / 255; +} + /** * \brief Process RkISP1 statistics, and run AGC operations * \param[in] context The shared IPA context @@ -438,7 +482,35 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, computeExposure(context, frameContext, yGain, iqMeanGain); frameCount_++; + expMeans_ = { params->ae.exp_mean, context.hw->numAeCells }; + + /* + * The Agc algorithm needs to know the effective exposure value that was + * applied to the sensor when the statistics were collected. + */ + utils::Duration exposureTime = context.configuration.sensor.lineDuration + * frameContext.sensor.exposure; + double analogueGain = frameContext.sensor.gain; + utils::Duration effectiveExposureValue = exposureTime * analogueGain; + + utils::Duration shutterTime; + double aGain, dGain; + std::tie(shutterTime, aGain, dGain) = + calculateNewEv(context.activeState.agc.constraintMode, + context.activeState.agc.exposureMode, + Histogram(hist), effectiveExposureValue); + + LOG(RkISP1Agc, Debug) + << "Divided up shutter, analogue gain and digital gain are " + << shutterTime << ", " << aGain << " and " << dGain; + + IPAActiveState &activeState = context.activeState; + /* Update the estimated exposure and gain. */ + activeState.agc.automatic.exposure = shutterTime / context.configuration.sensor.lineDuration; + activeState.agc.automatic.gain = aGain; + fillMetadata(context, frameContext, metadata); + expMeans_ = {}; } REGISTER_IPA_ALGORITHM(Agc, "Agc") diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h index fb82a33f..a8080228 100644 --- a/src/ipa/rkisp1/algorithms/agc.h +++ b/src/ipa/rkisp1/algorithms/agc.h @@ -14,18 +14,22 @@ #include +#include "libipa/agc_mean_luminance.h" +#include "libipa/histogram.h" + #include "algorithm.h" namespace libcamera { namespace ipa::rkisp1::algorithms { -class Agc : public Algorithm +class Agc : public Algorithm, public AgcMeanLuminance { public: Agc(); ~Agc() = default; + int init(IPAContext &context, const YamlObject &tuningData) override; int configure(IPAContext &context, const IPACameraSensorInfo &configInfo) override; void queueRequest(IPAContext &context, const uint32_t frame, @@ -47,10 +51,13 @@ private: double measureBrightness(Span hist) const; void fillMetadata(IPAContext &context, IPAFrameContext &frameContext, ControlList &metadata); + double estimateLuminance(double gain) override; uint64_t frameCount_; utils::Duration filteredExposure_; + + Span expMeans_; }; } /* namespace ipa::rkisp1::algorithms */ diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index 10d8f38c..256b75eb 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -12,6 +12,7 @@ #include +#include #include #include @@ -67,6 +68,8 @@ struct IPAActiveState { } automatic; bool autoEnabled; + uint32_t constraintMode; + uint32_t exposureMode; } agc; struct { @@ -151,6 +154,8 @@ struct IPAContext { IPAActiveState activeState; FCQueue frameContexts; + + ControlInfoMap::Map ctrlMap; }; } /* namespace ipa::rkisp1 */ diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 9dc5f53c..d8610095 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -119,7 +119,7 @@ const ControlInfoMap::Map rkisp1Controls{ } /* namespace */ IPARkISP1::IPARkISP1() - : context_({ {}, {}, {}, { kMaxFrameContexts } }) + : context_({ {}, {}, {}, { kMaxFrameContexts }, {} }) { } @@ -427,6 +427,7 @@ void IPARkISP1::updateControls(const IPACameraSensorInfo &sensorInfo, frameDurations[1], frameDurations[2]); + ctrlMap.merge(context_.ctrlMap); *ipaControls = ControlInfoMap(std::move(ctrlMap), controls::controls); } From patchwork Wed Apr 17 13:15:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19900 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 54D69C32C9 for ; Wed, 17 Apr 2024 13:16:10 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B756E63376; Wed, 17 Apr 2024 15:16:09 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="udBtvI3h"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9BA3963367 for ; Wed, 17 Apr 2024 15:15:57 +0200 (CEST) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 8CB0A63F; Wed, 17 Apr 2024 15:15:10 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1713359710; bh=shs/2pPUlOz5PmDROnvfEmksOQkRLkuVHnSctBwOJ7Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=udBtvI3hSpgcC/bOw3c/JnPp+wdbs68eBeR9a4nWwGWSWvC4Ljp2ih3SZycZ4ggJt JDDWXTDh7EZdMFfcrBH7+SUp1a/Rpb9zhZgpJPy4enBm9wWyzanU7L5467AQiDSjlS 7iwuk+WaKUGwo0S12XIFwBkLkEX0W+vbnYJzcKFc= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally , Stefan Klug Subject: [PATCH v2 8/8] ipa: rkisp1: Remove bespoke Agc functions Date: Wed, 17 Apr 2024 14:15:36 +0100 Message-Id: <20240417131536.484129-9-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240417131536.484129-1-dan.scally@ideasonboard.com> References: <20240417131536.484129-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that the rkisp1 Agc algorithm is a derivation of MeanLuminanceAgc we can remove the bespoke functions from the IPA's class. Reviewed-by: Stefan Klug Signed-off-by: Daniel Scally Reviewed-by: Paul Elder Reviewed-by: Jacopo Mondi --- Changes in v2: - Kept the documentation for estimateLuminance() src/ipa/rkisp1/algorithms/agc.cpp | 231 ++---------------------------- src/ipa/rkisp1/algorithms/agc.h | 9 -- 2 files changed, 14 insertions(+), 226 deletions(-) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 0d66fcca..27b6f2c1 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -36,30 +36,7 @@ namespace ipa::rkisp1::algorithms { LOG_DEFINE_CATEGORY(RkISP1Agc) -/* Minimum limit for analogue gain value */ -static constexpr double kMinAnalogueGain = 1.0; - -/* \todo Honour the FrameDurationLimits control instead of hardcoding a limit */ -static constexpr utils::Duration kMaxShutterSpeed = 60ms; - -/* Number of frames to wait before calculating stats on minimum exposure */ -static constexpr uint32_t kNumStartupFrames = 10; - -/* Target value to reach for the top 2% of the histogram */ -static constexpr double kEvGainTarget = 0.5; - -/* - * Relative luminance target. - * - * It's a number that's chosen so that, when the camera points at a grey - * target, the resulting image brightness is considered right. - * - * \todo Why is the value different between IPU3 and RkISP1 ? - */ -static constexpr double kRelativeLuminanceTarget = 0.4; - Agc::Agc() - : frameCount_(0), filteredExposure_(0s) { supportsRaw_ = true; } @@ -117,12 +94,6 @@ int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) context.configuration.agc.measureWindow.h_size = 3 * configInfo.outputSize.width / 4; context.configuration.agc.measureWindow.v_size = 3 * configInfo.outputSize.height / 4; - /* - * \todo Use the upcoming per-frame context API that will provide a - * frame index - */ - frameCount_ = 0; - /* \todo Run this again when FrameDurationLimits is passed in */ configureExposureModeHelpers(context.configuration.sensor.minShutterSpeed, context.configuration.sensor.maxShutterSpeed, @@ -224,122 +195,24 @@ void Agc::prepare(IPAContext &context, const uint32_t frame, params->module_en_update |= RKISP1_CIF_ISP_MODULE_HST; } -/** - * \brief Apply a filter on the exposure value to limit the speed of changes - * \param[in] exposureValue The target exposure from the AGC algorithm - * - * The speed of the filter is adaptive, and will produce the target quicker - * during startup, or when the target exposure is within 20% of the most recent - * filter output. - * - * \return The filtered exposure - */ -utils::Duration Agc::filterExposure(utils::Duration exposureValue) -{ - double speed = 0.2; - - /* Adapt instantly if we are in startup phase. */ - if (frameCount_ < kNumStartupFrames) - speed = 1.0; - - /* - * If we are close to the desired result, go faster to avoid making - * multiple micro-adjustments. - * \todo Make this customisable? - */ - if (filteredExposure_ < 1.2 * exposureValue && - filteredExposure_ > 0.8 * exposureValue) - speed = sqrt(speed); - - filteredExposure_ = speed * exposureValue + - filteredExposure_ * (1.0 - speed); - - LOG(RkISP1Agc, Debug) << "After filtering, exposure " << filteredExposure_; - - return filteredExposure_; -} - -/** - * \brief Estimate the new exposure and gain values - * \param[inout] context The shared IPA Context - * \param[in] frameContext The FrameContext for this frame - * \param[in] yGain The gain calculated on the current brightness level - * \param[in] iqMeanGain The gain calculated based on the relative luminance target - */ -void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, - double yGain, double iqMeanGain) +void Agc::fillMetadata(IPAContext &context, IPAFrameContext &frameContext, + ControlList &metadata) { - IPASessionConfiguration &configuration = context.configuration; - - /* Get the effective exposure and gain applied on the sensor. */ - uint32_t exposure = frameContext.sensor.exposure; - double analogueGain = frameContext.sensor.gain; - - /* Use the highest of the two gain estimates. */ - double evGain = std::max(yGain, iqMeanGain); - - utils::Duration minShutterSpeed = configuration.sensor.minShutterSpeed; - utils::Duration maxShutterSpeed = std::min(configuration.sensor.maxShutterSpeed, - kMaxShutterSpeed); - - double minAnalogueGain = std::max(configuration.sensor.minAnalogueGain, - kMinAnalogueGain); - double maxAnalogueGain = configuration.sensor.maxAnalogueGain; - - /* Consider within 1% of the target as correctly exposed. */ - if (utils::abs_diff(evGain, 1.0) < 0.01) - return; - - /* extracted from Rpi::Agc::computeTargetExposure. */ - - /* Calculate the shutter time in seconds. */ - utils::Duration currentShutter = exposure * configuration.sensor.lineDuration; - - /* - * Update the exposure value for the next computation using the values - * of exposure and gain really used by the sensor. - */ - utils::Duration effectiveExposureValue = currentShutter * analogueGain; - - LOG(RkISP1Agc, Debug) << "Actual total exposure " << currentShutter * analogueGain - << " Shutter speed " << currentShutter - << " Gain " << analogueGain - << " Needed ev gain " << evGain; - - /* - * Calculate the current exposure value for the scene as the latest - * exposure value applied multiplied by the new estimated gain. - */ - utils::Duration exposureValue = effectiveExposureValue * evGain; - - /* Clamp the exposure value to the min and max authorized. */ - utils::Duration maxTotalExposure = maxShutterSpeed * maxAnalogueGain; - exposureValue = std::min(exposureValue, maxTotalExposure); - LOG(RkISP1Agc, Debug) << "Target total exposure " << exposureValue - << ", maximum is " << maxTotalExposure; - - /* - * Divide the exposure value as new exposure and gain values. - * \todo estimate if we need to desaturate - */ - exposureValue = filterExposure(exposureValue); + utils::Duration exposureTime = context.configuration.sensor.lineDuration + * frameContext.sensor.exposure; + metadata.set(controls::AnalogueGain, frameContext.sensor.gain); + metadata.set(controls::ExposureTime, exposureTime.get()); - /* - * Push the shutter time up to the maximum first, and only then - * increase the gain. - */ - utils::Duration shutterTime = std::clamp(exposureValue / minAnalogueGain, - minShutterSpeed, maxShutterSpeed); - double stepGain = std::clamp(exposureValue / shutterTime, - minAnalogueGain, maxAnalogueGain); - LOG(RkISP1Agc, Debug) << "Divided up shutter and gain are " - << shutterTime << " and " - << stepGain; + /* \todo Use VBlank value calculated from each frame exposure. */ + uint32_t vTotal = context.configuration.sensor.size.height + + context.configuration.sensor.defVBlank; + utils::Duration frameDuration = context.configuration.sensor.lineDuration + * vTotal; + metadata.set(controls::FrameDuration, frameDuration.get()); } /** * \brief Estimate the relative luminance of the frame with a given gain - * \param[in] expMeans The mean luminance values, from the RkISP1 statistics * \param[in] gain The gain to apply to the frame * * This function estimates the average relative luminance of the frame that @@ -353,8 +226,6 @@ void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, * YUV doesn't take into account the fact that the R, G and B components * contribute differently to the relative luminance. * - * \todo Have a dedicated YUV algorithm ? - * * The values are normalized to the [0.0, 1.0] range, where 1.0 corresponds to a * theoretical perfect reflector of 100% reference white. * @@ -363,47 +234,6 @@ void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, * * \return The relative luminance */ -double Agc::estimateLuminance(Span expMeans, double gain) -{ - double ySum = 0.0; - - /* Sum the averages, saturated to 255. */ - for (uint8_t expMean : expMeans) - ySum += std::min(expMean * gain, 255.0); - - /* \todo Weight with the AWB gains */ - - return ySum / expMeans.size() / 255; -} - -/** - * \brief Estimate the mean value of the top 2% of the histogram - * \param[in] hist The histogram statistics computed by the RkISP1 - * \return The mean value of the top 2% of the histogram - */ -double Agc::measureBrightness(Span hist) const -{ - Histogram histogram{ hist }; - /* Estimate the quantile mean of the top 2% of the histogram. */ - return histogram.interQuantileMean(0.98, 1.0); -} - -void Agc::fillMetadata(IPAContext &context, IPAFrameContext &frameContext, - ControlList &metadata) -{ - utils::Duration exposureTime = context.configuration.sensor.lineDuration - * frameContext.sensor.exposure; - metadata.set(controls::AnalogueGain, frameContext.sensor.gain); - metadata.set(controls::ExposureTime, exposureTime.get()); - - /* \todo Use VBlank value calculated from each frame exposure. */ - uint32_t vTotal = context.configuration.sensor.size.height - + context.configuration.sensor.defVBlank; - utils::Duration frameDuration = context.configuration.sensor.lineDuration - * vTotal; - metadata.set(controls::FrameDuration, frameDuration.get()); -} - double Agc::estimateLuminance(double gain) { double ySum = 0.0; @@ -448,40 +278,7 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, const rkisp1_cif_isp_stat *params = &stats->params; ASSERT(stats->meas_type & RKISP1_CIF_ISP_STAT_AUTOEXP); - Span ae{ params->ae.exp_mean, context.hw->numAeCells }; - Span hist{ - params->hist.hist_bins, - context.hw->numHistogramBins - }; - - double iqMean = measureBrightness(hist); - double iqMeanGain = kEvGainTarget * hist.size() / iqMean; - - /* - * Estimate the gain needed to achieve a relative luminance target. To - * account for non-linearity caused by saturation, the value needs to be - * estimated in an iterative process, as multiplying by a gain will not - * increase the relative luminance by the same factor if some image - * regions are saturated. - */ - double yGain = 1.0; - double yTarget = kRelativeLuminanceTarget; - - for (unsigned int i = 0; i < 8; i++) { - double yValue = estimateLuminance(ae, yGain); - double extra_gain = std::min(10.0, yTarget / (yValue + .001)); - - yGain *= extra_gain; - LOG(RkISP1Agc, Debug) << "Y value: " << yValue - << ", Y target: " << yTarget - << ", gives gain " << yGain; - if (extra_gain < 1.01) - break; - } - - computeExposure(context, frameContext, yGain, iqMeanGain); - frameCount_++; - + Histogram hist({ params->hist.hist_bins, context.hw->numHistogramBins }); expMeans_ = { params->ae.exp_mean, context.hw->numAeCells }; /* @@ -498,7 +295,7 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, std::tie(shutterTime, aGain, dGain) = calculateNewEv(context.activeState.agc.constraintMode, context.activeState.agc.exposureMode, - Histogram(hist), effectiveExposureValue); + hist, effectiveExposureValue); LOG(RkISP1Agc, Debug) << "Divided up shutter, analogue gain and digital gain are " diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h index a8080228..a2c1f61a 100644 --- a/src/ipa/rkisp1/algorithms/agc.h +++ b/src/ipa/rkisp1/algorithms/agc.h @@ -44,19 +44,10 @@ public: ControlList &metadata) override; private: - void computeExposure(IPAContext &Context, IPAFrameContext &frameContext, - double yGain, double iqMeanGain); - utils::Duration filterExposure(utils::Duration exposureValue); - double estimateLuminance(Span expMeans, double gain); - double measureBrightness(Span hist) const; void fillMetadata(IPAContext &context, IPAFrameContext &frameContext, ControlList &metadata); double estimateLuminance(double gain) override; - uint64_t frameCount_; - - utils::Duration filteredExposure_; - Span expMeans_; };