From patchwork Fri Mar 22 13:14:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19787 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4A005C3272 for ; Fri, 22 Mar 2024 13:15:15 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 36A6263311; Fri, 22 Mar 2024 14:15:11 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="tbEJgqdX"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9FBBA63036 for ; Fri, 22 Mar 2024 14:15:07 +0100 (CET) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 916278CC; Fri, 22 Mar 2024 14:14:38 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1711113279; bh=nfi9Nt2Bx5i6Jnsj1gRsnK1v/XJKtmjPzmsM4qU/nvI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tbEJgqdXAIyOXkt3e1dPF1FvyF1AmFC/A4/Sjl6JD/TB4OUCevaRrRMPVp6PPPwBn E1Q0tGEmEeJd0N/8Z2C/G24UrnFhZepjgfhe6OQf4mMVsx6wniLzK8Tv0uJ3XvIPiz 7lHf+AYKqzBw1f7hUxaRKxf9WcuO/f+6HS+654MA= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally Subject: [PATCH 01/10] ipa: libipa: Allow creation of empty Histogram Date: Fri, 22 Mar 2024 13:14:42 +0000 Message-Id: <20240322131451.3092931-2-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240322131451.3092931-1-dan.scally@ideasonboard.com> References: <20240322131451.3092931-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" For convenience's sake allow the creation of empty Histograms so they can be embedded within other Classes and filled out with data at some later point in time. Signed-off-by: Daniel Scally Reviewed-by: Stefan Klug Reviewed-by: Paul Elder --- src/ipa/libipa/histogram.h | 1 + 1 file changed, 1 insertion(+) diff --git a/src/ipa/libipa/histogram.h b/src/ipa/libipa/histogram.h index 05bb4b80..db13c155 100644 --- a/src/ipa/libipa/histogram.h +++ b/src/ipa/libipa/histogram.h @@ -22,6 +22,7 @@ namespace ipa { class Histogram { public: + Histogram() { cumulative_.push_back(0); }; Histogram(Span data); size_t bins() const { return cumulative_.size() - 1; } uint64_t total() const { return cumulative_[cumulative_.size() - 1]; } From patchwork Fri Mar 22 13:14:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19788 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id F3B28C3274 for ; Fri, 22 Mar 2024 13:15:16 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 6FB9863075; Fri, 22 Mar 2024 14:15:12 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="cha/aXzh"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1641263036 for ; Fri, 22 Mar 2024 14:15:08 +0100 (CET) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 2C390842; Fri, 22 Mar 2024 14:14:39 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1711113279; bh=8NYpgW+u2h4uyxVkXYwlsAwOIZHi6oT2EGsDgM4I6lw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cha/aXzh3jxaY43Uc0Nmx1V4mgtneG/EC+0VfbalcL6/OOoSgZo3aeUX5mfKg4etM sTxmZtj+jlip6fPxS/sdjkCuaFlzvx8Z/Tat6nruSPTuk4tcfVnJct5HRNPPFm55oX kiiR42HtHaeRn8WWoQev3P9fEBSzcwUky+VYnO54= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally Subject: [PATCH 02/10] libcamera: controls: Generate enum value-name maps Date: Fri, 22 Mar 2024 13:14:43 +0000 Message-Id: <20240322131451.3092931-3-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240322131451.3092931-1-dan.scally@ideasonboard.com> References: <20240322131451.3092931-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Generate maps associating control enumerated control values with a string representing that value to provide a central reference for them across all pipeline handlers. Signed-off-by: Daniel Scally Reviewed-by: Stefan Klug Reviewed-by: Jacopo Mondi Reviewed-by: Paul Elder --- include/libcamera/control_ids.h.in | 2 ++ include/libcamera/property_ids.h.in | 2 ++ utils/gen-controls.py | 19 +++++++++++++++++++ 3 files changed, 23 insertions(+) diff --git a/include/libcamera/control_ids.h.in b/include/libcamera/control_ids.h.in index d53b1cf7..58dd48e1 100644 --- a/include/libcamera/control_ids.h.in +++ b/include/libcamera/control_ids.h.in @@ -10,7 +10,9 @@ #pragma once #include +#include #include +#include #include diff --git a/include/libcamera/property_ids.h.in b/include/libcamera/property_ids.h.in index 43372c71..f51ba028 100644 --- a/include/libcamera/property_ids.h.in +++ b/include/libcamera/property_ids.h.in @@ -9,7 +9,9 @@ #pragma once +#include #include +#include #include diff --git a/utils/gen-controls.py b/utils/gen-controls.py index 6cd5e362..4fe1e705 100755 --- a/utils/gen-controls.py +++ b/utils/gen-controls.py @@ -140,6 +140,12 @@ ${description} */''') enum_values_start = string.Template('''extern const std::array ${name}Values = {''') enum_values_values = string.Template('''\tstatic_cast(${name}),''') + name_value_map_doc = string.Template('''/** + * \\var ${name}NameValueMap + * \\brief Map of all $name supported value names (in std::string format) to value + */''') + name_value_map_start = string.Template('''extern const std::map ${name}NameValueMap = {''') + name_value_values = string.Template('''\t{ "${name}", ${name} },''') ctrls_doc = {} ctrls_def = {} @@ -183,6 +189,7 @@ ${description} values_info = { 'name': info['name'], + 'type': ctrl.type, 'size': num_entries, } target_doc.append(enum_values_doc.substitute(values_info)) @@ -194,6 +201,15 @@ ${description} target_def.append(enum_values_values.substitute(value_info)) target_def.append("};") + target_doc.append(name_value_map_doc.substitute(values_info)) + target_def.append(name_value_map_start.substitute(values_info)) + for enum in ctrl.enum_values: + value_info = { + 'name': enum.name + } + target_def.append(name_value_values.substitute(value_info)) + target_def.append("};") + target_doc.append(doc_template.substitute(info)) target_def.append(def_template.substitute(info)) @@ -231,6 +247,7 @@ def generate_h(controls, mode, ranges): enum_template_start = string.Template('''enum ${name}Enum {''') enum_value_template = string.Template('''\t${name} = ${value},''') enum_values_template = string.Template('''extern const std::array ${name}Values;''') + name_value_map_template = string.Template('''extern const std::map ${name}NameValueMap;''') template = string.Template('''extern const Control<${type}> ${name};''') ctrls = {} @@ -273,9 +290,11 @@ def generate_h(controls, mode, ranges): values_info = { 'name': info['name'], + 'type': ctrl.type, 'size': num_entries, } target_ctrls.append(enum_values_template.substitute(values_info)) + target_ctrls.append(name_value_map_template.substitute(values_info)) target_ctrls.append(template.substitute(info)) id_value[vendor] += 1 From patchwork Fri Mar 22 13:14:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19789 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 62621C3272 for ; Fri, 22 Mar 2024 13:15:18 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 18D8163079; Fri, 22 Mar 2024 14:15:13 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ak2J9ibh"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9B82761C45 for ; Fri, 22 Mar 2024 14:15:08 +0100 (CET) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 8ED6F8CC; Fri, 22 Mar 2024 14:14:39 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1711113279; bh=eo7TiyCG8+FJpjgxvEtbeh9ejggUlpeW4hnEZXI4KDk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ak2J9ibh8UvljWLh6giN8STz8rKZjJDR6L3Gv3lerfxrQRlCc7ESm6JOL0KqMuPC+ jR0W4+fhhzp1GTU8dvySaw8Xe3h26pru7YZd/aPFaEWBECdBLepuysAgUUbuk5MdU1 hr5d+FHKsUUGCs9+8zCAwF9i8WrM9wQkcaH20n2o= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Paul Elder , Daniel Scally Subject: [PATCH 03/10] ipa: libipa: Add ExposureModeHelper Date: Fri, 22 Mar 2024 13:14:44 +0000 Message-Id: <20240322131451.3092931-4-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240322131451.3092931-1-dan.scally@ideasonboard.com> References: <20240322131451.3092931-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Paul Elder Add a helper for managing exposure modes and splitting exposure times into shutter and gain values. Signed-off-by: Paul Elder Signed-off-by: Daniel Scally --- src/ipa/libipa/exposure_mode_helper.cpp | 307 ++++++++++++++++++++++++ src/ipa/libipa/exposure_mode_helper.h | 61 +++++ src/ipa/libipa/meson.build | 2 + 3 files changed, 370 insertions(+) create mode 100644 src/ipa/libipa/exposure_mode_helper.cpp create mode 100644 src/ipa/libipa/exposure_mode_helper.h diff --git a/src/ipa/libipa/exposure_mode_helper.cpp b/src/ipa/libipa/exposure_mode_helper.cpp new file mode 100644 index 00000000..9e01f908 --- /dev/null +++ b/src/ipa/libipa/exposure_mode_helper.cpp @@ -0,0 +1,307 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Paul Elder + * + * exposure_mode_helper.h - Helper class that performs computations relating to exposure + */ +#include "exposure_mode_helper.h" + +#include + +#include + +/** + * \file exposure_mode_helper.h + * \brief Helper class that performs computations relating to exposure + * + * Exposure modes contain a list of shutter and gain values to determine how to + * split the supplied exposure time into shutter and gain. As this function is + * expected to be replicated in various IPAs, this helper class factors it + * away. + */ + +namespace libcamera { + +LOG_DEFINE_CATEGORY(ExposureModeHelper) + +namespace ipa { + +/** + * \class ExposureModeHelper + * \brief Class for splitting exposure time into shutter and gain + */ + +/** + * \brief Initialize an ExposureModeHelper instance + */ +ExposureModeHelper::ExposureModeHelper() + : minShutter_(0), maxShutter_(0), minGain_(0), maxGain_(0) +{ +} + +ExposureModeHelper::~ExposureModeHelper() +{ +} + +/** + * \brief Initialize an ExposureModeHelper instance + * \param[in] shutter The list of shutter values + * \param[in] gain The list of gain values + * + * When splitting an exposure time into shutter and gain, the shutter will be + * increased first before increasing the gain. This is done in stages, where + * each stage is an index into both lists. Both lists consequently need to be + * the same length. + * + * \return Zero on success, negative error code otherwise + */ +int ExposureModeHelper::init(std::vector &shutter, std::vector &gain) +{ + if (shutter.size() != gain.size()) { + LOG(ExposureModeHelper, Error) + << "Invalid exposure mode:" + << " expected size of 'shutter' and 'gain to be equal," + << " got " << shutter.size() << " and " << gain.size() + << " respectively"; + return -EINVAL; + } + + std::copy(shutter.begin(), shutter.end(), std::back_inserter(shutters_)); + std::copy(gain.begin(), gain.end(), std::back_inserter(gains_)); + + /* + * Initialize the max shutter and gain if they aren't initialized yet. + * This is to protect against the event that configure() is not called + * before splitExposure(). + */ + if (!maxShutter_) { + if (shutters_.size() > 0) + maxShutter_ = shutter.back(); + } + + if (!maxGain_) { + if (gains_.size() > 0) + maxGain_ = gain.back(); + } + + return 0; +} + +/** + * \brief Configure the ExposureModeHelper + * \param[in] minShutter The minimum shutter time + * \param[in] maxShutter The maximum shutter time + * \param[in] minGain The minimum gain + * \param[in] maxGain The maximum gain + * + * Note that the ExposureModeHelper needs to be reconfigured when + * FrameDurationLimits is passed, and not just at IPA configuration time. + * + * This function configures the maximum shutter and maximum gain. + */ +void ExposureModeHelper::configure(utils::Duration minShutter, + utils::Duration maxShutter, + double minGain, + double maxGain) +{ + minShutter_ = minShutter; + maxShutter_ = maxShutter; + minGain_ = minGain; + maxGain_ = maxGain; +} + +utils::Duration ExposureModeHelper::clampShutter(utils::Duration shutter) +{ + return std::clamp(shutter, minShutter_, maxShutter_); +} + +double ExposureModeHelper::clampGain(double gain) +{ + return std::clamp(gain, minGain_, maxGain_); +} + +std::tuple +ExposureModeHelper::splitExposure(utils::Duration exposure, + utils::Duration shutter, bool shutterFixed, + double gain, bool gainFixed) +{ + shutter = clampShutter(shutter); + gain = clampGain(gain); + + /* Not sure why you'd want to do this... */ + if (shutterFixed && gainFixed) + return { shutter, gain, 1 }; + + /* Initial shutter and gain settings are sufficient */ + if (shutter * gain >= exposure) { + /* Both shutter and gain cannot go lower */ + if (shutter == minShutter_ && gain == minGain_) + return { shutter, gain, 1 }; + + /* Shutter cannot go lower */ + if (shutter == minShutter_ || shutterFixed) + return { shutter, + gainFixed ? gain : clampGain(exposure / shutter), + 1 }; + + /* Gain cannot go lower */ + if (gain == minGain_ || gainFixed) + return { + shutterFixed ? shutter : clampShutter(exposure / gain), + gain, + 1 + }; + + /* Both can go lower */ + return { clampShutter(exposure / minGain_), + exposure / clampShutter(exposure / minGain_), + 1 }; + } + + unsigned int stage; + utils::Duration stageShutter; + double stageGain; + double lastStageGain; + + /* We've already done stage 0 above so we start at 1 */ + for (stage = 1; stage < gains_.size(); stage++) { + stageShutter = shutterFixed ? shutter : clampShutter(shutters_[stage]); + stageGain = gainFixed ? gain : clampGain(gains_[stage]); + lastStageGain = gainFixed ? gain : clampGain(gains_[stage - 1]); + + /* + * If the product of the new stage shutter and the old stage + * gain is sufficient and we can change the shutter, reduce it. + */ + if (!shutterFixed && stageShutter * lastStageGain >= exposure) + return { clampShutter(exposure / lastStageGain), lastStageGain, 1 }; + + /* + * The new stage shutter with old stage gain were insufficient, + * so try the new stage shutter with new stage gain. If it is + * sufficient and we can change the shutter, reduce it. + */ + if (!shutterFixed && stageShutter * stageGain >= exposure) + return { clampShutter(exposure / stageGain), stageGain, 1 }; + + /* + * Same as above, but we can't change the shutter, so change + * the gain instead. + * + * Note that at least one of !shutterFixed and !gainFixed is + * guaranteed. + */ + if (!gainFixed && stageShutter * stageGain >= exposure) + return { stageShutter, clampGain(exposure / stageShutter), 1 }; + } + + /* From here on we're going to try to max out shutter then gain */ + shutter = shutterFixed ? shutter : maxShutter_; + gain = gainFixed ? gain : maxGain_; + + /* + * We probably don't want to use the actual maximum analogue gain (as + * it'll be unreasonably high), so we'll at least try to max out the + * shutter, which is expected to be a bit more reasonable, as it is + * limited by FrameDurationLimits and/or the sensor configuration. + */ + if (!shutterFixed && shutter * stageGain >= exposure) + return { clampShutter(exposure / stageGain), stageGain, 1 }; + + /* + * If that's still not enough exposure, or if shutter is fixed, then + * we'll max out the analogue gain before using digital gain. + */ + if (!gainFixed && shutter * gain >= exposure) + return { shutter, clampGain(exposure / shutter), 1 }; + + /* + * We're out of shutter time and analogue gain; send the rest of the + * exposure time to digital gain. + */ + return { shutter, gain, exposure / (shutter * gain) }; +} + +/** + * \brief Split exposure time into shutter and gain + * \param[in] exposure Exposure time + * \return Tuple of shutter time, analogue gain, and digital gain + */ +std::tuple +ExposureModeHelper::splitExposure(utils::Duration exposure) +{ + ASSERT(maxShutter_); + ASSERT(maxGain_); + utils::Duration shutter; + double gain; + + if (shutters_.size()) { + shutter = shutters_.at(0); + gain = gains_.at(0); + } else { + shutter = maxShutter_; + gain = maxGain_; + } + + return splitExposure(exposure, shutter, false, gain, false); +} + +/** + * \brief Split exposure time into shutter and gain, with fixed shutter + * \param[in] exposure Exposure time + * \param[in] fixedShutter Fixed shutter time + * + * Same as the base splitExposure, but with a fixed shutter (aka "shutter priority"). + * + * \return Tuple of shutter time, analogue gain, and digital gain + */ +std::tuple +ExposureModeHelper::splitExposure(utils::Duration exposure, utils::Duration fixedShutter) +{ + ASSERT(maxGain_); + double gain = gains_.size() ? gains_.at(0) : maxGain_; + + return splitExposure(exposure, fixedShutter, true, gain, false); +} + +/** + * \brief Split exposure time into shutter and gain, with fixed gain + * \param[in] exposure Exposure time + * \param[in] fixedGain Fixed gain + * + * Same as the base splitExposure, but with a fixed gain (aka "gain priority"). + * + * \return Tuple of shutter time, analogue gain, and digital gain + */ +std::tuple +ExposureModeHelper::splitExposure(utils::Duration exposure, double fixedGain) +{ + ASSERT(maxShutter_); + utils::Duration shutter = shutters_.size() ? shutters_.at(0) : maxShutter_; + + return splitExposure(exposure, shutter, false, fixedGain, true); +} + +/** + * \fn ExposureModeHelper::minShutter() + * \brief Retrieve the configured minimum shutter time + */ + +/** + * \fn ExposureModeHelper::maxShutter() + * \brief Retrieve the configured maximum shutter time + */ + +/** + * \fn ExposureModeHelper::minGain() + * \brief Retrieve the configured minimum gain + */ + +/** + * \fn ExposureModeHelper::maxGain() + * \brief Retrieve the configured maximum gain + */ + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/exposure_mode_helper.h b/src/ipa/libipa/exposure_mode_helper.h new file mode 100644 index 00000000..d576c952 --- /dev/null +++ b/src/ipa/libipa/exposure_mode_helper.h @@ -0,0 +1,61 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Paul Elder + * + * exposure_mode_helper.h - Helper class that performs computations relating to exposure + */ + +#pragma once + +#include +#include +#include + +#include + +namespace libcamera { + +namespace ipa { + +class ExposureModeHelper +{ +public: + ExposureModeHelper(); + ~ExposureModeHelper(); + + int init(std::vector &shutters, std::vector &gains); + void configure(utils::Duration minShutter, utils::Duration maxShutter, + double minGain, double maxGain); + + std::tuple splitExposure(utils::Duration exposure); + std::tuple splitExposure(utils::Duration exposure, + utils::Duration fixedShutter); + std::tuple splitExposure(utils::Duration exposure, + double fixedGain); + + utils::Duration minShutter() { return minShutter_; }; + utils::Duration maxShutter() { return maxShutter_; }; + double minGain() { return minGain_; }; + double maxGain() { return maxGain_; }; + +private: + utils::Duration clampShutter(utils::Duration shutter); + double clampGain(double gain); + + std::tuple + splitExposure(utils::Duration exposure, + utils::Duration shutter, bool shutterFixed, + double gain, bool gainFixed); + + std::vector shutters_; + std::vector gains_; + + utils::Duration minShutter_; + utils::Duration maxShutter_; + double minGain_; + double maxGain_; +}; + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index 016b8e0e..37fbd177 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -3,6 +3,7 @@ libipa_headers = files([ 'algorithm.h', 'camera_sensor_helper.h', + 'exposure_mode_helper.h', 'fc_queue.h', 'histogram.h', 'module.h', @@ -11,6 +12,7 @@ libipa_headers = files([ libipa_sources = files([ 'algorithm.cpp', 'camera_sensor_helper.cpp', + 'exposure_mode_helper.cpp', 'fc_queue.cpp', 'histogram.cpp', 'module.cpp', From patchwork Fri Mar 22 13:14:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19791 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id BC212C3274 for ; Fri, 22 Mar 2024 13:15:20 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C023263311; Fri, 22 Mar 2024 14:15:17 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="l7mKzC2V"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4592F63055 for ; Fri, 22 Mar 2024 14:15:09 +0100 (CET) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 2CA8C842; Fri, 22 Mar 2024 14:14:40 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1711113280; bh=S4cGW8CYQljeYizRjaOnjh9CA1KlaF98ppm+qKwygLQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=l7mKzC2VjeTUC6l/MdlIHZDeHiGDp2kg+FGWv+jm8LeVN0VYmZbArW3YWUFqONQQp wdXxt1QY5nEidkBMC+ZHdVU+pF+MmxrEDzQHvgl0oOiTjcUh2reea3RcEsdYSvRhEe hjd7xZO2dgic6Nl2j7hp0Mqgp0FHFGru8gFSiIBI= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally Subject: [PATCH 04/10] ipa: libipa: Add MeanLuminanceAgc base class Date: Fri, 22 Mar 2024 13:14:45 +0000 Message-Id: <20240322131451.3092931-5-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240322131451.3092931-1-dan.scally@ideasonboard.com> References: <20240322131451.3092931-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The Agc algorithms for the RkIsp1 and IPU3 IPAs do the same thing in very large part; following the Rpi IPA's algorithm in spirit with a few tunable values in that IPA being hardcoded in the libipa ones. Add a new base class for MeanLuminanceAgc which implements the same algorithm and additionally parses yaml tuning files to inform an IPA module's Agc algorithm about valid constraint and exposure modes and their associated bounds. Signed-off-by: Daniel Scally --- src/ipa/libipa/agc.cpp | 526 +++++++++++++++++++++++++++++++++++++ src/ipa/libipa/agc.h | 82 ++++++ src/ipa/libipa/meson.build | 2 + 3 files changed, 610 insertions(+) create mode 100644 src/ipa/libipa/agc.cpp create mode 100644 src/ipa/libipa/agc.h diff --git a/src/ipa/libipa/agc.cpp b/src/ipa/libipa/agc.cpp new file mode 100644 index 00000000..af57a571 --- /dev/null +++ b/src/ipa/libipa/agc.cpp @@ -0,0 +1,526 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Ideas on Board Oy + * + * agc.cpp - Base class for libipa-compliant AGC algorithms + */ + +#include "agc.h" + +#include + +#include +#include + +#include "exposure_mode_helper.h" + +using namespace libcamera::controls; + +/** + * \file agc.h + * \brief Base class implementing mean luminance AEGC. + */ + +namespace libcamera { + +using namespace std::literals::chrono_literals; + +LOG_DEFINE_CATEGORY(Agc) + +namespace ipa { + +/* + * Number of frames to wait before calculating stats on minimum exposure + * \todo should this be a tunable value? + */ +static constexpr uint32_t kNumStartupFrames = 10; + +/* + * Default relative luminance target + * + * This value should be chosen so that when the camera points at a grey target, + * the resulting image brightness looks "right". Custom values can be passed + * as the relativeLuminanceTarget value in sensor tuning files. + */ +static constexpr double kDefaultRelativeLuminanceTarget = 0.16; + +/** + * \struct MeanLuminanceAgc::AgcConstraint + * \brief The boundaries and target for an AeConstraintMode constraint + * + * This structure describes an AeConstraintMode constraint for the purposes of + * this algorithm. The algorithm will apply the constraints by calculating the + * Histogram's inter-quantile mean between the given quantiles and ensure that + * the resulting value is the right side of the given target (as defined by the + * boundary and luminance target). + */ + +/** + * \enum MeanLuminanceAgc::AgcConstraint::Bound + * \brief Specify whether the constraint defines a lower or upper bound + * \var MeanLuminanceAgc::AgcConstraint::LOWER + * \brief The constraint defines a lower bound + * \var MeanLuminanceAgc::AgcConstraint::UPPER + * \brief The constraint defines an upper bound + */ + +/** + * \var MeanLuminanceAgc::AgcConstraint::bound + * \brief The type of constraint bound + */ + +/** + * \var MeanLuminanceAgc::AgcConstraint::qLo + * \brief The lower quantile to use for the constraint + */ + +/** + * \var MeanLuminanceAgc::AgcConstraint::qHi + * \brief The upper quantile to use for the constraint + */ + +/** + * \var MeanLuminanceAgc::AgcConstraint::yTarget + * \brief The luminance target for the constraint + */ + +/** + * \class MeanLuminanceAgc + * \brief a mean-based auto-exposure algorithm + * + * This algorithm calculates a shutter time, analogue and digital gain such that + * the normalised mean luminance value of an image is driven towards a target, + * which itself is discovered from tuning data. The algorithm is a two-stage + * process: + * + * In the first stage, an initial gain value is derived by iteratively comparing + * the gain-adjusted mean luminance across an entire image against a target, and + * selecting a value which pushes it as closely as possible towards the target. + * + * In the second stage we calculate the gain required to drive the average of a + * section of a histogram to a target value, where the target and the boundaries + * of the section of the histogram used in the calculation are taken from the + * values defined for the currently configured AeConstraintMode within the + * tuning data. The gain from the first stage is then clamped to the gain from + * this stage. + * + * The final gain is used to adjust the effective exposure value of the image, + * and that new exposure value divided into shutter time, analogue gain and + * digital gain according to the selected AeExposureMode. + */ + +MeanLuminanceAgc::MeanLuminanceAgc() + : frameCount_(0), filteredExposure_(0s), relativeLuminanceTarget_(0) +{ +} + +/** + * \brief Parse the relative luminance target from the tuning data + * \param[in] tuningData The YamlObject holding the algorithm's tuning data + */ +void MeanLuminanceAgc::parseRelativeLuminanceTarget(const YamlObject &tuningData) +{ + relativeLuminanceTarget_ = + tuningData["relativeLuminanceTarget"].get(kDefaultRelativeLuminanceTarget); +} + +/** + * \brief Parse an AeConstraintMode constraint from tuning data + * \param[in] modeDict the YamlObject holding the constraint data + * \param[in] id The constraint ID from AeConstraintModeEnum + */ +void MeanLuminanceAgc::parseConstraint(const YamlObject &modeDict, int32_t id) +{ + for (const auto &[boundName, content] : modeDict.asDict()) { + if (boundName != "upper" && boundName != "lower") { + LOG(Agc, Warning) + << "Ignoring unknown constraint bound '" << boundName << "'"; + continue; + } + + unsigned int idx = static_cast(boundName == "upper"); + AgcConstraint::Bound bound = static_cast(idx); + double qLo = content["qLo"].get().value_or(0.98); + double qHi = content["qHi"].get().value_or(1.0); + double yTarget = + content["yTarget"].getList().value_or(std::vector{ 0.5 }).at(0); + + AgcConstraint constraint = { bound, qLo, qHi, yTarget }; + + if (!constraintModes_.count(id)) + constraintModes_[id] = {}; + + if (idx) + constraintModes_[id].push_back(constraint); + else + constraintModes_[id].insert(constraintModes_[id].begin(), constraint); + } +} + +/** + * \brief Parse tuning data file to populate AeConstraintMode control + * \param[in] tuningData the YamlObject representing the tuning data for Agc + * + * The Agc algorithm's tuning data should contain a dictionary called + * AeConstraintMode containing per-mode setting dictionaries with the key being + * a value from \ref controls::AeConstraintModeNameValueMap. Each mode dict may + * contain either a "lower" or "upper" key, or both, in this format: + * + * \code{.unparsed} + * algorithms: + * - Agc: + * AeConstraintMode: + * ConstraintNormal: + * lower: + * qLo: 0.98 + * qHi: 1.0 + * yTarget: 0.5 + * ConstraintHighlight: + * lower: + * qLo: 0.98 + * qHi: 1.0 + * yTarget: 0.5 + * upper: + * qLo: 0.98 + * qHi: 1.0 + * yTarget: 0.8 + * + * \endcode + * + * The parsed dictionaries are used to populate an array of available values for + * the AeConstraintMode control and stored for later use in the algorithm. + * + * \return -EINVAL Where a defined constraint mode is invalid + * \return 0 on success + */ +int MeanLuminanceAgc::parseConstraintModes(const YamlObject &tuningData) +{ + std::vector availableConstraintModes; + + const YamlObject &yamlConstraintModes = tuningData[controls::AeConstraintMode.name()]; + if (yamlConstraintModes.isDictionary()) { + for (const auto &[modeName, modeDict] : yamlConstraintModes.asDict()) { + if (AeConstraintModeNameValueMap.find(modeName) == + AeConstraintModeNameValueMap.end()) { + LOG(Agc, Warning) + << "Skipping unknown constraint mode '" << modeName << "'"; + continue; + } + + if (!modeDict.isDictionary()) { + LOG(Agc, Error) + << "Invalid constraint mode '" << modeName << "'"; + return -EINVAL; + } + + parseConstraint(modeDict, + AeConstraintModeNameValueMap.at(modeName)); + availableConstraintModes.push_back( + AeConstraintModeNameValueMap.at(modeName)); + } + } + + /* + * If the tuning data file contains no constraints then we use the + * default constraint that the various Agc algorithms were adhering to + * anyway before centralisation. + */ + if (constraintModes_.empty()) { + AgcConstraint constraint = { + AgcConstraint::Bound::LOWER, + 0.98, + 1.0, + 0.5 + }; + + constraintModes_[controls::ConstraintNormal].insert( + constraintModes_[controls::ConstraintNormal].begin(), + constraint); + availableConstraintModes.push_back( + AeConstraintModeNameValueMap.at("ConstraintNormal")); + } + + controls_[&controls::AeConstraintMode] = ControlInfo(availableConstraintModes); + + return 0; +} + +/** + * \brief Parse tuning data file to populate AeExposureMode control + * \param[in] tuningData the YamlObject representing the tuning data for Agc + * + * The Agc algorithm's tuning data should contain a dictionary called + * AeExposureMode containing per-mode setting dictionaries with the key being + * a value from \ref controls::AeExposureModeNameValueMap. Each mode dict should + * contain an array of shutter times with the key "shutter" and an array of gain + * values with the key "gain", in this format: + * + * \code{.unparsed} + * algorithms: + * - Agc: + * AeExposureMode: + * ExposureNormal: + * shutter: [ 100, 10000, 30000, 60000, 120000 ] + * gain: [ 1.0, 2.0, 4.0, 6.0, 6.0 ] + * ExposureShort: + * shutter: [ 100, 10000, 30000, 60000, 120000 ] + * gain: [ 1.0, 2.0, 4.0, 6.0, 6.0 ] + * + * \endcode + * + * The parsed dictionaries are used to populate an array of available values for + * the AeExposureMode control and to create ExposureModeHelpers + * + * \return -EINVAL Where a defined constraint mode is invalid + * \return 0 on success + */ +int MeanLuminanceAgc::parseExposureModes(const YamlObject &tuningData) +{ + std::vector availableExposureModes; + int ret; + + const YamlObject &yamlExposureModes = tuningData[controls::AeExposureMode.name()]; + if (yamlExposureModes.isDictionary()) { + for (const auto &[modeName, modeValues] : yamlExposureModes.asDict()) { + if (AeExposureModeNameValueMap.find(modeName) == + AeExposureModeNameValueMap.end()) { + LOG(Agc, Warning) + << "Skipping unknown exposure mode '" << modeName << "'"; + continue; + } + + if (!modeValues.isDictionary()) { + LOG(Agc, Error) + << "Invalid exposure mode '" << modeName << "'"; + return -EINVAL; + } + + std::vector shutters = + modeValues["shutter"].getList().value_or(std::vector{}); + std::vector gains = + modeValues["gain"].getList().value_or(std::vector{}); + + std::vector shutterDurations; + std::transform(shutters.begin(), shutters.end(), + std::back_inserter(shutterDurations), + [](uint32_t time) { return std::chrono::microseconds(time); }); + + std::shared_ptr helper = + std::make_shared(); + if ((ret = helper->init(shutterDurations, gains) < 0)) { + LOG(Agc, Error) + << "Failed to parse exposure mode '" << modeName << "'"; + return ret; + } + + exposureModeHelpers_[AeExposureModeNameValueMap.at(modeName)] = helper; + availableExposureModes.push_back(AeExposureModeNameValueMap.at(modeName)); + } + } + + /* + * If we don't have any exposure modes in the tuning data we create an + * ExposureModeHelper using empty shutter time and gain arrays, which + * will then internally simply drive the shutter as high as possible + * before touching gain + */ + if (availableExposureModes.empty()) { + int32_t exposureModeId = AeExposureModeNameValueMap.at("ExposureNormal"); + std::vector shutterDurations = {}; + std::vector gains = {}; + + std::shared_ptr helper = + std::make_shared(); + if ((ret = helper->init(shutterDurations, gains) < 0)) { + LOG(Agc, Error) + << "Failed to create default ExposureModeHelper"; + return ret; + } + + exposureModeHelpers_[exposureModeId] = helper; + availableExposureModes.push_back(exposureModeId); + } + + controls_[&controls::AeExposureMode] = ControlInfo(availableExposureModes); + + return 0; +} + +/** + * \fn MeanLuminanceAgc::constraintModes() + * \brief Get the constraint modes that have been parsed from tuning data + */ + +/** + * \fn MeanLuminanceAgc::exposureModeHelpers() + * \brief Get the ExposureModeHelpers that have been parsed from tuning data + */ + +/** + * \fn MeanLuminanceAgc::controls() + * \brief Get the controls that have been generated after parsing tuning data + */ + +/** + * \fn MeanLuminanceAgc::estimateLuminance(const double gain) + * \brief Estimate the luminance of an image, adjusted by a given gain + * \param[in] gain The gain with which to adjust the luminance estimate + * + * This function is a pure virtual function because estimation of luminance is a + * hardware-specific operation, which depends wholly on the format of the stats + * that are delivered to libcamera from the ISP. Derived classes must implement + * an overriding function that calculates the normalised mean luminance value + * across the entire image. + * + * \return The normalised relative luminance of the image + */ + +/** + * \brief Estimate the initial gain needed to achieve a relative luminance + * target + * + * To account for non-linearity caused by saturation, the value needs to be + * estimated in an iterative process, as multiplying by a gain will not increase + * the relative luminance by the same factor if some image regions are saturated + * + * \return The calculated initial gain + */ +double MeanLuminanceAgc::estimateInitialGain() +{ + double yTarget = relativeLuminanceTarget_; + double yGain = 1.0; + + for (unsigned int i = 0; i < 8; i++) { + double yValue = estimateLuminance(yGain); + double extra_gain = std::min(10.0, yTarget / (yValue + .001)); + + yGain *= extra_gain; + LOG(Agc, Debug) << "Y value: " << yValue + << ", Y target: " << yTarget + << ", gives gain " << yGain; + + if (utils::abs_diff(extra_gain, 1.0) < 0.01) + break; + } + + return yGain; +} + +/** + * \brief Clamp gain within the bounds of a defined constraint + * \param[in] constraintModeIndex The index of the constraint to adhere to + * \param[in] hist A histogram over which to calculate inter-quantile means + * \param[in] gain The gain to clamp + * + * \return The gain clamped within the constraint bounds + */ +double MeanLuminanceAgc::constraintClampGain(uint32_t constraintModeIndex, + const Histogram &hist, + double gain) +{ + std::vector &constraints = constraintModes_[constraintModeIndex]; + for (const AgcConstraint &constraint : constraints) { + double newGain = constraint.yTarget * hist.bins() / + hist.interQuantileMean(constraint.qLo, constraint.qHi); + + if (constraint.bound == AgcConstraint::Bound::LOWER && + newGain > gain) + gain = newGain; + + if (constraint.bound == AgcConstraint::Bound::UPPER && + newGain < gain) + gain = newGain; + } + + return gain; +} + +/** + * \brief Apply a filter on the exposure value to limit the speed of changes + * \param[in] exposureValue The target exposure from the AGC algorithm + * + * The speed of the filter is adaptive, and will produce the target quicker + * during startup, or when the target exposure is within 20% of the most recent + * filter output. + * + * \return The filtered exposure + */ +utils::Duration MeanLuminanceAgc::filterExposure(utils::Duration exposureValue) +{ + double speed = 0.2; + + /* Adapt instantly if we are in startup phase. */ + if (frameCount_ < kNumStartupFrames) + speed = 1.0; + + /* + * If we are close to the desired result, go faster to avoid making + * multiple micro-adjustments. + * \todo Make this customisable? + */ + if (filteredExposure_ < 1.2 * exposureValue && + filteredExposure_ > 0.8 * exposureValue) + speed = sqrt(speed); + + filteredExposure_ = speed * exposureValue + + filteredExposure_ * (1.0 - speed); + + return filteredExposure_; +} + +/** + * \brief Calculate the new exposure value + * \param[in] constraintModeIndex The index of the current constraint mode + * \param[in] exposureModeIndex The index of the current exposure mode + * \param[in] yHist A Histogram from the ISP statistics to use in constraining + * the calculated gain + * \param[in] effectiveExposureValue The EV applied to the frame from which the + * statistics in use derive + * + * Calculate a new exposure value to try to obtain the target. The calculated + * exposure value is filtered to prevent rapid changes from frame to frame, and + * divided into shutter time, analogue and digital gain. + * + * \return Tuple of shutter time, analogue gain, and digital gain + */ +std::tuple +MeanLuminanceAgc::calculateNewEv(uint32_t constraintModeIndex, + uint32_t exposureModeIndex, + const Histogram &yHist, + utils::Duration effectiveExposureValue) +{ + /* + * The pipeline handler should validate that we have received an allowed + * value for AeExposureMode. + */ + std::shared_ptr exposureModeHelper = + exposureModeHelpers_.at(exposureModeIndex); + + double gain = estimateInitialGain(); + gain = constraintClampGain(constraintModeIndex, yHist, gain); + + /* + * We don't check whether we're already close to the target, because + * even if the effective exposure value is the same as the last frame's + * we could have switched to an exposure mode that would require a new + * pass through the splitExposure() function. + */ + + utils::Duration newExposureValue = effectiveExposureValue * gain; + utils::Duration maxTotalExposure = exposureModeHelper->maxShutter() + * exposureModeHelper->maxGain(); + newExposureValue = std::min(newExposureValue, maxTotalExposure); + + /* + * We filter the exposure value to make sure changes are not too jarring + * from frame to frame. + */ + newExposureValue = filterExposure(newExposureValue); + + frameCount_++; + return exposureModeHelper->splitExposure(newExposureValue); +} + +}; /* namespace ipa */ + +}; /* namespace libcamera */ diff --git a/src/ipa/libipa/agc.h b/src/ipa/libipa/agc.h new file mode 100644 index 00000000..902a359a --- /dev/null +++ b/src/ipa/libipa/agc.h @@ -0,0 +1,82 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Ideas on Board Oy + * + agc.h - Base class for libipa-compliant AGC algorithms + */ + +#pragma once + +#include +#include + +#include + +#include "libcamera/internal/yaml_parser.h" + +#include "exposure_mode_helper.h" +#include "histogram.h" + +namespace libcamera { + +namespace ipa { + +class MeanLuminanceAgc +{ +public: + MeanLuminanceAgc(); + virtual ~MeanLuminanceAgc() = default; + + struct AgcConstraint { + enum class Bound { + LOWER = 0, + UPPER = 1 + }; + Bound bound; + double qLo; + double qHi; + double yTarget; + }; + + void parseRelativeLuminanceTarget(const YamlObject &tuningData); + void parseConstraint(const YamlObject &modeDict, int32_t id); + int parseConstraintModes(const YamlObject &tuningData); + int parseExposureModes(const YamlObject &tuningData); + + std::map> constraintModes() + { + return constraintModes_; + } + + std::map> exposureModeHelpers() + { + return exposureModeHelpers_; + } + + ControlInfoMap::Map controls() + { + return controls_; + } + + virtual double estimateLuminance(const double gain) = 0; + double estimateInitialGain(); + double constraintClampGain(uint32_t constraintModeIndex, + const Histogram &hist, + double gain); + utils::Duration filterExposure(utils::Duration exposureValue); + std::tuple + calculateNewEv(uint32_t constraintModeIndex, uint32_t exposureModeIndex, + const Histogram &yHist, utils::Duration effectiveExposureValue); +private: + uint64_t frameCount_; + utils::Duration filteredExposure_; + double relativeLuminanceTarget_; + + std::map> constraintModes_; + std::map> exposureModeHelpers_; + ControlInfoMap::Map controls_; +}; + +}; /* namespace ipa */ + +}; /* namespace libcamera */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index 37fbd177..31cc8d70 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -1,6 +1,7 @@ # SPDX-License-Identifier: CC0-1.0 libipa_headers = files([ + 'agc.h', 'algorithm.h', 'camera_sensor_helper.h', 'exposure_mode_helper.h', @@ -10,6 +11,7 @@ libipa_headers = files([ ]) libipa_sources = files([ + 'agc.cpp', 'algorithm.cpp', 'camera_sensor_helper.cpp', 'exposure_mode_helper.cpp', From patchwork Fri Mar 22 13:14:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19790 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id AD52EC32C3 for ; Fri, 22 Mar 2024 13:15:19 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5497F63354; Fri, 22 Mar 2024 14:15:16 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="jMwdG0Qe"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 82BA3632E7 for ; Fri, 22 Mar 2024 14:15:09 +0100 (CET) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 86991D50; Fri, 22 Mar 2024 14:14:40 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1711113280; bh=eBnNF8fPnn3dj5kMYozHbwOefdh6RqSKyouYwf3Qs1Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jMwdG0QeI/ox7dlXz7Hwv3ax6NQpDvaP4ESeJR5/n39/V+7uIlXpyWc7loyWn4zd+ i00Sre5/2rNWbhhnmg1qAhQwitdYbKy6xdViXdbS9urfguiCNWlSuYdw4I2k4p3AYk yRVHGtbKnRVMJwe2dJQ3L/sDaD3jlTgggLXFUHtA= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally Subject: [PATCH 05/10] ipa: ipu3: Parse and store Agc stats Date: Fri, 22 Mar 2024 13:14:46 +0000 Message-Id: <20240322131451.3092931-6-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240322131451.3092931-1-dan.scally@ideasonboard.com> References: <20240322131451.3092931-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In preparation for switching to a derivation of MeanLuminanceAgc, add a function to parse and store the statistics for easy retrieval in an overriding estimateLuminance() function. Signed-off-by: Daniel Scally --- src/ipa/ipu3/algorithms/agc.cpp | 33 +++++++++++++++++++++++++++++++++ src/ipa/ipu3/algorithms/agc.h | 8 ++++++++ 2 files changed, 41 insertions(+) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 606a237a..ee9a42cf 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -142,6 +142,39 @@ double Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, return Histogram(Span(hist)).interQuantileMean(0.98, 1.0); } +void Agc::parseStatistics(const ipu3_uapi_stats_3a *stats, + const ipu3_uapi_grid_config &grid) +{ + uint32_t hist[knumHistogramBins] = { 0 }; + + reds_.clear(); + greens_.clear(); + blues_.clear(); + + for (unsigned int cellY = 0; cellY < grid.height; cellY++) { + for (unsigned int cellX = 0; cellX < grid.width; cellX++) { + uint32_t cellPosition = cellY * stride_ + cellX; + + const ipu3_uapi_awb_set_item *cell = + reinterpret_cast( + &stats->awb_raw_buffer.meta_data[cellPosition]); + + reds_.push_back(cell->R_avg); + greens_.push_back((cell->Gr_avg + cell->Gb_avg) / 2); + blues_.push_back(cell->B_avg); + + /* + * Store the average green value to estimate the + * brightness. Even the overexposed pixels are + * taken into account. + */ + hist[(cell->Gr_avg + cell->Gb_avg) / 2]++; + } + } + + hist_ = Histogram(Span(hist)); +} + /** * \brief Apply a filter on the exposure value to limit the speed of changes * \param[in] exposureValue The target exposure from the AGC algorithm diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 9d6e3ff1..7ed0ef7a 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -13,6 +13,8 @@ #include +#include "libipa/histogram.h" + #include "algorithm.h" namespace libcamera { @@ -43,6 +45,8 @@ private: const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, double gain); + void parseStatistics(const ipu3_uapi_stats_3a *stats, + const ipu3_uapi_grid_config &grid); uint64_t frameCount_; @@ -55,6 +59,10 @@ private: utils::Duration filteredExposure_; uint32_t stride_; + std::vector reds_; + std::vector blues_; + std::vector greens_; + Histogram hist_; }; } /* namespace ipa::ipu3::algorithms */ From patchwork Fri Mar 22 13:14:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19792 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D48ECC32C4 for ; Fri, 22 Mar 2024 13:15:21 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D0A5363075; Fri, 22 Mar 2024 14:15:18 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="KpDHTMFe"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D41356330D for ; Fri, 22 Mar 2024 14:15:09 +0100 (CET) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id DD91313AB; Fri, 22 Mar 2024 14:14:40 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1711113281; bh=X9v72liXQVj9a2RAt2HzvwcVJtOFyAghxcn474kziEc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KpDHTMFeCfaZRakA91FEXqzi868C0aDlDhBikpB/dDOH1J0jjO3ja57YlzslBMxf/ E9E22N6GyP1jM1g+avpKR4MuawyX5OjqYRPsquwcIKTwANp4yJV0nXvsroddyhpPSj p1VsH8Tm0eqG9/SuPt07BDoS3bF3ysNqDVCzZGg4= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally Subject: [PATCH 06/10] ipa: ipu3: Derive ipu3::algorithms::Agc from MeanLuminanceAgc Date: Fri, 22 Mar 2024 13:14:47 +0000 Message-Id: <20240322131451.3092931-7-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240322131451.3092931-1-dan.scally@ideasonboard.com> References: <20240322131451.3092931-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that we have a MeanLuminanceAgc class that centralises our AEGC algorithm, derive the IPU3's Agc class from it and plumb in the necessary framework to enable it to be used. For simplicities sake this commit switches the algorithm to use the derived class, but does not remove the bespoke functions at this time. Signed-off-by: Daniel Scally --- src/ipa/ipu3/algorithms/agc.cpp | 98 ++++++++++++++++++++++++++++++--- src/ipa/ipu3/algorithms/agc.h | 6 +- src/ipa/ipu3/ipa_context.cpp | 3 + src/ipa/ipu3/ipa_context.h | 5 ++ src/ipa/ipu3/ipu3.cpp | 2 +- 5 files changed, 105 insertions(+), 9 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index ee9a42cf..a84534ea 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -71,11 +71,41 @@ static constexpr uint32_t kNumStartupFrames = 10; static constexpr double kRelativeLuminanceTarget = 0.16; Agc::Agc() - : frameCount_(0), minShutterSpeed_(0s), - maxShutterSpeed_(0s), filteredExposure_(0s) + : minShutterSpeed_(0s), maxShutterSpeed_(0s), context_(nullptr) { } +/** + * \brief Initialise the AGC algorith from tuning files + * + * \param[in] context The shared IPA context + * \param[in] tuningData The YamlObject containing Agc tuning data + * + * This function calls the base class' tuningData parsers to discover which + * control values are supported. + * + * \return 0 on success or errors from the base class + */ +int Agc::init(IPAContext &context, const YamlObject &tuningData) +{ + int ret; + + parseRelativeLuminanceTarget(tuningData); + + ret = parseConstraintModes(tuningData); + if (ret) + return ret; + + ret = parseExposureModes(tuningData); + if (ret) + return ret; + + context.ctrlMap.merge(controls()); + context_ = &context; + + return 0; +} + /** * \brief Configure the AGC given a configInfo * \param[in] context The shared IPA context @@ -103,6 +133,20 @@ int Agc::configure(IPAContext &context, activeState.agc.exposure = 10ms / configuration.sensor.lineDuration; frameCount_ = 0; + + /* + * \todo We should use the first available mode rather than assume that + * the "Normal" modes are present in tuning data. + */ + context.activeState.agc.constraintMode = controls::ConstraintNormal; + context.activeState.agc.exposureMode = controls::ExposureNormal; + + for (auto &[id, helper] : exposureModeHelpers()) { + /* \todo Run this again when FrameDurationLimits is passed in */ + helper->configure(minShutterSpeed_, maxShutterSpeed_, + minAnalogueGain_, maxAnalogueGain_); + } + return 0; } @@ -280,11 +324,6 @@ void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, LOG(IPU3Agc, Debug) << "Divided up shutter and gain are " << shutterTime << " and " << stepGain; - - IPAActiveState &activeState = context.activeState; - /* Update the estimated exposure and gain. */ - activeState.agc.exposure = shutterTime / configuration.sensor.lineDuration; - activeState.agc.gain = stepGain; } /** @@ -347,6 +386,26 @@ double Agc::estimateLuminance(IPAActiveState &activeState, return ySum / (grid.height * grid.width) / 255; } +double Agc::estimateLuminance(double gain) +{ + ASSERT(reds_.size() == greens_.size()); + ASSERT(greens_.size() == blues_.size()); + const ipu3_uapi_grid_config &grid = context_->configuration.grid.bdsGrid; + double redSum = 0, greenSum = 0, blueSum = 0; + + for (unsigned int i = 0; i < reds_.size(); i++) { + redSum += std::min(reds_[i] * gain, 255.0); + greenSum += std::min(greens_[i] * gain, 255.0); + blueSum += std::min(blues_[i] * gain, 255.0); + } + + double ySum = redSum * context_->activeState.awb.gains.red * 0.299 + + greenSum * context_->activeState.awb.gains.green * 0.587 + + blueSum * context_->activeState.awb.gains.blue * 0.114; + + return ySum / (grid.height * grid.width) / 255; +} + /** * \brief Process IPU3 statistics, and run AGC operations * \param[in] context The shared IPA context @@ -399,8 +458,33 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, computeExposure(context, frameContext, yGain, iqMeanGain); frameCount_++; + parseStatistics(stats, context.configuration.grid.bdsGrid); + + /* + * The Agc algorithm needs to know the effective exposure value that was + * applied to the sensor when the statistics were collected. + */ utils::Duration exposureTime = context.configuration.sensor.lineDuration * frameContext.sensor.exposure; + double analogueGain = frameContext.sensor.gain; + utils::Duration effectiveExposureValue = exposureTime * analogueGain; + + utils::Duration shutterTime; + double aGain, dGain; + std::tie(shutterTime, aGain, dGain) = + calculateNewEv(context.activeState.agc.constraintMode, + context.activeState.agc.exposureMode, hist_, + effectiveExposureValue); + + LOG(IPU3Agc, Debug) + << "Divided up shutter, analogue gain and digital gain are " + << shutterTime << ", " << aGain << " and " << dGain; + + IPAActiveState &activeState = context.activeState; + /* Update the estimated exposure and gain. */ + activeState.agc.exposure = shutterTime / context.configuration.sensor.lineDuration; + activeState.agc.gain = aGain; + metadata.set(controls::AnalogueGain, frameContext.sensor.gain); metadata.set(controls::ExposureTime, exposureTime.get()); diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 7ed0ef7a..8405da9d 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -13,6 +13,7 @@ #include +#include "libipa/agc.h" #include "libipa/histogram.h" #include "algorithm.h" @@ -23,12 +24,13 @@ struct IPACameraSensorInfo; namespace ipa::ipu3::algorithms { -class Agc : public Algorithm +class Agc : public Algorithm, public MeanLuminanceAgc { public: Agc(); ~Agc() = default; + int init(IPAContext &context, const YamlObject &tuningData) override; int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, @@ -45,6 +47,7 @@ private: const ipu3_uapi_grid_config &grid, const ipu3_uapi_stats_3a *stats, double gain); + double estimateLuminance(double gain) override; void parseStatistics(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid); @@ -59,6 +62,7 @@ private: utils::Duration filteredExposure_; uint32_t stride_; + IPAContext *context_; std::vector reds_; std::vector blues_; std::vector greens_; diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp index 959f314f..c4fb5642 100644 --- a/src/ipa/ipu3/ipa_context.cpp +++ b/src/ipa/ipu3/ipa_context.cpp @@ -47,6 +47,9 @@ namespace libcamera::ipa::ipu3 { * * \var IPAContext::activeState * \brief The current state of IPA algorithms + * + * \var IPAContext::ctrlMap + * \brief A ControlInfoMap::Map of controls populated by the algorithms */ /** diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index e9a3863b..a92cb6ce 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -12,6 +12,7 @@ #include +#include #include #include @@ -55,6 +56,8 @@ struct IPAActiveState { struct { uint32_t exposure; double gain; + uint32_t constraintMode; + uint32_t exposureMode; } agc; struct { @@ -85,6 +88,8 @@ struct IPAContext { IPAActiveState activeState; FCQueue frameContexts; + + ControlInfoMap::Map ctrlMap; }; } /* namespace ipa::ipu3 */ diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 08ee6eb3..2fcc0c91 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -189,7 +189,7 @@ private: }; IPAIPU3::IPAIPU3() - : context_({ {}, {}, { kMaxFrameContexts } }) + : context_({ {}, {}, { kMaxFrameContexts }, {} }) { } From patchwork Fri Mar 22 13:14:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19793 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C9944C3272 for ; Fri, 22 Mar 2024 13:15:22 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AA3386333B; Fri, 22 Mar 2024 14:15:20 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="soEqsDAV"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4A50861C45 for ; Fri, 22 Mar 2024 14:15:10 +0100 (CET) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 57673842; Fri, 22 Mar 2024 14:14:41 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1711113281; bh=OUI/NPlBNxWdhuGCcaP1NpaOJQeoz2eSzpnRacY+lNk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=soEqsDAVAoNyhSUVC2FFedaCwrghUqo8zoQ1hHKnzT1IsRqrOdEYtQc48kV4j+lpa 0caboVckfNJUZ6CZU6mjqoLiPwUuSani6u1v4v/17u4xgcHWjnE7kWm0JEmx4DIaIC hxzc6ShQqbAVD82sHTXqkg4FqLxkLmmAFQiH7XHg= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally Subject: [PATCH 07/10] ipa: ipu3: Remove bespoke AGC functions from IPU3 Date: Fri, 22 Mar 2024 13:14:48 +0000 Message-Id: <20240322131451.3092931-8-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240322131451.3092931-1-dan.scally@ideasonboard.com> References: <20240322131451.3092931-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that the IPU3's Agc is derived from MeanLuminanceAgc we can delete all the unecessary bespoke functions. Signed-off-by: Daniel Scally Reviewed-by: Stefan Klug --- src/ipa/ipu3/algorithms/agc.cpp | 241 -------------------------------- src/ipa/ipu3/algorithms/agc.h | 13 -- 2 files changed, 254 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index a84534ea..08deff0c 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -132,8 +132,6 @@ int Agc::configure(IPAContext &context, activeState.agc.gain = minAnalogueGain_; activeState.agc.exposure = 10ms / configuration.sensor.lineDuration; - frameCount_ = 0; - /* * \todo We should use the first available mode rather than assume that * the "Normal" modes are present in tuning data. @@ -150,42 +148,6 @@ int Agc::configure(IPAContext &context, return 0; } -/** - * \brief Estimate the mean value of the top 2% of the histogram - * \param[in] stats The statistics computed by the ImgU - * \param[in] grid The grid used to store the statistics in the IPU3 - * \return The mean value of the top 2% of the histogram - */ -double Agc::measureBrightness(const ipu3_uapi_stats_3a *stats, - const ipu3_uapi_grid_config &grid) const -{ - /* Initialise the histogram array */ - uint32_t hist[knumHistogramBins] = { 0 }; - - for (unsigned int cellY = 0; cellY < grid.height; cellY++) { - for (unsigned int cellX = 0; cellX < grid.width; cellX++) { - uint32_t cellPosition = cellY * stride_ + cellX; - - const ipu3_uapi_awb_set_item *cell = - reinterpret_cast( - &stats->awb_raw_buffer.meta_data[cellPosition] - ); - - uint8_t gr = cell->Gr_avg; - uint8_t gb = cell->Gb_avg; - /* - * Store the average green value to estimate the - * brightness. Even the overexposed pixels are - * taken into account. - */ - hist[(gr + gb) / 2]++; - } - } - - /* Estimate the quantile mean of the top 2% of the histogram. */ - return Histogram(Span(hist)).interQuantileMean(0.98, 1.0); -} - void Agc::parseStatistics(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid) { @@ -219,173 +181,6 @@ void Agc::parseStatistics(const ipu3_uapi_stats_3a *stats, hist_ = Histogram(Span(hist)); } -/** - * \brief Apply a filter on the exposure value to limit the speed of changes - * \param[in] exposureValue The target exposure from the AGC algorithm - * - * The speed of the filter is adaptive, and will produce the target quicker - * during startup, or when the target exposure is within 20% of the most recent - * filter output. - * - * \return The filtered exposure - */ -utils::Duration Agc::filterExposure(utils::Duration exposureValue) -{ - double speed = 0.2; - - /* Adapt instantly if we are in startup phase. */ - if (frameCount_ < kNumStartupFrames) - speed = 1.0; - - /* - * If we are close to the desired result, go faster to avoid making - * multiple micro-adjustments. - * \todo Make this customisable? - */ - if (filteredExposure_ < 1.2 * exposureValue && - filteredExposure_ > 0.8 * exposureValue) - speed = sqrt(speed); - - filteredExposure_ = speed * exposureValue + - filteredExposure_ * (1.0 - speed); - - LOG(IPU3Agc, Debug) << "After filtering, exposure " << filteredExposure_; - - return filteredExposure_; -} - -/** - * \brief Estimate the new exposure and gain values - * \param[inout] frameContext The shared IPA frame Context - * \param[in] yGain The gain calculated based on the relative luminance target - * \param[in] iqMeanGain The gain calculated based on the relative luminance target - */ -void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, - double yGain, double iqMeanGain) -{ - const IPASessionConfiguration &configuration = context.configuration; - /* Get the effective exposure and gain applied on the sensor. */ - uint32_t exposure = frameContext.sensor.exposure; - double analogueGain = frameContext.sensor.gain; - - /* Use the highest of the two gain estimates. */ - double evGain = std::max(yGain, iqMeanGain); - - /* Consider within 1% of the target as correctly exposed */ - if (utils::abs_diff(evGain, 1.0) < 0.01) - LOG(IPU3Agc, Debug) << "We are well exposed (evGain = " - << evGain << ")"; - - /* extracted from Rpi::Agc::computeTargetExposure */ - - /* Calculate the shutter time in seconds */ - utils::Duration currentShutter = exposure * configuration.sensor.lineDuration; - - /* - * Update the exposure value for the next computation using the values - * of exposure and gain really used by the sensor. - */ - utils::Duration effectiveExposureValue = currentShutter * analogueGain; - - LOG(IPU3Agc, Debug) << "Actual total exposure " << currentShutter * analogueGain - << " Shutter speed " << currentShutter - << " Gain " << analogueGain - << " Needed ev gain " << evGain; - - /* - * Calculate the current exposure value for the scene as the latest - * exposure value applied multiplied by the new estimated gain. - */ - utils::Duration exposureValue = effectiveExposureValue * evGain; - - /* Clamp the exposure value to the min and max authorized */ - utils::Duration maxTotalExposure = maxShutterSpeed_ * maxAnalogueGain_; - exposureValue = std::min(exposureValue, maxTotalExposure); - LOG(IPU3Agc, Debug) << "Target total exposure " << exposureValue - << ", maximum is " << maxTotalExposure; - - /* - * Filter the exposure. - * \todo estimate if we need to desaturate - */ - exposureValue = filterExposure(exposureValue); - - /* - * Divide the exposure value as new exposure and gain values. - * - * Push the shutter time up to the maximum first, and only then - * increase the gain. - */ - utils::Duration shutterTime = - std::clamp(exposureValue / minAnalogueGain_, - minShutterSpeed_, maxShutterSpeed_); - double stepGain = std::clamp(exposureValue / shutterTime, - minAnalogueGain_, maxAnalogueGain_); - LOG(IPU3Agc, Debug) << "Divided up shutter and gain are " - << shutterTime << " and " - << stepGain; -} - -/** - * \brief Estimate the relative luminance of the frame with a given gain - * \param[in] frameContext The shared IPA frame context - * \param[in] grid The grid used to store the statistics in the IPU3 - * \param[in] stats The IPU3 statistics and ISP results - * \param[in] gain The gain to apply to the frame - * \return The relative luminance - * - * This function estimates the average relative luminance of the frame that - * would be output by the sensor if an additional \a gain was applied. - * - * The estimation is based on the AWB statistics for the current frame. Red, - * green and blue averages for all cells are first multiplied by the gain, and - * then saturated to approximate the sensor behaviour at high brightness - * values. The approximation is quite rough, as it doesn't take into account - * non-linearities when approaching saturation. - * - * The relative luminance (Y) is computed from the linear RGB components using - * the Rec. 601 formula. The values are normalized to the [0.0, 1.0] range, - * where 1.0 corresponds to a theoretical perfect reflector of 100% reference - * white. - * - * More detailed information can be found in: - * https://en.wikipedia.org/wiki/Relative_luminance - */ -double Agc::estimateLuminance(IPAActiveState &activeState, - const ipu3_uapi_grid_config &grid, - const ipu3_uapi_stats_3a *stats, - double gain) -{ - double redSum = 0, greenSum = 0, blueSum = 0; - - /* Sum the per-channel averages, saturated to 255. */ - for (unsigned int cellY = 0; cellY < grid.height; cellY++) { - for (unsigned int cellX = 0; cellX < grid.width; cellX++) { - uint32_t cellPosition = cellY * stride_ + cellX; - - const ipu3_uapi_awb_set_item *cell = - reinterpret_cast( - &stats->awb_raw_buffer.meta_data[cellPosition] - ); - const uint8_t G_avg = (cell->Gr_avg + cell->Gb_avg) / 2; - - redSum += std::min(cell->R_avg * gain, 255.0); - greenSum += std::min(G_avg * gain, 255.0); - blueSum += std::min(cell->B_avg * gain, 255.0); - } - } - - /* - * Apply the AWB gains to approximate colours correctly, use the Rec. - * 601 formula to calculate the relative luminance, and normalize it. - */ - double ySum = redSum * activeState.awb.gains.red * 0.299 - + greenSum * activeState.awb.gains.green * 0.587 - + blueSum * activeState.awb.gains.blue * 0.114; - - return ySum / (grid.height * grid.width) / 255; -} - double Agc::estimateLuminance(double gain) { ASSERT(reds_.size() == greens_.size()); @@ -422,42 +217,6 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, const ipu3_uapi_stats_3a *stats, ControlList &metadata) { - /* - * Estimate the gain needed to have the proportion of pixels in a given - * desired range. iqMean is the mean value of the top 2% of the - * cumulative histogram, and we want it to be as close as possible to a - * configured target. - */ - double iqMean = measureBrightness(stats, context.configuration.grid.bdsGrid); - double iqMeanGain = kEvGainTarget * knumHistogramBins / iqMean; - - /* - * Estimate the gain needed to achieve a relative luminance target. To - * account for non-linearity caused by saturation, the value needs to be - * estimated in an iterative process, as multiplying by a gain will not - * increase the relative luminance by the same factor if some image - * regions are saturated. - */ - double yGain = 1.0; - double yTarget = kRelativeLuminanceTarget; - - for (unsigned int i = 0; i < 8; i++) { - double yValue = estimateLuminance(context.activeState, - context.configuration.grid.bdsGrid, - stats, yGain); - double extraGain = std::min(10.0, yTarget / (yValue + .001)); - - yGain *= extraGain; - LOG(IPU3Agc, Debug) << "Y value: " << yValue - << ", Y target: " << yTarget - << ", gives gain " << yGain; - if (extraGain < 1.01) - break; - } - - computeExposure(context, frameContext, yGain, iqMeanGain); - frameCount_++; - parseStatistics(stats, context.configuration.grid.bdsGrid); /* diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 8405da9d..78fa3c75 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -38,29 +38,16 @@ public: ControlList &metadata) override; private: - double measureBrightness(const ipu3_uapi_stats_3a *stats, - const ipu3_uapi_grid_config &grid) const; - utils::Duration filterExposure(utils::Duration currentExposure); - void computeExposure(IPAContext &context, IPAFrameContext &frameContext, - double yGain, double iqMeanGain); - double estimateLuminance(IPAActiveState &activeState, - const ipu3_uapi_grid_config &grid, - const ipu3_uapi_stats_3a *stats, - double gain); double estimateLuminance(double gain) override; void parseStatistics(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid); - uint64_t frameCount_; - utils::Duration minShutterSpeed_; utils::Duration maxShutterSpeed_; double minAnalogueGain_; double maxAnalogueGain_; - utils::Duration filteredExposure_; - uint32_t stride_; IPAContext *context_; std::vector reds_; From patchwork Fri Mar 22 13:14:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19794 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id AF684C32C5 for ; Fri, 22 Mar 2024 13:15:23 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BF3FF63362; Fri, 22 Mar 2024 14:15:21 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="WcjB1xek"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B16286307E for ; Fri, 22 Mar 2024 14:15:10 +0100 (CET) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C5FE18CC; Fri, 22 Mar 2024 14:14:41 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1711113282; bh=hHcYPOoPDFIRKQJ+tlevepPv4PcfaBaUHMkH9zmq1+0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WcjB1xekyu/8sv/+W7ZoKz9DQQmFXOBdO2KHzyhZ1ubFntn2Pgp4IBjc7WTXF/M6e Bk2waAo1hVt2WXm+0l/V8nbDdPxGnWRVTjCAvqLH3wS7yyxiglK61/AkYdkPtoT5EP 3pfetoF76sesBg2P6JzSav2gvnIux/vLsnar8l2A= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally Subject: [PATCH 08/10] ipa: rkisp1: Add parseStatistics() to Agc Date: Fri, 22 Mar 2024 13:14:49 +0000 Message-Id: <20240322131451.3092931-9-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240322131451.3092931-1-dan.scally@ideasonboard.com> References: <20240322131451.3092931-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In preparation for switching the RkISP1 Agc to a derivation of MeanLuminanceAgc, add a function that parses the statistics and stores the values for easy retrieval later. Signed-off-by: Daniel Scally --- src/ipa/rkisp1/algorithms/agc.cpp | 10 ++++++++++ src/ipa/rkisp1/algorithms/agc.h | 7 +++++++ 2 files changed, 17 insertions(+) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 47a6f7b2..d380194d 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -373,6 +373,16 @@ void Agc::fillMetadata(IPAContext &context, IPAFrameContext &frameContext, metadata.set(controls::FrameDuration, frameDuration.get()); } +void Agc::parseStatistics(const rkisp1_stat_buffer *stats, + IPAContext &context) +{ + const rkisp1_cif_isp_stat *params = &stats->params; + + expMeans_ = { params->ae.exp_mean, context.hw->numAeCells }; + hist_ = Histogram(Span(params->hist.hist_bins, + context.hw->numHistogramBins)); +} + /** * \brief Process RkISP1 statistics, and run AGC operations * \param[in] context The shared IPA context diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h index fb82a33f..b0de4898 100644 --- a/src/ipa/rkisp1/algorithms/agc.h +++ b/src/ipa/rkisp1/algorithms/agc.h @@ -14,6 +14,8 @@ #include +#include "libipa/histogram.h" + #include "algorithm.h" namespace libcamera { @@ -47,10 +49,15 @@ private: double measureBrightness(Span hist) const; void fillMetadata(IPAContext &context, IPAFrameContext &frameContext, ControlList &metadata); + void parseStatistics(const rkisp1_stat_buffer *stats, + IPAContext &context); uint64_t frameCount_; utils::Duration filteredExposure_; + + Histogram hist_; + Span expMeans_; }; } /* namespace ipa::rkisp1::algorithms */ From patchwork Fri Mar 22 13:14:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19795 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A3073C32C6 for ; Fri, 22 Mar 2024 13:15:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 51E1A63365; Fri, 22 Mar 2024 14:15:23 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="czVbsI9M"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3791A6331B for ; Fri, 22 Mar 2024 14:15:11 +0100 (CET) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 42AF0842; Fri, 22 Mar 2024 14:14:42 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1711113282; bh=MchdhWt7VO2UZ5A91x/o2dRbQJp5CMuOhmRxDyMDCNo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=czVbsI9MRmDwhyTI40r35SArHaeespK5UCnHhzNHV2ovGOC2OXJl6SBD++1CDkILA IU5Pc6PQwdCQ6s+C7hBogLi6adE/vw5LFTqnYfKI5vh44pQh1eY3zl9cJRSAPDT5Kh koofIjh7l5bhO3E0h7alnWzd2MyKeP4Ubj5N3Gz4= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally Subject: [PATCH 09/10] ipa: rkisp1: Derive rkisp1::algorithms::Agc from MeanLuminanceAgc Date: Fri, 22 Mar 2024 13:14:50 +0000 Message-Id: <20240322131451.3092931-10-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240322131451.3092931-1-dan.scally@ideasonboard.com> References: <20240322131451.3092931-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that we have a MeanLuminanceAgc class that centralises our AEGC algorithm, derive the RkISP1's Agc class from it and plumb in the necessary framework to enable it to be used. For simplicities sake this commit switches the algorithm to use the derived class, but does not remove the bespoke functions at this time. Signed-off-by: Daniel Scally --- src/ipa/rkisp1/algorithms/agc.cpp | 91 +++++++++++++++++++++++++++++-- src/ipa/rkisp1/algorithms/agc.h | 5 +- src/ipa/rkisp1/ipa_context.h | 5 ++ src/ipa/rkisp1/rkisp1.cpp | 2 +- 4 files changed, 96 insertions(+), 7 deletions(-) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index d380194d..3389c471 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -64,6 +64,36 @@ Agc::Agc() supportsRaw_ = true; } +/** + * \brief Initialise the AGC algorithm from tuning files + * + * \param[in] context The shared IPA context + * \param[in] tuningData The YamlObject containing Agc tuning data + * + * This function calls the base class' tuningData parsers to discover which + * control values are supported. + * + * \return 0 on success or errors from the base class + */ +int Agc::init(IPAContext &context, const YamlObject &tuningData) +{ + int ret; + + parseRelativeLuminanceTarget(tuningData); + + ret = parseConstraintModes(tuningData); + if (ret) + return ret; + + ret = parseExposureModes(tuningData); + if (ret) + return ret; + + context.ctrlMap.merge(controls()); + + return 0; +} + /** * \brief Configure the AGC given a configInfo * \param[in] context The shared IPA context @@ -81,6 +111,13 @@ int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) context.activeState.agc.manual.exposure = context.activeState.agc.automatic.exposure; context.activeState.agc.autoEnabled = !context.configuration.raw; + /* + * \todo We should use the first available mode rather than assume that + * the "Normal" modes are present in tuning data. + */ + context.activeState.agc.constraintMode = controls::ConstraintNormal; + context.activeState.agc.exposureMode = controls::ExposureNormal; + /* * Define the measurement window for AGC as a centered rectangle * covering 3/4 of the image width and height. @@ -95,6 +132,15 @@ int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) * frame index */ frameCount_ = 0; + + for (auto &[id, helper] : exposureModeHelpers()) { + /* \todo Run this again when FrameDurationLimits is passed in */ + helper->configure(context.configuration.sensor.minShutterSpeed, + context.configuration.sensor.maxShutterSpeed, + context.configuration.sensor.minAnalogueGain, + context.configuration.sensor.maxAnalogueGain); + } + return 0; } @@ -234,7 +280,6 @@ void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, double yGain, double iqMeanGain) { IPASessionConfiguration &configuration = context.configuration; - IPAActiveState &activeState = context.activeState; /* Get the effective exposure and gain applied on the sensor. */ uint32_t exposure = frameContext.sensor.exposure; @@ -300,10 +345,6 @@ void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, LOG(RkISP1Agc, Debug) << "Divided up shutter and gain are " << shutterTime << " and " << stepGain; - - /* Update the estimated exposure and gain. */ - activeState.agc.automatic.exposure = shutterTime / configuration.sensor.lineDuration; - activeState.agc.automatic.gain = stepGain; } /** @@ -383,6 +424,19 @@ void Agc::parseStatistics(const rkisp1_stat_buffer *stats, context.hw->numHistogramBins)); } +double Agc::estimateLuminance(double gain) +{ + double ySum = 0.0; + + /* Sum the averages, saturated to 255. */ + for (uint8_t expMean : expMeans_) + ySum += std::min(expMean * gain, 255.0); + + /* \todo Weight with the AWB gains */ + + return ySum / expMeans_.size() / 255; +} + /** * \brief Process RkISP1 statistics, and run AGC operations * \param[in] context The shared IPA context @@ -448,6 +502,33 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, computeExposure(context, frameContext, yGain, iqMeanGain); frameCount_++; + parseStatistics(stats, context); + + /* + * The Agc algorithm needs to know the effective exposure value that was + * applied to the sensor when the statistics were collected. + */ + utils::Duration exposureTime = context.configuration.sensor.lineDuration + * frameContext.sensor.exposure; + double analogueGain = frameContext.sensor.gain; + utils::Duration effectiveExposureValue = exposureTime * analogueGain; + + utils::Duration shutterTime; + double aGain, dGain; + std::tie(shutterTime, aGain, dGain) = + calculateNewEv(context.activeState.agc.constraintMode, + context.activeState.agc.exposureMode, hist_, + effectiveExposureValue); + + LOG(RkISP1Agc, Debug) + << "Divided up shutter, analogue gain and digital gain are " + << shutterTime << ", " << aGain << " and " << dGain; + + IPAActiveState &activeState = context.activeState; + /* Update the estimated exposure and gain. */ + activeState.agc.automatic.exposure = shutterTime / context.configuration.sensor.lineDuration; + activeState.agc.automatic.gain = aGain; + fillMetadata(context, frameContext, metadata); } diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h index b0de4898..1271741e 100644 --- a/src/ipa/rkisp1/algorithms/agc.h +++ b/src/ipa/rkisp1/algorithms/agc.h @@ -14,6 +14,7 @@ #include +#include "libipa/agc.h" #include "libipa/histogram.h" #include "algorithm.h" @@ -22,12 +23,13 @@ namespace libcamera { namespace ipa::rkisp1::algorithms { -class Agc : public Algorithm +class Agc : public Algorithm, public MeanLuminanceAgc { public: Agc(); ~Agc() = default; + int init(IPAContext &context, const YamlObject &tuningData) override; int configure(IPAContext &context, const IPACameraSensorInfo &configInfo) override; void queueRequest(IPAContext &context, const uint32_t frame, @@ -51,6 +53,7 @@ private: ControlList &metadata); void parseStatistics(const rkisp1_stat_buffer *stats, IPAContext &context); + double estimateLuminance(double gain) override; uint64_t frameCount_; diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index 10d8f38c..256b75eb 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -12,6 +12,7 @@ #include +#include #include #include @@ -67,6 +68,8 @@ struct IPAActiveState { } automatic; bool autoEnabled; + uint32_t constraintMode; + uint32_t exposureMode; } agc; struct { @@ -151,6 +154,8 @@ struct IPAContext { IPAActiveState activeState; FCQueue frameContexts; + + ControlInfoMap::Map ctrlMap; }; } /* namespace ipa::rkisp1 */ diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 9dc5f53c..5f74762f 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -119,7 +119,7 @@ const ControlInfoMap::Map rkisp1Controls{ } /* namespace */ IPARkISP1::IPARkISP1() - : context_({ {}, {}, {}, { kMaxFrameContexts } }) + : context_({ {}, {}, {}, { kMaxFrameContexts }, {} }) { } From patchwork Fri Mar 22 13:14:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 19796 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 29C3AC32C3 for ; Fri, 22 Mar 2024 13:15:25 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 76A6F63366; Fri, 22 Mar 2024 14:15:24 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="iWo+TmaH"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A038562826 for ; Fri, 22 Mar 2024 14:15:11 +0100 (CET) Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id B192C8CC; Fri, 22 Mar 2024 14:14:42 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1711113282; bh=NaZogK4c4M+TDkMR+cp0HMtMQ7yJX+85J2p9vUG4Gps=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iWo+TmaHfletIN4kMLPYNCRAqGveZ1UKfCOInC8ptJeo0QmJxrl4dkmZ4awVsP/uF q57eQN4RdfnfOny3Y1d/nHAYoElT08/3ujaIIiw+0/RHsQRjtbP3rAl9x8bmLvwKte yMvu/M0r4Ithye3ppPAkB5dPVsWdUJu6s8wb8epU= From: Daniel Scally To: libcamera-devel@lists.libcamera.org Cc: Daniel Scally Subject: [PATCH 10/10] ipa: rkisp1: Remove bespoke Agc functions Date: Fri, 22 Mar 2024 13:14:51 +0000 Message-Id: <20240322131451.3092931-11-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240322131451.3092931-1-dan.scally@ideasonboard.com> References: <20240322131451.3092931-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that the rkisp1 Agc algorithm is a derivation of MeanLuminanceAgc we can remove the bespoke functions from the IPA's class. Signed-off-by: Daniel Scally Reviewed-by: Stefan Klug --- src/ipa/rkisp1/algorithms/agc.cpp | 222 ------------------------------ src/ipa/rkisp1/algorithms/agc.h | 9 -- 2 files changed, 231 deletions(-) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 3389c471..5e6a8ba0 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -42,24 +42,7 @@ static constexpr double kMinAnalogueGain = 1.0; /* \todo Honour the FrameDurationLimits control instead of hardcoding a limit */ static constexpr utils::Duration kMaxShutterSpeed = 60ms; -/* Number of frames to wait before calculating stats on minimum exposure */ -static constexpr uint32_t kNumStartupFrames = 10; - -/* Target value to reach for the top 2% of the histogram */ -static constexpr double kEvGainTarget = 0.5; - -/* - * Relative luminance target. - * - * It's a number that's chosen so that, when the camera points at a grey - * target, the resulting image brightness is considered right. - * - * \todo Why is the value different between IPU3 and RkISP1 ? - */ -static constexpr double kRelativeLuminanceTarget = 0.4; - Agc::Agc() - : frameCount_(0), filteredExposure_(0s) { supportsRaw_ = true; } @@ -127,12 +110,6 @@ int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) context.configuration.agc.measureWindow.h_size = 3 * configInfo.outputSize.width / 4; context.configuration.agc.measureWindow.v_size = 3 * configInfo.outputSize.height / 4; - /* - * \todo Use the upcoming per-frame context API that will provide a - * frame index - */ - frameCount_ = 0; - for (auto &[id, helper] : exposureModeHelpers()) { /* \todo Run this again when FrameDurationLimits is passed in */ helper->configure(context.configuration.sensor.minShutterSpeed, @@ -234,170 +211,6 @@ void Agc::prepare(IPAContext &context, const uint32_t frame, params->module_en_update |= RKISP1_CIF_ISP_MODULE_HST; } -/** - * \brief Apply a filter on the exposure value to limit the speed of changes - * \param[in] exposureValue The target exposure from the AGC algorithm - * - * The speed of the filter is adaptive, and will produce the target quicker - * during startup, or when the target exposure is within 20% of the most recent - * filter output. - * - * \return The filtered exposure - */ -utils::Duration Agc::filterExposure(utils::Duration exposureValue) -{ - double speed = 0.2; - - /* Adapt instantly if we are in startup phase. */ - if (frameCount_ < kNumStartupFrames) - speed = 1.0; - - /* - * If we are close to the desired result, go faster to avoid making - * multiple micro-adjustments. - * \todo Make this customisable? - */ - if (filteredExposure_ < 1.2 * exposureValue && - filteredExposure_ > 0.8 * exposureValue) - speed = sqrt(speed); - - filteredExposure_ = speed * exposureValue + - filteredExposure_ * (1.0 - speed); - - LOG(RkISP1Agc, Debug) << "After filtering, exposure " << filteredExposure_; - - return filteredExposure_; -} - -/** - * \brief Estimate the new exposure and gain values - * \param[inout] context The shared IPA Context - * \param[in] frameContext The FrameContext for this frame - * \param[in] yGain The gain calculated on the current brightness level - * \param[in] iqMeanGain The gain calculated based on the relative luminance target - */ -void Agc::computeExposure(IPAContext &context, IPAFrameContext &frameContext, - double yGain, double iqMeanGain) -{ - IPASessionConfiguration &configuration = context.configuration; - - /* Get the effective exposure and gain applied on the sensor. */ - uint32_t exposure = frameContext.sensor.exposure; - double analogueGain = frameContext.sensor.gain; - - /* Use the highest of the two gain estimates. */ - double evGain = std::max(yGain, iqMeanGain); - - utils::Duration minShutterSpeed = configuration.sensor.minShutterSpeed; - utils::Duration maxShutterSpeed = std::min(configuration.sensor.maxShutterSpeed, - kMaxShutterSpeed); - - double minAnalogueGain = std::max(configuration.sensor.minAnalogueGain, - kMinAnalogueGain); - double maxAnalogueGain = configuration.sensor.maxAnalogueGain; - - /* Consider within 1% of the target as correctly exposed. */ - if (utils::abs_diff(evGain, 1.0) < 0.01) - return; - - /* extracted from Rpi::Agc::computeTargetExposure. */ - - /* Calculate the shutter time in seconds. */ - utils::Duration currentShutter = exposure * configuration.sensor.lineDuration; - - /* - * Update the exposure value for the next computation using the values - * of exposure and gain really used by the sensor. - */ - utils::Duration effectiveExposureValue = currentShutter * analogueGain; - - LOG(RkISP1Agc, Debug) << "Actual total exposure " << currentShutter * analogueGain - << " Shutter speed " << currentShutter - << " Gain " << analogueGain - << " Needed ev gain " << evGain; - - /* - * Calculate the current exposure value for the scene as the latest - * exposure value applied multiplied by the new estimated gain. - */ - utils::Duration exposureValue = effectiveExposureValue * evGain; - - /* Clamp the exposure value to the min and max authorized. */ - utils::Duration maxTotalExposure = maxShutterSpeed * maxAnalogueGain; - exposureValue = std::min(exposureValue, maxTotalExposure); - LOG(RkISP1Agc, Debug) << "Target total exposure " << exposureValue - << ", maximum is " << maxTotalExposure; - - /* - * Divide the exposure value as new exposure and gain values. - * \todo estimate if we need to desaturate - */ - exposureValue = filterExposure(exposureValue); - - /* - * Push the shutter time up to the maximum first, and only then - * increase the gain. - */ - utils::Duration shutterTime = std::clamp(exposureValue / minAnalogueGain, - minShutterSpeed, maxShutterSpeed); - double stepGain = std::clamp(exposureValue / shutterTime, - minAnalogueGain, maxAnalogueGain); - LOG(RkISP1Agc, Debug) << "Divided up shutter and gain are " - << shutterTime << " and " - << stepGain; -} - -/** - * \brief Estimate the relative luminance of the frame with a given gain - * \param[in] expMeans The mean luminance values, from the RkISP1 statistics - * \param[in] gain The gain to apply to the frame - * - * This function estimates the average relative luminance of the frame that - * would be output by the sensor if an additional \a gain was applied. - * - * The estimation is based on the AE statistics for the current frame. Y - * averages for all cells are first multiplied by the gain, and then saturated - * to approximate the sensor behaviour at high brightness values. The - * approximation is quite rough, as it doesn't take into account non-linearities - * when approaching saturation. In this case, saturating after the conversion to - * YUV doesn't take into account the fact that the R, G and B components - * contribute differently to the relative luminance. - * - * \todo Have a dedicated YUV algorithm ? - * - * The values are normalized to the [0.0, 1.0] range, where 1.0 corresponds to a - * theoretical perfect reflector of 100% reference white. - * - * More detailed information can be found in: - * https://en.wikipedia.org/wiki/Relative_luminance - * - * \return The relative luminance - */ -double Agc::estimateLuminance(Span expMeans, double gain) -{ - double ySum = 0.0; - - /* Sum the averages, saturated to 255. */ - for (uint8_t expMean : expMeans) - ySum += std::min(expMean * gain, 255.0); - - /* \todo Weight with the AWB gains */ - - return ySum / expMeans.size() / 255; -} - -/** - * \brief Estimate the mean value of the top 2% of the histogram - * \param[in] hist The histogram statistics computed by the RkISP1 - * \return The mean value of the top 2% of the histogram - */ -double Agc::measureBrightness(Span hist) const -{ - Histogram histogram{ hist }; - /* Estimate the quantile mean of the top 2% of the histogram. */ - return histogram.interQuantileMean(0.98, 1.0); -} - void Agc::fillMetadata(IPAContext &context, IPAFrameContext &frameContext, ControlList &metadata) { @@ -465,43 +278,8 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, * we receive), but is important in manual mode. */ - const rkisp1_cif_isp_stat *params = &stats->params; ASSERT(stats->meas_type & RKISP1_CIF_ISP_STAT_AUTOEXP); - Span ae{ params->ae.exp_mean, context.hw->numAeCells }; - Span hist{ - params->hist.hist_bins, - context.hw->numHistogramBins - }; - - double iqMean = measureBrightness(hist); - double iqMeanGain = kEvGainTarget * hist.size() / iqMean; - - /* - * Estimate the gain needed to achieve a relative luminance target. To - * account for non-linearity caused by saturation, the value needs to be - * estimated in an iterative process, as multiplying by a gain will not - * increase the relative luminance by the same factor if some image - * regions are saturated. - */ - double yGain = 1.0; - double yTarget = kRelativeLuminanceTarget; - - for (unsigned int i = 0; i < 8; i++) { - double yValue = estimateLuminance(ae, yGain); - double extra_gain = std::min(10.0, yTarget / (yValue + .001)); - - yGain *= extra_gain; - LOG(RkISP1Agc, Debug) << "Y value: " << yValue - << ", Y target: " << yTarget - << ", gives gain " << yGain; - if (extra_gain < 1.01) - break; - } - - computeExposure(context, frameContext, yGain, iqMeanGain); - frameCount_++; - parseStatistics(stats, context); /* diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h index 1271741e..311d4e94 100644 --- a/src/ipa/rkisp1/algorithms/agc.h +++ b/src/ipa/rkisp1/algorithms/agc.h @@ -44,21 +44,12 @@ public: ControlList &metadata) override; private: - void computeExposure(IPAContext &Context, IPAFrameContext &frameContext, - double yGain, double iqMeanGain); - utils::Duration filterExposure(utils::Duration exposureValue); - double estimateLuminance(Span expMeans, double gain); - double measureBrightness(Span hist) const; void fillMetadata(IPAContext &context, IPAFrameContext &frameContext, ControlList &metadata); void parseStatistics(const rkisp1_stat_buffer *stats, IPAContext &context); double estimateLuminance(double gain) override; - uint64_t frameCount_; - - utils::Duration filteredExposure_; - Histogram hist_; Span expMeans_; };