[{"id":29250,"web_url":"https://patchwork.libcamera.org/comment/29250/","msgid":"<20240418074823.l2k273uo5fx2nmwp@jasper>","date":"2024-04-18T07:48:23","subject":"Re: [PATCH v2 4/8] ipa: libipa: Add MeanLuminanceAgc base class","submitter":{"id":184,"url":"https://patchwork.libcamera.org/api/people/184/","name":"Stefan Klug","email":"stefan.klug@ideasonboard.com"},"content":"Hi Dan,\n\nthank you for the patch.\n\nOn Wed, Apr 17, 2024 at 02:15:32PM +0100, Daniel Scally wrote:\n> The Agc algorithms for the RkIsp1 and IPU3 IPAs do the same thing in\n> very large part; following the Rpi IPA's algorithm in spirit with a\n> few tunable values in that IPA being hardcoded in the libipa ones.\n> Add a new base class for MeanLuminanceAgc which implements the same\n> algorithm and additionally parses yaml tuning files to inform an IPA\n> module's Agc algorithm about valid constraint and exposure modes and\n> their associated bounds.\n> \n> Signed-off-by: Daniel Scally \n> ---\n> Changes in v2:\n> \n> \t- Renamed the class and files\n> \t- Expanded the documentation\n> \t- Added parseTuningData() so derived classes can call a single function\n> \t to cover all the parsing in ::init().\n> \n> src/ipa/libipa/agc_mean_luminance.cpp | 581 ++++++++++++++++++++++++++\n> src/ipa/libipa/agc_mean_luminance.h | 91 ++++\n> src/ipa/libipa/meson.build | 2 +\n> 3 files changed, 674 insertions(+)\n> create mode 100644 src/ipa/libipa/agc_mean_luminance.cpp\n> create mode 100644 src/ipa/libipa/agc_mean_luminance.h\n> \n> diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp\n> new file mode 100644\n> index 00000000..02e223cf\n> --- /dev/null\n> +++ b/src/ipa/libipa/agc_mean_luminance.cpp\n> @@ -0,0 +1,581 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2024 Ideas on Board Oy\n> + *\n> + * agc_mean_luminance.cpp - Base class for mean luminance AGC algorithms\n> + */\n> +\n> +#include \"agc_mean_luminance.h\"\n> +\n> +#include \n> +\n> +#include \n> +#include \n> +\n> +#include \"exposure_mode_helper.h\"\n> +\n> +using namespace libcamera::controls;\n> +\n> +/**\n> + * \\file agc_mean_luminance.h\n> + * \\brief Base class implementing mean luminance AEGC\n> + */\n> +\n> +namespace libcamera {\n> +\n> +using namespace std::literals::chrono_literals;\n> +\n> +LOG_DEFINE_CATEGORY(AgcMeanLuminance)\n> +\n> +namespace ipa {\n> +\n> +/*\n> + * Number of frames for which to run the algorithm at full speed, before slowing\n> + * down to prevent large and jarring changes in exposure from frame to frame.\n> + */\n> +static constexpr uint32_t kNumStartupFrames = 10;\n> +\n> +/*\n> + * Default relative luminance target\n> + *\n> + * This value should be chosen so that when the camera points at a grey target,\n> + * the resulting image brightness looks \"right\". Custom values can be passed\n> + * as the relativeLuminanceTarget value in sensor tuning files.\n> + */\n> +static constexpr double kDefaultRelativeLuminanceTarget = 0.16;\n> +\n> +/**\n> + * \\struct AgcMeanLuminance::AgcConstraint\n> + * \\brief The boundaries and target for an AeConstraintMode constraint\n> + *\n> + * This structure describes an AeConstraintMode constraint for the purposes of\n> + * this algorithm. The algorithm will apply the constraints by calculating the\n> + * Histogram's inter-quantile mean between the given quantiles and ensure that\n> + * the resulting value is the right side of the given target (as defined by the\n> + * boundary and luminance target).\n> + */\n> +\n> +/**\n> + * \\enum AgcMeanLuminance::AgcConstraint::Bound\n> + * \\brief Specify whether the constraint defines a lower or upper bound\n> + * \\var AgcMeanLuminance::AgcConstraint::lower\n> + * \\brief The constraint defines a lower bound\n> + * \\var AgcMeanLuminance::AgcConstraint::upper\n> + * \\brief The constraint defines an upper bound\n> + */\n> +\n> +/**\n> + * \\var AgcMeanLuminance::AgcConstraint::bound\n> + * \\brief The type of constraint bound\n> + */\n> +\n> +/**\n> + * \\var AgcMeanLuminance::AgcConstraint::qLo\n> + * \\brief The lower quantile to use for the constraint\n> + */\n> +\n> +/**\n> + * \\var AgcMeanLuminance::AgcConstraint::qHi\n> + * \\brief The upper quantile to use for the constraint\n> + */\n> +\n> +/**\n> + * \\var AgcMeanLuminance::AgcConstraint::yTarget\n> + * \\brief The luminance target for the constraint\n> + */\n> +\n> +/**\n> + * \\class AgcMeanLuminance\n> + * \\brief A mean-based auto-exposure algorithm\n> + *\n> + * This algorithm calculates a shutter time, analogue and digital gain such that\n> + * the normalised mean luminance value of an image is driven towards a target,\n> + * which itself is discovered from tuning data. The algorithm is a two-stage\n> + * process.\n> + *\n> + * In the first stage, an initial gain value is derived by iteratively comparing\n> + * the gain-adjusted mean luminance across an entire image against a target, and\n> + * selecting a value which pushes it as closely as possible towards the target.\n> + *\n> + * In the second stage we calculate the gain required to drive the average of a\n> + * section of a histogram to a target value, where the target and the boundaries\n> + * of the section of the histogram used in the calculation are taken from the\n> + * values defined for the currently configured AeConstraintMode within the\n> + * tuning data. This class provides a helper function to parse those tuning data\n> + * to discover the constraints, and so requires a specific format for those\n> + * data which is described in \\ref parseTuningData(). The gain from the first\n> + * stage is then clamped to the gain from this stage.\n> + *\n> + * The final gain is used to adjust the effective exposure value of the image,\n> + * and that new exposure value is divided into shutter time, analogue gain and\n> + * digital gain according to the selected AeExposureMode. This class expects to\n> + * use the \\ref ExposureModeHelper class to assist in that division, and expects\n> + * the data needed to initialise that class to be present in tuning data in a\n> + * format described in \\ref parseTuningData().\n> + *\n> + * In order to be able to derive an AGC implementation from this class, an IPA\n> + * needs to be able to do the following:\n> + *\n> + * 1. Provide a luminance estimation across an entire image.\n> + * 2. Provide a luminance Histogram for the image to use in calculating\n> + * constraint compliance. The precision of the Histogram that is available\n> + * will determine the supportable precision of the constraints.\n> + */\n> +\n> +AgcMeanLuminance::AgcMeanLuminance()\n> +\t: frameCount_(0), filteredExposure_(0s), relativeLuminanceTarget_(0)\n> +{\n> +}\n> +\n> +/**\n> + * \\brief Parse the relative luminance target from the tuning data\n> + * \\param[in] tuningData The YamlObject holding the algorithm's tuning data\n> + */\n> +void AgcMeanLuminance::parseRelativeLuminanceTarget(const YamlObject &tuningData)\n> +{\n> +\trelativeLuminanceTarget_ =\n> +\t\ttuningData[\"relativeLuminanceTarget\"].get(kDefaultRelativeLuminanceTarget);\n> +}\n> +\n> +/**\n> + * \\brief Parse an AeConstraintMode constraint from tuning data\n> + * \\param[in] modeDict the YamlObject holding the constraint data\n> + * \\param[in] id The constraint ID from AeConstraintModeEnum\n> + */\n> +void AgcMeanLuminance::parseConstraint(const YamlObject &modeDict, int32_t id)\n> +{\n> +\tfor (const auto &[boundName, content] : modeDict.asDict()) {\n> +\t\tif (boundName != \"upper\" && boundName != \"lower\") {\n> +\t\t\tLOG(AgcMeanLuminance, Warning)\n> +\t\t\t\t<< \"Ignoring unknown constraint bound '\" << boundName << \"'\";\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tunsigned int idx = static_cast(boundName == \"upper\");\n> +\t\tAgcConstraint::Bound bound = static_cast(idx);\n> +\t\tdouble qLo = content[\"qLo\"].get().value_or(0.98);\n> +\t\tdouble qHi = content[\"qHi\"].get().value_or(1.0);\n> +\t\tdouble yTarget =\n> +\t\t\tcontent[\"yTarget\"].getList().value_or(std::vector{ 0.5 }).at(0);\n> +\n> +\t\tAgcConstraint constraint = { bound, qLo, qHi, yTarget };\n> +\n> +\t\tif (!constraintModes_.count(id))\n> +\t\t\tconstraintModes_[id] = {};\n> +\n> +\t\tif (idx)\n> +\t\t\tconstraintModes_[id].push_back(constraint);\n> +\t\telse\n> +\t\t\tconstraintModes_[id].insert(constraintModes_[id].begin(), constraint);\n> +\t}\n> +}\n> +\n> +int AgcMeanLuminance::parseConstraintModes(const YamlObject &tuningData)\n> +{\n> +\tstd::vector availableConstraintModes;\n> +\n> +\tconst YamlObject &yamlConstraintModes = tuningData[controls::AeConstraintMode.name()];\n> +\tif (yamlConstraintModes.isDictionary()) {\n> +\t\tfor (const auto &[modeName, modeDict] : yamlConstraintModes.asDict()) {\n> +\t\t\tif (AeConstraintModeNameValueMap.find(modeName) ==\n> +\t\t\t AeConstraintModeNameValueMap.end()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Warning)\n> +\t\t\t\t\t<< \"Skipping unknown constraint mode '\" << modeName << \"'\";\n> +\t\t\t\tcontinue;\n> +\t\t\t}\n> +\n> +\t\t\tif (!modeDict.isDictionary()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> +\t\t\t\t\t<< \"Invalid constraint mode '\" << modeName << \"'\";\n> +\t\t\t\treturn -EINVAL;\n> +\t\t\t}\n> +\n> +\t\t\tparseConstraint(modeDict,\n> +\t\t\t\t\tAeConstraintModeNameValueMap.at(modeName));\n> +\t\t\tavailableConstraintModes.push_back(\n> +\t\t\t\tAeConstraintModeNameValueMap.at(modeName));\n> +\t\t}\n> +\t}\n> +\n> +\t/*\n> +\t * If the tuning data file contains no constraints then we use the\n> +\t * default constraint that the various Agc algorithms were adhering to\n> +\t * anyway before centralisation.\n> +\t */\n> +\tif (constraintModes_.empty()) {\n> +\t\tAgcConstraint constraint = {\n> +\t\t\tAgcConstraint::Bound::lower,\n> +\t\t\t0.98,\n> +\t\t\t1.0,\n> +\t\t\t0.5\n> +\t\t};\n> +\n> +\t\tconstraintModes_[controls::ConstraintNormal].insert(\n> +\t\t\tconstraintModes_[controls::ConstraintNormal].begin(),\n> +\t\t\tconstraint);\n> +\t\tavailableConstraintModes.push_back(\n> +\t\t\tAeConstraintModeNameValueMap.at(\"ConstraintNormal\"));\n> +\t}\n> +\n> +\tcontrols_[&controls::AeConstraintMode] = ControlInfo(availableConstraintModes);\n> +\n> +\treturn 0;\n> +}\n> +\n> +int AgcMeanLuminance::parseExposureModes(const YamlObject &tuningData)\n> +{\n> +\tstd::vector availableExposureModes;\n> +\n> +\tconst YamlObject &yamlExposureModes = tuningData[controls::AeExposureMode.name()];\n> +\tif (yamlExposureModes.isDictionary()) {\n> +\t\tfor (const auto &[modeName, modeValues] : yamlExposureModes.asDict()) {\n> +\t\t\tif (AeExposureModeNameValueMap.find(modeName) ==\n> +\t\t\t AeExposureModeNameValueMap.end()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Warning)\n> +\t\t\t\t\t<< \"Skipping unknown exposure mode '\" << modeName << \"'\";\n> +\t\t\t\tcontinue;\n> +\t\t\t}\n> +\n> +\t\t\tif (!modeValues.isDictionary()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> +\t\t\t\t\t<< \"Invalid exposure mode '\" << modeName << \"'\";\n> +\t\t\t\treturn -EINVAL;\n> +\t\t\t}\n> +\n> +\t\t\tstd::vector shutters =\n> +\t\t\t\tmodeValues[\"shutter\"].getList().value_or(std::vector{});\n> +\t\t\tstd::vector gains =\n> +\t\t\t\tmodeValues[\"gain\"].getList().value_or(std::vector{});\n> +\n> +\t\t\tif (shutters.size() != gains.size()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> +\t\t\t\t\t<< \"Shutter and gain array sizes unequal\";\n> +\t\t\t\treturn -EINVAL;\n> +\t\t\t}\n> +\n> +\t\t\tif (shutters.empty()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> +\t\t\t\t\t<< \"Shutter and gain arrays are empty\";\n> +\t\t\t\treturn -EINVAL;\n> +\t\t\t}\n> +\n> +\t\t\tstd::vector> stages;\n> +\t\t\tfor (unsigned int i = 0; i < shutters.size(); i++) {\n> +\t\t\t\tstages.push_back({\n> +\t\t\t\t\tstd::chrono::microseconds(shutters[i]),\n> +\t\t\t\t\tgains[i]\n> +\t\t\t\t});\n> +\t\t\t}\n> +\n> +\t\t\tstd::shared_ptr helper =\n> +\t\t\t\tstd::make_shared();\n> +\t\t\thelper->init(stages);\n> +\n> +\t\t\texposureModeHelpers_[AeExposureModeNameValueMap.at(modeName)] = helper;\n> +\t\t\tavailableExposureModes.push_back(AeExposureModeNameValueMap.at(modeName));\n> +\t\t}\n> +\t}\n> +\n> +\t/*\n> +\t * If we don't have any exposure modes in the tuning data we create an\n> +\t * ExposureModeHelper using an empty vector of stages. This will result\n> +\t * in the ExposureModeHelper simply driving the shutter as high as\n> +\t * possible before touching gain.\n> +\t */\n> +\tif (availableExposureModes.empty()) {\n> +\t\tint32_t exposureModeId = AeExposureModeNameValueMap.at(\"ExposureNormal\");\n> +\t\tstd::vector> stages = { };\n> +\n> +\t\tstd::shared_ptr helper =\n> +\t\t\tstd::make_shared();\n> +\t\thelper->init(stages);\n> +\n> +\t\texposureModeHelpers_[exposureModeId] = helper;\n> +\t\tavailableExposureModes.push_back(exposureModeId);\n> +\t}\n> +\n> +\tcontrols_[&controls::AeExposureMode] = ControlInfo(availableExposureModes);\n> +\n> +\treturn 0;\n> +}\n> +\n> +/**\n> + * \\brief Parse tuning data for AeConstraintMode and AeExposureMode controls\n> + * \\param[in] tuningData the YamlObject representing the tuning data\n> + *\n> + * This function parses tuning data to build the list of allowed values for the\n> + * AeConstraintMode and AeExposureMode controls. Those tuning data must provide\n> + * the data in a specific format; the Agc algorithm's tuning data should contain\n> + * a dictionary called AeConstraintMode containing per-mode setting dictionaries\n> + * with the key being a value from \\ref controls::AeConstraintModeNameValueMap.\n> + * Each mode dict may contain either a \"lower\" or \"upper\" key or both, for\n> + * example:\n> + *\n> + * \\code{.unparsed}\n> + * algorithms:\n> + * - Agc:\n> + * AeConstraintMode:\n> + * ConstraintNormal:\n> + * lower:\n> + * qLo: 0.98\n> + * qHi: 1.0\n> + * yTarget: 0.5\n> + * ConstraintHighlight:\n> + * lower:\n> + * qLo: 0.98\n> + * qHi: 1.0\n> + * yTarget: 0.5\n> + * upper:\n> + * qLo: 0.98\n> + * qHi: 1.0\n> + * yTarget: 0.8\n> + *\n> + * \\endcode\n> + *\n> + * For the AeExposureMode control the data should contain a dictionary called\n> + * AeExposureMode containing per-mode setting dictionaries with the key being a\n> + * value from \\ref controls::AeExposureModeNameValueMap. Each mode dict should\n> + * contain an array of shutter times with the key \"shutter\" and an array of gain\n> + * values with the key \"gain\", in this format:\n> + *\n> + * \\code{.unparsed}\n> + * algorithms:\n> + * - Agc:\n> + * AeExposureMode:\n> + * ExposureNormal:\n> + * shutter: [ 100, 10000, 30000, 60000, 120000 ]\n> + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ]\n> + * ExposureShort:\n> + * shutter: [ 100, 10000, 30000, 60000, 120000 ]\n> + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ]\n> + *\n> + * \\endcode\n> + *\n> + * @return 0 on success or a negative error code\n> + */\n> +int AgcMeanLuminance::parseTuningData(const YamlObject &tuningData)\n> +{\n> +\tint ret;\n> +\n> +\tparseRelativeLuminanceTarget(tuningData);\n> +\n> +\tret = parseConstraintModes(tuningData);\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n> +\tret = parseExposureModes(tuningData);\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n> +\treturn 0;\n> +}\n> +\n> +/**\n> + * \\brief configure the ExposureModeHelpers for this class\n> + * \\param[in] minShutter Minimum shutter time to allow\n> + * \\param[in] maxShutter Maximum shutter time to allow\n> + * \\param[in] minGain Minimum gain to allow\n> + * \\param[in] maxGain Maximum gain to allow\n> + *\n> + * This function calls \\ref ExposureModeHelper::setShutterGainLimits() for each\n> + * ExposureModeHelper that has been created for this class.\n> + */\n> +void AgcMeanLuminance::configureExposureModeHelpers(utils::Duration minShutter,\n> +\t\t\t\t\t\t utils::Duration maxShutter,\n> +\t\t\t\t\t\t double minGain,\n> +\t\t\t\t\t\t double maxGain)\n> +{\n> +\tfor (auto &[id, helper] : exposureModeHelpers_)\n> +\t\thelper->setShutterGainLimits(minShutter, maxShutter, minGain, maxGain);\n> +}\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::constraintModes()\n> + * \\brief Get the constraint modes that have been parsed from tuning data\n> + */\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::exposureModeHelpers()\n> + * \\brief Get the ExposureModeHelpers that have been parsed from tuning data\n> + */\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::controls()\n> + * \\brief Get the controls that have been generated after parsing tuning data\n> + */\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::estimateLuminance(const double gain)\n> + * \\brief Estimate the luminance of an image, adjusted by a given gain\n> + * \\param[in] gain The gain with which to adjust the luminance estimate\n> + *\n> + * This function estimates the average relative luminance of the frame that\n> + * would be output by the sensor if an additional \\a gain was applied. It is a\n> + * pure virtual function because estimation of luminance is a hardware-specific\n> + * operation, which depends wholly on the format of the stats that are delivered\n> + * to libcamera from the ISP. Derived classes must implement an overriding\n> + * function that calculates the normalised mean luminance value across the\n> + * entire image.\n> + *\n> + * \\return The normalised relative luminance of the image\n> + */\n> +\n> +/**\n> + * \\brief Estimate the initial gain needed to achieve a relative luminance\n> + * target\n> + *\n> + * To account for non-linearity caused by saturation, the value needs to be\n> + * estimated in an iterative process, as multiplying by a gain will not increase\n> + * the relative luminance by the same factor if some image regions are saturated\n> + *\n> + * \\return The calculated initial gain\n> + */\n> +double AgcMeanLuminance::estimateInitialGain()\n> +{\n> +\tdouble yTarget = relativeLuminanceTarget_;\n> +\tdouble yGain = 1.0;\n> +\n> +\tfor (unsigned int i = 0; i < 8; i++) {\n> +\t\tdouble yValue = estimateLuminance(yGain);\n> +\t\tdouble extra_gain = std::min(10.0, yTarget / (yValue + .001));\n> +\n> +\t\tyGain *= extra_gain;\n> +\t\tLOG(AgcMeanLuminance, Debug) << \"Y value: \" << yValue\n> +\t\t\t\t<< \", Y target: \" << yTarget\n> +\t\t\t\t<< \", gives gain \" << yGain;\n> +\n> +\t\tif (utils::abs_diff(extra_gain, 1.0) < 0.01)\n> +\t\t\tbreak;\n> +\t}\n> +\n> +\treturn yGain;\n> +}\n> +\n> +/**\n> + * \\brief Clamp gain within the bounds of a defined constraint\n> + * \\param[in] constraintModeIndex The index of the constraint to adhere to\n> + * \\param[in] hist A histogram over which to calculate inter-quantile means\n> + * \\param[in] gain The gain to clamp\n> + *\n> + * \\return The gain clamped within the constraint bounds\n> + */\n> +double AgcMeanLuminance::constraintClampGain(uint32_t constraintModeIndex,\n> +\t\t\t\t\t const Histogram &hist,\n> +\t\t\t\t\t double gain)\n> +{\n> +\tstd::vector &constraints = constraintModes_[constraintModeIndex];\n> +\tfor (const AgcConstraint &constraint : constraints) {\n> +\t\tdouble newGain = constraint.yTarget * hist.bins() /\n> +\t\t\t\t hist.interQuantileMean(constraint.qLo, constraint.qHi);\n> +\n> +\t\tif (constraint.bound == AgcConstraint::Bound::lower &&\n> +\t\t newGain > gain)\n> +\t\t\tgain = newGain;\n> +\n> +\t\tif (constraint.bound == AgcConstraint::Bound::upper &&\n> +\t\t newGain < gain)\n> +\t\t\tgain = newGain;\n> +\t}\n> +\n> +\treturn gain;\n> +}\n> +\n> +/**\n> + * \\brief Apply a filter on the exposure value to limit the speed of changes\n> + * \\param[in] exposureValue The target exposure from the AGC algorithm\n> + *\n> + * The speed of the filter is adaptive, and will produce the target quicker\n> + * during startup, or when the target exposure is within 20% of the most recent\n> + * filter output.\n> + *\n> + * \\return The filtered exposure\n> + */\n> +utils::Duration AgcMeanLuminance::filterExposure(utils::Duration exposureValue)\n> +{\n> +\tdouble speed = 0.2;\n> +\n> +\t/* Adapt instantly if we are in startup phase. */\n> +\tif (frameCount_ < kNumStartupFrames)\n> +\t\tspeed = 1.0;\n> +\n> +\t/*\n> +\t * If we are close to the desired result, go faster to avoid making\n> +\t * multiple micro-adjustments.\n> +\t * \\todo Make this customisable?\n> +\t */\n> +\tif (filteredExposure_ < 1.2 * exposureValue &&\n> +\t filteredExposure_ > 0.8 * exposureValue)\n> +\t\tspeed = sqrt(speed);\n> +\n> +\tfilteredExposure_ = speed * exposureValue +\n> +\t\t\t filteredExposure_ * (1.0 - speed);\n> +\n> +\treturn filteredExposure_;\n> +}\n> +\n> +/**\n> + * \\brief Calculate the new exposure value\n> + * \\param[in] constraintModeIndex The index of the current constraint mode\n> + * \\param[in] exposureModeIndex The index of the current exposure mode\n> + * \\param[in] yHist A Histogram from the ISP statistics to use in constraining\n> + *\t the calculated gain\n\nnit: no indentation\n\n> + * \\param[in] effectiveExposureValue The EV applied to the frame from which the\n> + *\t statistics in use derive\n\nnit: no indentation\n\n> + *\n> + * Calculate a new exposure value to try to obtain the target. The calculated\n> + * exposure value is filtered to prevent rapid changes from frame to frame, and\n> + * divided into shutter time, analogue and digital gain.\n> + *\n> + * \\return Tuple of shutter time, analogue gain, and digital gain\n> + */\n> +std::tuple\n> +AgcMeanLuminance::calculateNewEv(uint32_t constraintModeIndex,\n> +\t\t\t\t uint32_t exposureModeIndex,\n> +\t\t\t\t const Histogram &yHist,\n> +\t\t\t\t utils::Duration effectiveExposureValue)\n> +{\n> +\t/*\n> +\t * The pipeline handler should validate that we have received an allowed\n> +\t * value for AeExposureMode.\n> +\t */\n> +\tstd::shared_ptr exposureModeHelper =\n> +\t\texposureModeHelpers_.at(exposureModeIndex);\n> +\n> +\tdouble gain = estimateInitialGain();\n> +\tgain = constraintClampGain(constraintModeIndex, yHist, gain);\n> +\n> +\t/*\n> +\t * We don't check whether we're already close to the target, because\n> +\t * even if the effective exposure value is the same as the last frame's\n> +\t * we could have switched to an exposure mode that would require a new\n> +\t * pass through the splitExposure() function.\n> +\t */\n> +\n> +\tutils::Duration newExposureValue = effectiveExposureValue * gain;\n> +\tutils::Duration maxTotalExposure = exposureModeHelper->maxShutter()\n> +\t\t\t\t\t * exposureModeHelper->maxGain();\n> +\tnewExposureValue = std::min(newExposureValue, maxTotalExposure);\n> +\n> +\t/*\n> +\t * We filter the exposure value to make sure changes are not too jarring\n> +\t * from frame to frame.\n> +\t */\n> +\tnewExposureValue = filterExposure(newExposureValue);\n> +\n> +\tframeCount_++;\n> +\treturn exposureModeHelper->splitExposure(newExposureValue);\n> +}\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::resetFrameCount()\n> + * \\brief Reset the frame counter\n> + *\n> + * This function resets the internal frame counter, which exists to help the\n> + * algorithm decide whether it should respond instantly or not. The expectation\n> + * is for derived classes to call this function before each camera start call,\n> + * either in configure() or queueRequest() if the frame number is zero.\n> + */\n> +\n> +}; /* namespace ipa */\n> +\n> +}; /* namespace libcamera */\n> diff --git a/src/ipa/libipa/agc_mean_luminance.h b/src/ipa/libipa/agc_mean_luminance.h\n> new file mode 100644\n> index 00000000..e48dc498\n> --- /dev/null\n> +++ b/src/ipa/libipa/agc_mean_luminance.h\n> @@ -0,0 +1,91 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2024 Ideas on Board Oy\n> + *\n> + agc_mean_luminance.h - Base class for mean luminance AGC algorithms\n> + */\n> +\n> +#pragma once\n> +\n> +#include \n> +#include \n> +\n> +#include \n> +\n> +#include \"libcamera/internal/yaml_parser.h\"\n> +\n> +#include \"exposure_mode_helper.h\"\n> +#include \"histogram.h\"\n> +\n> +namespace libcamera {\n> +\n> +namespace ipa {\n> +\n> +class AgcMeanLuminance\n> +{\n> +public:\n> +\tAgcMeanLuminance();\n> +\tvirtual ~AgcMeanLuminance() = default;\n\nThere were a few small comments from Laurent that got lost\n * destructor in cpp\n * code sytel in enum\n * missing line\n\nAside from that, I think we should merge it in.\n\nReviewed-by: Stefan Klug \n\nCheers,\nStefan\n\n> +\n> +\tstruct AgcConstraint {\n> +\t\tenum class Bound {\n> +\t\t\tlower = 0,\n> +\t\t\tupper = 1\n> +\t\t};\n> +\t\tBound bound;\n> +\t\tdouble qLo;\n> +\t\tdouble qHi;\n> +\t\tdouble yTarget;\n> +\t};\n> +\n> +\tint parseTuningData(const YamlObject &tuningData);\n> +\n> +\tvoid configureExposureModeHelpers(utils::Duration minShutter,\n> +\t\t\t\t\t utils::Duration maxShutter,\n> +\t\t\t\t\t double minGain,\n> +\t\t\t\t\t double maxGain);\n> +\n> +\tstd::map> constraintModes()\n> +\t{\n> +\t\treturn constraintModes_;\n> +\t}\n> +\n> +\tstd::map> exposureModeHelpers()\n> +\t{\n> +\t\treturn exposureModeHelpers_;\n> +\t}\n> +\n> +\tControlInfoMap::Map controls()\n> +\t{\n> +\t\treturn controls_;\n> +\t}\n> +\n> +\tdouble estimateInitialGain();\n> +\tdouble constraintClampGain(uint32_t constraintModeIndex,\n> +\t\t\t\t const Histogram &hist,\n> +\t\t\t\t double gain);\n> +\tutils::Duration filterExposure(utils::Duration exposureValue);\n> +\tstd::tuple\n> +\tcalculateNewEv(uint32_t constraintModeIndex, uint32_t exposureModeIndex,\n> +\t\t const Histogram &yHist, utils::Duration effectiveExposureValue);\n> +\tvoid resetFrameCount() { frameCount_ = 0; }\n> +private:\n> +\tvirtual double estimateLuminance(const double gain) = 0;\n> +\n> +\tvoid parseRelativeLuminanceTarget(const YamlObject &tuningData);\n> +\tvoid parseConstraint(const YamlObject &modeDict, int32_t id);\n> +\tint parseConstraintModes(const YamlObject &tuningData);\n> +\tint parseExposureModes(const YamlObject &tuningData);\n> +\n> +\tuint64_t frameCount_;\n> +\tutils::Duration filteredExposure_;\n> +\tdouble relativeLuminanceTarget_;\n> +\n> +\tstd::map> constraintModes_;\n> +\tstd::map> exposureModeHelpers_;\n> +\tControlInfoMap::Map controls_;\n> +};\n> +\n> +}; /* namespace ipa */\n> +\n> +}; /* namespace libcamera */\n> diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build\n> index 37fbd177..7ce885da 100644\n> --- a/src/ipa/libipa/meson.build\n> +++ b/src/ipa/libipa/meson.build\n> @@ -1,6 +1,7 @@\n> # SPDX-License-Identifier: CC0-1.0\n> \n> libipa_headers = files([\n> + 'agc_mean_luminance.h',\n> 'algorithm.h',\n> 'camera_sensor_helper.h',\n> 'exposure_mode_helper.h',\n> @@ -10,6 +11,7 @@ libipa_headers = files([\n> ])\n> \n> libipa_sources = files([\n> + 'agc_mean_luminance.cpp',\n> 'algorithm.cpp',\n> 'camera_sensor_helper.cpp',\n> 'exposure_mode_helper.cpp',\n> -- \n> 2.34.1\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 275CFBE08B\n\tfor ;\n\tThu, 18 Apr 2024 07:48:29 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 4392E633EE;\n\tThu, 18 Apr 2024 09:48:28 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 05C8661B43\n\tfor ;\n\tThu, 18 Apr 2024 09:48:25 +0200 (CEST)","from ideasonboard.com (unknown\n\t[IPv6:2a00:6020:448c:6c00:c257:f1f7:8fc8:140f])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 5D30927C;\n\tThu, 18 Apr 2024 09:47:38 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"orM/jcTS\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1713426458;\n\tbh=lH8yvAgL2NezFLLNnb+YbHBJ7O8LzDk/wx+ibqQI0UA=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=orM/jcTS2sTMVE/8HDXh3ZEVXPDnZBshPtbhVjm47voz9ToI74EKyLY4aVrJrOBPY\n\t8GZWxqgrhC+F3g0v41KfNGDhbHxqEqZDNXiPbl6x8qDdu3dMloNLc5NHLVo6iBbI/D\n\tRsWpOVEJpDsWrlVElZYJu6mWLDoKSfevHmmB2kNM=","Date":"Thu, 18 Apr 2024 09:48:23 +0200","From":"Stefan Klug ","To":"Daniel Scally ","Cc":"libcamera-devel@lists.libcamera.org","Subject":"Re: [PATCH v2 4/8] ipa: libipa: Add MeanLuminanceAgc base class","Message-ID":"<20240418074823.l2k273uo5fx2nmwp@jasper>","References":"<20240417131536.484129-1-dan.scally@ideasonboard.com>\n\t<20240417131536.484129-5-dan.scally@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20240417131536.484129-5-dan.scally@ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":29262,"web_url":"https://patchwork.libcamera.org/comment/29262/","msgid":"","date":"2024-04-18T12:49:34","subject":"Re: [PATCH v2 4/8] ipa: libipa: Add MeanLuminanceAgc base class","submitter":{"id":17,"url":"https://patchwork.libcamera.org/api/people/17/","name":"Paul Elder","email":"paul.elder@ideasonboard.com"},"content":"Hi Dan,\n\nOn Wed, Apr 17, 2024 at 02:15:32PM +0100, Daniel Scally wrote:\n> The Agc algorithms for the RkIsp1 and IPU3 IPAs do the same thing in\n> very large part; following the Rpi IPA's algorithm in spirit with a\n> few tunable values in that IPA being hardcoded in the libipa ones.\n> Add a new base class for MeanLuminanceAgc which implements the same\n> algorithm and additionally parses yaml tuning files to inform an IPA\n> module's Agc algorithm about valid constraint and exposure modes and\n> their associated bounds.\n> \n> Signed-off-by: Daniel Scally \n> ---\n> Changes in v2:\n> \n> \t- Renamed the class and files\n> \t- Expanded the documentation\n> \t- Added parseTuningData() so derived classes can call a single function\n> \t to cover all the parsing in ::init().\n> \n> src/ipa/libipa/agc_mean_luminance.cpp | 581 ++++++++++++++++++++++++++\n> src/ipa/libipa/agc_mean_luminance.h | 91 ++++\n> src/ipa/libipa/meson.build | 2 +\n> 3 files changed, 674 insertions(+)\n> create mode 100644 src/ipa/libipa/agc_mean_luminance.cpp\n> create mode 100644 src/ipa/libipa/agc_mean_luminance.h\n> \n> diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp\n> new file mode 100644\n> index 00000000..02e223cf\n> --- /dev/null\n> +++ b/src/ipa/libipa/agc_mean_luminance.cpp\n> @@ -0,0 +1,581 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2024 Ideas on Board Oy\n> + *\n> + * agc_mean_luminance.cpp - Base class for mean luminance AGC algorithms\n> + */\n> +\n> +#include \"agc_mean_luminance.h\"\n> +\n> +#include \n> +\n> +#include \n> +#include \n> +\n> +#include \"exposure_mode_helper.h\"\n> +\n> +using namespace libcamera::controls;\n> +\n> +/**\n> + * \\file agc_mean_luminance.h\n> + * \\brief Base class implementing mean luminance AEGC\n> + */\n> +\n> +namespace libcamera {\n> +\n> +using namespace std::literals::chrono_literals;\n> +\n> +LOG_DEFINE_CATEGORY(AgcMeanLuminance)\n> +\n> +namespace ipa {\n> +\n> +/*\n> + * Number of frames for which to run the algorithm at full speed, before slowing\n> + * down to prevent large and jarring changes in exposure from frame to frame.\n> + */\n> +static constexpr uint32_t kNumStartupFrames = 10;\n> +\n> +/*\n> + * Default relative luminance target\n> + *\n> + * This value should be chosen so that when the camera points at a grey target,\n> + * the resulting image brightness looks \"right\". Custom values can be passed\n> + * as the relativeLuminanceTarget value in sensor tuning files.\n> + */\n> +static constexpr double kDefaultRelativeLuminanceTarget = 0.16;\n> +\n> +/**\n> + * \\struct AgcMeanLuminance::AgcConstraint\n> + * \\brief The boundaries and target for an AeConstraintMode constraint\n> + *\n> + * This structure describes an AeConstraintMode constraint for the purposes of\n> + * this algorithm. The algorithm will apply the constraints by calculating the\n> + * Histogram's inter-quantile mean between the given quantiles and ensure that\n> + * the resulting value is the right side of the given target (as defined by the\n> + * boundary and luminance target).\n> + */\n> +\n> +/**\n> + * \\enum AgcMeanLuminance::AgcConstraint::Bound\n> + * \\brief Specify whether the constraint defines a lower or upper bound\n> + * \\var AgcMeanLuminance::AgcConstraint::lower\n> + * \\brief The constraint defines a lower bound\n> + * \\var AgcMeanLuminance::AgcConstraint::upper\n> + * \\brief The constraint defines an upper bound\n> + */\n> +\n> +/**\n> + * \\var AgcMeanLuminance::AgcConstraint::bound\n> + * \\brief The type of constraint bound\n> + */\n> +\n> +/**\n> + * \\var AgcMeanLuminance::AgcConstraint::qLo\n> + * \\brief The lower quantile to use for the constraint\n> + */\n> +\n> +/**\n> + * \\var AgcMeanLuminance::AgcConstraint::qHi\n> + * \\brief The upper quantile to use for the constraint\n> + */\n> +\n> +/**\n> + * \\var AgcMeanLuminance::AgcConstraint::yTarget\n> + * \\brief The luminance target for the constraint\n> + */\n> +\n> +/**\n> + * \\class AgcMeanLuminance\n> + * \\brief A mean-based auto-exposure algorithm\n> + *\n> + * This algorithm calculates a shutter time, analogue and digital gain such that\n> + * the normalised mean luminance value of an image is driven towards a target,\n> + * which itself is discovered from tuning data. The algorithm is a two-stage\n> + * process.\n> + *\n> + * In the first stage, an initial gain value is derived by iteratively comparing\n> + * the gain-adjusted mean luminance across an entire image against a target, and\n> + * selecting a value which pushes it as closely as possible towards the target.\n> + *\n> + * In the second stage we calculate the gain required to drive the average of a\n> + * section of a histogram to a target value, where the target and the boundaries\n> + * of the section of the histogram used in the calculation are taken from the\n> + * values defined for the currently configured AeConstraintMode within the\n> + * tuning data. This class provides a helper function to parse those tuning data\n> + * to discover the constraints, and so requires a specific format for those\n> + * data which is described in \\ref parseTuningData(). The gain from the first\n> + * stage is then clamped to the gain from this stage.\n> + *\n> + * The final gain is used to adjust the effective exposure value of the image,\n> + * and that new exposure value is divided into shutter time, analogue gain and\n> + * digital gain according to the selected AeExposureMode. This class expects to\n> + * use the \\ref ExposureModeHelper class to assist in that division, and expects\n> + * the data needed to initialise that class to be present in tuning data in a\n> + * format described in \\ref parseTuningData().\n> + *\n> + * In order to be able to derive an AGC implementation from this class, an IPA\n> + * needs to be able to do the following:\n> + *\n> + * 1. Provide a luminance estimation across an entire image.\n> + * 2. Provide a luminance Histogram for the image to use in calculating\n> + * constraint compliance. The precision of the Histogram that is available\n> + * will determine the supportable precision of the constraints.\n> + */\n> +\n> +AgcMeanLuminance::AgcMeanLuminance()\n> +\t: frameCount_(0), filteredExposure_(0s), relativeLuminanceTarget_(0)\n> +{\n> +}\n> +\n> +/**\n> + * \\brief Parse the relative luminance target from the tuning data\n> + * \\param[in] tuningData The YamlObject holding the algorithm's tuning data\n> + */\n> +void AgcMeanLuminance::parseRelativeLuminanceTarget(const YamlObject &tuningData)\n> +{\n> +\trelativeLuminanceTarget_ =\n> +\t\ttuningData[\"relativeLuminanceTarget\"].get(kDefaultRelativeLuminanceTarget);\n> +}\n> +\n> +/**\n> + * \\brief Parse an AeConstraintMode constraint from tuning data\n> + * \\param[in] modeDict the YamlObject holding the constraint data\n> + * \\param[in] id The constraint ID from AeConstraintModeEnum\n> + */\n> +void AgcMeanLuminance::parseConstraint(const YamlObject &modeDict, int32_t id)\n> +{\n> +\tfor (const auto &[boundName, content] : modeDict.asDict()) {\n> +\t\tif (boundName != \"upper\" && boundName != \"lower\") {\n> +\t\t\tLOG(AgcMeanLuminance, Warning)\n> +\t\t\t\t<< \"Ignoring unknown constraint bound '\" << boundName << \"'\";\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tunsigned int idx = static_cast(boundName == \"upper\");\n> +\t\tAgcConstraint::Bound bound = static_cast(idx);\n> +\t\tdouble qLo = content[\"qLo\"].get().value_or(0.98);\n> +\t\tdouble qHi = content[\"qHi\"].get().value_or(1.0);\n> +\t\tdouble yTarget =\n> +\t\t\tcontent[\"yTarget\"].getList().value_or(std::vector{ 0.5 }).at(0);\n> +\n> +\t\tAgcConstraint constraint = { bound, qLo, qHi, yTarget };\n> +\n> +\t\tif (!constraintModes_.count(id))\n> +\t\t\tconstraintModes_[id] = {};\n> +\n> +\t\tif (idx)\n> +\t\t\tconstraintModes_[id].push_back(constraint);\n> +\t\telse\n> +\t\t\tconstraintModes_[id].insert(constraintModes_[id].begin(), constraint);\n> +\t}\n> +}\n> +\n> +int AgcMeanLuminance::parseConstraintModes(const YamlObject &tuningData)\n> +{\n> +\tstd::vector availableConstraintModes;\n> +\n> +\tconst YamlObject &yamlConstraintModes = tuningData[controls::AeConstraintMode.name()];\n> +\tif (yamlConstraintModes.isDictionary()) {\n> +\t\tfor (const auto &[modeName, modeDict] : yamlConstraintModes.asDict()) {\n> +\t\t\tif (AeConstraintModeNameValueMap.find(modeName) ==\n> +\t\t\t AeConstraintModeNameValueMap.end()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Warning)\n> +\t\t\t\t\t<< \"Skipping unknown constraint mode '\" << modeName << \"'\";\n> +\t\t\t\tcontinue;\n> +\t\t\t}\n> +\n> +\t\t\tif (!modeDict.isDictionary()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> +\t\t\t\t\t<< \"Invalid constraint mode '\" << modeName << \"'\";\n> +\t\t\t\treturn -EINVAL;\n> +\t\t\t}\n> +\n> +\t\t\tparseConstraint(modeDict,\n> +\t\t\t\t\tAeConstraintModeNameValueMap.at(modeName));\n> +\t\t\tavailableConstraintModes.push_back(\n> +\t\t\t\tAeConstraintModeNameValueMap.at(modeName));\n> +\t\t}\n> +\t}\n> +\n> +\t/*\n> +\t * If the tuning data file contains no constraints then we use the\n> +\t * default constraint that the various Agc algorithms were adhering to\n> +\t * anyway before centralisation.\n> +\t */\n> +\tif (constraintModes_.empty()) {\n> +\t\tAgcConstraint constraint = {\n> +\t\t\tAgcConstraint::Bound::lower,\n> +\t\t\t0.98,\n> +\t\t\t1.0,\n> +\t\t\t0.5\n> +\t\t};\n> +\n> +\t\tconstraintModes_[controls::ConstraintNormal].insert(\n> +\t\t\tconstraintModes_[controls::ConstraintNormal].begin(),\n> +\t\t\tconstraint);\n> +\t\tavailableConstraintModes.push_back(\n> +\t\t\tAeConstraintModeNameValueMap.at(\"ConstraintNormal\"));\n> +\t}\n> +\n> +\tcontrols_[&controls::AeConstraintMode] = ControlInfo(availableConstraintModes);\n> +\n> +\treturn 0;\n> +}\n> +\n> +int AgcMeanLuminance::parseExposureModes(const YamlObject &tuningData)\n> +{\n> +\tstd::vector availableExposureModes;\n> +\n> +\tconst YamlObject &yamlExposureModes = tuningData[controls::AeExposureMode.name()];\n> +\tif (yamlExposureModes.isDictionary()) {\n> +\t\tfor (const auto &[modeName, modeValues] : yamlExposureModes.asDict()) {\n> +\t\t\tif (AeExposureModeNameValueMap.find(modeName) ==\n> +\t\t\t AeExposureModeNameValueMap.end()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Warning)\n> +\t\t\t\t\t<< \"Skipping unknown exposure mode '\" << modeName << \"'\";\n> +\t\t\t\tcontinue;\n> +\t\t\t}\n> +\n> +\t\t\tif (!modeValues.isDictionary()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> +\t\t\t\t\t<< \"Invalid exposure mode '\" << modeName << \"'\";\n> +\t\t\t\treturn -EINVAL;\n> +\t\t\t}\n> +\n> +\t\t\tstd::vector shutters =\n> +\t\t\t\tmodeValues[\"shutter\"].getList().value_or(std::vector{});\n> +\t\t\tstd::vector gains =\n> +\t\t\t\tmodeValues[\"gain\"].getList().value_or(std::vector{});\n> +\n> +\t\t\tif (shutters.size() != gains.size()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> +\t\t\t\t\t<< \"Shutter and gain array sizes unequal\";\n> +\t\t\t\treturn -EINVAL;\n> +\t\t\t}\n> +\n> +\t\t\tif (shutters.empty()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> +\t\t\t\t\t<< \"Shutter and gain arrays are empty\";\n> +\t\t\t\treturn -EINVAL;\n> +\t\t\t}\n> +\n> +\t\t\tstd::vector> stages;\n> +\t\t\tfor (unsigned int i = 0; i < shutters.size(); i++) {\n> +\t\t\t\tstages.push_back({\n> +\t\t\t\t\tstd::chrono::microseconds(shutters[i]),\n> +\t\t\t\t\tgains[i]\n> +\t\t\t\t});\n> +\t\t\t}\n\nI was wondering if we could move a significant portion of this to\nExposureModeHelper::readYaml() but since this is the only user (so far)\nof it I suppose it doesn't really matter.\n\nReviewed-by: Paul Elder \n\n> +\n> +\t\t\tstd::shared_ptr helper =\n> +\t\t\t\tstd::make_shared();\n> +\t\t\thelper->init(stages);\n> +\n> +\t\t\texposureModeHelpers_[AeExposureModeNameValueMap.at(modeName)] = helper;\n> +\t\t\tavailableExposureModes.push_back(AeExposureModeNameValueMap.at(modeName));\n> +\t\t}\n> +\t}\n> +\n> +\t/*\n> +\t * If we don't have any exposure modes in the tuning data we create an\n> +\t * ExposureModeHelper using an empty vector of stages. This will result\n> +\t * in the ExposureModeHelper simply driving the shutter as high as\n> +\t * possible before touching gain.\n> +\t */\n> +\tif (availableExposureModes.empty()) {\n> +\t\tint32_t exposureModeId = AeExposureModeNameValueMap.at(\"ExposureNormal\");\n> +\t\tstd::vector> stages = { };\n> +\n> +\t\tstd::shared_ptr helper =\n> +\t\t\tstd::make_shared();\n> +\t\thelper->init(stages);\n> +\n> +\t\texposureModeHelpers_[exposureModeId] = helper;\n> +\t\tavailableExposureModes.push_back(exposureModeId);\n> +\t}\n> +\n> +\tcontrols_[&controls::AeExposureMode] = ControlInfo(availableExposureModes);\n> +\n> +\treturn 0;\n> +}\n> +\n> +/**\n> + * \\brief Parse tuning data for AeConstraintMode and AeExposureMode controls\n> + * \\param[in] tuningData the YamlObject representing the tuning data\n> + *\n> + * This function parses tuning data to build the list of allowed values for the\n> + * AeConstraintMode and AeExposureMode controls. Those tuning data must provide\n> + * the data in a specific format; the Agc algorithm's tuning data should contain\n> + * a dictionary called AeConstraintMode containing per-mode setting dictionaries\n> + * with the key being a value from \\ref controls::AeConstraintModeNameValueMap.\n> + * Each mode dict may contain either a \"lower\" or \"upper\" key or both, for\n> + * example:\n> + *\n> + * \\code{.unparsed}\n> + * algorithms:\n> + * - Agc:\n> + * AeConstraintMode:\n> + * ConstraintNormal:\n> + * lower:\n> + * qLo: 0.98\n> + * qHi: 1.0\n> + * yTarget: 0.5\n> + * ConstraintHighlight:\n> + * lower:\n> + * qLo: 0.98\n> + * qHi: 1.0\n> + * yTarget: 0.5\n> + * upper:\n> + * qLo: 0.98\n> + * qHi: 1.0\n> + * yTarget: 0.8\n> + *\n> + * \\endcode\n> + *\n> + * For the AeExposureMode control the data should contain a dictionary called\n> + * AeExposureMode containing per-mode setting dictionaries with the key being a\n> + * value from \\ref controls::AeExposureModeNameValueMap. Each mode dict should\n> + * contain an array of shutter times with the key \"shutter\" and an array of gain\n> + * values with the key \"gain\", in this format:\n> + *\n> + * \\code{.unparsed}\n> + * algorithms:\n> + * - Agc:\n> + * AeExposureMode:\n> + * ExposureNormal:\n> + * shutter: [ 100, 10000, 30000, 60000, 120000 ]\n> + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ]\n> + * ExposureShort:\n> + * shutter: [ 100, 10000, 30000, 60000, 120000 ]\n> + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ]\n> + *\n> + * \\endcode\n> + *\n> + * @return 0 on success or a negative error code\n> + */\n> +int AgcMeanLuminance::parseTuningData(const YamlObject &tuningData)\n> +{\n> +\tint ret;\n> +\n> +\tparseRelativeLuminanceTarget(tuningData);\n> +\n> +\tret = parseConstraintModes(tuningData);\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n> +\tret = parseExposureModes(tuningData);\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n> +\treturn 0;\n> +}\n> +\n> +/**\n> + * \\brief configure the ExposureModeHelpers for this class\n> + * \\param[in] minShutter Minimum shutter time to allow\n> + * \\param[in] maxShutter Maximum shutter time to allow\n> + * \\param[in] minGain Minimum gain to allow\n> + * \\param[in] maxGain Maximum gain to allow\n> + *\n> + * This function calls \\ref ExposureModeHelper::setShutterGainLimits() for each\n> + * ExposureModeHelper that has been created for this class.\n> + */\n> +void AgcMeanLuminance::configureExposureModeHelpers(utils::Duration minShutter,\n> +\t\t\t\t\t\t utils::Duration maxShutter,\n> +\t\t\t\t\t\t double minGain,\n> +\t\t\t\t\t\t double maxGain)\n> +{\n> +\tfor (auto &[id, helper] : exposureModeHelpers_)\n> +\t\thelper->setShutterGainLimits(minShutter, maxShutter, minGain, maxGain);\n> +}\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::constraintModes()\n> + * \\brief Get the constraint modes that have been parsed from tuning data\n> + */\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::exposureModeHelpers()\n> + * \\brief Get the ExposureModeHelpers that have been parsed from tuning data\n> + */\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::controls()\n> + * \\brief Get the controls that have been generated after parsing tuning data\n> + */\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::estimateLuminance(const double gain)\n> + * \\brief Estimate the luminance of an image, adjusted by a given gain\n> + * \\param[in] gain The gain with which to adjust the luminance estimate\n> + *\n> + * This function estimates the average relative luminance of the frame that\n> + * would be output by the sensor if an additional \\a gain was applied. It is a\n> + * pure virtual function because estimation of luminance is a hardware-specific\n> + * operation, which depends wholly on the format of the stats that are delivered\n> + * to libcamera from the ISP. Derived classes must implement an overriding\n> + * function that calculates the normalised mean luminance value across the\n> + * entire image.\n> + *\n> + * \\return The normalised relative luminance of the image\n> + */\n> +\n> +/**\n> + * \\brief Estimate the initial gain needed to achieve a relative luminance\n> + * target\n> + *\n> + * To account for non-linearity caused by saturation, the value needs to be\n> + * estimated in an iterative process, as multiplying by a gain will not increase\n> + * the relative luminance by the same factor if some image regions are saturated\n> + *\n> + * \\return The calculated initial gain\n> + */\n> +double AgcMeanLuminance::estimateInitialGain()\n> +{\n> +\tdouble yTarget = relativeLuminanceTarget_;\n> +\tdouble yGain = 1.0;\n> +\n> +\tfor (unsigned int i = 0; i < 8; i++) {\n> +\t\tdouble yValue = estimateLuminance(yGain);\n> +\t\tdouble extra_gain = std::min(10.0, yTarget / (yValue + .001));\n> +\n> +\t\tyGain *= extra_gain;\n> +\t\tLOG(AgcMeanLuminance, Debug) << \"Y value: \" << yValue\n> +\t\t\t\t<< \", Y target: \" << yTarget\n> +\t\t\t\t<< \", gives gain \" << yGain;\n> +\n> +\t\tif (utils::abs_diff(extra_gain, 1.0) < 0.01)\n> +\t\t\tbreak;\n> +\t}\n> +\n> +\treturn yGain;\n> +}\n> +\n> +/**\n> + * \\brief Clamp gain within the bounds of a defined constraint\n> + * \\param[in] constraintModeIndex The index of the constraint to adhere to\n> + * \\param[in] hist A histogram over which to calculate inter-quantile means\n> + * \\param[in] gain The gain to clamp\n> + *\n> + * \\return The gain clamped within the constraint bounds\n> + */\n> +double AgcMeanLuminance::constraintClampGain(uint32_t constraintModeIndex,\n> +\t\t\t\t\t const Histogram &hist,\n> +\t\t\t\t\t double gain)\n> +{\n> +\tstd::vector &constraints = constraintModes_[constraintModeIndex];\n> +\tfor (const AgcConstraint &constraint : constraints) {\n> +\t\tdouble newGain = constraint.yTarget * hist.bins() /\n> +\t\t\t\t hist.interQuantileMean(constraint.qLo, constraint.qHi);\n> +\n> +\t\tif (constraint.bound == AgcConstraint::Bound::lower &&\n> +\t\t newGain > gain)\n> +\t\t\tgain = newGain;\n> +\n> +\t\tif (constraint.bound == AgcConstraint::Bound::upper &&\n> +\t\t newGain < gain)\n> +\t\t\tgain = newGain;\n> +\t}\n> +\n> +\treturn gain;\n> +}\n> +\n> +/**\n> + * \\brief Apply a filter on the exposure value to limit the speed of changes\n> + * \\param[in] exposureValue The target exposure from the AGC algorithm\n> + *\n> + * The speed of the filter is adaptive, and will produce the target quicker\n> + * during startup, or when the target exposure is within 20% of the most recent\n> + * filter output.\n> + *\n> + * \\return The filtered exposure\n> + */\n> +utils::Duration AgcMeanLuminance::filterExposure(utils::Duration exposureValue)\n> +{\n> +\tdouble speed = 0.2;\n> +\n> +\t/* Adapt instantly if we are in startup phase. */\n> +\tif (frameCount_ < kNumStartupFrames)\n> +\t\tspeed = 1.0;\n> +\n> +\t/*\n> +\t * If we are close to the desired result, go faster to avoid making\n> +\t * multiple micro-adjustments.\n> +\t * \\todo Make this customisable?\n> +\t */\n> +\tif (filteredExposure_ < 1.2 * exposureValue &&\n> +\t filteredExposure_ > 0.8 * exposureValue)\n> +\t\tspeed = sqrt(speed);\n> +\n> +\tfilteredExposure_ = speed * exposureValue +\n> +\t\t\t filteredExposure_ * (1.0 - speed);\n> +\n> +\treturn filteredExposure_;\n> +}\n> +\n> +/**\n> + * \\brief Calculate the new exposure value\n> + * \\param[in] constraintModeIndex The index of the current constraint mode\n> + * \\param[in] exposureModeIndex The index of the current exposure mode\n> + * \\param[in] yHist A Histogram from the ISP statistics to use in constraining\n> + *\t the calculated gain\n> + * \\param[in] effectiveExposureValue The EV applied to the frame from which the\n> + *\t statistics in use derive\n> + *\n> + * Calculate a new exposure value to try to obtain the target. The calculated\n> + * exposure value is filtered to prevent rapid changes from frame to frame, and\n> + * divided into shutter time, analogue and digital gain.\n> + *\n> + * \\return Tuple of shutter time, analogue gain, and digital gain\n> + */\n> +std::tuple\n> +AgcMeanLuminance::calculateNewEv(uint32_t constraintModeIndex,\n> +\t\t\t\t uint32_t exposureModeIndex,\n> +\t\t\t\t const Histogram &yHist,\n> +\t\t\t\t utils::Duration effectiveExposureValue)\n> +{\n> +\t/*\n> +\t * The pipeline handler should validate that we have received an allowed\n> +\t * value for AeExposureMode.\n> +\t */\n> +\tstd::shared_ptr exposureModeHelper =\n> +\t\texposureModeHelpers_.at(exposureModeIndex);\n> +\n> +\tdouble gain = estimateInitialGain();\n> +\tgain = constraintClampGain(constraintModeIndex, yHist, gain);\n> +\n> +\t/*\n> +\t * We don't check whether we're already close to the target, because\n> +\t * even if the effective exposure value is the same as the last frame's\n> +\t * we could have switched to an exposure mode that would require a new\n> +\t * pass through the splitExposure() function.\n> +\t */\n> +\n> +\tutils::Duration newExposureValue = effectiveExposureValue * gain;\n> +\tutils::Duration maxTotalExposure = exposureModeHelper->maxShutter()\n> +\t\t\t\t\t * exposureModeHelper->maxGain();\n> +\tnewExposureValue = std::min(newExposureValue, maxTotalExposure);\n> +\n> +\t/*\n> +\t * We filter the exposure value to make sure changes are not too jarring\n> +\t * from frame to frame.\n> +\t */\n> +\tnewExposureValue = filterExposure(newExposureValue);\n> +\n> +\tframeCount_++;\n> +\treturn exposureModeHelper->splitExposure(newExposureValue);\n> +}\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::resetFrameCount()\n> + * \\brief Reset the frame counter\n> + *\n> + * This function resets the internal frame counter, which exists to help the\n> + * algorithm decide whether it should respond instantly or not. The expectation\n> + * is for derived classes to call this function before each camera start call,\n> + * either in configure() or queueRequest() if the frame number is zero.\n> + */\n> +\n> +}; /* namespace ipa */\n> +\n> +}; /* namespace libcamera */\n> diff --git a/src/ipa/libipa/agc_mean_luminance.h b/src/ipa/libipa/agc_mean_luminance.h\n> new file mode 100644\n> index 00000000..e48dc498\n> --- /dev/null\n> +++ b/src/ipa/libipa/agc_mean_luminance.h\n> @@ -0,0 +1,91 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2024 Ideas on Board Oy\n> + *\n> + agc_mean_luminance.h - Base class for mean luminance AGC algorithms\n> + */\n> +\n> +#pragma once\n> +\n> +#include \n> +#include \n> +\n> +#include \n> +\n> +#include \"libcamera/internal/yaml_parser.h\"\n> +\n> +#include \"exposure_mode_helper.h\"\n> +#include \"histogram.h\"\n> +\n> +namespace libcamera {\n> +\n> +namespace ipa {\n> +\n> +class AgcMeanLuminance\n> +{\n> +public:\n> +\tAgcMeanLuminance();\n> +\tvirtual ~AgcMeanLuminance() = default;\n> +\n> +\tstruct AgcConstraint {\n> +\t\tenum class Bound {\n> +\t\t\tlower = 0,\n> +\t\t\tupper = 1\n> +\t\t};\n> +\t\tBound bound;\n> +\t\tdouble qLo;\n> +\t\tdouble qHi;\n> +\t\tdouble yTarget;\n> +\t};\n> +\n> +\tint parseTuningData(const YamlObject &tuningData);\n> +\n> +\tvoid configureExposureModeHelpers(utils::Duration minShutter,\n> +\t\t\t\t\t utils::Duration maxShutter,\n> +\t\t\t\t\t double minGain,\n> +\t\t\t\t\t double maxGain);\n> +\n> +\tstd::map> constraintModes()\n> +\t{\n> +\t\treturn constraintModes_;\n> +\t}\n> +\n> +\tstd::map> exposureModeHelpers()\n> +\t{\n> +\t\treturn exposureModeHelpers_;\n> +\t}\n> +\n> +\tControlInfoMap::Map controls()\n> +\t{\n> +\t\treturn controls_;\n> +\t}\n> +\n> +\tdouble estimateInitialGain();\n> +\tdouble constraintClampGain(uint32_t constraintModeIndex,\n> +\t\t\t\t const Histogram &hist,\n> +\t\t\t\t double gain);\n> +\tutils::Duration filterExposure(utils::Duration exposureValue);\n> +\tstd::tuple\n> +\tcalculateNewEv(uint32_t constraintModeIndex, uint32_t exposureModeIndex,\n> +\t\t const Histogram &yHist, utils::Duration effectiveExposureValue);\n> +\tvoid resetFrameCount() { frameCount_ = 0; }\n> +private:\n> +\tvirtual double estimateLuminance(const double gain) = 0;\n> +\n> +\tvoid parseRelativeLuminanceTarget(const YamlObject &tuningData);\n> +\tvoid parseConstraint(const YamlObject &modeDict, int32_t id);\n> +\tint parseConstraintModes(const YamlObject &tuningData);\n> +\tint parseExposureModes(const YamlObject &tuningData);\n> +\n> +\tuint64_t frameCount_;\n> +\tutils::Duration filteredExposure_;\n> +\tdouble relativeLuminanceTarget_;\n> +\n> +\tstd::map> constraintModes_;\n> +\tstd::map> exposureModeHelpers_;\n> +\tControlInfoMap::Map controls_;\n> +};\n> +\n> +}; /* namespace ipa */\n> +\n> +}; /* namespace libcamera */\n> diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build\n> index 37fbd177..7ce885da 100644\n> --- a/src/ipa/libipa/meson.build\n> +++ b/src/ipa/libipa/meson.build\n> @@ -1,6 +1,7 @@\n> # SPDX-License-Identifier: CC0-1.0\n> \n> libipa_headers = files([\n> + 'agc_mean_luminance.h',\n> 'algorithm.h',\n> 'camera_sensor_helper.h',\n> 'exposure_mode_helper.h',\n> @@ -10,6 +11,7 @@ libipa_headers = files([\n> ])\n> \n> libipa_sources = files([\n> + 'agc_mean_luminance.cpp',\n> 'algorithm.cpp',\n> 'camera_sensor_helper.cpp',\n> 'exposure_mode_helper.cpp',\n> -- \n> 2.34.1\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 5C500BE08B\n\tfor ;\n\tThu, 18 Apr 2024 12:49:43 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 4F208633F5;\n\tThu, 18 Apr 2024 14:49:42 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id D4D8F6334D\n\tfor ;\n\tThu, 18 Apr 2024 14:49:40 +0200 (CEST)","from pyrite.rasen.tech (h175-177-049-156.catv02.itscom.jp\n\t[175.177.49.156])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id D0E7827C;\n\tThu, 18 Apr 2024 14:48:51 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"gDEadDYf\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1713444533;\n\tbh=KOWndEG4I7AQ2vrnELy9l+d2UTTn27yQA87TDHjKU6I=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=gDEadDYflDOnxspxrKFJkSJ3ZK29Xmh4dgg2UdzHfxXoKgwp1LWG8CJ7jbuG8G3CB\n\tVV4Vv5yyACPWmHhy6Ft2tllsbKDJhfVabZPTeBzNb5Cx0iSmdpWC/L+yE7hOn9JJll\n\tWr0QsCG9jrHL8bWbHNhOs4/0Bc0VzluKB7rJ/6LI=","Date":"Thu, 18 Apr 2024 21:49:34 +0900","From":"Paul Elder ","To":"Daniel Scally ","Cc":"libcamera-devel@lists.libcamera.org","Subject":"Re: [PATCH v2 4/8] ipa: libipa: Add MeanLuminanceAgc base class","Message-ID":"","References":"<20240417131536.484129-1-dan.scally@ideasonboard.com>\n\t<20240417131536.484129-5-dan.scally@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"<20240417131536.484129-5-dan.scally@ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":29265,"web_url":"https://patchwork.libcamera.org/comment/29265/","msgid":"","date":"2024-04-18T15:03:24","subject":"Re: [PATCH v2 4/8] ipa: libipa: Add MeanLuminanceAgc base class","submitter":{"id":156,"url":"https://patchwork.libcamera.org/api/people/156/","name":"Dan Scally","email":"dan.scally@ideasonboard.com"},"content":"Hi Paul\n\nOn 17/04/2024 14:15, Daniel Scally wrote:\n> The Agc algorithms for the RkIsp1 and IPU3 IPAs do the same thing in\n> very large part; following the Rpi IPA's algorithm in spirit with a\n> few tunable values in that IPA being hardcoded in the libipa ones.\n> Add a new base class for MeanLuminanceAgc which implements the same\n> algorithm and additionally parses yaml tuning files to inform an IPA\n> module's Agc algorithm about valid constraint and exposure modes and\n> their associated bounds.\n>\n> Signed-off-by: Daniel Scally \n> ---\n> Changes in v2:\n>\n> \t- Renamed the class and files\n> \t- Expanded the documentation\n> \t- Added parseTuningData() so derived classes can call a single function\n> \t to cover all the parsing in ::init().\n>\n> src/ipa/libipa/agc_mean_luminance.cpp | 581 ++++++++++++++++++++++++++\n> src/ipa/libipa/agc_mean_luminance.h | 91 ++++\n> src/ipa/libipa/meson.build | 2 +\n> 3 files changed, 674 insertions(+)\n> create mode 100644 src/ipa/libipa/agc_mean_luminance.cpp\n> create mode 100644 src/ipa/libipa/agc_mean_luminance.h\n>\n> diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp\n> new file mode 100644\n> index 00000000..02e223cf\n> --- /dev/null\n> +++ b/src/ipa/libipa/agc_mean_luminance.cpp\n> @@ -0,0 +1,581 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2024 Ideas on Board Oy\n> + *\n> + * agc_mean_luminance.cpp - Base class for mean luminance AGC algorithms\n> + */\n> +\n> +#include \"agc_mean_luminance.h\"\n> +\n> +#include \n> +\n> +#include \n> +#include \n> +\n> +#include \"exposure_mode_helper.h\"\n> +\n> +using namespace libcamera::controls;\n> +\n> +/**\n> + * \\file agc_mean_luminance.h\n> + * \\brief Base class implementing mean luminance AEGC\n> + */\n> +\n> +namespace libcamera {\n> +\n> +using namespace std::literals::chrono_literals;\n> +\n> +LOG_DEFINE_CATEGORY(AgcMeanLuminance)\n> +\n> +namespace ipa {\n> +\n> +/*\n> + * Number of frames for which to run the algorithm at full speed, before slowing\n> + * down to prevent large and jarring changes in exposure from frame to frame.\n> + */\n> +static constexpr uint32_t kNumStartupFrames = 10;\n> +\n> +/*\n> + * Default relative luminance target\n> + *\n> + * This value should be chosen so that when the camera points at a grey target,\n> + * the resulting image brightness looks \"right\". Custom values can be passed\n> + * as the relativeLuminanceTarget value in sensor tuning files.\n> + */\n> +static constexpr double kDefaultRelativeLuminanceTarget = 0.16;\n> +\n> +/**\n> + * \\struct AgcMeanLuminance::AgcConstraint\n> + * \\brief The boundaries and target for an AeConstraintMode constraint\n> + *\n> + * This structure describes an AeConstraintMode constraint for the purposes of\n> + * this algorithm. The algorithm will apply the constraints by calculating the\n> + * Histogram's inter-quantile mean between the given quantiles and ensure that\n> + * the resulting value is the right side of the given target (as defined by the\n> + * boundary and luminance target).\n> + */\n> +\n> +/**\n> + * \\enum AgcMeanLuminance::AgcConstraint::Bound\n> + * \\brief Specify whether the constraint defines a lower or upper bound\n> + * \\var AgcMeanLuminance::AgcConstraint::lower\n> + * \\brief The constraint defines a lower bound\n> + * \\var AgcMeanLuminance::AgcConstraint::upper\n> + * \\brief The constraint defines an upper bound\n> + */\n> +\n> +/**\n> + * \\var AgcMeanLuminance::AgcConstraint::bound\n> + * \\brief The type of constraint bound\n> + */\n> +\n> +/**\n> + * \\var AgcMeanLuminance::AgcConstraint::qLo\n> + * \\brief The lower quantile to use for the constraint\n> + */\n> +\n> +/**\n> + * \\var AgcMeanLuminance::AgcConstraint::qHi\n> + * \\brief The upper quantile to use for the constraint\n> + */\n> +\n> +/**\n> + * \\var AgcMeanLuminance::AgcConstraint::yTarget\n> + * \\brief The luminance target for the constraint\n> + */\n> +\n> +/**\n> + * \\class AgcMeanLuminance\n> + * \\brief A mean-based auto-exposure algorithm\n> + *\n> + * This algorithm calculates a shutter time, analogue and digital gain such that\n> + * the normalised mean luminance value of an image is driven towards a target,\n> + * which itself is discovered from tuning data. The algorithm is a two-stage\n> + * process.\n> + *\n> + * In the first stage, an initial gain value is derived by iteratively comparing\n> + * the gain-adjusted mean luminance across an entire image against a target, and\n> + * selecting a value which pushes it as closely as possible towards the target.\n> + *\n> + * In the second stage we calculate the gain required to drive the average of a\n> + * section of a histogram to a target value, where the target and the boundaries\n> + * of the section of the histogram used in the calculation are taken from the\n> + * values defined for the currently configured AeConstraintMode within the\n> + * tuning data. This class provides a helper function to parse those tuning data\n> + * to discover the constraints, and so requires a specific format for those\n> + * data which is described in \\ref parseTuningData(). The gain from the first\n> + * stage is then clamped to the gain from this stage.\n> + *\n> + * The final gain is used to adjust the effective exposure value of the image,\n> + * and that new exposure value is divided into shutter time, analogue gain and\n> + * digital gain according to the selected AeExposureMode. This class expects to\n> + * use the \\ref ExposureModeHelper class to assist in that division, and expects\n> + * the data needed to initialise that class to be present in tuning data in a\n> + * format described in \\ref parseTuningData().\n> + *\n> + * In order to be able to derive an AGC implementation from this class, an IPA\n> + * needs to be able to do the following:\n> + *\n> + * 1. Provide a luminance estimation across an entire image.\n> + * 2. Provide a luminance Histogram for the image to use in calculating\n> + * constraint compliance. The precision of the Histogram that is available\n> + * will determine the supportable precision of the constraints.\n> + */\n> +\n> +AgcMeanLuminance::AgcMeanLuminance()\n> +\t: frameCount_(0), filteredExposure_(0s), relativeLuminanceTarget_(0)\n> +{\n> +}\n> +\n> +/**\n> + * \\brief Parse the relative luminance target from the tuning data\n> + * \\param[in] tuningData The YamlObject holding the algorithm's tuning data\n> + */\n> +void AgcMeanLuminance::parseRelativeLuminanceTarget(const YamlObject &tuningData)\n> +{\n> +\trelativeLuminanceTarget_ =\n> +\t\ttuningData[\"relativeLuminanceTarget\"].get(kDefaultRelativeLuminanceTarget);\n> +}\n> +\n> +/**\n> + * \\brief Parse an AeConstraintMode constraint from tuning data\n> + * \\param[in] modeDict the YamlObject holding the constraint data\n> + * \\param[in] id The constraint ID from AeConstraintModeEnum\n> + */\n> +void AgcMeanLuminance::parseConstraint(const YamlObject &modeDict, int32_t id)\n> +{\n> +\tfor (const auto &[boundName, content] : modeDict.asDict()) {\n> +\t\tif (boundName != \"upper\" && boundName != \"lower\") {\n> +\t\t\tLOG(AgcMeanLuminance, Warning)\n> +\t\t\t\t<< \"Ignoring unknown constraint bound '\" << boundName << \"'\";\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tunsigned int idx = static_cast(boundName == \"upper\");\n> +\t\tAgcConstraint::Bound bound = static_cast(idx);\n> +\t\tdouble qLo = content[\"qLo\"].get().value_or(0.98);\n> +\t\tdouble qHi = content[\"qHi\"].get().value_or(1.0);\n> +\t\tdouble yTarget =\n> +\t\t\tcontent[\"yTarget\"].getList().value_or(std::vector{ 0.5 }).at(0);\n> +\n\n\nSorry, I did see your point in the last thread that this should be a piecewise linear function but I \nforgot to respond to it. I do wonder if it's actually necessary for this one to be a piecewise \nlinear function since, although Rpi treats it as a PWL, the points they define in their tuning data \nare such that it's actually a static value whatever the lux...but perhaps it's better to be \nconsistent in how we treat it.\n\n> +\t\tAgcConstraint constraint = { bound, qLo, qHi, yTarget };\n> +\n> +\t\tif (!constraintModes_.count(id))\n> +\t\t\tconstraintModes_[id] = {};\n> +\n> +\t\tif (idx)\n> +\t\t\tconstraintModes_[id].push_back(constraint);\n> +\t\telse\n> +\t\t\tconstraintModes_[id].insert(constraintModes_[id].begin(), constraint);\n> +\t}\n> +}\n> +\n> +int AgcMeanLuminance::parseConstraintModes(const YamlObject &tuningData)\n> +{\n> +\tstd::vector availableConstraintModes;\n> +\n> +\tconst YamlObject &yamlConstraintModes = tuningData[controls::AeConstraintMode.name()];\n> +\tif (yamlConstraintModes.isDictionary()) {\n> +\t\tfor (const auto &[modeName, modeDict] : yamlConstraintModes.asDict()) {\n> +\t\t\tif (AeConstraintModeNameValueMap.find(modeName) ==\n> +\t\t\t AeConstraintModeNameValueMap.end()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Warning)\n> +\t\t\t\t\t<< \"Skipping unknown constraint mode '\" << modeName << \"'\";\n> +\t\t\t\tcontinue;\n> +\t\t\t}\n> +\n> +\t\t\tif (!modeDict.isDictionary()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> +\t\t\t\t\t<< \"Invalid constraint mode '\" << modeName << \"'\";\n> +\t\t\t\treturn -EINVAL;\n> +\t\t\t}\n> +\n> +\t\t\tparseConstraint(modeDict,\n> +\t\t\t\t\tAeConstraintModeNameValueMap.at(modeName));\n> +\t\t\tavailableConstraintModes.push_back(\n> +\t\t\t\tAeConstraintModeNameValueMap.at(modeName));\n> +\t\t}\n> +\t}\n> +\n> +\t/*\n> +\t * If the tuning data file contains no constraints then we use the\n> +\t * default constraint that the various Agc algorithms were adhering to\n> +\t * anyway before centralisation.\n> +\t */\n> +\tif (constraintModes_.empty()) {\n> +\t\tAgcConstraint constraint = {\n> +\t\t\tAgcConstraint::Bound::lower,\n> +\t\t\t0.98,\n> +\t\t\t1.0,\n> +\t\t\t0.5\n> +\t\t};\n> +\n> +\t\tconstraintModes_[controls::ConstraintNormal].insert(\n> +\t\t\tconstraintModes_[controls::ConstraintNormal].begin(),\n> +\t\t\tconstraint);\n> +\t\tavailableConstraintModes.push_back(\n> +\t\t\tAeConstraintModeNameValueMap.at(\"ConstraintNormal\"));\n> +\t}\n> +\n> +\tcontrols_[&controls::AeConstraintMode] = ControlInfo(availableConstraintModes);\n> +\n> +\treturn 0;\n> +}\n> +\n> +int AgcMeanLuminance::parseExposureModes(const YamlObject &tuningData)\n> +{\n> +\tstd::vector availableExposureModes;\n> +\n> +\tconst YamlObject &yamlExposureModes = tuningData[controls::AeExposureMode.name()];\n> +\tif (yamlExposureModes.isDictionary()) {\n> +\t\tfor (const auto &[modeName, modeValues] : yamlExposureModes.asDict()) {\n> +\t\t\tif (AeExposureModeNameValueMap.find(modeName) ==\n> +\t\t\t AeExposureModeNameValueMap.end()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Warning)\n> +\t\t\t\t\t<< \"Skipping unknown exposure mode '\" << modeName << \"'\";\n> +\t\t\t\tcontinue;\n> +\t\t\t}\n> +\n> +\t\t\tif (!modeValues.isDictionary()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> +\t\t\t\t\t<< \"Invalid exposure mode '\" << modeName << \"'\";\n> +\t\t\t\treturn -EINVAL;\n> +\t\t\t}\n> +\n> +\t\t\tstd::vector shutters =\n> +\t\t\t\tmodeValues[\"shutter\"].getList().value_or(std::vector{});\n> +\t\t\tstd::vector gains =\n> +\t\t\t\tmodeValues[\"gain\"].getList().value_or(std::vector{});\n> +\n> +\t\t\tif (shutters.size() != gains.size()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> +\t\t\t\t\t<< \"Shutter and gain array sizes unequal\";\n> +\t\t\t\treturn -EINVAL;\n> +\t\t\t}\n> +\n> +\t\t\tif (shutters.empty()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> +\t\t\t\t\t<< \"Shutter and gain arrays are empty\";\n> +\t\t\t\treturn -EINVAL;\n> +\t\t\t}\n> +\n> +\t\t\tstd::vector> stages;\n> +\t\t\tfor (unsigned int i = 0; i < shutters.size(); i++) {\n> +\t\t\t\tstages.push_back({\n> +\t\t\t\t\tstd::chrono::microseconds(shutters[i]),\n> +\t\t\t\t\tgains[i]\n> +\t\t\t\t});\n> +\t\t\t}\n> +\n> +\t\t\tstd::shared_ptr helper =\n> +\t\t\t\tstd::make_shared();\n> +\t\t\thelper->init(stages);\n> +\n> +\t\t\texposureModeHelpers_[AeExposureModeNameValueMap.at(modeName)] = helper;\n> +\t\t\tavailableExposureModes.push_back(AeExposureModeNameValueMap.at(modeName));\n> +\t\t}\n> +\t}\n> +\n> +\t/*\n> +\t * If we don't have any exposure modes in the tuning data we create an\n> +\t * ExposureModeHelper using an empty vector of stages. This will result\n> +\t * in the ExposureModeHelper simply driving the shutter as high as\n> +\t * possible before touching gain.\n> +\t */\n> +\tif (availableExposureModes.empty()) {\n> +\t\tint32_t exposureModeId = AeExposureModeNameValueMap.at(\"ExposureNormal\");\n> +\t\tstd::vector> stages = { };\n> +\n> +\t\tstd::shared_ptr helper =\n> +\t\t\tstd::make_shared();\n> +\t\thelper->init(stages);\n> +\n> +\t\texposureModeHelpers_[exposureModeId] = helper;\n> +\t\tavailableExposureModes.push_back(exposureModeId);\n> +\t}\n> +\n> +\tcontrols_[&controls::AeExposureMode] = ControlInfo(availableExposureModes);\n> +\n> +\treturn 0;\n> +}\n> +\n> +/**\n> + * \\brief Parse tuning data for AeConstraintMode and AeExposureMode controls\n> + * \\param[in] tuningData the YamlObject representing the tuning data\n> + *\n> + * This function parses tuning data to build the list of allowed values for the\n> + * AeConstraintMode and AeExposureMode controls. Those tuning data must provide\n> + * the data in a specific format; the Agc algorithm's tuning data should contain\n> + * a dictionary called AeConstraintMode containing per-mode setting dictionaries\n> + * with the key being a value from \\ref controls::AeConstraintModeNameValueMap.\n> + * Each mode dict may contain either a \"lower\" or \"upper\" key or both, for\n> + * example:\n> + *\n> + * \\code{.unparsed}\n> + * algorithms:\n> + * - Agc:\n> + * AeConstraintMode:\n> + * ConstraintNormal:\n> + * lower:\n> + * qLo: 0.98\n> + * qHi: 1.0\n> + * yTarget: 0.5\n> + * ConstraintHighlight:\n> + * lower:\n> + * qLo: 0.98\n> + * qHi: 1.0\n> + * yTarget: 0.5\n> + * upper:\n> + * qLo: 0.98\n> + * qHi: 1.0\n> + * yTarget: 0.8\n> + *\n> + * \\endcode\n> + *\n> + * For the AeExposureMode control the data should contain a dictionary called\n> + * AeExposureMode containing per-mode setting dictionaries with the key being a\n> + * value from \\ref controls::AeExposureModeNameValueMap. Each mode dict should\n> + * contain an array of shutter times with the key \"shutter\" and an array of gain\n> + * values with the key \"gain\", in this format:\n> + *\n> + * \\code{.unparsed}\n> + * algorithms:\n> + * - Agc:\n> + * AeExposureMode:\n> + * ExposureNormal:\n> + * shutter: [ 100, 10000, 30000, 60000, 120000 ]\n> + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ]\n> + * ExposureShort:\n> + * shutter: [ 100, 10000, 30000, 60000, 120000 ]\n> + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ]\n> + *\n> + * \\endcode\n> + *\n> + * @return 0 on success or a negative error code\n> + */\n> +int AgcMeanLuminance::parseTuningData(const YamlObject &tuningData)\n> +{\n> +\tint ret;\n> +\n> +\tparseRelativeLuminanceTarget(tuningData);\n> +\n> +\tret = parseConstraintModes(tuningData);\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n> +\tret = parseExposureModes(tuningData);\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n> +\treturn 0;\n> +}\n> +\n> +/**\n> + * \\brief configure the ExposureModeHelpers for this class\n> + * \\param[in] minShutter Minimum shutter time to allow\n> + * \\param[in] maxShutter Maximum shutter time to allow\n> + * \\param[in] minGain Minimum gain to allow\n> + * \\param[in] maxGain Maximum gain to allow\n> + *\n> + * This function calls \\ref ExposureModeHelper::setShutterGainLimits() for each\n> + * ExposureModeHelper that has been created for this class.\n> + */\n> +void AgcMeanLuminance::configureExposureModeHelpers(utils::Duration minShutter,\n> +\t\t\t\t\t\t utils::Duration maxShutter,\n> +\t\t\t\t\t\t double minGain,\n> +\t\t\t\t\t\t double maxGain)\n> +{\n> +\tfor (auto &[id, helper] : exposureModeHelpers_)\n> +\t\thelper->setShutterGainLimits(minShutter, maxShutter, minGain, maxGain);\n> +}\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::constraintModes()\n> + * \\brief Get the constraint modes that have been parsed from tuning data\n> + */\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::exposureModeHelpers()\n> + * \\brief Get the ExposureModeHelpers that have been parsed from tuning data\n> + */\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::controls()\n> + * \\brief Get the controls that have been generated after parsing tuning data\n> + */\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::estimateLuminance(const double gain)\n> + * \\brief Estimate the luminance of an image, adjusted by a given gain\n> + * \\param[in] gain The gain with which to adjust the luminance estimate\n> + *\n> + * This function estimates the average relative luminance of the frame that\n> + * would be output by the sensor if an additional \\a gain was applied. It is a\n> + * pure virtual function because estimation of luminance is a hardware-specific\n> + * operation, which depends wholly on the format of the stats that are delivered\n> + * to libcamera from the ISP. Derived classes must implement an overriding\n> + * function that calculates the normalised mean luminance value across the\n> + * entire image.\n> + *\n> + * \\return The normalised relative luminance of the image\n> + */\n> +\n> +/**\n> + * \\brief Estimate the initial gain needed to achieve a relative luminance\n> + * target\n> + *\n> + * To account for non-linearity caused by saturation, the value needs to be\n> + * estimated in an iterative process, as multiplying by a gain will not increase\n> + * the relative luminance by the same factor if some image regions are saturated\n> + *\n> + * \\return The calculated initial gain\n> + */\n> +double AgcMeanLuminance::estimateInitialGain()\n> +{\n> +\tdouble yTarget = relativeLuminanceTarget_;\n> +\tdouble yGain = 1.0;\n> +\n> +\tfor (unsigned int i = 0; i < 8; i++) {\n> +\t\tdouble yValue = estimateLuminance(yGain);\n> +\t\tdouble extra_gain = std::min(10.0, yTarget / (yValue + .001));\n> +\n> +\t\tyGain *= extra_gain;\n> +\t\tLOG(AgcMeanLuminance, Debug) << \"Y value: \" << yValue\n> +\t\t\t\t<< \", Y target: \" << yTarget\n> +\t\t\t\t<< \", gives gain \" << yGain;\n> +\n> +\t\tif (utils::abs_diff(extra_gain, 1.0) < 0.01)\n> +\t\t\tbreak;\n> +\t}\n> +\n> +\treturn yGain;\n> +}\n> +\n> +/**\n> + * \\brief Clamp gain within the bounds of a defined constraint\n> + * \\param[in] constraintModeIndex The index of the constraint to adhere to\n> + * \\param[in] hist A histogram over which to calculate inter-quantile means\n> + * \\param[in] gain The gain to clamp\n> + *\n> + * \\return The gain clamped within the constraint bounds\n> + */\n> +double AgcMeanLuminance::constraintClampGain(uint32_t constraintModeIndex,\n> +\t\t\t\t\t const Histogram &hist,\n> +\t\t\t\t\t double gain)\n> +{\n> +\tstd::vector &constraints = constraintModes_[constraintModeIndex];\n> +\tfor (const AgcConstraint &constraint : constraints) {\n> +\t\tdouble newGain = constraint.yTarget * hist.bins() /\n> +\t\t\t\t hist.interQuantileMean(constraint.qLo, constraint.qHi);\n> +\n> +\t\tif (constraint.bound == AgcConstraint::Bound::lower &&\n> +\t\t newGain > gain)\n> +\t\t\tgain = newGain;\n> +\n> +\t\tif (constraint.bound == AgcConstraint::Bound::upper &&\n> +\t\t newGain < gain)\n> +\t\t\tgain = newGain;\n> +\t}\n> +\n> +\treturn gain;\n> +}\n> +\n> +/**\n> + * \\brief Apply a filter on the exposure value to limit the speed of changes\n> + * \\param[in] exposureValue The target exposure from the AGC algorithm\n> + *\n> + * The speed of the filter is adaptive, and will produce the target quicker\n> + * during startup, or when the target exposure is within 20% of the most recent\n> + * filter output.\n> + *\n> + * \\return The filtered exposure\n> + */\n> +utils::Duration AgcMeanLuminance::filterExposure(utils::Duration exposureValue)\n> +{\n> +\tdouble speed = 0.2;\n> +\n> +\t/* Adapt instantly if we are in startup phase. */\n> +\tif (frameCount_ < kNumStartupFrames)\n> +\t\tspeed = 1.0;\n> +\n> +\t/*\n> +\t * If we are close to the desired result, go faster to avoid making\n> +\t * multiple micro-adjustments.\n> +\t * \\todo Make this customisable?\n> +\t */\n> +\tif (filteredExposure_ < 1.2 * exposureValue &&\n> +\t filteredExposure_ > 0.8 * exposureValue)\n> +\t\tspeed = sqrt(speed);\n> +\n> +\tfilteredExposure_ = speed * exposureValue +\n> +\t\t\t filteredExposure_ * (1.0 - speed);\n> +\n> +\treturn filteredExposure_;\n> +}\n> +\n> +/**\n> + * \\brief Calculate the new exposure value\n> + * \\param[in] constraintModeIndex The index of the current constraint mode\n> + * \\param[in] exposureModeIndex The index of the current exposure mode\n> + * \\param[in] yHist A Histogram from the ISP statistics to use in constraining\n> + *\t the calculated gain\n> + * \\param[in] effectiveExposureValue The EV applied to the frame from which the\n> + *\t statistics in use derive\n> + *\n> + * Calculate a new exposure value to try to obtain the target. The calculated\n> + * exposure value is filtered to prevent rapid changes from frame to frame, and\n> + * divided into shutter time, analogue and digital gain.\n> + *\n> + * \\return Tuple of shutter time, analogue gain, and digital gain\n> + */\n> +std::tuple\n> +AgcMeanLuminance::calculateNewEv(uint32_t constraintModeIndex,\n> +\t\t\t\t uint32_t exposureModeIndex,\n> +\t\t\t\t const Histogram &yHist,\n> +\t\t\t\t utils::Duration effectiveExposureValue)\n> +{\n> +\t/*\n> +\t * The pipeline handler should validate that we have received an allowed\n> +\t * value for AeExposureMode.\n> +\t */\n> +\tstd::shared_ptr exposureModeHelper =\n> +\t\texposureModeHelpers_.at(exposureModeIndex);\n> +\n> +\tdouble gain = estimateInitialGain();\n> +\tgain = constraintClampGain(constraintModeIndex, yHist, gain);\n> +\n> +\t/*\n> +\t * We don't check whether we're already close to the target, because\n> +\t * even if the effective exposure value is the same as the last frame's\n> +\t * we could have switched to an exposure mode that would require a new\n> +\t * pass through the splitExposure() function.\n> +\t */\n> +\n> +\tutils::Duration newExposureValue = effectiveExposureValue * gain;\n> +\tutils::Duration maxTotalExposure = exposureModeHelper->maxShutter()\n> +\t\t\t\t\t * exposureModeHelper->maxGain();\n> +\tnewExposureValue = std::min(newExposureValue, maxTotalExposure);\n> +\n> +\t/*\n> +\t * We filter the exposure value to make sure changes are not too jarring\n> +\t * from frame to frame.\n> +\t */\n> +\tnewExposureValue = filterExposure(newExposureValue);\n> +\n> +\tframeCount_++;\n> +\treturn exposureModeHelper->splitExposure(newExposureValue);\n> +}\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::resetFrameCount()\n> + * \\brief Reset the frame counter\n> + *\n> + * This function resets the internal frame counter, which exists to help the\n> + * algorithm decide whether it should respond instantly or not. The expectation\n> + * is for derived classes to call this function before each camera start call,\n> + * either in configure() or queueRequest() if the frame number is zero.\n> + */\n> +\n> +}; /* namespace ipa */\n> +\n> +}; /* namespace libcamera */\n> diff --git a/src/ipa/libipa/agc_mean_luminance.h b/src/ipa/libipa/agc_mean_luminance.h\n> new file mode 100644\n> index 00000000..e48dc498\n> --- /dev/null\n> +++ b/src/ipa/libipa/agc_mean_luminance.h\n> @@ -0,0 +1,91 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2024 Ideas on Board Oy\n> + *\n> + agc_mean_luminance.h - Base class for mean luminance AGC algorithms\n> + */\n> +\n> +#pragma once\n> +\n> +#include \n> +#include \n> +\n> +#include \n> +\n> +#include \"libcamera/internal/yaml_parser.h\"\n> +\n> +#include \"exposure_mode_helper.h\"\n> +#include \"histogram.h\"\n> +\n> +namespace libcamera {\n> +\n> +namespace ipa {\n> +\n> +class AgcMeanLuminance\n> +{\n> +public:\n> +\tAgcMeanLuminance();\n> +\tvirtual ~AgcMeanLuminance() = default;\n> +\n> +\tstruct AgcConstraint {\n> +\t\tenum class Bound {\n> +\t\t\tlower = 0,\n> +\t\t\tupper = 1\n> +\t\t};\n> +\t\tBound bound;\n> +\t\tdouble qLo;\n> +\t\tdouble qHi;\n> +\t\tdouble yTarget;\n> +\t};\n> +\n> +\tint parseTuningData(const YamlObject &tuningData);\n> +\n> +\tvoid configureExposureModeHelpers(utils::Duration minShutter,\n> +\t\t\t\t\t utils::Duration maxShutter,\n> +\t\t\t\t\t double minGain,\n> +\t\t\t\t\t double maxGain);\n> +\n> +\tstd::map> constraintModes()\n> +\t{\n> +\t\treturn constraintModes_;\n> +\t}\n> +\n> +\tstd::map> exposureModeHelpers()\n> +\t{\n> +\t\treturn exposureModeHelpers_;\n> +\t}\n> +\n> +\tControlInfoMap::Map controls()\n> +\t{\n> +\t\treturn controls_;\n> +\t}\n> +\n> +\tdouble estimateInitialGain();\n> +\tdouble constraintClampGain(uint32_t constraintModeIndex,\n> +\t\t\t\t const Histogram &hist,\n> +\t\t\t\t double gain);\n> +\tutils::Duration filterExposure(utils::Duration exposureValue);\n> +\tstd::tuple\n> +\tcalculateNewEv(uint32_t constraintModeIndex, uint32_t exposureModeIndex,\n> +\t\t const Histogram &yHist, utils::Duration effectiveExposureValue);\n> +\tvoid resetFrameCount() { frameCount_ = 0; }\n> +private:\n> +\tvirtual double estimateLuminance(const double gain) = 0;\n> +\n> +\tvoid parseRelativeLuminanceTarget(const YamlObject &tuningData);\n> +\tvoid parseConstraint(const YamlObject &modeDict, int32_t id);\n> +\tint parseConstraintModes(const YamlObject &tuningData);\n> +\tint parseExposureModes(const YamlObject &tuningData);\n> +\n> +\tuint64_t frameCount_;\n> +\tutils::Duration filteredExposure_;\n> +\tdouble relativeLuminanceTarget_;\n> +\n> +\tstd::map> constraintModes_;\n> +\tstd::map> exposureModeHelpers_;\n> +\tControlInfoMap::Map controls_;\n> +};\n> +\n> +}; /* namespace ipa */\n> +\n> +}; /* namespace libcamera */\n> diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build\n> index 37fbd177..7ce885da 100644\n> --- a/src/ipa/libipa/meson.build\n> +++ b/src/ipa/libipa/meson.build\n> @@ -1,6 +1,7 @@\n> # SPDX-License-Identifier: CC0-1.0\n> \n> libipa_headers = files([\n> + 'agc_mean_luminance.h',\n> 'algorithm.h',\n> 'camera_sensor_helper.h',\n> 'exposure_mode_helper.h',\n> @@ -10,6 +11,7 @@ libipa_headers = files([\n> ])\n> \n> libipa_sources = files([\n> + 'agc_mean_luminance.cpp',\n> 'algorithm.cpp',\n> 'camera_sensor_helper.cpp',\n> 'exposure_mode_helper.cpp',","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 33109C3200\n\tfor ;\n\tThu, 18 Apr 2024 15:03:30 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 590706334D;\n\tThu, 18 Apr 2024 17:03:29 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 5EB1E61B4F\n\tfor ;\n\tThu, 18 Apr 2024 17:03:27 +0200 (CEST)","from [192.168.0.43]\n\t(cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 6EAD9802;\n\tThu, 18 Apr 2024 17:02:39 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"JSjfWe/E\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1713452559;\n\tbh=UBGvqnuGtzxh9m+3uHiw+5HExO/3IIqO14gv5OkFO4o=;\n\th=Date:Subject:To:References:Cc:From:In-Reply-To:From;\n\tb=JSjfWe/EHGXkFR38CFxJZfe1uEdqQZuNFcmlxUC/wupkClPgY6Goj7cT3CeBEuEZm\n\tZp7y71E1UaMD7mt1tnV3/0Ls9MF8+M/Ry73PGFeuLRYNvCdAhSNE6WGPbu/bghMp6P\n\tP/c1ZW3NpAEVqde6uIV5NzOd68PmWJkjkpwjARpA=","Message-ID":"","Date":"Thu, 18 Apr 2024 16:03:24 +0100","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH v2 4/8] ipa: libipa: Add MeanLuminanceAgc base class","To":"Paul Elder ","References":"<20240417131536.484129-1-dan.scally@ideasonboard.com>\n\t<20240417131536.484129-5-dan.scally@ideasonboard.com>","Content-Language":"en-US","Cc":"Jacopo Mondi via libcamera-devel ","From":"Dan Scally ","Autocrypt":"addr=dan.scally@ideasonboard.com; keydata=\n\txsFNBGLydlEBEADa5O2s0AbUguprfvXOQun/0a8y2Vk6BqkQALgeD6KnXSWwaoCULp18etYW\n\tB31bfgrdphXQ5kUQibB0ADK8DERB4wrzrUb5CMxLBFE7mQty+v5NsP0OFNK9XTaAOcmD+Ove\n\teIjYvqurAaro91jrRVrS1gBRxIFqyPgNvwwL+alMZhn3/2jU2uvBmuRrgnc/e9cHKiuT3Dtq\n\tMHGPKL2m+plk+7tjMoQFfexoQ1JKugHAjxAhJfrkXh6uS6rc01bYCyo7ybzg53m1HLFJdNGX\n\tsUKR+dQpBs3SY4s66tc1sREJqdYyTsSZf80HjIeJjU/hRunRo4NjRIJwhvnK1GyjOvvuCKVU\n\tRWpY8dNjNu5OeAfdrlvFJOxIE9M8JuYCQTMULqd1NuzbpFMjc9524U3Cngs589T7qUMPb1H1\n\tNTA81LmtJ6Y+IV5/kiTUANflpzBwhu18Ok7kGyCq2a2jsOcVmk8gZNs04gyjuj8JziYwwLbf\n\tvzABwpFVcS8aR+nHIZV1HtOzyw8CsL8OySc3K9y+Y0NRpziMRvutrppzgyMb9V+N31mK9Mxl\n\t1YkgaTl4ciNWpdfUe0yxH03OCuHi3922qhPLF4XX5LN+NaVw5Xz2o3eeWklXdouxwV7QlN33\n\tu4+u2FWzKxDqO6WLQGjxPE0mVB4Gh5Pa1Vb0ct9Ctg0qElvtGQARAQABzShEYW4gU2NhbGx5\n\tIDxkYW4uc2NhbGx5QGlkZWFzb25ib2FyZC5jb20+wsGNBBMBCAA3FiEEsdtt8OWP7+8SNfQe\n\tkiQuh/L+GMQFAmLydlIFCQWjmoACGwMECwkIBwUVCAkKCwUWAgMBAAAKCRCSJC6H8v4YxDI2\n\tEAC2Gz0iyaXJkPInyshrREEWbo0CA6v5KKf3I/HlMPqkZ48bmGoYm4mEQGFWZJAT3K4ir8bg\n\tcEfs9V54gpbrZvdwS4abXbUK4WjKwEs8HK3XJv1WXUN2bsz5oEJWZUImh9gD3naiLLI9QMMm\n\tw/aZkT+NbN5/2KvChRWhdcha7+2Te4foOY66nIM+pw2FZM6zIkInLLUik2zXOhaZtqdeJZQi\n\tHSPU9xu7TRYN4cvdZAnSpG7gQqmLm5/uGZN1/sB3kHTustQtSXKMaIcD/DMNI3JN/t+RJVS7\n\tc0Jh/ThzTmhHyhxx3DRnDIy7kwMI4CFvmhkVC2uNs9kWsj1DuX5kt8513mvfw2OcX9UnNKmZ\n\tnhNCuF6DxVrL8wjOPuIpiEj3V+K7DFF1Cxw1/yrLs8dYdYh8T8vCY2CHBMsqpESROnTazboh\n\tAiQ2xMN1cyXtX11Qwqm5U3sykpLbx2BcmUUUEAKNsM//Zn81QXKG8vOx0ZdMfnzsCaCzt8f6\n\t9dcDBBI3tJ0BI9ByiocqUoL6759LM8qm18x3FYlxvuOs4wSGPfRVaA4yh0pgI+ModVC2Pu3y\n\tejE/IxeatGqJHh6Y+iJzskdi27uFkRixl7YJZvPJAbEn7kzSi98u/5ReEA8Qhc8KO/B7wprj\n\txjNMZNYd0Eth8+WkixHYj752NT5qshKJXcyUU87BTQRi8nZSARAAx0BJayh1Fhwbf4zoY56x\n\txHEpT6DwdTAYAetd3yiKClLVJadYxOpuqyWa1bdfQWPb+h4MeXbWw/53PBgn7gI2EA7ebIRC\n\tPJJhAIkeym7hHZoxqDQTGDJjxFEL11qF+U3rhWiL2Zt0Pl+zFq0eWYYVNiXjsIS4FI2+4m16\n\ttPbDWZFJnSZ828VGtRDQdhXfx3zyVX21lVx1bX4/OZvIET7sVUufkE4hrbqrrufre7wsjD1t\n\t8MQKSapVrr1RltpzPpScdoxknOSBRwOvpp57pJJe5A0L7+WxJ+vQoQXj0j+5tmIWOAV1qBQp\n\thyoyUk9JpPfntk2EKnZHWaApFp5TcL6c5LhUvV7F6XwOjGPuGlZQCWXee9dr7zym8iR3irWT\n\t+49bIh5PMlqSLXJDYbuyFQHFxoiNdVvvf7etvGfqFYVMPVjipqfEQ38ST2nkzx+KBICz7uwj\n\tJwLBdTXzGFKHQNckGMl7F5QdO/35An/QcxBnHVMXqaSd12tkJmoRVWduwuuoFfkTY5mUV3uX\n\txGj3iVCK4V+ezOYA7c2YolfRCNMTza6vcK/P4tDjjsyBBZrCCzhBvd4VVsnnlZhVaIxoky4K\n\taL+AP+zcQrUZmXmgZjXOLryGnsaeoVrIFyrU6ly90s1y3KLoPsDaTBMtnOdwxPmo1xisH8oL\n\ta/VRgpFBfojLPxMAEQEAAcLBfAQYAQgAJhYhBLHbbfDlj+/vEjX0HpIkLofy/hjEBQJi8nZT\n\tBQkFo5qAAhsMAAoJEJIkLofy/hjEXPcQAMIPNqiWiz/HKu9W4QIf1OMUpKn3YkVIj3p3gvfM\n\tRes4fGX94Ji599uLNrPoxKyaytC4R6BTxVriTJjWK8mbo9jZIRM4vkwkZZ2bu98EweSucxbp\n\tvjESsvMXGgxniqV/RQ/3T7LABYRoIUutARYq58p5HwSP0frF0fdFHYdTa2g7MYZl1ur2JzOC\n\tFHRpGadlNzKDE3fEdoMobxHB3Lm6FDml5GyBAA8+dQYVI0oDwJ3gpZPZ0J5Vx9RbqXe8RDuR\n\tdu90hvCJkq7/tzSQ0GeD3BwXb9/R/A4dVXhaDd91Q1qQXidI+2jwhx8iqiYxbT+DoAUkQRQy\n\txBtoCM1CxH7u45URUgD//fxYr3D4B1SlonA6vdaEdHZOGwECnDpTxecENMbz/Bx7qfrmd901\n\tD+N9SjIwrbVhhSyUXYnSUb8F+9g2RDY42Sk7GcYxIeON4VzKqWM7hpkXZ47pkK0YodO+dRKM\n\tyMcoUWrTK0Uz6UzUGKoJVbxmSW/EJLEGoI5p3NWxWtScEVv8mO49gqQdrRIOheZycDmHnItt\n\t9Qjv00uFhEwv2YfiyGk6iGF2W40s2pH2t6oeuGgmiZ7g6d0MEK8Ql/4zPItvr1c1rpwpXUC1\n\tu1kQWgtnNjFHX3KiYdqjcZeRBiry1X0zY+4Y24wUU0KsEewJwjhmCKAsju1RpdlPg2kC","In-Reply-To":"<20240417131536.484129-5-dan.scally@ideasonboard.com>","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"7bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":29270,"web_url":"https://patchwork.libcamera.org/comment/29270/","msgid":"","date":"2024-04-19T04:51:06","subject":"Re: [PATCH v2 4/8] ipa: libipa: Add MeanLuminanceAgc base class","submitter":{"id":17,"url":"https://patchwork.libcamera.org/api/people/17/","name":"Paul Elder","email":"paul.elder@ideasonboard.com"},"content":"It just occurred to me that the commit message still says\nMeanLuminanceAgc instead of AgcMeanLuminance.\n\n\nPaul\n\nOn Thu, Apr 18, 2024 at 09:49:34PM +0900, Paul Elder wrote:\n> Hi Dan,\n> \n> On Wed, Apr 17, 2024 at 02:15:32PM +0100, Daniel Scally wrote:\n> > The Agc algorithms for the RkIsp1 and IPU3 IPAs do the same thing in\n> > very large part; following the Rpi IPA's algorithm in spirit with a\n> > few tunable values in that IPA being hardcoded in the libipa ones.\n> > Add a new base class for MeanLuminanceAgc which implements the same\n> > algorithm and additionally parses yaml tuning files to inform an IPA\n> > module's Agc algorithm about valid constraint and exposure modes and\n> > their associated bounds.\n> > \n> > Signed-off-by: Daniel Scally \n> > ---\n> > Changes in v2:\n> > \n> > \t- Renamed the class and files\n> > \t- Expanded the documentation\n> > \t- Added parseTuningData() so derived classes can call a single function\n> > \t to cover all the parsing in ::init().\n> > \n> > src/ipa/libipa/agc_mean_luminance.cpp | 581 ++++++++++++++++++++++++++\n> > src/ipa/libipa/agc_mean_luminance.h | 91 ++++\n> > src/ipa/libipa/meson.build | 2 +\n> > 3 files changed, 674 insertions(+)\n> > create mode 100644 src/ipa/libipa/agc_mean_luminance.cpp\n> > create mode 100644 src/ipa/libipa/agc_mean_luminance.h\n> > \n> > diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp\n> > new file mode 100644\n> > index 00000000..02e223cf\n> > --- /dev/null\n> > +++ b/src/ipa/libipa/agc_mean_luminance.cpp\n> > @@ -0,0 +1,581 @@\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) 2024 Ideas on Board Oy\n> > + *\n> > + * agc_mean_luminance.cpp - Base class for mean luminance AGC algorithms\n> > + */\n> > +\n> > +#include \"agc_mean_luminance.h\"\n> > +\n> > +#include \n> > +\n> > +#include \n> > +#include \n> > +\n> > +#include \"exposure_mode_helper.h\"\n> > +\n> > +using namespace libcamera::controls;\n> > +\n> > +/**\n> > + * \\file agc_mean_luminance.h\n> > + * \\brief Base class implementing mean luminance AEGC\n> > + */\n> > +\n> > +namespace libcamera {\n> > +\n> > +using namespace std::literals::chrono_literals;\n> > +\n> > +LOG_DEFINE_CATEGORY(AgcMeanLuminance)\n> > +\n> > +namespace ipa {\n> > +\n> > +/*\n> > + * Number of frames for which to run the algorithm at full speed, before slowing\n> > + * down to prevent large and jarring changes in exposure from frame to frame.\n> > + */\n> > +static constexpr uint32_t kNumStartupFrames = 10;\n> > +\n> > +/*\n> > + * Default relative luminance target\n> > + *\n> > + * This value should be chosen so that when the camera points at a grey target,\n> > + * the resulting image brightness looks \"right\". Custom values can be passed\n> > + * as the relativeLuminanceTarget value in sensor tuning files.\n> > + */\n> > +static constexpr double kDefaultRelativeLuminanceTarget = 0.16;\n> > +\n> > +/**\n> > + * \\struct AgcMeanLuminance::AgcConstraint\n> > + * \\brief The boundaries and target for an AeConstraintMode constraint\n> > + *\n> > + * This structure describes an AeConstraintMode constraint for the purposes of\n> > + * this algorithm. The algorithm will apply the constraints by calculating the\n> > + * Histogram's inter-quantile mean between the given quantiles and ensure that\n> > + * the resulting value is the right side of the given target (as defined by the\n> > + * boundary and luminance target).\n> > + */\n> > +\n> > +/**\n> > + * \\enum AgcMeanLuminance::AgcConstraint::Bound\n> > + * \\brief Specify whether the constraint defines a lower or upper bound\n> > + * \\var AgcMeanLuminance::AgcConstraint::lower\n> > + * \\brief The constraint defines a lower bound\n> > + * \\var AgcMeanLuminance::AgcConstraint::upper\n> > + * \\brief The constraint defines an upper bound\n> > + */\n> > +\n> > +/**\n> > + * \\var AgcMeanLuminance::AgcConstraint::bound\n> > + * \\brief The type of constraint bound\n> > + */\n> > +\n> > +/**\n> > + * \\var AgcMeanLuminance::AgcConstraint::qLo\n> > + * \\brief The lower quantile to use for the constraint\n> > + */\n> > +\n> > +/**\n> > + * \\var AgcMeanLuminance::AgcConstraint::qHi\n> > + * \\brief The upper quantile to use for the constraint\n> > + */\n> > +\n> > +/**\n> > + * \\var AgcMeanLuminance::AgcConstraint::yTarget\n> > + * \\brief The luminance target for the constraint\n> > + */\n> > +\n> > +/**\n> > + * \\class AgcMeanLuminance\n> > + * \\brief A mean-based auto-exposure algorithm\n> > + *\n> > + * This algorithm calculates a shutter time, analogue and digital gain such that\n> > + * the normalised mean luminance value of an image is driven towards a target,\n> > + * which itself is discovered from tuning data. The algorithm is a two-stage\n> > + * process.\n> > + *\n> > + * In the first stage, an initial gain value is derived by iteratively comparing\n> > + * the gain-adjusted mean luminance across an entire image against a target, and\n> > + * selecting a value which pushes it as closely as possible towards the target.\n> > + *\n> > + * In the second stage we calculate the gain required to drive the average of a\n> > + * section of a histogram to a target value, where the target and the boundaries\n> > + * of the section of the histogram used in the calculation are taken from the\n> > + * values defined for the currently configured AeConstraintMode within the\n> > + * tuning data. This class provides a helper function to parse those tuning data\n> > + * to discover the constraints, and so requires a specific format for those\n> > + * data which is described in \\ref parseTuningData(). The gain from the first\n> > + * stage is then clamped to the gain from this stage.\n> > + *\n> > + * The final gain is used to adjust the effective exposure value of the image,\n> > + * and that new exposure value is divided into shutter time, analogue gain and\n> > + * digital gain according to the selected AeExposureMode. This class expects to\n> > + * use the \\ref ExposureModeHelper class to assist in that division, and expects\n> > + * the data needed to initialise that class to be present in tuning data in a\n> > + * format described in \\ref parseTuningData().\n> > + *\n> > + * In order to be able to derive an AGC implementation from this class, an IPA\n> > + * needs to be able to do the following:\n> > + *\n> > + * 1. Provide a luminance estimation across an entire image.\n> > + * 2. Provide a luminance Histogram for the image to use in calculating\n> > + * constraint compliance. The precision of the Histogram that is available\n> > + * will determine the supportable precision of the constraints.\n> > + */\n> > +\n> > +AgcMeanLuminance::AgcMeanLuminance()\n> > +\t: frameCount_(0), filteredExposure_(0s), relativeLuminanceTarget_(0)\n> > +{\n> > +}\n> > +\n> > +/**\n> > + * \\brief Parse the relative luminance target from the tuning data\n> > + * \\param[in] tuningData The YamlObject holding the algorithm's tuning data\n> > + */\n> > +void AgcMeanLuminance::parseRelativeLuminanceTarget(const YamlObject &tuningData)\n> > +{\n> > +\trelativeLuminanceTarget_ =\n> > +\t\ttuningData[\"relativeLuminanceTarget\"].get(kDefaultRelativeLuminanceTarget);\n> > +}\n> > +\n> > +/**\n> > + * \\brief Parse an AeConstraintMode constraint from tuning data\n> > + * \\param[in] modeDict the YamlObject holding the constraint data\n> > + * \\param[in] id The constraint ID from AeConstraintModeEnum\n> > + */\n> > +void AgcMeanLuminance::parseConstraint(const YamlObject &modeDict, int32_t id)\n> > +{\n> > +\tfor (const auto &[boundName, content] : modeDict.asDict()) {\n> > +\t\tif (boundName != \"upper\" && boundName != \"lower\") {\n> > +\t\t\tLOG(AgcMeanLuminance, Warning)\n> > +\t\t\t\t<< \"Ignoring unknown constraint bound '\" << boundName << \"'\";\n> > +\t\t\tcontinue;\n> > +\t\t}\n> > +\n> > +\t\tunsigned int idx = static_cast(boundName == \"upper\");\n> > +\t\tAgcConstraint::Bound bound = static_cast(idx);\n> > +\t\tdouble qLo = content[\"qLo\"].get().value_or(0.98);\n> > +\t\tdouble qHi = content[\"qHi\"].get().value_or(1.0);\n> > +\t\tdouble yTarget =\n> > +\t\t\tcontent[\"yTarget\"].getList().value_or(std::vector{ 0.5 }).at(0);\n> > +\n> > +\t\tAgcConstraint constraint = { bound, qLo, qHi, yTarget };\n> > +\n> > +\t\tif (!constraintModes_.count(id))\n> > +\t\t\tconstraintModes_[id] = {};\n> > +\n> > +\t\tif (idx)\n> > +\t\t\tconstraintModes_[id].push_back(constraint);\n> > +\t\telse\n> > +\t\t\tconstraintModes_[id].insert(constraintModes_[id].begin(), constraint);\n> > +\t}\n> > +}\n> > +\n> > +int AgcMeanLuminance::parseConstraintModes(const YamlObject &tuningData)\n> > +{\n> > +\tstd::vector availableConstraintModes;\n> > +\n> > +\tconst YamlObject &yamlConstraintModes = tuningData[controls::AeConstraintMode.name()];\n> > +\tif (yamlConstraintModes.isDictionary()) {\n> > +\t\tfor (const auto &[modeName, modeDict] : yamlConstraintModes.asDict()) {\n> > +\t\t\tif (AeConstraintModeNameValueMap.find(modeName) ==\n> > +\t\t\t AeConstraintModeNameValueMap.end()) {\n> > +\t\t\t\tLOG(AgcMeanLuminance, Warning)\n> > +\t\t\t\t\t<< \"Skipping unknown constraint mode '\" << modeName << \"'\";\n> > +\t\t\t\tcontinue;\n> > +\t\t\t}\n> > +\n> > +\t\t\tif (!modeDict.isDictionary()) {\n> > +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> > +\t\t\t\t\t<< \"Invalid constraint mode '\" << modeName << \"'\";\n> > +\t\t\t\treturn -EINVAL;\n> > +\t\t\t}\n> > +\n> > +\t\t\tparseConstraint(modeDict,\n> > +\t\t\t\t\tAeConstraintModeNameValueMap.at(modeName));\n> > +\t\t\tavailableConstraintModes.push_back(\n> > +\t\t\t\tAeConstraintModeNameValueMap.at(modeName));\n> > +\t\t}\n> > +\t}\n> > +\n> > +\t/*\n> > +\t * If the tuning data file contains no constraints then we use the\n> > +\t * default constraint that the various Agc algorithms were adhering to\n> > +\t * anyway before centralisation.\n> > +\t */\n> > +\tif (constraintModes_.empty()) {\n> > +\t\tAgcConstraint constraint = {\n> > +\t\t\tAgcConstraint::Bound::lower,\n> > +\t\t\t0.98,\n> > +\t\t\t1.0,\n> > +\t\t\t0.5\n> > +\t\t};\n> > +\n> > +\t\tconstraintModes_[controls::ConstraintNormal].insert(\n> > +\t\t\tconstraintModes_[controls::ConstraintNormal].begin(),\n> > +\t\t\tconstraint);\n> > +\t\tavailableConstraintModes.push_back(\n> > +\t\t\tAeConstraintModeNameValueMap.at(\"ConstraintNormal\"));\n> > +\t}\n> > +\n> > +\tcontrols_[&controls::AeConstraintMode] = ControlInfo(availableConstraintModes);\n> > +\n> > +\treturn 0;\n> > +}\n> > +\n> > +int AgcMeanLuminance::parseExposureModes(const YamlObject &tuningData)\n> > +{\n> > +\tstd::vector availableExposureModes;\n> > +\n> > +\tconst YamlObject &yamlExposureModes = tuningData[controls::AeExposureMode.name()];\n> > +\tif (yamlExposureModes.isDictionary()) {\n> > +\t\tfor (const auto &[modeName, modeValues] : yamlExposureModes.asDict()) {\n> > +\t\t\tif (AeExposureModeNameValueMap.find(modeName) ==\n> > +\t\t\t AeExposureModeNameValueMap.end()) {\n> > +\t\t\t\tLOG(AgcMeanLuminance, Warning)\n> > +\t\t\t\t\t<< \"Skipping unknown exposure mode '\" << modeName << \"'\";\n> > +\t\t\t\tcontinue;\n> > +\t\t\t}\n> > +\n> > +\t\t\tif (!modeValues.isDictionary()) {\n> > +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> > +\t\t\t\t\t<< \"Invalid exposure mode '\" << modeName << \"'\";\n> > +\t\t\t\treturn -EINVAL;\n> > +\t\t\t}\n> > +\n> > +\t\t\tstd::vector shutters =\n> > +\t\t\t\tmodeValues[\"shutter\"].getList().value_or(std::vector{});\n> > +\t\t\tstd::vector gains =\n> > +\t\t\t\tmodeValues[\"gain\"].getList().value_or(std::vector{});\n> > +\n> > +\t\t\tif (shutters.size() != gains.size()) {\n> > +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> > +\t\t\t\t\t<< \"Shutter and gain array sizes unequal\";\n> > +\t\t\t\treturn -EINVAL;\n> > +\t\t\t}\n> > +\n> > +\t\t\tif (shutters.empty()) {\n> > +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> > +\t\t\t\t\t<< \"Shutter and gain arrays are empty\";\n> > +\t\t\t\treturn -EINVAL;\n> > +\t\t\t}\n> > +\n> > +\t\t\tstd::vector> stages;\n> > +\t\t\tfor (unsigned int i = 0; i < shutters.size(); i++) {\n> > +\t\t\t\tstages.push_back({\n> > +\t\t\t\t\tstd::chrono::microseconds(shutters[i]),\n> > +\t\t\t\t\tgains[i]\n> > +\t\t\t\t});\n> > +\t\t\t}\n> \n> I was wondering if we could move a significant portion of this to\n> ExposureModeHelper::readYaml() but since this is the only user (so far)\n> of it I suppose it doesn't really matter.\n> \n> Reviewed-by: Paul Elder \n> \n> > +\n> > +\t\t\tstd::shared_ptr helper =\n> > +\t\t\t\tstd::make_shared();\n> > +\t\t\thelper->init(stages);\n> > +\n> > +\t\t\texposureModeHelpers_[AeExposureModeNameValueMap.at(modeName)] = helper;\n> > +\t\t\tavailableExposureModes.push_back(AeExposureModeNameValueMap.at(modeName));\n> > +\t\t}\n> > +\t}\n> > +\n> > +\t/*\n> > +\t * If we don't have any exposure modes in the tuning data we create an\n> > +\t * ExposureModeHelper using an empty vector of stages. This will result\n> > +\t * in the ExposureModeHelper simply driving the shutter as high as\n> > +\t * possible before touching gain.\n> > +\t */\n> > +\tif (availableExposureModes.empty()) {\n> > +\t\tint32_t exposureModeId = AeExposureModeNameValueMap.at(\"ExposureNormal\");\n> > +\t\tstd::vector> stages = { };\n> > +\n> > +\t\tstd::shared_ptr helper =\n> > +\t\t\tstd::make_shared();\n> > +\t\thelper->init(stages);\n> > +\n> > +\t\texposureModeHelpers_[exposureModeId] = helper;\n> > +\t\tavailableExposureModes.push_back(exposureModeId);\n> > +\t}\n> > +\n> > +\tcontrols_[&controls::AeExposureMode] = ControlInfo(availableExposureModes);\n> > +\n> > +\treturn 0;\n> > +}\n> > +\n> > +/**\n> > + * \\brief Parse tuning data for AeConstraintMode and AeExposureMode controls\n> > + * \\param[in] tuningData the YamlObject representing the tuning data\n> > + *\n> > + * This function parses tuning data to build the list of allowed values for the\n> > + * AeConstraintMode and AeExposureMode controls. Those tuning data must provide\n> > + * the data in a specific format; the Agc algorithm's tuning data should contain\n> > + * a dictionary called AeConstraintMode containing per-mode setting dictionaries\n> > + * with the key being a value from \\ref controls::AeConstraintModeNameValueMap.\n> > + * Each mode dict may contain either a \"lower\" or \"upper\" key or both, for\n> > + * example:\n> > + *\n> > + * \\code{.unparsed}\n> > + * algorithms:\n> > + * - Agc:\n> > + * AeConstraintMode:\n> > + * ConstraintNormal:\n> > + * lower:\n> > + * qLo: 0.98\n> > + * qHi: 1.0\n> > + * yTarget: 0.5\n> > + * ConstraintHighlight:\n> > + * lower:\n> > + * qLo: 0.98\n> > + * qHi: 1.0\n> > + * yTarget: 0.5\n> > + * upper:\n> > + * qLo: 0.98\n> > + * qHi: 1.0\n> > + * yTarget: 0.8\n> > + *\n> > + * \\endcode\n> > + *\n> > + * For the AeExposureMode control the data should contain a dictionary called\n> > + * AeExposureMode containing per-mode setting dictionaries with the key being a\n> > + * value from \\ref controls::AeExposureModeNameValueMap. Each mode dict should\n> > + * contain an array of shutter times with the key \"shutter\" and an array of gain\n> > + * values with the key \"gain\", in this format:\n> > + *\n> > + * \\code{.unparsed}\n> > + * algorithms:\n> > + * - Agc:\n> > + * AeExposureMode:\n> > + * ExposureNormal:\n> > + * shutter: [ 100, 10000, 30000, 60000, 120000 ]\n> > + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ]\n> > + * ExposureShort:\n> > + * shutter: [ 100, 10000, 30000, 60000, 120000 ]\n> > + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ]\n> > + *\n> > + * \\endcode\n> > + *\n> > + * @return 0 on success or a negative error code\n> > + */\n> > +int AgcMeanLuminance::parseTuningData(const YamlObject &tuningData)\n> > +{\n> > +\tint ret;\n> > +\n> > +\tparseRelativeLuminanceTarget(tuningData);\n> > +\n> > +\tret = parseConstraintModes(tuningData);\n> > +\tif (ret)\n> > +\t\treturn ret;\n> > +\n> > +\tret = parseExposureModes(tuningData);\n> > +\tif (ret)\n> > +\t\treturn ret;\n> > +\n> > +\treturn 0;\n> > +}\n> > +\n> > +/**\n> > + * \\brief configure the ExposureModeHelpers for this class\n> > + * \\param[in] minShutter Minimum shutter time to allow\n> > + * \\param[in] maxShutter Maximum shutter time to allow\n> > + * \\param[in] minGain Minimum gain to allow\n> > + * \\param[in] maxGain Maximum gain to allow\n> > + *\n> > + * This function calls \\ref ExposureModeHelper::setShutterGainLimits() for each\n> > + * ExposureModeHelper that has been created for this class.\n> > + */\n> > +void AgcMeanLuminance::configureExposureModeHelpers(utils::Duration minShutter,\n> > +\t\t\t\t\t\t utils::Duration maxShutter,\n> > +\t\t\t\t\t\t double minGain,\n> > +\t\t\t\t\t\t double maxGain)\n> > +{\n> > +\tfor (auto &[id, helper] : exposureModeHelpers_)\n> > +\t\thelper->setShutterGainLimits(minShutter, maxShutter, minGain, maxGain);\n> > +}\n> > +\n> > +/**\n> > + * \\fn AgcMeanLuminance::constraintModes()\n> > + * \\brief Get the constraint modes that have been parsed from tuning data\n> > + */\n> > +\n> > +/**\n> > + * \\fn AgcMeanLuminance::exposureModeHelpers()\n> > + * \\brief Get the ExposureModeHelpers that have been parsed from tuning data\n> > + */\n> > +\n> > +/**\n> > + * \\fn AgcMeanLuminance::controls()\n> > + * \\brief Get the controls that have been generated after parsing tuning data\n> > + */\n> > +\n> > +/**\n> > + * \\fn AgcMeanLuminance::estimateLuminance(const double gain)\n> > + * \\brief Estimate the luminance of an image, adjusted by a given gain\n> > + * \\param[in] gain The gain with which to adjust the luminance estimate\n> > + *\n> > + * This function estimates the average relative luminance of the frame that\n> > + * would be output by the sensor if an additional \\a gain was applied. It is a\n> > + * pure virtual function because estimation of luminance is a hardware-specific\n> > + * operation, which depends wholly on the format of the stats that are delivered\n> > + * to libcamera from the ISP. Derived classes must implement an overriding\n> > + * function that calculates the normalised mean luminance value across the\n> > + * entire image.\n> > + *\n> > + * \\return The normalised relative luminance of the image\n> > + */\n> > +\n> > +/**\n> > + * \\brief Estimate the initial gain needed to achieve a relative luminance\n> > + * target\n> > + *\n> > + * To account for non-linearity caused by saturation, the value needs to be\n> > + * estimated in an iterative process, as multiplying by a gain will not increase\n> > + * the relative luminance by the same factor if some image regions are saturated\n> > + *\n> > + * \\return The calculated initial gain\n> > + */\n> > +double AgcMeanLuminance::estimateInitialGain()\n> > +{\n> > +\tdouble yTarget = relativeLuminanceTarget_;\n> > +\tdouble yGain = 1.0;\n> > +\n> > +\tfor (unsigned int i = 0; i < 8; i++) {\n> > +\t\tdouble yValue = estimateLuminance(yGain);\n> > +\t\tdouble extra_gain = std::min(10.0, yTarget / (yValue + .001));\n> > +\n> > +\t\tyGain *= extra_gain;\n> > +\t\tLOG(AgcMeanLuminance, Debug) << \"Y value: \" << yValue\n> > +\t\t\t\t<< \", Y target: \" << yTarget\n> > +\t\t\t\t<< \", gives gain \" << yGain;\n> > +\n> > +\t\tif (utils::abs_diff(extra_gain, 1.0) < 0.01)\n> > +\t\t\tbreak;\n> > +\t}\n> > +\n> > +\treturn yGain;\n> > +}\n> > +\n> > +/**\n> > + * \\brief Clamp gain within the bounds of a defined constraint\n> > + * \\param[in] constraintModeIndex The index of the constraint to adhere to\n> > + * \\param[in] hist A histogram over which to calculate inter-quantile means\n> > + * \\param[in] gain The gain to clamp\n> > + *\n> > + * \\return The gain clamped within the constraint bounds\n> > + */\n> > +double AgcMeanLuminance::constraintClampGain(uint32_t constraintModeIndex,\n> > +\t\t\t\t\t const Histogram &hist,\n> > +\t\t\t\t\t double gain)\n> > +{\n> > +\tstd::vector &constraints = constraintModes_[constraintModeIndex];\n> > +\tfor (const AgcConstraint &constraint : constraints) {\n> > +\t\tdouble newGain = constraint.yTarget * hist.bins() /\n> > +\t\t\t\t hist.interQuantileMean(constraint.qLo, constraint.qHi);\n> > +\n> > +\t\tif (constraint.bound == AgcConstraint::Bound::lower &&\n> > +\t\t newGain > gain)\n> > +\t\t\tgain = newGain;\n> > +\n> > +\t\tif (constraint.bound == AgcConstraint::Bound::upper &&\n> > +\t\t newGain < gain)\n> > +\t\t\tgain = newGain;\n> > +\t}\n> > +\n> > +\treturn gain;\n> > +}\n> > +\n> > +/**\n> > + * \\brief Apply a filter on the exposure value to limit the speed of changes\n> > + * \\param[in] exposureValue The target exposure from the AGC algorithm\n> > + *\n> > + * The speed of the filter is adaptive, and will produce the target quicker\n> > + * during startup, or when the target exposure is within 20% of the most recent\n> > + * filter output.\n> > + *\n> > + * \\return The filtered exposure\n> > + */\n> > +utils::Duration AgcMeanLuminance::filterExposure(utils::Duration exposureValue)\n> > +{\n> > +\tdouble speed = 0.2;\n> > +\n> > +\t/* Adapt instantly if we are in startup phase. */\n> > +\tif (frameCount_ < kNumStartupFrames)\n> > +\t\tspeed = 1.0;\n> > +\n> > +\t/*\n> > +\t * If we are close to the desired result, go faster to avoid making\n> > +\t * multiple micro-adjustments.\n> > +\t * \\todo Make this customisable?\n> > +\t */\n> > +\tif (filteredExposure_ < 1.2 * exposureValue &&\n> > +\t filteredExposure_ > 0.8 * exposureValue)\n> > +\t\tspeed = sqrt(speed);\n> > +\n> > +\tfilteredExposure_ = speed * exposureValue +\n> > +\t\t\t filteredExposure_ * (1.0 - speed);\n> > +\n> > +\treturn filteredExposure_;\n> > +}\n> > +\n> > +/**\n> > + * \\brief Calculate the new exposure value\n> > + * \\param[in] constraintModeIndex The index of the current constraint mode\n> > + * \\param[in] exposureModeIndex The index of the current exposure mode\n> > + * \\param[in] yHist A Histogram from the ISP statistics to use in constraining\n> > + *\t the calculated gain\n> > + * \\param[in] effectiveExposureValue The EV applied to the frame from which the\n> > + *\t statistics in use derive\n> > + *\n> > + * Calculate a new exposure value to try to obtain the target. The calculated\n> > + * exposure value is filtered to prevent rapid changes from frame to frame, and\n> > + * divided into shutter time, analogue and digital gain.\n> > + *\n> > + * \\return Tuple of shutter time, analogue gain, and digital gain\n> > + */\n> > +std::tuple\n> > +AgcMeanLuminance::calculateNewEv(uint32_t constraintModeIndex,\n> > +\t\t\t\t uint32_t exposureModeIndex,\n> > +\t\t\t\t const Histogram &yHist,\n> > +\t\t\t\t utils::Duration effectiveExposureValue)\n> > +{\n> > +\t/*\n> > +\t * The pipeline handler should validate that we have received an allowed\n> > +\t * value for AeExposureMode.\n> > +\t */\n> > +\tstd::shared_ptr exposureModeHelper =\n> > +\t\texposureModeHelpers_.at(exposureModeIndex);\n> > +\n> > +\tdouble gain = estimateInitialGain();\n> > +\tgain = constraintClampGain(constraintModeIndex, yHist, gain);\n> > +\n> > +\t/*\n> > +\t * We don't check whether we're already close to the target, because\n> > +\t * even if the effective exposure value is the same as the last frame's\n> > +\t * we could have switched to an exposure mode that would require a new\n> > +\t * pass through the splitExposure() function.\n> > +\t */\n> > +\n> > +\tutils::Duration newExposureValue = effectiveExposureValue * gain;\n> > +\tutils::Duration maxTotalExposure = exposureModeHelper->maxShutter()\n> > +\t\t\t\t\t * exposureModeHelper->maxGain();\n> > +\tnewExposureValue = std::min(newExposureValue, maxTotalExposure);\n> > +\n> > +\t/*\n> > +\t * We filter the exposure value to make sure changes are not too jarring\n> > +\t * from frame to frame.\n> > +\t */\n> > +\tnewExposureValue = filterExposure(newExposureValue);\n> > +\n> > +\tframeCount_++;\n> > +\treturn exposureModeHelper->splitExposure(newExposureValue);\n> > +}\n> > +\n> > +/**\n> > + * \\fn AgcMeanLuminance::resetFrameCount()\n> > + * \\brief Reset the frame counter\n> > + *\n> > + * This function resets the internal frame counter, which exists to help the\n> > + * algorithm decide whether it should respond instantly or not. The expectation\n> > + * is for derived classes to call this function before each camera start call,\n> > + * either in configure() or queueRequest() if the frame number is zero.\n> > + */\n> > +\n> > +}; /* namespace ipa */\n> > +\n> > +}; /* namespace libcamera */\n> > diff --git a/src/ipa/libipa/agc_mean_luminance.h b/src/ipa/libipa/agc_mean_luminance.h\n> > new file mode 100644\n> > index 00000000..e48dc498\n> > --- /dev/null\n> > +++ b/src/ipa/libipa/agc_mean_luminance.h\n> > @@ -0,0 +1,91 @@\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) 2024 Ideas on Board Oy\n> > + *\n> > + agc_mean_luminance.h - Base class for mean luminance AGC algorithms\n> > + */\n> > +\n> > +#pragma once\n> > +\n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +\n> > +#include \"libcamera/internal/yaml_parser.h\"\n> > +\n> > +#include \"exposure_mode_helper.h\"\n> > +#include \"histogram.h\"\n> > +\n> > +namespace libcamera {\n> > +\n> > +namespace ipa {\n> > +\n> > +class AgcMeanLuminance\n> > +{\n> > +public:\n> > +\tAgcMeanLuminance();\n> > +\tvirtual ~AgcMeanLuminance() = default;\n> > +\n> > +\tstruct AgcConstraint {\n> > +\t\tenum class Bound {\n> > +\t\t\tlower = 0,\n> > +\t\t\tupper = 1\n> > +\t\t};\n> > +\t\tBound bound;\n> > +\t\tdouble qLo;\n> > +\t\tdouble qHi;\n> > +\t\tdouble yTarget;\n> > +\t};\n> > +\n> > +\tint parseTuningData(const YamlObject &tuningData);\n> > +\n> > +\tvoid configureExposureModeHelpers(utils::Duration minShutter,\n> > +\t\t\t\t\t utils::Duration maxShutter,\n> > +\t\t\t\t\t double minGain,\n> > +\t\t\t\t\t double maxGain);\n> > +\n> > +\tstd::map> constraintModes()\n> > +\t{\n> > +\t\treturn constraintModes_;\n> > +\t}\n> > +\n> > +\tstd::map> exposureModeHelpers()\n> > +\t{\n> > +\t\treturn exposureModeHelpers_;\n> > +\t}\n> > +\n> > +\tControlInfoMap::Map controls()\n> > +\t{\n> > +\t\treturn controls_;\n> > +\t}\n> > +\n> > +\tdouble estimateInitialGain();\n> > +\tdouble constraintClampGain(uint32_t constraintModeIndex,\n> > +\t\t\t\t const Histogram &hist,\n> > +\t\t\t\t double gain);\n> > +\tutils::Duration filterExposure(utils::Duration exposureValue);\n> > +\tstd::tuple\n> > +\tcalculateNewEv(uint32_t constraintModeIndex, uint32_t exposureModeIndex,\n> > +\t\t const Histogram &yHist, utils::Duration effectiveExposureValue);\n> > +\tvoid resetFrameCount() { frameCount_ = 0; }\n> > +private:\n> > +\tvirtual double estimateLuminance(const double gain) = 0;\n> > +\n> > +\tvoid parseRelativeLuminanceTarget(const YamlObject &tuningData);\n> > +\tvoid parseConstraint(const YamlObject &modeDict, int32_t id);\n> > +\tint parseConstraintModes(const YamlObject &tuningData);\n> > +\tint parseExposureModes(const YamlObject &tuningData);\n> > +\n> > +\tuint64_t frameCount_;\n> > +\tutils::Duration filteredExposure_;\n> > +\tdouble relativeLuminanceTarget_;\n> > +\n> > +\tstd::map> constraintModes_;\n> > +\tstd::map> exposureModeHelpers_;\n> > +\tControlInfoMap::Map controls_;\n> > +};\n> > +\n> > +}; /* namespace ipa */\n> > +\n> > +}; /* namespace libcamera */\n> > diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build\n> > index 37fbd177..7ce885da 100644\n> > --- a/src/ipa/libipa/meson.build\n> > +++ b/src/ipa/libipa/meson.build\n> > @@ -1,6 +1,7 @@\n> > # SPDX-License-Identifier: CC0-1.0\n> > \n> > libipa_headers = files([\n> > + 'agc_mean_luminance.h',\n> > 'algorithm.h',\n> > 'camera_sensor_helper.h',\n> > 'exposure_mode_helper.h',\n> > @@ -10,6 +11,7 @@ libipa_headers = files([\n> > ])\n> > \n> > libipa_sources = files([\n> > + 'agc_mean_luminance.cpp',\n> > 'algorithm.cpp',\n> > 'camera_sensor_helper.cpp',\n> > 'exposure_mode_helper.cpp',\n> > -- \n> > 2.34.1\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id B9745BE08B\n\tfor ;\n\tFri, 19 Apr 2024 04:51:16 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 7C03D633F3;\n\tFri, 19 Apr 2024 06:51:15 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 670C061B17\n\tfor ;\n\tFri, 19 Apr 2024 06:51:13 +0200 (CEST)","from pyrite.rasen.tech (h175-177-049-156.catv02.itscom.jp\n\t[175.177.49.156])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id D8BBB802;\n\tFri, 19 Apr 2024 06:50:23 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"lB+CT6ds\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1713502225;\n\tbh=70QOgDj2jflXDpthtQl0/PnHphiIhMgpmDx0ey4aMQM=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=lB+CT6dsV8EdnqGVhNA2ZjN/W7HoagUHgY4DfMDEhi6+AmWgOp/GOXMFGtlshImfW\n\tylMBz+29jLgASumNqjCN/mrlR3EhUVvo+A7kt+/vcGU3zeZcfkSUC/95Aym7GqPV+o\n\tP1RzhRMhzrGhWQe6OysYqIxFfcGGYI4WRa0U9Ygk=","Date":"Fri, 19 Apr 2024 13:51:06 +0900","From":"Paul Elder ","To":"Daniel Scally ","Cc":"libcamera-devel@lists.libcamera.org","Subject":"Re: [PATCH v2 4/8] ipa: libipa: Add MeanLuminanceAgc base class","Message-ID":"","References":"<20240417131536.484129-1-dan.scally@ideasonboard.com>\n\t<20240417131536.484129-5-dan.scally@ideasonboard.com>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":29277,"web_url":"https://patchwork.libcamera.org/comment/29277/","msgid":"<42axtt2x6v4deeuzolzgsw3ksyio3emfsx26ynfonfuh7nvs2w@m5a5ox6pohmn>","date":"2024-04-19T14:34:24","subject":"Re: [PATCH v2 4/8] ipa: libipa: Add MeanLuminanceAgc base class","submitter":{"id":143,"url":"https://patchwork.libcamera.org/api/people/143/","name":"Jacopo Mondi","email":"jacopo.mondi@ideasonboard.com"},"content":"Hi Dan\n\nOn Wed, Apr 17, 2024 at 02:15:32PM +0100, Daniel Scally wrote:\n> The Agc algorithms for the RkIsp1 and IPU3 IPAs do the same thing in\n> very large part; following the Rpi IPA's algorithm in spirit with a\n> few tunable values in that IPA being hardcoded in the libipa ones.\n> Add a new base class for MeanLuminanceAgc which implements the same\n> algorithm and additionally parses yaml tuning files to inform an IPA\n> module's Agc algorithm about valid constraint and exposure modes and\n> their associated bounds.\n>\n> Signed-off-by: Daniel Scally \n> ---\n> Changes in v2:\n>\n> \t- Renamed the class and files\n> \t- Expanded the documentation\n> \t- Added parseTuningData() so derived classes can call a single function\n> \t to cover all the parsing in ::init().\n>\n> src/ipa/libipa/agc_mean_luminance.cpp | 581 ++++++++++++++++++++++++++\n> src/ipa/libipa/agc_mean_luminance.h | 91 ++++\n> src/ipa/libipa/meson.build | 2 +\n> 3 files changed, 674 insertions(+)\n> create mode 100644 src/ipa/libipa/agc_mean_luminance.cpp\n> create mode 100644 src/ipa/libipa/agc_mean_luminance.h\n>\n> diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp\n> new file mode 100644\n> index 00000000..02e223cf\n> --- /dev/null\n> +++ b/src/ipa/libipa/agc_mean_luminance.cpp\n> @@ -0,0 +1,581 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2024 Ideas on Board Oy\n> + *\n> + * agc_mean_luminance.cpp - Base class for mean luminance AGC algorithms\n> + */\n> +\n> +#include \"agc_mean_luminance.h\"\n> +\n> +#include \n> +\n> +#include \n> +#include \n> +\n> +#include \"exposure_mode_helper.h\"\n> +\n> +using namespace libcamera::controls;\n> +\n> +/**\n> + * \\file agc_mean_luminance.h\n> + * \\brief Base class implementing mean luminance AEGC\n> + */\n> +\n> +namespace libcamera {\n> +\n> +using namespace std::literals::chrono_literals;\n> +\n> +LOG_DEFINE_CATEGORY(AgcMeanLuminance)\n> +\n> +namespace ipa {\n> +\n> +/*\n> + * Number of frames for which to run the algorithm at full speed, before slowing\n> + * down to prevent large and jarring changes in exposure from frame to frame.\n> + */\n> +static constexpr uint32_t kNumStartupFrames = 10;\n> +\n> +/*\n> + * Default relative luminance target\n> + *\n> + * This value should be chosen so that when the camera points at a grey target,\n> + * the resulting image brightness looks \"right\". Custom values can be passed\n> + * as the relativeLuminanceTarget value in sensor tuning files.\n> + */\n> +static constexpr double kDefaultRelativeLuminanceTarget = 0.16;\n> +\n> +/**\n> + * \\struct AgcMeanLuminance::AgcConstraint\n> + * \\brief The boundaries and target for an AeConstraintMode constraint\n> + *\n> + * This structure describes an AeConstraintMode constraint for the purposes of\n> + * this algorithm. The algorithm will apply the constraints by calculating the\n> + * Histogram's inter-quantile mean between the given quantiles and ensure that\n> + * the resulting value is the right side of the given target (as defined by the\n> + * boundary and luminance target).\n\nI would appreciate a more generic overview of the what these\nconstraints are and what they express, instead of specifying how the\nalgorithm uses them (that's more appropriate for a comment in the code\nif you like).\n\n> + */\n> +\n> +/**\n> + * \\enum AgcMeanLuminance::AgcConstraint::Bound\n> + * \\brief Specify whether the constraint defines a lower or upper bound\n> + * \\var AgcMeanLuminance::AgcConstraint::lower\n> + * \\brief The constraint defines a lower bound\n> + * \\var AgcMeanLuminance::AgcConstraint::upper\n> + * \\brief The constraint defines an upper bound\n> + */\n> +\n> +/**\n> + * \\var AgcMeanLuminance::AgcConstraint::bound\n> + * \\brief The type of constraint bound\n> + */\n> +\n> +/**\n> + * \\var AgcMeanLuminance::AgcConstraint::qLo\n> + * \\brief The lower quantile to use for the constraint\n> + */\n> +\n> +/**\n> + * \\var AgcMeanLuminance::AgcConstraint::qHi\n> + * \\brief The upper quantile to use for the constraint\n> + */\n> +\n> +/**\n> + * \\var AgcMeanLuminance::AgcConstraint::yTarget\n> + * \\brief The luminance target for the constraint\n> + */\n> +\n> +/**\n> + * \\class AgcMeanLuminance\n> + * \\brief A mean-based auto-exposure algorithm\n> + *\n> + * This algorithm calculates a shutter time, analogue and digital gain such that\n> + * the normalised mean luminance value of an image is driven towards a target,\n> + * which itself is discovered from tuning data. The algorithm is a two-stage\n> + * process.\n> + *\n> + * In the first stage, an initial gain value is derived by iteratively comparing\n> + * the gain-adjusted mean luminance across an entire image against a target, and\n> + * selecting a value which pushes it as closely as possible towards the target.\n> + *\n> + * In the second stage we calculate the gain required to drive the average of a\n> + * section of a histogram to a target value, where the target and the boundaries\n> + * of the section of the histogram used in the calculation are taken from the\n> + * values defined for the currently configured AeConstraintMode within the\n> + * tuning data. This class provides a helper function to parse those tuning data\n> + * to discover the constraints, and so requires a specific format for those\n> + * data which is described in \\ref parseTuningData(). The gain from the first\n> + * stage is then clamped to the gain from this stage.\n\nThis is a nice description of the internal working, and it has value\nindeed.\n\nHowever, the class API is not described, nor how IPA modules are\nexpected to use this class in their implementation. Do you think this\nwould have any value ?\n\n> + *\n> + * The final gain is used to adjust the effective exposure value of the image,\n> + * and that new exposure value is divided into shutter time, analogue gain and\n> + * digital gain according to the selected AeExposureMode. This class expects to\n\nWhy does it expects to ? doesn't it just do that ?\n\n> + * use the \\ref ExposureModeHelper class to assist in that division, and expects\n> + * the data needed to initialise that class to be present in tuning data in a\n> + * format described in \\ref parseTuningData().\n> + *\n> + * In order to be able to derive an AGC implementation from this class, an IPA\n\nAh here you go, so IPA modules are expected to derive this class and\nnot use it by composition it seems. A little more details on this\nmight help IPA implementers to figure this out.\n\n> + * needs to be able to do the following:\n> + *\n> + * 1. Provide a luminance estimation across an entire image.\n> + * 2. Provide a luminance Histogram for the image to use in calculating\n> + * constraint compliance. The precision of the Histogram that is available\n> + * will determine the supportable precision of the constraints.\n> + */\n> +\n> +AgcMeanLuminance::AgcMeanLuminance()\n> +\t: frameCount_(0), filteredExposure_(0s), relativeLuminanceTarget_(0)\n> +{\n> +}\n> +\n> +/**\n> + * \\brief Parse the relative luminance target from the tuning data\n> + * \\param[in] tuningData The YamlObject holding the algorithm's tuning data\n> + */\n> +void AgcMeanLuminance::parseRelativeLuminanceTarget(const YamlObject &tuningData)\n> +{\n> +\trelativeLuminanceTarget_ =\n> +\t\ttuningData[\"relativeLuminanceTarget\"].get(kDefaultRelativeLuminanceTarget);\n> +}\n> +\n> +/**\n> + * \\brief Parse an AeConstraintMode constraint from tuning data\n> + * \\param[in] modeDict the YamlObject holding the constraint data\n> + * \\param[in] id The constraint ID from AeConstraintModeEnum\n> + */\n> +void AgcMeanLuminance::parseConstraint(const YamlObject &modeDict, int32_t id)\n> +{\n> +\tfor (const auto &[boundName, content] : modeDict.asDict()) {\n> +\t\tif (boundName != \"upper\" && boundName != \"lower\") {\n> +\t\t\tLOG(AgcMeanLuminance, Warning)\n> +\t\t\t\t<< \"Ignoring unknown constraint bound '\" << boundName << \"'\";\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tunsigned int idx = static_cast(boundName == \"upper\");\n> +\t\tAgcConstraint::Bound bound = static_cast(idx);\n> +\t\tdouble qLo = content[\"qLo\"].get().value_or(0.98);\n> +\t\tdouble qHi = content[\"qHi\"].get().value_or(1.0);\n> +\t\tdouble yTarget =\n> +\t\t\tcontent[\"yTarget\"].getList().value_or(std::vector{ 0.5 }).at(0);\n> +\n> +\t\tAgcConstraint constraint = { bound, qLo, qHi, yTarget };\n> +\n> +\t\tif (!constraintModes_.count(id))\n> +\t\t\tconstraintModes_[id] = {};\n> +\n> +\t\tif (idx)\n> +\t\t\tconstraintModes_[id].push_back(constraint);\n> +\t\telse\n> +\t\t\tconstraintModes_[id].insert(constraintModes_[id].begin(), constraint);\n> +\t}\n> +}\n> +\n> +int AgcMeanLuminance::parseConstraintModes(const YamlObject &tuningData)\n> +{\n> +\tstd::vector availableConstraintModes;\n> +\n> +\tconst YamlObject &yamlConstraintModes = tuningData[controls::AeConstraintMode.name()];\n> +\tif (yamlConstraintModes.isDictionary()) {\n> +\t\tfor (const auto &[modeName, modeDict] : yamlConstraintModes.asDict()) {\n> +\t\t\tif (AeConstraintModeNameValueMap.find(modeName) ==\n> +\t\t\t AeConstraintModeNameValueMap.end()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Warning)\n> +\t\t\t\t\t<< \"Skipping unknown constraint mode '\" << modeName << \"'\";\n> +\t\t\t\tcontinue;\n> +\t\t\t}\n> +\n> +\t\t\tif (!modeDict.isDictionary()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> +\t\t\t\t\t<< \"Invalid constraint mode '\" << modeName << \"'\";\n> +\t\t\t\treturn -EINVAL;\n> +\t\t\t}\n> +\n> +\t\t\tparseConstraint(modeDict,\n> +\t\t\t\t\tAeConstraintModeNameValueMap.at(modeName));\n> +\t\t\tavailableConstraintModes.push_back(\n> +\t\t\t\tAeConstraintModeNameValueMap.at(modeName));\n> +\t\t}\n> +\t}\n> +\n> +\t/*\n> +\t * If the tuning data file contains no constraints then we use the\n> +\t * default constraint that the various Agc algorithms were adhering to\n> +\t * anyway before centralisation.\n\nIf you read this in 2 years, without knowing there was a\ncentralization and what \"various algorithms\" are, you want be able to\nmake a sense out of this.\n\nWhat is needed here is to explain what the below values represent I\nguess..\n\n> +\t */\n> +\tif (constraintModes_.empty()) {\n> +\t\tAgcConstraint constraint = {\n> +\t\t\tAgcConstraint::Bound::lower,\n> +\t\t\t0.98,\n> +\t\t\t1.0,\n> +\t\t\t0.5\n> +\t\t};\n> +\n> +\t\tconstraintModes_[controls::ConstraintNormal].insert(\n> +\t\t\tconstraintModes_[controls::ConstraintNormal].begin(),\n> +\t\t\tconstraint);\n> +\t\tavailableConstraintModes.push_back(\n> +\t\t\tAeConstraintModeNameValueMap.at(\"ConstraintNormal\"));\n> +\t}\n> +\n> +\tcontrols_[&controls::AeConstraintMode] = ControlInfo(availableConstraintModes);\n> +\n> +\treturn 0;\n> +}\n> +\n> +int AgcMeanLuminance::parseExposureModes(const YamlObject &tuningData)\n> +{\n> +\tstd::vector availableExposureModes;\n> +\n> +\tconst YamlObject &yamlExposureModes = tuningData[controls::AeExposureMode.name()];\n> +\tif (yamlExposureModes.isDictionary()) {\n> +\t\tfor (const auto &[modeName, modeValues] : yamlExposureModes.asDict()) {\n> +\t\t\tif (AeExposureModeNameValueMap.find(modeName) ==\n> +\t\t\t AeExposureModeNameValueMap.end()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Warning)\n> +\t\t\t\t\t<< \"Skipping unknown exposure mode '\" << modeName << \"'\";\n> +\t\t\t\tcontinue;\n> +\t\t\t}\n> +\n> +\t\t\tif (!modeValues.isDictionary()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> +\t\t\t\t\t<< \"Invalid exposure mode '\" << modeName << \"'\";\n> +\t\t\t\treturn -EINVAL;\n> +\t\t\t}\n> +\n> +\t\t\tstd::vector shutters =\n> +\t\t\t\tmodeValues[\"shutter\"].getList().value_or(std::vector{});\n> +\t\t\tstd::vector gains =\n> +\t\t\t\tmodeValues[\"gain\"].getList().value_or(std::vector{});\n> +\n> +\t\t\tif (shutters.size() != gains.size()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> +\t\t\t\t\t<< \"Shutter and gain array sizes unequal\";\n> +\t\t\t\treturn -EINVAL;\n> +\t\t\t}\n> +\n> +\t\t\tif (shutters.empty()) {\n> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> +\t\t\t\t\t<< \"Shutter and gain arrays are empty\";\n> +\t\t\t\treturn -EINVAL;\n> +\t\t\t}\n> +\n> +\t\t\tstd::vector> stages;\n> +\t\t\tfor (unsigned int i = 0; i < shutters.size(); i++) {\n> +\t\t\t\tstages.push_back({\n> +\t\t\t\t\tstd::chrono::microseconds(shutters[i]),\n> +\t\t\t\t\tgains[i]\n> +\t\t\t\t});\n> +\t\t\t}\n> +\n> +\t\t\tstd::shared_ptr helper =\n> +\t\t\t\tstd::make_shared();\n> +\t\t\thelper->init(stages);\n> +\n> +\t\t\texposureModeHelpers_[AeExposureModeNameValueMap.at(modeName)] = helper;\n> +\t\t\tavailableExposureModes.push_back(AeExposureModeNameValueMap.at(modeName));\n> +\t\t}\n> +\t}\n> +\n> +\t/*\n> +\t * If we don't have any exposure modes in the tuning data we create an\n> +\t * ExposureModeHelper using an empty vector of stages. This will result\n> +\t * in the ExposureModeHelper simply driving the shutter as high as\n> +\t * possible before touching gain.\n> +\t */\n> +\tif (availableExposureModes.empty()) {\n> +\t\tint32_t exposureModeId = AeExposureModeNameValueMap.at(\"ExposureNormal\");\n> +\t\tstd::vector> stages = { };\n> +\n> +\t\tstd::shared_ptr helper =\n> +\t\t\tstd::make_shared();\n> +\t\thelper->init(stages);\n> +\n> +\t\texposureModeHelpers_[exposureModeId] = helper;\n> +\t\tavailableExposureModes.push_back(exposureModeId);\n> +\t}\n> +\n> +\tcontrols_[&controls::AeExposureMode] = ControlInfo(availableExposureModes);\n> +\n> +\treturn 0;\n> +}\n> +\n> +/**\n> + * \\brief Parse tuning data for AeConstraintMode and AeExposureMode controls\n> + * \\param[in] tuningData the YamlObject representing the tuning data\n> + *\n> + * This function parses tuning data to build the list of allowed values for the\n> + * AeConstraintMode and AeExposureMode controls. Those tuning data must provide\n> + * the data in a specific format; the Agc algorithm's tuning data should contain\n> + * a dictionary called AeConstraintMode containing per-mode setting dictionaries\n> + * with the key being a value from \\ref controls::AeConstraintModeNameValueMap.\n> + * Each mode dict may contain either a \"lower\" or \"upper\" key or both, for\n> + * example:\n> + *\n> + * \\code{.unparsed}\n> + * algorithms:\n> + * - Agc:\n> + * AeConstraintMode:\n> + * ConstraintNormal:\n> + * lower:\n> + * qLo: 0.98\n> + * qHi: 1.0\n> + * yTarget: 0.5\n> + * ConstraintHighlight:\n> + * lower:\n> + * qLo: 0.98\n> + * qHi: 1.0\n> + * yTarget: 0.5\n> + * upper:\n> + * qLo: 0.98\n> + * qHi: 1.0\n> + * yTarget: 0.8\n> + *\n> + * \\endcode\n> + *\n> + * For the AeExposureMode control the data should contain a dictionary called\n> + * AeExposureMode containing per-mode setting dictionaries with the key being a\n> + * value from \\ref controls::AeExposureModeNameValueMap. Each mode dict should\n> + * contain an array of shutter times with the key \"shutter\" and an array of gain\n> + * values with the key \"gain\", in this format:\n> + *\n> + * \\code{.unparsed}\n> + * algorithms:\n> + * - Agc:\n> + * AeExposureMode:\n> + * ExposureNormal:\n> + * shutter: [ 100, 10000, 30000, 60000, 120000 ]\n> + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ]\n> + * ExposureShort:\n> + * shutter: [ 100, 10000, 30000, 60000, 120000 ]\n> + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ]\n> + *\n> + * \\endcode\n> + *\n> + * @return 0 on success or a negative error code\n> + */\n> +int AgcMeanLuminance::parseTuningData(const YamlObject &tuningData)\n> +{\n> +\tint ret;\n> +\n> +\tparseRelativeLuminanceTarget(tuningData);\n> +\n> +\tret = parseConstraintModes(tuningData);\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n> +\tret = parseExposureModes(tuningData);\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n> +\treturn 0;\n> +}\n> +\n> +/**\n> + * \\brief configure the ExposureModeHelpers for this class\n> + * \\param[in] minShutter Minimum shutter time to allow\n> + * \\param[in] maxShutter Maximum shutter time to allow\n> + * \\param[in] minGain Minimum gain to allow\n> + * \\param[in] maxGain Maximum gain to allow\n> + *\n> + * This function calls \\ref ExposureModeHelper::setShutterGainLimits() for each\n> + * ExposureModeHelper that has been created for this class.\n> + */\n> +void AgcMeanLuminance::configureExposureModeHelpers(utils::Duration minShutter,\n> +\t\t\t\t\t\t utils::Duration maxShutter,\n> +\t\t\t\t\t\t double minGain,\n> +\t\t\t\t\t\t double maxGain)\n> +{\n> +\tfor (auto &[id, helper] : exposureModeHelpers_)\n> +\t\thelper->setShutterGainLimits(minShutter, maxShutter, minGain, maxGain);\n> +}\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::constraintModes()\n> + * \\brief Get the constraint modes that have been parsed from tuning data\n> + */\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::exposureModeHelpers()\n> + * \\brief Get the ExposureModeHelpers that have been parsed from tuning data\n> + */\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::controls()\n> + * \\brief Get the controls that have been generated after parsing tuning data\n> + */\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::estimateLuminance(const double gain)\n> + * \\brief Estimate the luminance of an image, adjusted by a given gain\n> + * \\param[in] gain The gain with which to adjust the luminance estimate\n> + *\n> + * This function estimates the average relative luminance of the frame that\n> + * would be output by the sensor if an additional \\a gain was applied. It is a\n> + * pure virtual function because estimation of luminance is a hardware-specific\n> + * operation, which depends wholly on the format of the stats that are delivered\n> + * to libcamera from the ISP. Derived classes must implement an overriding\n\nor just \"must override\"\n\n> + * function that calculates the normalised mean luminance value across the\n> + * entire image.\n> + *\n> + * \\return The normalised relative luminance of the image\n> + */\n> +\n> +/**\n> + * \\brief Estimate the initial gain needed to achieve a relative luminance\n> + * target\n> + *\n> + * To account for non-linearity caused by saturation, the value needs to be\n> + * estimated in an iterative process, as multiplying by a gain will not increase\n> + * the relative luminance by the same factor if some image regions are saturated\n\nThis seems more a comment to the above for loop than documentation\n\n> + *\n> + * \\return The calculated initial gain\n> + */\n> +double AgcMeanLuminance::estimateInitialGain()\n> +{\n> +\tdouble yTarget = relativeLuminanceTarget_;\n> +\tdouble yGain = 1.0;\n> +\n> +\tfor (unsigned int i = 0; i < 8; i++) {\n> +\t\tdouble yValue = estimateLuminance(yGain);\n> +\t\tdouble extra_gain = std::min(10.0, yTarget / (yValue + .001));\n> +\n> +\t\tyGain *= extra_gain;\n> +\t\tLOG(AgcMeanLuminance, Debug) << \"Y value: \" << yValue\n> +\t\t\t\t<< \", Y target: \" << yTarget\n> +\t\t\t\t<< \", gives gain \" << yGain;\n> +\n> +\t\tif (utils::abs_diff(extra_gain, 1.0) < 0.01)\n> +\t\t\tbreak;\n> +\t}\n> +\n> +\treturn yGain;\n> +}\n> +\n> +/**\n> + * \\brief Clamp gain within the bounds of a defined constraint\n> + * \\param[in] constraintModeIndex The index of the constraint to adhere to\n> + * \\param[in] hist A histogram over which to calculate inter-quantile means\n> + * \\param[in] gain The gain to clamp\n> + *\n> + * \\return The gain clamped within the constraint bounds\n> + */\n> +double AgcMeanLuminance::constraintClampGain(uint32_t constraintModeIndex,\n> +\t\t\t\t\t const Histogram &hist,\n> +\t\t\t\t\t double gain)\n> +{\n> +\tstd::vector &constraints = constraintModes_[constraintModeIndex];\n> +\tfor (const AgcConstraint &constraint : constraints) {\n> +\t\tdouble newGain = constraint.yTarget * hist.bins() /\n> +\t\t\t\t hist.interQuantileMean(constraint.qLo, constraint.qHi);\n> +\n> +\t\tif (constraint.bound == AgcConstraint::Bound::lower &&\n> +\t\t newGain > gain)\n> +\t\t\tgain = newGain;\n> +\n> +\t\tif (constraint.bound == AgcConstraint::Bound::upper &&\n> +\t\t newGain < gain)\n> +\t\t\tgain = newGain;\n> +\t}\n> +\n> +\treturn gain;\n> +}\n> +\n> +/**\n> + * \\brief Apply a filter on the exposure value to limit the speed of changes\n> + * \\param[in] exposureValue The target exposure from the AGC algorithm\n> + *\n> + * The speed of the filter is adaptive, and will produce the target quicker\n> + * during startup, or when the target exposure is within 20% of the most recent\n> + * filter output.\n> + *\n> + * \\return The filtered exposure\n> + */\n> +utils::Duration AgcMeanLuminance::filterExposure(utils::Duration exposureValue)\n> +{\n> +\tdouble speed = 0.2;\n> +\n> +\t/* Adapt instantly if we are in startup phase. */\n> +\tif (frameCount_ < kNumStartupFrames)\n> +\t\tspeed = 1.0;\n> +\n> +\t/*\n> +\t * If we are close to the desired result, go faster to avoid making\n> +\t * multiple micro-adjustments.\n> +\t * \\todo Make this customisable?\n> +\t */\n> +\tif (filteredExposure_ < 1.2 * exposureValue &&\n> +\t filteredExposure_ > 0.8 * exposureValue)\n> +\t\tspeed = sqrt(speed);\n> +\n> +\tfilteredExposure_ = speed * exposureValue +\n> +\t\t\t filteredExposure_ * (1.0 - speed);\n> +\n> +\treturn filteredExposure_;\n> +}\n> +\n> +/**\n> + * \\brief Calculate the new exposure value\n> + * \\param[in] constraintModeIndex The index of the current constraint mode\n> + * \\param[in] exposureModeIndex The index of the current exposure mode\n> + * \\param[in] yHist A Histogram from the ISP statistics to use in constraining\n> + *\t the calculated gain\n> + * \\param[in] effectiveExposureValue The EV applied to the frame from which the\n> + *\t statistics in use derive\n> + *\n> + * Calculate a new exposure value to try to obtain the target. The calculated\n> + * exposure value is filtered to prevent rapid changes from frame to frame, and\n> + * divided into shutter time, analogue and digital gain.\n> + *\n> + * \\return Tuple of shutter time, analogue gain, and digital gain\n> + */\n> +std::tuple\n> +AgcMeanLuminance::calculateNewEv(uint32_t constraintModeIndex,\n> +\t\t\t\t uint32_t exposureModeIndex,\n> +\t\t\t\t const Histogram &yHist,\n> +\t\t\t\t utils::Duration effectiveExposureValue)\n> +{\n> +\t/*\n> +\t * The pipeline handler should validate that we have received an allowed\n> +\t * value for AeExposureMode.\n\nHowever, a check wouldn't hurt\n\n> +\t */\n> +\tstd::shared_ptr exposureModeHelper =\n> +\t\texposureModeHelpers_.at(exposureModeIndex);\n> +\n> +\tdouble gain = estimateInitialGain();\n> +\tgain = constraintClampGain(constraintModeIndex, yHist, gain);\n> +\n> +\t/*\n> +\t * We don't check whether we're already close to the target, because\n> +\t * even if the effective exposure value is the same as the last frame's\n> +\t * we could have switched to an exposure mode that would require a new\n> +\t * pass through the splitExposure() function.\n> +\t */\n> +\n> +\tutils::Duration newExposureValue = effectiveExposureValue * gain;\n> +\tutils::Duration maxTotalExposure = exposureModeHelper->maxShutter()\n> +\t\t\t\t\t * exposureModeHelper->maxGain();\n> +\tnewExposureValue = std::min(newExposureValue, maxTotalExposure);\n> +\n> +\t/*\n> +\t * We filter the exposure value to make sure changes are not too jarring\n> +\t * from frame to frame.\n> +\t */\n> +\tnewExposureValue = filterExposure(newExposureValue);\n> +\n> +\tframeCount_++;\n> +\treturn exposureModeHelper->splitExposure(newExposureValue);\n\nI wonder if this class shouldn't instead just calculate the exposure\nvalue and let the IPA divide it up as it likes.. Just thinking out\nloud and no real preference..\n\n> +}\n> +\n> +/**\n> + * \\fn AgcMeanLuminance::resetFrameCount()\n> + * \\brief Reset the frame counter\n> + *\n> + * This function resets the internal frame counter, which exists to help the\n> + * algorithm decide whether it should respond instantly or not. The expectation\n> + * is for derived classes to call this function before each camera start call,\n> + * either in configure() or queueRequest() if the frame number is zero.\n> + */\n> +\n> +}; /* namespace ipa */\n> +\n> +}; /* namespace libcamera */\n> diff --git a/src/ipa/libipa/agc_mean_luminance.h b/src/ipa/libipa/agc_mean_luminance.h\n> new file mode 100644\n> index 00000000..e48dc498\n> --- /dev/null\n> +++ b/src/ipa/libipa/agc_mean_luminance.h\n> @@ -0,0 +1,91 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2024 Ideas on Board Oy\n> + *\n> + agc_mean_luminance.h - Base class for mean luminance AGC algorithms\n> + */\n> +\n> +#pragma once\n> +\n> +#include \n> +#include \n> +\n> +#include \n> +\n> +#include \"libcamera/internal/yaml_parser.h\"\n> +\n> +#include \"exposure_mode_helper.h\"\n> +#include \"histogram.h\"\n> +\n> +namespace libcamera {\n> +\n> +namespace ipa {\n> +\n> +class AgcMeanLuminance\n> +{\n> +public:\n> +\tAgcMeanLuminance();\n> +\tvirtual ~AgcMeanLuminance() = default;\n> +\n> +\tstruct AgcConstraint {\n> +\t\tenum class Bound {\n> +\t\t\tlower = 0,\n> +\t\t\tupper = 1\n> +\t\t};\n> +\t\tBound bound;\n> +\t\tdouble qLo;\n> +\t\tdouble qHi;\n> +\t\tdouble yTarget;\n> +\t};\n> +\n> +\tint parseTuningData(const YamlObject &tuningData);\n> +\n> +\tvoid configureExposureModeHelpers(utils::Duration minShutter,\n> +\t\t\t\t\t utils::Duration maxShutter,\n> +\t\t\t\t\t double minGain,\n> +\t\t\t\t\t double maxGain);\n> +\n> +\tstd::map> constraintModes()\n> +\t{\n> +\t\treturn constraintModes_;\n> +\t}\n> +\n> +\tstd::map> exposureModeHelpers()\n> +\t{\n> +\t\treturn exposureModeHelpers_;\n> +\t}\n> +\n> +\tControlInfoMap::Map controls()\n> +\t{\n> +\t\treturn controls_;\n> +\t}\n> +\n> +\tdouble estimateInitialGain();\n> +\tdouble constraintClampGain(uint32_t constraintModeIndex,\n> +\t\t\t\t const Histogram &hist,\n> +\t\t\t\t double gain);\n> +\tutils::Duration filterExposure(utils::Duration exposureValue);\n\nIs filterExposure and clamps used outside of this class ?\n\n> +\tstd::tuple\n> +\tcalculateNewEv(uint32_t constraintModeIndex, uint32_t exposureModeIndex,\n> +\t\t const Histogram &yHist, utils::Duration effectiveExposureValue);\n> +\tvoid resetFrameCount() { frameCount_ = 0; }\n\nEmpty line\n\n> +private:\n> +\tvirtual double estimateLuminance(const double gain) = 0;\n> +\n> +\tvoid parseRelativeLuminanceTarget(const YamlObject &tuningData);\n> +\tvoid parseConstraint(const YamlObject &modeDict, int32_t id);\n> +\tint parseConstraintModes(const YamlObject &tuningData);\n> +\tint parseExposureModes(const YamlObject &tuningData);\n> +\n> +\tuint64_t frameCount_;\n> +\tutils::Duration filteredExposure_;\n> +\tdouble relativeLuminanceTarget_;\n> +\n> +\tstd::map> constraintModes_;\n> +\tstd::map> exposureModeHelpers_;\n> +\tControlInfoMap::Map controls_;\n> +};\n> +\n> +}; /* namespace ipa */\n> +\n> +}; /* namespace libcamera */\n> diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build\n> index 37fbd177..7ce885da 100644\n> --- a/src/ipa/libipa/meson.build\n> +++ b/src/ipa/libipa/meson.build\n> @@ -1,6 +1,7 @@\n> # SPDX-License-Identifier: CC0-1.0\n>\n> libipa_headers = files([\n> + 'agc_mean_luminance.h',\n> 'algorithm.h',\n> 'camera_sensor_helper.h',\n> 'exposure_mode_helper.h',\n> @@ -10,6 +11,7 @@ libipa_headers = files([\n> ])\n>\n> libipa_sources = files([\n> + 'agc_mean_luminance.cpp',\n> 'algorithm.cpp',\n> 'camera_sensor_helper.cpp',\n> 'exposure_mode_helper.cpp',\n> --\n> 2.34.1\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 48A44C3200\n\tfor ;\n\tFri, 19 Apr 2024 14:34:30 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 592C8633F3;\n\tFri, 19 Apr 2024 16:34:29 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 89C1F61C15\n\tfor ;\n\tFri, 19 Apr 2024 16:34:27 +0200 (CEST)","from ideasonboard.com (93-61-96-190.ip145.fastwebnet.it\n\t[93.61.96.190])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id DDF5563B;\n\tFri, 19 Apr 2024 16:33:38 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"VHe/4PbO\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1713537219;\n\tbh=vc7jzLtEkRc4xq+1sOqFK2Pse/4sPiY0++Wu/9x/sqA=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=VHe/4PbO9wziGsLTNNRFbDQPBiTC6uy6s/T4RCeMQUfBQFoj1ODoLrtRUYbxKLR0g\n\tPBqrAvrcWZcPtXzZwJxVf1k8ANvHihiUg14VfXZNLyzuUilVBXYRj1aRN0Xv55rwU4\n\tSWd2Pphgk3Byf+CoMBgejVAoWNM81PQ14ZTU6kEE=","Date":"Fri, 19 Apr 2024 16:34:24 +0200","From":"Jacopo Mondi ","To":"Daniel Scally ","Cc":"libcamera-devel@lists.libcamera.org","Subject":"Re: [PATCH v2 4/8] ipa: libipa: Add MeanLuminanceAgc base class","Message-ID":"<42axtt2x6v4deeuzolzgsw3ksyio3emfsx26ynfonfuh7nvs2w@m5a5ox6pohmn>","References":"<20240417131536.484129-1-dan.scally@ideasonboard.com>\n\t<20240417131536.484129-5-dan.scally@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20240417131536.484129-5-dan.scally@ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":29328,"web_url":"https://patchwork.libcamera.org/comment/29328/","msgid":"<0c4c9897-6474-4b9f-b24c-fb8671f74ab2@ideasonboard.com>","date":"2024-04-24T08:51:58","subject":"Re: [PATCH v2 4/8] ipa: libipa: Add MeanLuminanceAgc base class","submitter":{"id":156,"url":"https://patchwork.libcamera.org/api/people/156/","name":"Dan Scally","email":"dan.scally@ideasonboard.com"},"content":"Hi Stefan and Laurent\n\nOn 18/04/2024 08:48, Stefan Klug wrote:\n> Hi Dan,\n>\n> thank you for the patch.\n>\n> On Wed, Apr 17, 2024 at 02:15:32PM +0100, Daniel Scally wrote:\n>> The Agc algorithms for the RkIsp1 and IPU3 IPAs do the same thing in\n>> very large part; following the Rpi IPA's algorithm in spirit with a\n>> few tunable values in that IPA being hardcoded in the libipa ones.\n>> Add a new base class for MeanLuminanceAgc which implements the same\n>> algorithm and additionally parses yaml tuning files to inform an IPA\n>> module's Agc algorithm about valid constraint and exposure modes and\n>> their associated bounds.\n>>\n>> Signed-off-by: Daniel Scally \n>> ---\n>> Changes in v2:\n>>\n>> \t- Renamed the class and files\n>> \t- Expanded the documentation\n>> \t- Added parseTuningData() so derived classes can call a single function\n>> \t to cover all the parsing in ::init().\n>>\n>> src/ipa/libipa/agc_mean_luminance.cpp | 581 ++++++++++++++++++++++++++\n>> src/ipa/libipa/agc_mean_luminance.h | 91 ++++\n>> src/ipa/libipa/meson.build | 2 +\n>> 3 files changed, 674 insertions(+)\n>> create mode 100644 src/ipa/libipa/agc_mean_luminance.cpp\n>> create mode 100644 src/ipa/libipa/agc_mean_luminance.h\n>>\n>> diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp\n>> new file mode 100644\n>> index 00000000..02e223cf\n>> --- /dev/null\n>> +++ b/src/ipa/libipa/agc_mean_luminance.cpp\n>> @@ -0,0 +1,581 @@\n>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n>> +/*\n>> + * Copyright (C) 2024 Ideas on Board Oy\n>> + *\n>> + * agc_mean_luminance.cpp - Base class for mean luminance AGC algorithms\n>> + */\n>> +\n>> +#include \"agc_mean_luminance.h\"\n>> +\n>> +#include \n>> +\n>> +#include \n>> +#include \n>> +\n>> +#include \"exposure_mode_helper.h\"\n>> +\n>> +using namespace libcamera::controls;\n>> +\n>> +/**\n>> + * \\file agc_mean_luminance.h\n>> + * \\brief Base class implementing mean luminance AEGC\n>> + */\n>> +\n>> +namespace libcamera {\n>> +\n>> +using namespace std::literals::chrono_literals;\n>> +\n>> +LOG_DEFINE_CATEGORY(AgcMeanLuminance)\n>> +\n>> +namespace ipa {\n>> +\n>> +/*\n>> + * Number of frames for which to run the algorithm at full speed, before slowing\n>> + * down to prevent large and jarring changes in exposure from frame to frame.\n>> + */\n>> +static constexpr uint32_t kNumStartupFrames = 10;\n>> +\n>> +/*\n>> + * Default relative luminance target\n>> + *\n>> + * This value should be chosen so that when the camera points at a grey target,\n>> + * the resulting image brightness looks \"right\". Custom values can be passed\n>> + * as the relativeLuminanceTarget value in sensor tuning files.\n>> + */\n>> +static constexpr double kDefaultRelativeLuminanceTarget = 0.16;\n>> +\n>> +/**\n>> + * \\struct AgcMeanLuminance::AgcConstraint\n>> + * \\brief The boundaries and target for an AeConstraintMode constraint\n>> + *\n>> + * This structure describes an AeConstraintMode constraint for the purposes of\n>> + * this algorithm. The algorithm will apply the constraints by calculating the\n>> + * Histogram's inter-quantile mean between the given quantiles and ensure that\n>> + * the resulting value is the right side of the given target (as defined by the\n>> + * boundary and luminance target).\n>> + */\n>> +\n>> +/**\n>> + * \\enum AgcMeanLuminance::AgcConstraint::Bound\n>> + * \\brief Specify whether the constraint defines a lower or upper bound\n>> + * \\var AgcMeanLuminance::AgcConstraint::lower\n>> + * \\brief The constraint defines a lower bound\n>> + * \\var AgcMeanLuminance::AgcConstraint::upper\n>> + * \\brief The constraint defines an upper bound\n>> + */\n>> +\n>> +/**\n>> + * \\var AgcMeanLuminance::AgcConstraint::bound\n>> + * \\brief The type of constraint bound\n>> + */\n>> +\n>> +/**\n>> + * \\var AgcMeanLuminance::AgcConstraint::qLo\n>> + * \\brief The lower quantile to use for the constraint\n>> + */\n>> +\n>> +/**\n>> + * \\var AgcMeanLuminance::AgcConstraint::qHi\n>> + * \\brief The upper quantile to use for the constraint\n>> + */\n>> +\n>> +/**\n>> + * \\var AgcMeanLuminance::AgcConstraint::yTarget\n>> + * \\brief The luminance target for the constraint\n>> + */\n>> +\n>> +/**\n>> + * \\class AgcMeanLuminance\n>> + * \\brief A mean-based auto-exposure algorithm\n>> + *\n>> + * This algorithm calculates a shutter time, analogue and digital gain such that\n>> + * the normalised mean luminance value of an image is driven towards a target,\n>> + * which itself is discovered from tuning data. The algorithm is a two-stage\n>> + * process.\n>> + *\n>> + * In the first stage, an initial gain value is derived by iteratively comparing\n>> + * the gain-adjusted mean luminance across an entire image against a target, and\n>> + * selecting a value which pushes it as closely as possible towards the target.\n>> + *\n>> + * In the second stage we calculate the gain required to drive the average of a\n>> + * section of a histogram to a target value, where the target and the boundaries\n>> + * of the section of the histogram used in the calculation are taken from the\n>> + * values defined for the currently configured AeConstraintMode within the\n>> + * tuning data. This class provides a helper function to parse those tuning data\n>> + * to discover the constraints, and so requires a specific format for those\n>> + * data which is described in \\ref parseTuningData(). The gain from the first\n>> + * stage is then clamped to the gain from this stage.\n>> + *\n>> + * The final gain is used to adjust the effective exposure value of the image,\n>> + * and that new exposure value is divided into shutter time, analogue gain and\n>> + * digital gain according to the selected AeExposureMode. This class expects to\n>> + * use the \\ref ExposureModeHelper class to assist in that division, and expects\n>> + * the data needed to initialise that class to be present in tuning data in a\n>> + * format described in \\ref parseTuningData().\n>> + *\n>> + * In order to be able to derive an AGC implementation from this class, an IPA\n>> + * needs to be able to do the following:\n>> + *\n>> + * 1. Provide a luminance estimation across an entire image.\n>> + * 2. Provide a luminance Histogram for the image to use in calculating\n>> + * constraint compliance. The precision of the Histogram that is available\n>> + * will determine the supportable precision of the constraints.\n>> + */\n>> +\n>> +AgcMeanLuminance::AgcMeanLuminance()\n>> +\t: frameCount_(0), filteredExposure_(0s), relativeLuminanceTarget_(0)\n>> +{\n>> +}\n>> +\n>> +/**\n>> + * \\brief Parse the relative luminance target from the tuning data\n>> + * \\param[in] tuningData The YamlObject holding the algorithm's tuning data\n>> + */\n>> +void AgcMeanLuminance::parseRelativeLuminanceTarget(const YamlObject &tuningData)\n>> +{\n>> +\trelativeLuminanceTarget_ =\n>> +\t\ttuningData[\"relativeLuminanceTarget\"].get(kDefaultRelativeLuminanceTarget);\n>> +}\n>> +\n>> +/**\n>> + * \\brief Parse an AeConstraintMode constraint from tuning data\n>> + * \\param[in] modeDict the YamlObject holding the constraint data\n>> + * \\param[in] id The constraint ID from AeConstraintModeEnum\n>> + */\n>> +void AgcMeanLuminance::parseConstraint(const YamlObject &modeDict, int32_t id)\n>> +{\n>> +\tfor (const auto &[boundName, content] : modeDict.asDict()) {\n>> +\t\tif (boundName != \"upper\" && boundName != \"lower\") {\n>> +\t\t\tLOG(AgcMeanLuminance, Warning)\n>> +\t\t\t\t<< \"Ignoring unknown constraint bound '\" << boundName << \"'\";\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\n>> +\t\tunsigned int idx = static_cast(boundName == \"upper\");\n>> +\t\tAgcConstraint::Bound bound = static_cast(idx);\n>> +\t\tdouble qLo = content[\"qLo\"].get().value_or(0.98);\n>> +\t\tdouble qHi = content[\"qHi\"].get().value_or(1.0);\n>> +\t\tdouble yTarget =\n>> +\t\t\tcontent[\"yTarget\"].getList().value_or(std::vector{ 0.5 }).at(0);\n>> +\n>> +\t\tAgcConstraint constraint = { bound, qLo, qHi, yTarget };\n>> +\n>> +\t\tif (!constraintModes_.count(id))\n>> +\t\t\tconstraintModes_[id] = {};\n>> +\n>> +\t\tif (idx)\n>> +\t\t\tconstraintModes_[id].push_back(constraint);\n>> +\t\telse\n>> +\t\t\tconstraintModes_[id].insert(constraintModes_[id].begin(), constraint);\n>> +\t}\n>> +}\n>> +\n>> +int AgcMeanLuminance::parseConstraintModes(const YamlObject &tuningData)\n>> +{\n>> +\tstd::vector availableConstraintModes;\n>> +\n>> +\tconst YamlObject &yamlConstraintModes = tuningData[controls::AeConstraintMode.name()];\n>> +\tif (yamlConstraintModes.isDictionary()) {\n>> +\t\tfor (const auto &[modeName, modeDict] : yamlConstraintModes.asDict()) {\n>> +\t\t\tif (AeConstraintModeNameValueMap.find(modeName) ==\n>> +\t\t\t AeConstraintModeNameValueMap.end()) {\n>> +\t\t\t\tLOG(AgcMeanLuminance, Warning)\n>> +\t\t\t\t\t<< \"Skipping unknown constraint mode '\" << modeName << \"'\";\n>> +\t\t\t\tcontinue;\n>> +\t\t\t}\n>> +\n>> +\t\t\tif (!modeDict.isDictionary()) {\n>> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n>> +\t\t\t\t\t<< \"Invalid constraint mode '\" << modeName << \"'\";\n>> +\t\t\t\treturn -EINVAL;\n>> +\t\t\t}\n>> +\n>> +\t\t\tparseConstraint(modeDict,\n>> +\t\t\t\t\tAeConstraintModeNameValueMap.at(modeName));\n>> +\t\t\tavailableConstraintModes.push_back(\n>> +\t\t\t\tAeConstraintModeNameValueMap.at(modeName));\n>> +\t\t}\n>> +\t}\n>> +\n>> +\t/*\n>> +\t * If the tuning data file contains no constraints then we use the\n>> +\t * default constraint that the various Agc algorithms were adhering to\n>> +\t * anyway before centralisation.\n>> +\t */\n>> +\tif (constraintModes_.empty()) {\n>> +\t\tAgcConstraint constraint = {\n>> +\t\t\tAgcConstraint::Bound::lower,\n>> +\t\t\t0.98,\n>> +\t\t\t1.0,\n>> +\t\t\t0.5\n>> +\t\t};\n>> +\n>> +\t\tconstraintModes_[controls::ConstraintNormal].insert(\n>> +\t\t\tconstraintModes_[controls::ConstraintNormal].begin(),\n>> +\t\t\tconstraint);\n>> +\t\tavailableConstraintModes.push_back(\n>> +\t\t\tAeConstraintModeNameValueMap.at(\"ConstraintNormal\"));\n>> +\t}\n>> +\n>> +\tcontrols_[&controls::AeConstraintMode] = ControlInfo(availableConstraintModes);\n>> +\n>> +\treturn 0;\n>> +}\n>> +\n>> +int AgcMeanLuminance::parseExposureModes(const YamlObject &tuningData)\n>> +{\n>> +\tstd::vector availableExposureModes;\n>> +\n>> +\tconst YamlObject &yamlExposureModes = tuningData[controls::AeExposureMode.name()];\n>> +\tif (yamlExposureModes.isDictionary()) {\n>> +\t\tfor (const auto &[modeName, modeValues] : yamlExposureModes.asDict()) {\n>> +\t\t\tif (AeExposureModeNameValueMap.find(modeName) ==\n>> +\t\t\t AeExposureModeNameValueMap.end()) {\n>> +\t\t\t\tLOG(AgcMeanLuminance, Warning)\n>> +\t\t\t\t\t<< \"Skipping unknown exposure mode '\" << modeName << \"'\";\n>> +\t\t\t\tcontinue;\n>> +\t\t\t}\n>> +\n>> +\t\t\tif (!modeValues.isDictionary()) {\n>> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n>> +\t\t\t\t\t<< \"Invalid exposure mode '\" << modeName << \"'\";\n>> +\t\t\t\treturn -EINVAL;\n>> +\t\t\t}\n>> +\n>> +\t\t\tstd::vector shutters =\n>> +\t\t\t\tmodeValues[\"shutter\"].getList().value_or(std::vector{});\n>> +\t\t\tstd::vector gains =\n>> +\t\t\t\tmodeValues[\"gain\"].getList().value_or(std::vector{});\n>> +\n>> +\t\t\tif (shutters.size() != gains.size()) {\n>> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n>> +\t\t\t\t\t<< \"Shutter and gain array sizes unequal\";\n>> +\t\t\t\treturn -EINVAL;\n>> +\t\t\t}\n>> +\n>> +\t\t\tif (shutters.empty()) {\n>> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n>> +\t\t\t\t\t<< \"Shutter and gain arrays are empty\";\n>> +\t\t\t\treturn -EINVAL;\n>> +\t\t\t}\n>> +\n>> +\t\t\tstd::vector> stages;\n>> +\t\t\tfor (unsigned int i = 0; i < shutters.size(); i++) {\n>> +\t\t\t\tstages.push_back({\n>> +\t\t\t\t\tstd::chrono::microseconds(shutters[i]),\n>> +\t\t\t\t\tgains[i]\n>> +\t\t\t\t});\n>> +\t\t\t}\n>> +\n>> +\t\t\tstd::shared_ptr helper =\n>> +\t\t\t\tstd::make_shared();\n>> +\t\t\thelper->init(stages);\n>> +\n>> +\t\t\texposureModeHelpers_[AeExposureModeNameValueMap.at(modeName)] = helper;\n>> +\t\t\tavailableExposureModes.push_back(AeExposureModeNameValueMap.at(modeName));\n>> +\t\t}\n>> +\t}\n>> +\n>> +\t/*\n>> +\t * If we don't have any exposure modes in the tuning data we create an\n>> +\t * ExposureModeHelper using an empty vector of stages. This will result\n>> +\t * in the ExposureModeHelper simply driving the shutter as high as\n>> +\t * possible before touching gain.\n>> +\t */\n>> +\tif (availableExposureModes.empty()) {\n>> +\t\tint32_t exposureModeId = AeExposureModeNameValueMap.at(\"ExposureNormal\");\n>> +\t\tstd::vector> stages = { };\n>> +\n>> +\t\tstd::shared_ptr helper =\n>> +\t\t\tstd::make_shared();\n>> +\t\thelper->init(stages);\n>> +\n>> +\t\texposureModeHelpers_[exposureModeId] = helper;\n>> +\t\tavailableExposureModes.push_back(exposureModeId);\n>> +\t}\n>> +\n>> +\tcontrols_[&controls::AeExposureMode] = ControlInfo(availableExposureModes);\n>> +\n>> +\treturn 0;\n>> +}\n>> +\n>> +/**\n>> + * \\brief Parse tuning data for AeConstraintMode and AeExposureMode controls\n>> + * \\param[in] tuningData the YamlObject representing the tuning data\n>> + *\n>> + * This function parses tuning data to build the list of allowed values for the\n>> + * AeConstraintMode and AeExposureMode controls. Those tuning data must provide\n>> + * the data in a specific format; the Agc algorithm's tuning data should contain\n>> + * a dictionary called AeConstraintMode containing per-mode setting dictionaries\n>> + * with the key being a value from \\ref controls::AeConstraintModeNameValueMap.\n>> + * Each mode dict may contain either a \"lower\" or \"upper\" key or both, for\n>> + * example:\n>> + *\n>> + * \\code{.unparsed}\n>> + * algorithms:\n>> + * - Agc:\n>> + * AeConstraintMode:\n>> + * ConstraintNormal:\n>> + * lower:\n>> + * qLo: 0.98\n>> + * qHi: 1.0\n>> + * yTarget: 0.5\n>> + * ConstraintHighlight:\n>> + * lower:\n>> + * qLo: 0.98\n>> + * qHi: 1.0\n>> + * yTarget: 0.5\n>> + * upper:\n>> + * qLo: 0.98\n>> + * qHi: 1.0\n>> + * yTarget: 0.8\n>> + *\n>> + * \\endcode\n>> + *\n>> + * For the AeExposureMode control the data should contain a dictionary called\n>> + * AeExposureMode containing per-mode setting dictionaries with the key being a\n>> + * value from \\ref controls::AeExposureModeNameValueMap. Each mode dict should\n>> + * contain an array of shutter times with the key \"shutter\" and an array of gain\n>> + * values with the key \"gain\", in this format:\n>> + *\n>> + * \\code{.unparsed}\n>> + * algorithms:\n>> + * - Agc:\n>> + * AeExposureMode:\n>> + * ExposureNormal:\n>> + * shutter: [ 100, 10000, 30000, 60000, 120000 ]\n>> + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ]\n>> + * ExposureShort:\n>> + * shutter: [ 100, 10000, 30000, 60000, 120000 ]\n>> + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ]\n>> + *\n>> + * \\endcode\n>> + *\n>> + * @return 0 on success or a negative error code\n>> + */\n>> +int AgcMeanLuminance::parseTuningData(const YamlObject &tuningData)\n>> +{\n>> +\tint ret;\n>> +\n>> +\tparseRelativeLuminanceTarget(tuningData);\n>> +\n>> +\tret = parseConstraintModes(tuningData);\n>> +\tif (ret)\n>> +\t\treturn ret;\n>> +\n>> +\tret = parseExposureModes(tuningData);\n>> +\tif (ret)\n>> +\t\treturn ret;\n>> +\n>> +\treturn 0;\n>> +}\n>> +\n>> +/**\n>> + * \\brief configure the ExposureModeHelpers for this class\n>> + * \\param[in] minShutter Minimum shutter time to allow\n>> + * \\param[in] maxShutter Maximum shutter time to allow\n>> + * \\param[in] minGain Minimum gain to allow\n>> + * \\param[in] maxGain Maximum gain to allow\n>> + *\n>> + * This function calls \\ref ExposureModeHelper::setShutterGainLimits() for each\n>> + * ExposureModeHelper that has been created for this class.\n>> + */\n>> +void AgcMeanLuminance::configureExposureModeHelpers(utils::Duration minShutter,\n>> +\t\t\t\t\t\t utils::Duration maxShutter,\n>> +\t\t\t\t\t\t double minGain,\n>> +\t\t\t\t\t\t double maxGain)\n>> +{\n>> +\tfor (auto &[id, helper] : exposureModeHelpers_)\n>> +\t\thelper->setShutterGainLimits(minShutter, maxShutter, minGain, maxGain);\n>> +}\n>> +\n>> +/**\n>> + * \\fn AgcMeanLuminance::constraintModes()\n>> + * \\brief Get the constraint modes that have been parsed from tuning data\n>> + */\n>> +\n>> +/**\n>> + * \\fn AgcMeanLuminance::exposureModeHelpers()\n>> + * \\brief Get the ExposureModeHelpers that have been parsed from tuning data\n>> + */\n>> +\n>> +/**\n>> + * \\fn AgcMeanLuminance::controls()\n>> + * \\brief Get the controls that have been generated after parsing tuning data\n>> + */\n>> +\n>> +/**\n>> + * \\fn AgcMeanLuminance::estimateLuminance(const double gain)\n>> + * \\brief Estimate the luminance of an image, adjusted by a given gain\n>> + * \\param[in] gain The gain with which to adjust the luminance estimate\n>> + *\n>> + * This function estimates the average relative luminance of the frame that\n>> + * would be output by the sensor if an additional \\a gain was applied. It is a\n>> + * pure virtual function because estimation of luminance is a hardware-specific\n>> + * operation, which depends wholly on the format of the stats that are delivered\n>> + * to libcamera from the ISP. Derived classes must implement an overriding\n>> + * function that calculates the normalised mean luminance value across the\n>> + * entire image.\n>> + *\n>> + * \\return The normalised relative luminance of the image\n>> + */\n>> +\n>> +/**\n>> + * \\brief Estimate the initial gain needed to achieve a relative luminance\n>> + * target\n>> + *\n>> + * To account for non-linearity caused by saturation, the value needs to be\n>> + * estimated in an iterative process, as multiplying by a gain will not increase\n>> + * the relative luminance by the same factor if some image regions are saturated\n>> + *\n>> + * \\return The calculated initial gain\n>> + */\n>> +double AgcMeanLuminance::estimateInitialGain()\n>> +{\n>> +\tdouble yTarget = relativeLuminanceTarget_;\n>> +\tdouble yGain = 1.0;\n>> +\n>> +\tfor (unsigned int i = 0; i < 8; i++) {\n>> +\t\tdouble yValue = estimateLuminance(yGain);\n>> +\t\tdouble extra_gain = std::min(10.0, yTarget / (yValue + .001));\n>> +\n>> +\t\tyGain *= extra_gain;\n>> +\t\tLOG(AgcMeanLuminance, Debug) << \"Y value: \" << yValue\n>> +\t\t\t\t<< \", Y target: \" << yTarget\n>> +\t\t\t\t<< \", gives gain \" << yGain;\n>> +\n>> +\t\tif (utils::abs_diff(extra_gain, 1.0) < 0.01)\n>> +\t\t\tbreak;\n>> +\t}\n>> +\n>> +\treturn yGain;\n>> +}\n>> +\n>> +/**\n>> + * \\brief Clamp gain within the bounds of a defined constraint\n>> + * \\param[in] constraintModeIndex The index of the constraint to adhere to\n>> + * \\param[in] hist A histogram over which to calculate inter-quantile means\n>> + * \\param[in] gain The gain to clamp\n>> + *\n>> + * \\return The gain clamped within the constraint bounds\n>> + */\n>> +double AgcMeanLuminance::constraintClampGain(uint32_t constraintModeIndex,\n>> +\t\t\t\t\t const Histogram &hist,\n>> +\t\t\t\t\t double gain)\n>> +{\n>> +\tstd::vector &constraints = constraintModes_[constraintModeIndex];\n>> +\tfor (const AgcConstraint &constraint : constraints) {\n>> +\t\tdouble newGain = constraint.yTarget * hist.bins() /\n>> +\t\t\t\t hist.interQuantileMean(constraint.qLo, constraint.qHi);\n>> +\n>> +\t\tif (constraint.bound == AgcConstraint::Bound::lower &&\n>> +\t\t newGain > gain)\n>> +\t\t\tgain = newGain;\n>> +\n>> +\t\tif (constraint.bound == AgcConstraint::Bound::upper &&\n>> +\t\t newGain < gain)\n>> +\t\t\tgain = newGain;\n>> +\t}\n>> +\n>> +\treturn gain;\n>> +}\n>> +\n>> +/**\n>> + * \\brief Apply a filter on the exposure value to limit the speed of changes\n>> + * \\param[in] exposureValue The target exposure from the AGC algorithm\n>> + *\n>> + * The speed of the filter is adaptive, and will produce the target quicker\n>> + * during startup, or when the target exposure is within 20% of the most recent\n>> + * filter output.\n>> + *\n>> + * \\return The filtered exposure\n>> + */\n>> +utils::Duration AgcMeanLuminance::filterExposure(utils::Duration exposureValue)\n>> +{\n>> +\tdouble speed = 0.2;\n>> +\n>> +\t/* Adapt instantly if we are in startup phase. */\n>> +\tif (frameCount_ < kNumStartupFrames)\n>> +\t\tspeed = 1.0;\n>> +\n>> +\t/*\n>> +\t * If we are close to the desired result, go faster to avoid making\n>> +\t * multiple micro-adjustments.\n>> +\t * \\todo Make this customisable?\n>> +\t */\n>> +\tif (filteredExposure_ < 1.2 * exposureValue &&\n>> +\t filteredExposure_ > 0.8 * exposureValue)\n>> +\t\tspeed = sqrt(speed);\n>> +\n>> +\tfilteredExposure_ = speed * exposureValue +\n>> +\t\t\t filteredExposure_ * (1.0 - speed);\n>> +\n>> +\treturn filteredExposure_;\n>> +}\n>> +\n>> +/**\n>> + * \\brief Calculate the new exposure value\n>> + * \\param[in] constraintModeIndex The index of the current constraint mode\n>> + * \\param[in] exposureModeIndex The index of the current exposure mode\n>> + * \\param[in] yHist A Histogram from the ISP statistics to use in constraining\n>> + *\t the calculated gain\n> nit: no indentation\n>\n>> + * \\param[in] effectiveExposureValue The EV applied to the frame from which the\n>> + *\t statistics in use derive\n> nit: no indentation\n>\n>> + *\n>> + * Calculate a new exposure value to try to obtain the target. The calculated\n>> + * exposure value is filtered to prevent rapid changes from frame to frame, and\n>> + * divided into shutter time, analogue and digital gain.\n>> + *\n>> + * \\return Tuple of shutter time, analogue gain, and digital gain\n>> + */\n>> +std::tuple\n>> +AgcMeanLuminance::calculateNewEv(uint32_t constraintModeIndex,\n>> +\t\t\t\t uint32_t exposureModeIndex,\n>> +\t\t\t\t const Histogram &yHist,\n>> +\t\t\t\t utils::Duration effectiveExposureValue)\n>> +{\n>> +\t/*\n>> +\t * The pipeline handler should validate that we have received an allowed\n>> +\t * value for AeExposureMode.\n>> +\t */\n>> +\tstd::shared_ptr exposureModeHelper =\n>> +\t\texposureModeHelpers_.at(exposureModeIndex);\n>> +\n>> +\tdouble gain = estimateInitialGain();\n>> +\tgain = constraintClampGain(constraintModeIndex, yHist, gain);\n>> +\n>> +\t/*\n>> +\t * We don't check whether we're already close to the target, because\n>> +\t * even if the effective exposure value is the same as the last frame's\n>> +\t * we could have switched to an exposure mode that would require a new\n>> +\t * pass through the splitExposure() function.\n>> +\t */\n>> +\n>> +\tutils::Duration newExposureValue = effectiveExposureValue * gain;\n>> +\tutils::Duration maxTotalExposure = exposureModeHelper->maxShutter()\n>> +\t\t\t\t\t * exposureModeHelper->maxGain();\n>> +\tnewExposureValue = std::min(newExposureValue, maxTotalExposure);\n>> +\n>> +\t/*\n>> +\t * We filter the exposure value to make sure changes are not too jarring\n>> +\t * from frame to frame.\n>> +\t */\n>> +\tnewExposureValue = filterExposure(newExposureValue);\n>> +\n>> +\tframeCount_++;\n>> +\treturn exposureModeHelper->splitExposure(newExposureValue);\n>> +}\n>> +\n>> +/**\n>> + * \\fn AgcMeanLuminance::resetFrameCount()\n>> + * \\brief Reset the frame counter\n>> + *\n>> + * This function resets the internal frame counter, which exists to help the\n>> + * algorithm decide whether it should respond instantly or not. The expectation\n>> + * is for derived classes to call this function before each camera start call,\n>> + * either in configure() or queueRequest() if the frame number is zero.\n>> + */\n>> +\n>> +}; /* namespace ipa */\n>> +\n>> +}; /* namespace libcamera */\n>> diff --git a/src/ipa/libipa/agc_mean_luminance.h b/src/ipa/libipa/agc_mean_luminance.h\n>> new file mode 100644\n>> index 00000000..e48dc498\n>> --- /dev/null\n>> +++ b/src/ipa/libipa/agc_mean_luminance.h\n>> @@ -0,0 +1,91 @@\n>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n>> +/*\n>> + * Copyright (C) 2024 Ideas on Board Oy\n>> + *\n>> + agc_mean_luminance.h - Base class for mean luminance AGC algorithms\n>> + */\n>> +\n>> +#pragma once\n>> +\n>> +#include \n>> +#include \n>> +\n>> +#include \n>> +\n>> +#include \"libcamera/internal/yaml_parser.h\"\n>> +\n>> +#include \"exposure_mode_helper.h\"\n>> +#include \"histogram.h\"\n>> +\n>> +namespace libcamera {\n>> +\n>> +namespace ipa {\n>> +\n>> +class AgcMeanLuminance\n>> +{\n>> +public:\n>> +\tAgcMeanLuminance();\n>> +\tvirtual ~AgcMeanLuminance() = default;\n> There were a few small comments from Laurent that got lost\n> * destructor in cpp\n\n\nThe compiler says I can't follow this suggestion; since the estimateLuminance() function is virtual \nI apparently need a virtual destructor:\n\n\nclass libcamera::ipa::AgcMeanLuminance’ has virtual functions and accessible non-virtual destructor\n\n\nOr am I doing something wrong?\n\n> * code sytel in enum\n\n\nIsn't this addressed by making them lowercase?\n\n> * missing line\n>\n> Aside from that, I think we should merge it in.\n>\n> Reviewed-by: Stefan Klug \n>\n> Cheers,\n> Stefan\n>\n>> +\n>> +\tstruct AgcConstraint {\n>> +\t\tenum class Bound {\n>> +\t\t\tlower = 0,\n>> +\t\t\tupper = 1\n>> +\t\t};\n>> +\t\tBound bound;\n>> +\t\tdouble qLo;\n>> +\t\tdouble qHi;\n>> +\t\tdouble yTarget;\n>> +\t};\n>> +\n>> +\tint parseTuningData(const YamlObject &tuningData);\n>> +\n>> +\tvoid configureExposureModeHelpers(utils::Duration minShutter,\n>> +\t\t\t\t\t utils::Duration maxShutter,\n>> +\t\t\t\t\t double minGain,\n>> +\t\t\t\t\t double maxGain);\n>> +\n>> +\tstd::map> constraintModes()\n>> +\t{\n>> +\t\treturn constraintModes_;\n>> +\t}\n>> +\n>> +\tstd::map> exposureModeHelpers()\n>> +\t{\n>> +\t\treturn exposureModeHelpers_;\n>> +\t}\n>> +\n>> +\tControlInfoMap::Map controls()\n>> +\t{\n>> +\t\treturn controls_;\n>> +\t}\n>> +\n>> +\tdouble estimateInitialGain();\n>> +\tdouble constraintClampGain(uint32_t constraintModeIndex,\n>> +\t\t\t\t const Histogram &hist,\n>> +\t\t\t\t double gain);\n>> +\tutils::Duration filterExposure(utils::Duration exposureValue);\n>> +\tstd::tuple\n>> +\tcalculateNewEv(uint32_t constraintModeIndex, uint32_t exposureModeIndex,\n>> +\t\t const Histogram &yHist, utils::Duration effectiveExposureValue);\n>> +\tvoid resetFrameCount() { frameCount_ = 0; }\n>> +private:\n>> +\tvirtual double estimateLuminance(const double gain) = 0;\n>> +\n>> +\tvoid parseRelativeLuminanceTarget(const YamlObject &tuningData);\n>> +\tvoid parseConstraint(const YamlObject &modeDict, int32_t id);\n>> +\tint parseConstraintModes(const YamlObject &tuningData);\n>> +\tint parseExposureModes(const YamlObject &tuningData);\n>> +\n>> +\tuint64_t frameCount_;\n>> +\tutils::Duration filteredExposure_;\n>> +\tdouble relativeLuminanceTarget_;\n>> +\n>> +\tstd::map> constraintModes_;\n>> +\tstd::map> exposureModeHelpers_;\n>> +\tControlInfoMap::Map controls_;\n>> +};\n>> +\n>> +}; /* namespace ipa */\n>> +\n>> +}; /* namespace libcamera */\n>> diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build\n>> index 37fbd177..7ce885da 100644\n>> --- a/src/ipa/libipa/meson.build\n>> +++ b/src/ipa/libipa/meson.build\n>> @@ -1,6 +1,7 @@\n>> # SPDX-License-Identifier: CC0-1.0\n>> \n>> libipa_headers = files([\n>> + 'agc_mean_luminance.h',\n>> 'algorithm.h',\n>> 'camera_sensor_helper.h',\n>> 'exposure_mode_helper.h',\n>> @@ -10,6 +11,7 @@ libipa_headers = files([\n>> ])\n>> \n>> libipa_sources = files([\n>> + 'agc_mean_luminance.cpp',\n>> 'algorithm.cpp',\n>> 'camera_sensor_helper.cpp',\n>> 'exposure_mode_helper.cpp',\n>> -- \n>> 2.34.1\n>>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id A9AB2BE08B\n\tfor ;\n\tWed, 24 Apr 2024 08:52:03 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id BBE1B633EB;\n\tWed, 24 Apr 2024 10:52:02 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 4F99C61B14\n\tfor ;\n\tWed, 24 Apr 2024 10:52:01 +0200 (CEST)","from [192.168.0.43]\n\t(cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 5FC2066B;\n\tWed, 24 Apr 2024 10:51:09 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"ZCdH0KdO\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1713948669;\n\tbh=WJgOCMko80ZYRIWR0fInhY1fT7RcdREmYgAKPBmeidE=;\n\th=Date:Subject:To:Cc:References:From:In-Reply-To:From;\n\tb=ZCdH0KdOLtpDOgo8GHtijEM1Mveuk42Xm4ITmDb4ppjbz35K54TyvmnPRSAR2ZGmT\n\t6vwNUaGjS+MK7ElEiL7o3wY14jiCFIQwrKf4T4wd/ATt7jgz0kGj4/EqQP63mjfkBM\n\trS4T11QGuGb0TWfOgeCiGdDPnNvVYV2blnzphHfc=","Message-ID":"<0c4c9897-6474-4b9f-b24c-fb8671f74ab2@ideasonboard.com>","Date":"Wed, 24 Apr 2024 09:51:58 +0100","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH v2 4/8] ipa: libipa: Add MeanLuminanceAgc base class","To":"Stefan Klug ,\n\tLaurent Pinchart ","Cc":"libcamera-devel@lists.libcamera.org","References":"<20240417131536.484129-1-dan.scally@ideasonboard.com>\n\t<20240417131536.484129-5-dan.scally@ideasonboard.com>\n\t<20240418074823.l2k273uo5fx2nmwp@jasper>","Content-Language":"en-US","From":"Dan Scally ","Autocrypt":"addr=dan.scally@ideasonboard.com; keydata=\n\txsFNBGLydlEBEADa5O2s0AbUguprfvXOQun/0a8y2Vk6BqkQALgeD6KnXSWwaoCULp18etYW\n\tB31bfgrdphXQ5kUQibB0ADK8DERB4wrzrUb5CMxLBFE7mQty+v5NsP0OFNK9XTaAOcmD+Ove\n\teIjYvqurAaro91jrRVrS1gBRxIFqyPgNvwwL+alMZhn3/2jU2uvBmuRrgnc/e9cHKiuT3Dtq\n\tMHGPKL2m+plk+7tjMoQFfexoQ1JKugHAjxAhJfrkXh6uS6rc01bYCyo7ybzg53m1HLFJdNGX\n\tsUKR+dQpBs3SY4s66tc1sREJqdYyTsSZf80HjIeJjU/hRunRo4NjRIJwhvnK1GyjOvvuCKVU\n\tRWpY8dNjNu5OeAfdrlvFJOxIE9M8JuYCQTMULqd1NuzbpFMjc9524U3Cngs589T7qUMPb1H1\n\tNTA81LmtJ6Y+IV5/kiTUANflpzBwhu18Ok7kGyCq2a2jsOcVmk8gZNs04gyjuj8JziYwwLbf\n\tvzABwpFVcS8aR+nHIZV1HtOzyw8CsL8OySc3K9y+Y0NRpziMRvutrppzgyMb9V+N31mK9Mxl\n\t1YkgaTl4ciNWpdfUe0yxH03OCuHi3922qhPLF4XX5LN+NaVw5Xz2o3eeWklXdouxwV7QlN33\n\tu4+u2FWzKxDqO6WLQGjxPE0mVB4Gh5Pa1Vb0ct9Ctg0qElvtGQARAQABzShEYW4gU2NhbGx5\n\tIDxkYW4uc2NhbGx5QGlkZWFzb25ib2FyZC5jb20+wsGNBBMBCAA3FiEEsdtt8OWP7+8SNfQe\n\tkiQuh/L+GMQFAmLydlIFCQWjmoACGwMECwkIBwUVCAkKCwUWAgMBAAAKCRCSJC6H8v4YxDI2\n\tEAC2Gz0iyaXJkPInyshrREEWbo0CA6v5KKf3I/HlMPqkZ48bmGoYm4mEQGFWZJAT3K4ir8bg\n\tcEfs9V54gpbrZvdwS4abXbUK4WjKwEs8HK3XJv1WXUN2bsz5oEJWZUImh9gD3naiLLI9QMMm\n\tw/aZkT+NbN5/2KvChRWhdcha7+2Te4foOY66nIM+pw2FZM6zIkInLLUik2zXOhaZtqdeJZQi\n\tHSPU9xu7TRYN4cvdZAnSpG7gQqmLm5/uGZN1/sB3kHTustQtSXKMaIcD/DMNI3JN/t+RJVS7\n\tc0Jh/ThzTmhHyhxx3DRnDIy7kwMI4CFvmhkVC2uNs9kWsj1DuX5kt8513mvfw2OcX9UnNKmZ\n\tnhNCuF6DxVrL8wjOPuIpiEj3V+K7DFF1Cxw1/yrLs8dYdYh8T8vCY2CHBMsqpESROnTazboh\n\tAiQ2xMN1cyXtX11Qwqm5U3sykpLbx2BcmUUUEAKNsM//Zn81QXKG8vOx0ZdMfnzsCaCzt8f6\n\t9dcDBBI3tJ0BI9ByiocqUoL6759LM8qm18x3FYlxvuOs4wSGPfRVaA4yh0pgI+ModVC2Pu3y\n\tejE/IxeatGqJHh6Y+iJzskdi27uFkRixl7YJZvPJAbEn7kzSi98u/5ReEA8Qhc8KO/B7wprj\n\txjNMZNYd0Eth8+WkixHYj752NT5qshKJXcyUU87BTQRi8nZSARAAx0BJayh1Fhwbf4zoY56x\n\txHEpT6DwdTAYAetd3yiKClLVJadYxOpuqyWa1bdfQWPb+h4MeXbWw/53PBgn7gI2EA7ebIRC\n\tPJJhAIkeym7hHZoxqDQTGDJjxFEL11qF+U3rhWiL2Zt0Pl+zFq0eWYYVNiXjsIS4FI2+4m16\n\ttPbDWZFJnSZ828VGtRDQdhXfx3zyVX21lVx1bX4/OZvIET7sVUufkE4hrbqrrufre7wsjD1t\n\t8MQKSapVrr1RltpzPpScdoxknOSBRwOvpp57pJJe5A0L7+WxJ+vQoQXj0j+5tmIWOAV1qBQp\n\thyoyUk9JpPfntk2EKnZHWaApFp5TcL6c5LhUvV7F6XwOjGPuGlZQCWXee9dr7zym8iR3irWT\n\t+49bIh5PMlqSLXJDYbuyFQHFxoiNdVvvf7etvGfqFYVMPVjipqfEQ38ST2nkzx+KBICz7uwj\n\tJwLBdTXzGFKHQNckGMl7F5QdO/35An/QcxBnHVMXqaSd12tkJmoRVWduwuuoFfkTY5mUV3uX\n\txGj3iVCK4V+ezOYA7c2YolfRCNMTza6vcK/P4tDjjsyBBZrCCzhBvd4VVsnnlZhVaIxoky4K\n\taL+AP+zcQrUZmXmgZjXOLryGnsaeoVrIFyrU6ly90s1y3KLoPsDaTBMtnOdwxPmo1xisH8oL\n\ta/VRgpFBfojLPxMAEQEAAcLBfAQYAQgAJhYhBLHbbfDlj+/vEjX0HpIkLofy/hjEBQJi8nZT\n\tBQkFo5qAAhsMAAoJEJIkLofy/hjEXPcQAMIPNqiWiz/HKu9W4QIf1OMUpKn3YkVIj3p3gvfM\n\tRes4fGX94Ji599uLNrPoxKyaytC4R6BTxVriTJjWK8mbo9jZIRM4vkwkZZ2bu98EweSucxbp\n\tvjESsvMXGgxniqV/RQ/3T7LABYRoIUutARYq58p5HwSP0frF0fdFHYdTa2g7MYZl1ur2JzOC\n\tFHRpGadlNzKDE3fEdoMobxHB3Lm6FDml5GyBAA8+dQYVI0oDwJ3gpZPZ0J5Vx9RbqXe8RDuR\n\tdu90hvCJkq7/tzSQ0GeD3BwXb9/R/A4dVXhaDd91Q1qQXidI+2jwhx8iqiYxbT+DoAUkQRQy\n\txBtoCM1CxH7u45URUgD//fxYr3D4B1SlonA6vdaEdHZOGwECnDpTxecENMbz/Bx7qfrmd901\n\tD+N9SjIwrbVhhSyUXYnSUb8F+9g2RDY42Sk7GcYxIeON4VzKqWM7hpkXZ47pkK0YodO+dRKM\n\tyMcoUWrTK0Uz6UzUGKoJVbxmSW/EJLEGoI5p3NWxWtScEVv8mO49gqQdrRIOheZycDmHnItt\n\t9Qjv00uFhEwv2YfiyGk6iGF2W40s2pH2t6oeuGgmiZ7g6d0MEK8Ql/4zPItvr1c1rpwpXUC1\n\tu1kQWgtnNjFHX3KiYdqjcZeRBiry1X0zY+4Y24wUU0KsEewJwjhmCKAsju1RpdlPg2kC","In-Reply-To":"<20240418074823.l2k273uo5fx2nmwp@jasper>","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"8bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":29334,"web_url":"https://patchwork.libcamera.org/comment/29334/","msgid":"<20240424125609.GF18608@pendragon.ideasonboard.com>","date":"2024-04-24T12:56:09","subject":"Re: [PATCH v2 4/8] ipa: libipa: Add MeanLuminanceAgc base class","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Dan,\n\nOn Wed, Apr 24, 2024 at 09:51:58AM +0100, Daniel Scally wrote:\n> On 18/04/2024 08:48, Stefan Klug wrote:\n> > On Wed, Apr 17, 2024 at 02:15:32PM +0100, Daniel Scally wrote:\n> >> The Agc algorithms for the RkIsp1 and IPU3 IPAs do the same thing in\n> >> very large part; following the Rpi IPA's algorithm in spirit with a\n> >> few tunable values in that IPA being hardcoded in the libipa ones.\n> >> Add a new base class for MeanLuminanceAgc which implements the same\n> >> algorithm and additionally parses yaml tuning files to inform an IPA\n> >> module's Agc algorithm about valid constraint and exposure modes and\n> >> their associated bounds.\n> >>\n> >> Signed-off-by: Daniel Scally \n> >> ---\n> >> Changes in v2:\n> >>\n> >> \t- Renamed the class and files\n> >> \t- Expanded the documentation\n> >> \t- Added parseTuningData() so derived classes can call a single function\n> >> \t to cover all the parsing in ::init().\n> >>\n> >> src/ipa/libipa/agc_mean_luminance.cpp | 581 ++++++++++++++++++++++++++\n> >> src/ipa/libipa/agc_mean_luminance.h | 91 ++++\n> >> src/ipa/libipa/meson.build | 2 +\n> >> 3 files changed, 674 insertions(+)\n> >> create mode 100644 src/ipa/libipa/agc_mean_luminance.cpp\n> >> create mode 100644 src/ipa/libipa/agc_mean_luminance.h\n> >>\n> >> diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp\n> >> new file mode 100644\n> >> index 00000000..02e223cf\n> >> --- /dev/null\n> >> +++ b/src/ipa/libipa/agc_mean_luminance.cpp\n> >> @@ -0,0 +1,581 @@\n> >> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> >> +/*\n> >> + * Copyright (C) 2024 Ideas on Board Oy\n> >> + *\n> >> + * agc_mean_luminance.cpp - Base class for mean luminance AGC algorithms\n> >> + */\n> >> +\n> >> +#include \"agc_mean_luminance.h\"\n> >> +\n> >> +#include \n> >> +\n> >> +#include \n> >> +#include \n> >> +\n> >> +#include \"exposure_mode_helper.h\"\n> >> +\n> >> +using namespace libcamera::controls;\n> >> +\n> >> +/**\n> >> + * \\file agc_mean_luminance.h\n> >> + * \\brief Base class implementing mean luminance AEGC\n> >> + */\n> >> +\n> >> +namespace libcamera {\n> >> +\n> >> +using namespace std::literals::chrono_literals;\n> >> +\n> >> +LOG_DEFINE_CATEGORY(AgcMeanLuminance)\n> >> +\n> >> +namespace ipa {\n> >> +\n> >> +/*\n> >> + * Number of frames for which to run the algorithm at full speed, before slowing\n> >> + * down to prevent large and jarring changes in exposure from frame to frame.\n> >> + */\n> >> +static constexpr uint32_t kNumStartupFrames = 10;\n> >> +\n> >> +/*\n> >> + * Default relative luminance target\n> >> + *\n> >> + * This value should be chosen so that when the camera points at a grey target,\n> >> + * the resulting image brightness looks \"right\". Custom values can be passed\n> >> + * as the relativeLuminanceTarget value in sensor tuning files.\n> >> + */\n> >> +static constexpr double kDefaultRelativeLuminanceTarget = 0.16;\n> >> +\n> >> +/**\n> >> + * \\struct AgcMeanLuminance::AgcConstraint\n> >> + * \\brief The boundaries and target for an AeConstraintMode constraint\n> >> + *\n> >> + * This structure describes an AeConstraintMode constraint for the purposes of\n> >> + * this algorithm. The algorithm will apply the constraints by calculating the\n> >> + * Histogram's inter-quantile mean between the given quantiles and ensure that\n> >> + * the resulting value is the right side of the given target (as defined by the\n> >> + * boundary and luminance target).\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\enum AgcMeanLuminance::AgcConstraint::Bound\n> >> + * \\brief Specify whether the constraint defines a lower or upper bound\n> >> + * \\var AgcMeanLuminance::AgcConstraint::lower\n> >> + * \\brief The constraint defines a lower bound\n> >> + * \\var AgcMeanLuminance::AgcConstraint::upper\n> >> + * \\brief The constraint defines an upper bound\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\var AgcMeanLuminance::AgcConstraint::bound\n> >> + * \\brief The type of constraint bound\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\var AgcMeanLuminance::AgcConstraint::qLo\n> >> + * \\brief The lower quantile to use for the constraint\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\var AgcMeanLuminance::AgcConstraint::qHi\n> >> + * \\brief The upper quantile to use for the constraint\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\var AgcMeanLuminance::AgcConstraint::yTarget\n> >> + * \\brief The luminance target for the constraint\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\class AgcMeanLuminance\n> >> + * \\brief A mean-based auto-exposure algorithm\n> >> + *\n> >> + * This algorithm calculates a shutter time, analogue and digital gain such that\n> >> + * the normalised mean luminance value of an image is driven towards a target,\n> >> + * which itself is discovered from tuning data. The algorithm is a two-stage\n> >> + * process.\n> >> + *\n> >> + * In the first stage, an initial gain value is derived by iteratively comparing\n> >> + * the gain-adjusted mean luminance across an entire image against a target, and\n> >> + * selecting a value which pushes it as closely as possible towards the target.\n> >> + *\n> >> + * In the second stage we calculate the gain required to drive the average of a\n> >> + * section of a histogram to a target value, where the target and the boundaries\n> >> + * of the section of the histogram used in the calculation are taken from the\n> >> + * values defined for the currently configured AeConstraintMode within the\n> >> + * tuning data. This class provides a helper function to parse those tuning data\n> >> + * to discover the constraints, and so requires a specific format for those\n> >> + * data which is described in \\ref parseTuningData(). The gain from the first\n> >> + * stage is then clamped to the gain from this stage.\n> >> + *\n> >> + * The final gain is used to adjust the effective exposure value of the image,\n> >> + * and that new exposure value is divided into shutter time, analogue gain and\n> >> + * digital gain according to the selected AeExposureMode. This class expects to\n> >> + * use the \\ref ExposureModeHelper class to assist in that division, and expects\n> >> + * the data needed to initialise that class to be present in tuning data in a\n> >> + * format described in \\ref parseTuningData().\n> >> + *\n> >> + * In order to be able to derive an AGC implementation from this class, an IPA\n> >> + * needs to be able to do the following:\n> >> + *\n> >> + * 1. Provide a luminance estimation across an entire image.\n> >> + * 2. Provide a luminance Histogram for the image to use in calculating\n> >> + * constraint compliance. The precision of the Histogram that is available\n> >> + * will determine the supportable precision of the constraints.\n> >> + */\n> >> +\n> >> +AgcMeanLuminance::AgcMeanLuminance()\n> >> +\t: frameCount_(0), filteredExposure_(0s), relativeLuminanceTarget_(0)\n> >> +{\n> >> +}\n> >> +\n> >> +/**\n> >> + * \\brief Parse the relative luminance target from the tuning data\n> >> + * \\param[in] tuningData The YamlObject holding the algorithm's tuning data\n> >> + */\n> >> +void AgcMeanLuminance::parseRelativeLuminanceTarget(const YamlObject &tuningData)\n> >> +{\n> >> +\trelativeLuminanceTarget_ =\n> >> +\t\ttuningData[\"relativeLuminanceTarget\"].get(kDefaultRelativeLuminanceTarget);\n> >> +}\n> >> +\n> >> +/**\n> >> + * \\brief Parse an AeConstraintMode constraint from tuning data\n> >> + * \\param[in] modeDict the YamlObject holding the constraint data\n> >> + * \\param[in] id The constraint ID from AeConstraintModeEnum\n> >> + */\n> >> +void AgcMeanLuminance::parseConstraint(const YamlObject &modeDict, int32_t id)\n> >> +{\n> >> +\tfor (const auto &[boundName, content] : modeDict.asDict()) {\n> >> +\t\tif (boundName != \"upper\" && boundName != \"lower\") {\n> >> +\t\t\tLOG(AgcMeanLuminance, Warning)\n> >> +\t\t\t\t<< \"Ignoring unknown constraint bound '\" << boundName << \"'\";\n> >> +\t\t\tcontinue;\n> >> +\t\t}\n> >> +\n> >> +\t\tunsigned int idx = static_cast(boundName == \"upper\");\n> >> +\t\tAgcConstraint::Bound bound = static_cast(idx);\n> >> +\t\tdouble qLo = content[\"qLo\"].get().value_or(0.98);\n> >> +\t\tdouble qHi = content[\"qHi\"].get().value_or(1.0);\n> >> +\t\tdouble yTarget =\n> >> +\t\t\tcontent[\"yTarget\"].getList().value_or(std::vector{ 0.5 }).at(0);\n> >> +\n> >> +\t\tAgcConstraint constraint = { bound, qLo, qHi, yTarget };\n> >> +\n> >> +\t\tif (!constraintModes_.count(id))\n> >> +\t\t\tconstraintModes_[id] = {};\n> >> +\n> >> +\t\tif (idx)\n> >> +\t\t\tconstraintModes_[id].push_back(constraint);\n> >> +\t\telse\n> >> +\t\t\tconstraintModes_[id].insert(constraintModes_[id].begin(), constraint);\n> >> +\t}\n> >> +}\n> >> +\n> >> +int AgcMeanLuminance::parseConstraintModes(const YamlObject &tuningData)\n> >> +{\n> >> +\tstd::vector availableConstraintModes;\n> >> +\n> >> +\tconst YamlObject &yamlConstraintModes = tuningData[controls::AeConstraintMode.name()];\n> >> +\tif (yamlConstraintModes.isDictionary()) {\n> >> +\t\tfor (const auto &[modeName, modeDict] : yamlConstraintModes.asDict()) {\n> >> +\t\t\tif (AeConstraintModeNameValueMap.find(modeName) ==\n> >> +\t\t\t AeConstraintModeNameValueMap.end()) {\n> >> +\t\t\t\tLOG(AgcMeanLuminance, Warning)\n> >> +\t\t\t\t\t<< \"Skipping unknown constraint mode '\" << modeName << \"'\";\n> >> +\t\t\t\tcontinue;\n> >> +\t\t\t}\n> >> +\n> >> +\t\t\tif (!modeDict.isDictionary()) {\n> >> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> >> +\t\t\t\t\t<< \"Invalid constraint mode '\" << modeName << \"'\";\n> >> +\t\t\t\treturn -EINVAL;\n> >> +\t\t\t}\n> >> +\n> >> +\t\t\tparseConstraint(modeDict,\n> >> +\t\t\t\t\tAeConstraintModeNameValueMap.at(modeName));\n> >> +\t\t\tavailableConstraintModes.push_back(\n> >> +\t\t\t\tAeConstraintModeNameValueMap.at(modeName));\n> >> +\t\t}\n> >> +\t}\n> >> +\n> >> +\t/*\n> >> +\t * If the tuning data file contains no constraints then we use the\n> >> +\t * default constraint that the various Agc algorithms were adhering to\n> >> +\t * anyway before centralisation.\n> >> +\t */\n> >> +\tif (constraintModes_.empty()) {\n> >> +\t\tAgcConstraint constraint = {\n> >> +\t\t\tAgcConstraint::Bound::lower,\n> >> +\t\t\t0.98,\n> >> +\t\t\t1.0,\n> >> +\t\t\t0.5\n> >> +\t\t};\n> >> +\n> >> +\t\tconstraintModes_[controls::ConstraintNormal].insert(\n> >> +\t\t\tconstraintModes_[controls::ConstraintNormal].begin(),\n> >> +\t\t\tconstraint);\n> >> +\t\tavailableConstraintModes.push_back(\n> >> +\t\t\tAeConstraintModeNameValueMap.at(\"ConstraintNormal\"));\n> >> +\t}\n> >> +\n> >> +\tcontrols_[&controls::AeConstraintMode] = ControlInfo(availableConstraintModes);\n> >> +\n> >> +\treturn 0;\n> >> +}\n> >> +\n> >> +int AgcMeanLuminance::parseExposureModes(const YamlObject &tuningData)\n> >> +{\n> >> +\tstd::vector availableExposureModes;\n> >> +\n> >> +\tconst YamlObject &yamlExposureModes = tuningData[controls::AeExposureMode.name()];\n> >> +\tif (yamlExposureModes.isDictionary()) {\n> >> +\t\tfor (const auto &[modeName, modeValues] : yamlExposureModes.asDict()) {\n> >> +\t\t\tif (AeExposureModeNameValueMap.find(modeName) ==\n> >> +\t\t\t AeExposureModeNameValueMap.end()) {\n> >> +\t\t\t\tLOG(AgcMeanLuminance, Warning)\n> >> +\t\t\t\t\t<< \"Skipping unknown exposure mode '\" << modeName << \"'\";\n> >> +\t\t\t\tcontinue;\n> >> +\t\t\t}\n> >> +\n> >> +\t\t\tif (!modeValues.isDictionary()) {\n> >> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> >> +\t\t\t\t\t<< \"Invalid exposure mode '\" << modeName << \"'\";\n> >> +\t\t\t\treturn -EINVAL;\n> >> +\t\t\t}\n> >> +\n> >> +\t\t\tstd::vector shutters =\n> >> +\t\t\t\tmodeValues[\"shutter\"].getList().value_or(std::vector{});\n> >> +\t\t\tstd::vector gains =\n> >> +\t\t\t\tmodeValues[\"gain\"].getList().value_or(std::vector{});\n> >> +\n> >> +\t\t\tif (shutters.size() != gains.size()) {\n> >> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> >> +\t\t\t\t\t<< \"Shutter and gain array sizes unequal\";\n> >> +\t\t\t\treturn -EINVAL;\n> >> +\t\t\t}\n> >> +\n> >> +\t\t\tif (shutters.empty()) {\n> >> +\t\t\t\tLOG(AgcMeanLuminance, Error)\n> >> +\t\t\t\t\t<< \"Shutter and gain arrays are empty\";\n> >> +\t\t\t\treturn -EINVAL;\n> >> +\t\t\t}\n> >> +\n> >> +\t\t\tstd::vector> stages;\n> >> +\t\t\tfor (unsigned int i = 0; i < shutters.size(); i++) {\n> >> +\t\t\t\tstages.push_back({\n> >> +\t\t\t\t\tstd::chrono::microseconds(shutters[i]),\n> >> +\t\t\t\t\tgains[i]\n> >> +\t\t\t\t});\n> >> +\t\t\t}\n> >> +\n> >> +\t\t\tstd::shared_ptr helper =\n> >> +\t\t\t\tstd::make_shared();\n> >> +\t\t\thelper->init(stages);\n> >> +\n> >> +\t\t\texposureModeHelpers_[AeExposureModeNameValueMap.at(modeName)] = helper;\n> >> +\t\t\tavailableExposureModes.push_back(AeExposureModeNameValueMap.at(modeName));\n> >> +\t\t}\n> >> +\t}\n> >> +\n> >> +\t/*\n> >> +\t * If we don't have any exposure modes in the tuning data we create an\n> >> +\t * ExposureModeHelper using an empty vector of stages. This will result\n> >> +\t * in the ExposureModeHelper simply driving the shutter as high as\n> >> +\t * possible before touching gain.\n> >> +\t */\n> >> +\tif (availableExposureModes.empty()) {\n> >> +\t\tint32_t exposureModeId = AeExposureModeNameValueMap.at(\"ExposureNormal\");\n> >> +\t\tstd::vector> stages = { };\n> >> +\n> >> +\t\tstd::shared_ptr helper =\n> >> +\t\t\tstd::make_shared();\n> >> +\t\thelper->init(stages);\n> >> +\n> >> +\t\texposureModeHelpers_[exposureModeId] = helper;\n> >> +\t\tavailableExposureModes.push_back(exposureModeId);\n> >> +\t}\n> >> +\n> >> +\tcontrols_[&controls::AeExposureMode] = ControlInfo(availableExposureModes);\n> >> +\n> >> +\treturn 0;\n> >> +}\n> >> +\n> >> +/**\n> >> + * \\brief Parse tuning data for AeConstraintMode and AeExposureMode controls\n> >> + * \\param[in] tuningData the YamlObject representing the tuning data\n> >> + *\n> >> + * This function parses tuning data to build the list of allowed values for the\n> >> + * AeConstraintMode and AeExposureMode controls. Those tuning data must provide\n> >> + * the data in a specific format; the Agc algorithm's tuning data should contain\n> >> + * a dictionary called AeConstraintMode containing per-mode setting dictionaries\n> >> + * with the key being a value from \\ref controls::AeConstraintModeNameValueMap.\n> >> + * Each mode dict may contain either a \"lower\" or \"upper\" key or both, for\n> >> + * example:\n> >> + *\n> >> + * \\code{.unparsed}\n> >> + * algorithms:\n> >> + * - Agc:\n> >> + * AeConstraintMode:\n> >> + * ConstraintNormal:\n> >> + * lower:\n> >> + * qLo: 0.98\n> >> + * qHi: 1.0\n> >> + * yTarget: 0.5\n> >> + * ConstraintHighlight:\n> >> + * lower:\n> >> + * qLo: 0.98\n> >> + * qHi: 1.0\n> >> + * yTarget: 0.5\n> >> + * upper:\n> >> + * qLo: 0.98\n> >> + * qHi: 1.0\n> >> + * yTarget: 0.8\n> >> + *\n> >> + * \\endcode\n> >> + *\n> >> + * For the AeExposureMode control the data should contain a dictionary called\n> >> + * AeExposureMode containing per-mode setting dictionaries with the key being a\n> >> + * value from \\ref controls::AeExposureModeNameValueMap. Each mode dict should\n> >> + * contain an array of shutter times with the key \"shutter\" and an array of gain\n> >> + * values with the key \"gain\", in this format:\n> >> + *\n> >> + * \\code{.unparsed}\n> >> + * algorithms:\n> >> + * - Agc:\n> >> + * AeExposureMode:\n> >> + * ExposureNormal:\n> >> + * shutter: [ 100, 10000, 30000, 60000, 120000 ]\n> >> + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ]\n> >> + * ExposureShort:\n> >> + * shutter: [ 100, 10000, 30000, 60000, 120000 ]\n> >> + * gain: [ 2.0, 4.0, 6.0, 8.0, 10.0 ]\n> >> + *\n> >> + * \\endcode\n> >> + *\n> >> + * @return 0 on success or a negative error code\n> >> + */\n> >> +int AgcMeanLuminance::parseTuningData(const YamlObject &tuningData)\n> >> +{\n> >> +\tint ret;\n> >> +\n> >> +\tparseRelativeLuminanceTarget(tuningData);\n> >> +\n> >> +\tret = parseConstraintModes(tuningData);\n> >> +\tif (ret)\n> >> +\t\treturn ret;\n> >> +\n> >> +\tret = parseExposureModes(tuningData);\n> >> +\tif (ret)\n> >> +\t\treturn ret;\n> >> +\n> >> +\treturn 0;\n> >> +}\n> >> +\n> >> +/**\n> >> + * \\brief configure the ExposureModeHelpers for this class\n> >> + * \\param[in] minShutter Minimum shutter time to allow\n> >> + * \\param[in] maxShutter Maximum shutter time to allow\n> >> + * \\param[in] minGain Minimum gain to allow\n> >> + * \\param[in] maxGain Maximum gain to allow\n> >> + *\n> >> + * This function calls \\ref ExposureModeHelper::setShutterGainLimits() for each\n> >> + * ExposureModeHelper that has been created for this class.\n> >> + */\n> >> +void AgcMeanLuminance::configureExposureModeHelpers(utils::Duration minShutter,\n> >> +\t\t\t\t\t\t utils::Duration maxShutter,\n> >> +\t\t\t\t\t\t double minGain,\n> >> +\t\t\t\t\t\t double maxGain)\n> >> +{\n> >> +\tfor (auto &[id, helper] : exposureModeHelpers_)\n> >> +\t\thelper->setShutterGainLimits(minShutter, maxShutter, minGain, maxGain);\n> >> +}\n> >> +\n> >> +/**\n> >> + * \\fn AgcMeanLuminance::constraintModes()\n> >> + * \\brief Get the constraint modes that have been parsed from tuning data\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn AgcMeanLuminance::exposureModeHelpers()\n> >> + * \\brief Get the ExposureModeHelpers that have been parsed from tuning data\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn AgcMeanLuminance::controls()\n> >> + * \\brief Get the controls that have been generated after parsing tuning data\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn AgcMeanLuminance::estimateLuminance(const double gain)\n> >> + * \\brief Estimate the luminance of an image, adjusted by a given gain\n> >> + * \\param[in] gain The gain with which to adjust the luminance estimate\n> >> + *\n> >> + * This function estimates the average relative luminance of the frame that\n> >> + * would be output by the sensor if an additional \\a gain was applied. It is a\n> >> + * pure virtual function because estimation of luminance is a hardware-specific\n> >> + * operation, which depends wholly on the format of the stats that are delivered\n> >> + * to libcamera from the ISP. Derived classes must implement an overriding\n> >> + * function that calculates the normalised mean luminance value across the\n> >> + * entire image.\n> >> + *\n> >> + * \\return The normalised relative luminance of the image\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\brief Estimate the initial gain needed to achieve a relative luminance\n> >> + * target\n> >> + *\n> >> + * To account for non-linearity caused by saturation, the value needs to be\n> >> + * estimated in an iterative process, as multiplying by a gain will not increase\n> >> + * the relative luminance by the same factor if some image regions are saturated\n> >> + *\n> >> + * \\return The calculated initial gain\n> >> + */\n> >> +double AgcMeanLuminance::estimateInitialGain()\n> >> +{\n> >> +\tdouble yTarget = relativeLuminanceTarget_;\n> >> +\tdouble yGain = 1.0;\n> >> +\n> >> +\tfor (unsigned int i = 0; i < 8; i++) {\n> >> +\t\tdouble yValue = estimateLuminance(yGain);\n> >> +\t\tdouble extra_gain = std::min(10.0, yTarget / (yValue + .001));\n> >> +\n> >> +\t\tyGain *= extra_gain;\n> >> +\t\tLOG(AgcMeanLuminance, Debug) << \"Y value: \" << yValue\n> >> +\t\t\t\t<< \", Y target: \" << yTarget\n> >> +\t\t\t\t<< \", gives gain \" << yGain;\n> >> +\n> >> +\t\tif (utils::abs_diff(extra_gain, 1.0) < 0.01)\n> >> +\t\t\tbreak;\n> >> +\t}\n> >> +\n> >> +\treturn yGain;\n> >> +}\n> >> +\n> >> +/**\n> >> + * \\brief Clamp gain within the bounds of a defined constraint\n> >> + * \\param[in] constraintModeIndex The index of the constraint to adhere to\n> >> + * \\param[in] hist A histogram over which to calculate inter-quantile means\n> >> + * \\param[in] gain The gain to clamp\n> >> + *\n> >> + * \\return The gain clamped within the constraint bounds\n> >> + */\n> >> +double AgcMeanLuminance::constraintClampGain(uint32_t constraintModeIndex,\n> >> +\t\t\t\t\t const Histogram &hist,\n> >> +\t\t\t\t\t double gain)\n> >> +{\n> >> +\tstd::vector &constraints = constraintModes_[constraintModeIndex];\n> >> +\tfor (const AgcConstraint &constraint : constraints) {\n> >> +\t\tdouble newGain = constraint.yTarget * hist.bins() /\n> >> +\t\t\t\t hist.interQuantileMean(constraint.qLo, constraint.qHi);\n> >> +\n> >> +\t\tif (constraint.bound == AgcConstraint::Bound::lower &&\n> >> +\t\t newGain > gain)\n> >> +\t\t\tgain = newGain;\n> >> +\n> >> +\t\tif (constraint.bound == AgcConstraint::Bound::upper &&\n> >> +\t\t newGain < gain)\n> >> +\t\t\tgain = newGain;\n> >> +\t}\n> >> +\n> >> +\treturn gain;\n> >> +}\n> >> +\n> >> +/**\n> >> + * \\brief Apply a filter on the exposure value to limit the speed of changes\n> >> + * \\param[in] exposureValue The target exposure from the AGC algorithm\n> >> + *\n> >> + * The speed of the filter is adaptive, and will produce the target quicker\n> >> + * during startup, or when the target exposure is within 20% of the most recent\n> >> + * filter output.\n> >> + *\n> >> + * \\return The filtered exposure\n> >> + */\n> >> +utils::Duration AgcMeanLuminance::filterExposure(utils::Duration exposureValue)\n> >> +{\n> >> +\tdouble speed = 0.2;\n> >> +\n> >> +\t/* Adapt instantly if we are in startup phase. */\n> >> +\tif (frameCount_ < kNumStartupFrames)\n> >> +\t\tspeed = 1.0;\n> >> +\n> >> +\t/*\n> >> +\t * If we are close to the desired result, go faster to avoid making\n> >> +\t * multiple micro-adjustments.\n> >> +\t * \\todo Make this customisable?\n> >> +\t */\n> >> +\tif (filteredExposure_ < 1.2 * exposureValue &&\n> >> +\t filteredExposure_ > 0.8 * exposureValue)\n> >> +\t\tspeed = sqrt(speed);\n> >> +\n> >> +\tfilteredExposure_ = speed * exposureValue +\n> >> +\t\t\t filteredExposure_ * (1.0 - speed);\n> >> +\n> >> +\treturn filteredExposure_;\n> >> +}\n> >> +\n> >> +/**\n> >> + * \\brief Calculate the new exposure value\n> >> + * \\param[in] constraintModeIndex The index of the current constraint mode\n> >> + * \\param[in] exposureModeIndex The index of the current exposure mode\n> >> + * \\param[in] yHist A Histogram from the ISP statistics to use in constraining\n> >> + *\t the calculated gain\n> > nit: no indentation\n> >\n> >> + * \\param[in] effectiveExposureValue The EV applied to the frame from which the\n> >> + *\t statistics in use derive\n> > nit: no indentation\n> >\n> >> + *\n> >> + * Calculate a new exposure value to try to obtain the target. The calculated\n> >> + * exposure value is filtered to prevent rapid changes from frame to frame, and\n> >> + * divided into shutter time, analogue and digital gain.\n> >> + *\n> >> + * \\return Tuple of shutter time, analogue gain, and digital gain\n> >> + */\n> >> +std::tuple\n> >> +AgcMeanLuminance::calculateNewEv(uint32_t constraintModeIndex,\n> >> +\t\t\t\t uint32_t exposureModeIndex,\n> >> +\t\t\t\t const Histogram &yHist,\n> >> +\t\t\t\t utils::Duration effectiveExposureValue)\n> >> +{\n> >> +\t/*\n> >> +\t * The pipeline handler should validate that we have received an allowed\n> >> +\t * value for AeExposureMode.\n> >> +\t */\n> >> +\tstd::shared_ptr exposureModeHelper =\n> >> +\t\texposureModeHelpers_.at(exposureModeIndex);\n> >> +\n> >> +\tdouble gain = estimateInitialGain();\n> >> +\tgain = constraintClampGain(constraintModeIndex, yHist, gain);\n> >> +\n> >> +\t/*\n> >> +\t * We don't check whether we're already close to the target, because\n> >> +\t * even if the effective exposure value is the same as the last frame's\n> >> +\t * we could have switched to an exposure mode that would require a new\n> >> +\t * pass through the splitExposure() function.\n> >> +\t */\n> >> +\n> >> +\tutils::Duration newExposureValue = effectiveExposureValue * gain;\n> >> +\tutils::Duration maxTotalExposure = exposureModeHelper->maxShutter()\n> >> +\t\t\t\t\t * exposureModeHelper->maxGain();\n> >> +\tnewExposureValue = std::min(newExposureValue, maxTotalExposure);\n> >> +\n> >> +\t/*\n> >> +\t * We filter the exposure value to make sure changes are not too jarring\n> >> +\t * from frame to frame.\n> >> +\t */\n> >> +\tnewExposureValue = filterExposure(newExposureValue);\n> >> +\n> >> +\tframeCount_++;\n> >> +\treturn exposureModeHelper->splitExposure(newExposureValue);\n> >> +}\n> >> +\n> >> +/**\n> >> + * \\fn AgcMeanLuminance::resetFrameCount()\n> >> + * \\brief Reset the frame counter\n> >> + *\n> >> + * This function resets the internal frame counter, which exists to help the\n> >> + * algorithm decide whether it should respond instantly or not. The expectation\n> >> + * is for derived classes to call this function before each camera start call,\n> >> + * either in configure() or queueRequest() if the frame number is zero.\n> >> + */\n> >> +\n> >> +}; /* namespace ipa */\n> >> +\n> >> +}; /* namespace libcamera */\n> >> diff --git a/src/ipa/libipa/agc_mean_luminance.h b/src/ipa/libipa/agc_mean_luminance.h\n> >> new file mode 100644\n> >> index 00000000..e48dc498\n> >> --- /dev/null\n> >> +++ b/src/ipa/libipa/agc_mean_luminance.h\n> >> @@ -0,0 +1,91 @@\n> >> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> >> +/*\n> >> + * Copyright (C) 2024 Ideas on Board Oy\n> >> + *\n> >> + agc_mean_luminance.h - Base class for mean luminance AGC algorithms\n> >> + */\n> >> +\n> >> +#pragma once\n> >> +\n> >> +#include \n> >> +#include \n> >> +\n> >> +#include \n> >> +\n> >> +#include \"libcamera/internal/yaml_parser.h\"\n> >> +\n> >> +#include \"exposure_mode_helper.h\"\n> >> +#include \"histogram.h\"\n> >> +\n> >> +namespace libcamera {\n> >> +\n> >> +namespace ipa {\n> >> +\n> >> +class AgcMeanLuminance\n> >> +{\n> >> +public:\n> >> +\tAgcMeanLuminance();\n> >> +\tvirtual ~AgcMeanLuminance() = default;\n> > There were a few small comments from Laurent that got lost\n> > * destructor in cpp\n> \n> \n> The compiler says I can't follow this suggestion; since the estimateLuminance() function is virtual \n> I apparently need a virtual destructor:\n> \n> \n> class libcamera::ipa::AgcMeanLuminance’ has virtual functions and accessible non-virtual destructor\n> \n> \n> Or am I doing something wrong?\n\nIn the .h file,\n\nclass AgcMeanLuminance\n{\npublic:\n\t...\n\tvirtual ~AgcMeanLuminance();\n\t...\n};\n\nIn the .cpp file,\n\nAgcMeanLuminance::~AgcMeanLuminance() = default;\n\n> > * code sytel in enum\n> \n> \n> Isn't this addressed by making them lowercase?\n> \n> > * missing line\n> >\n> > Aside from that, I think we should merge it in.\n> >\n> > Reviewed-by: Stefan Klug \n> >\n> > Cheers,\n> > Stefan\n> >\n> >> +\n> >> +\tstruct AgcConstraint {\n> >> +\t\tenum class Bound {\n> >> +\t\t\tlower = 0,\n> >> +\t\t\tupper = 1\n> >> +\t\t};\n> >> +\t\tBound bound;\n> >> +\t\tdouble qLo;\n> >> +\t\tdouble qHi;\n> >> +\t\tdouble yTarget;\n> >> +\t};\n> >> +\n> >> +\tint parseTuningData(const YamlObject &tuningData);\n> >> +\n> >> +\tvoid configureExposureModeHelpers(utils::Duration minShutter,\n> >> +\t\t\t\t\t utils::Duration maxShutter,\n> >> +\t\t\t\t\t double minGain,\n> >> +\t\t\t\t\t double maxGain);\n> >> +\n> >> +\tstd::map> constraintModes()\n> >> +\t{\n> >> +\t\treturn constraintModes_;\n> >> +\t}\n> >> +\n> >> +\tstd::map> exposureModeHelpers()\n> >> +\t{\n> >> +\t\treturn exposureModeHelpers_;\n> >> +\t}\n> >> +\n> >> +\tControlInfoMap::Map controls()\n> >> +\t{\n> >> +\t\treturn controls_;\n> >> +\t}\n> >> +\n> >> +\tdouble estimateInitialGain();\n> >> +\tdouble constraintClampGain(uint32_t constraintModeIndex,\n> >> +\t\t\t\t const Histogram &hist,\n> >> +\t\t\t\t double gain);\n> >> +\tutils::Duration filterExposure(utils::Duration exposureValue);\n> >> +\tstd::tuple\n> >> +\tcalculateNewEv(uint32_t constraintModeIndex, uint32_t exposureModeIndex,\n> >> +\t\t const Histogram &yHist, utils::Duration effectiveExposureValue);\n> >> +\tvoid resetFrameCount() { frameCount_ = 0; }\n> >> +private:\n> >> +\tvirtual double estimateLuminance(const double gain) = 0;\n> >> +\n> >> +\tvoid parseRelativeLuminanceTarget(const YamlObject &tuningData);\n> >> +\tvoid parseConstraint(const YamlObject &modeDict, int32_t id);\n> >> +\tint parseConstraintModes(const YamlObject &tuningData);\n> >> +\tint parseExposureModes(const YamlObject &tuningData);\n> >> +\n> >> +\tuint64_t frameCount_;\n> >> +\tutils::Duration filteredExposure_;\n> >> +\tdouble relativeLuminanceTarget_;\n> >> +\n> >> +\tstd::map> constraintModes_;\n> >> +\tstd::map> exposureModeHelpers_;\n> >> +\tControlInfoMap::Map controls_;\n> >> +};\n> >> +\n> >> +}; /* namespace ipa */\n> >> +\n> >> +}; /* namespace libcamera */\n> >> diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build\n> >> index 37fbd177..7ce885da 100644\n> >> --- a/src/ipa/libipa/meson.build\n> >> +++ b/src/ipa/libipa/meson.build\n> >> @@ -1,6 +1,7 @@\n> >> # SPDX-License-Identifier: CC0-1.0\n> >> \n> >> libipa_headers = files([\n> >> + 'agc_mean_luminance.h',\n> >> 'algorithm.h',\n> >> 'camera_sensor_helper.h',\n> >> 'exposure_mode_helper.h',\n> >> @@ -10,6 +11,7 @@ libipa_headers = files([\n> >> ])\n> >> \n> >> libipa_sources = files([\n> >> + 'agc_mean_luminance.cpp',\n> >> 'algorithm.cpp',\n> >> 'camera_sensor_helper.cpp',\n> >> 'exposure_mode_helper.cpp',","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 30DBCBE08B\n\tfor ;\n\tWed, 24 Apr 2024 12:56:20 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id EA518633F3;\n\tWed, 24 Apr 2024 14:56:18 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id EC1D461B43\n\tfor ;\n\tWed, 24 Apr 2024 14:56:16 +0200 (CEST)","from pendragon.ideasonboard.com\n\t(117.145-247-81.adsl-dyn.isp.belgacom.be [81.247.145.117])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id C74EF6B3;\n\tWed, 24 Apr 2024 14:55:24 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"RypbIWFn\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1713963324;\n\tbh=3dIMrJfVx7E+Lvdh9GLVJe5wMMcl3eERPbYT14LU44A=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=RypbIWFnqtZpDbSxHhVJ7ULXUazkGU+ht78nVPd5fTpwu5gvmtvlkT6nJH75WLcDC\n\tcdYjFjR7fAD6ZGOcU/auBqzj1ih5qr9Yw5ZS1NZKZ3LD+u4RejiJMb48FkeobDwsVi\n\t1ylfeQYuLQUEx1eX7ESJ4IA54xtUGp50IXgn9klw=","Date":"Wed, 24 Apr 2024 15:56:09 +0300","From":"Laurent Pinchart ","To":"Dan Scally ","Cc":"Stefan Klug ,\n\tlibcamera-devel@lists.libcamera.org","Subject":"Re: [PATCH v2 4/8] ipa: libipa: Add MeanLuminanceAgc base class","Message-ID":"<20240424125609.GF18608@pendragon.ideasonboard.com>","References":"<20240417131536.484129-1-dan.scally@ideasonboard.com>\n\t<20240417131536.484129-5-dan.scally@ideasonboard.com>\n\t<20240418074823.l2k273uo5fx2nmwp@jasper>\n\t<0c4c9897-6474-4b9f-b24c-fb8671f74ab2@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<0c4c9897-6474-4b9f-b24c-fb8671f74ab2@ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]