{"id":14002,"url":"https://patchwork.libcamera.org/api/1.1/patches/14002/?format=json","web_url":"https://patchwork.libcamera.org/patch/14002/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20210930095513.76213-7-jeanmichel.hautbois@ideasonboard.com>","date":"2021-09-30T09:55:01","name":"[libcamera-devel,06/18] ipa: ipu3: agc: Document AGC mean-based algorithm","commit_ref":null,"pull_url":null,"state":"changes-requested","archived":false,"hash":"ecc092c67375959d1d30af46032c20e4a951e169","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/1.1/people/75/?format=json","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/14002/mbox/","series":[{"id":2588,"url":"https://patchwork.libcamera.org/api/1.1/series/2588/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=2588","date":"2021-09-30T09:54:55","name":"Document IPU3 IPA","version":1,"mbox":"https://patchwork.libcamera.org/series/2588/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/14002/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/14002/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 9E2C9C3243\n\tfor ;\n\tThu, 30 Sep 2021 09:55:29 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 8828E691C8;\n\tThu, 30 Sep 2021 11:55:26 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 8212A691A3\n\tfor ;\n\tThu, 30 Sep 2021 11:55:18 +0200 (CEST)","from tatooine.ideasonboard.com (unknown\n\t[IPv6:2a01:e0a:169:7140:bab4:22c5:662d:e478])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 3CF162A8;\n\tThu, 30 Sep 2021 11:55:18 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"YUJblwbb\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1632995718;\n\tbh=pzB+SmLd/iPDdUY9sZq09VPRGurVnZ0DZvxabPCWeEc=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=YUJblwbb0SEOeCtqnOPg+jZGD8mvVdKV3By88VFuvMr998Quu0VUXsmJY3DgVoRIG\n\tejso4LvopnHQ9VxXAP8Ejg4/WfJ8UzqQK8YIl9NpQQ9spOQypkLH+VGwIVCL0Wqimx\n\tV5/PqGqvXPb0c1RZ6BcIP6SLIUL90ukmZ64PDcDw=","From":"Jean-Michel Hautbois ","To":"libcamera-devel@lists.libcamera.org","Date":"Thu, 30 Sep 2021 11:55:01 +0200","Message-Id":"<20210930095513.76213-7-jeanmichel.hautbois@ideasonboard.com>","X-Mailer":"git-send-email 2.30.2","In-Reply-To":"<20210930095513.76213-1-jeanmichel.hautbois@ideasonboard.com>","References":"<20210930095513.76213-1-jeanmichel.hautbois@ideasonboard.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH 06/18] ipa: ipu3: agc: Document AGC\n\tmean-based algorithm","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"The AGC class was not documented while developing. Extend that to\nreference the origins of the implementation, and improve the\ndescriptions on how the algorithm operates internally.\n\nSigned-off-by: Jean-Michel Hautbois \nSigned-off-by: Kieran Bingham \n---\n src/ipa/ipu3/algorithms/agc.cpp | 102 ++++++++++++++++++++++++++++----\n src/ipa/ipu3/algorithms/agc.h | 2 +-\n 2 files changed, 90 insertions(+), 14 deletions(-)","diff":"diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp\nindex 44008632..2ddbf438 100644\n--- a/src/ipa/ipu3/algorithms/agc.cpp\n+++ b/src/ipa/ipu3/algorithms/agc.cpp\n@@ -2,7 +2,7 @@\n /*\n * Copyright (C) 2021, Ideas On Board\n *\n- * ipu3_agc.cpp - AGC/AEC control algorithm\n+ * ipu3_agc.cpp - AGC/AEC mean-based control algorithm\n */\n \n #include \"agc.h\"\n@@ -17,27 +17,49 @@\n \n #include \"libipa/histogram.h\"\n \n+/**\n+ * \\file agc.h\n+ */\n+\n namespace libcamera {\n \n using namespace std::literals::chrono_literals;\n \n namespace ipa::ipu3::algorithms {\n \n+/**\n+ * \\class Agc\n+ * \\brief A mean-based auto-exposure algorithm\n+ *\n+ * This algorithm calculates a shutter time and a gain so that the average value\n+ * of the green channel of the brightest 2% of pixels approaches 0.5. The AWB\n+ * gains are not used here, and all cells in the grid have the same weight, like\n+ * an average-metering case. In this metering mode, the camera uses light\n+ * information from the entire scene and creates an average for the final\n+ * exposure setting, giving no weighting to any particular portion of the\n+ * metered area.\n+ *\n+ * Reference: Battiato, Messina & Castorina. (2008). Exposure\n+ * Correction for Imaging Devices: An Overview. 10.1201/9781420054538.ch12.\n+ */\n+\n LOG_DEFINE_CATEGORY(IPU3Agc)\n \n /* Number of frames to wait before calculating stats on minimum exposure */\n static constexpr uint32_t kInitialFrameMinAECount = 4;\n-/* Number of frames to wait between new gain/exposure estimations */\n+/* Number of frames to wait between new gain/shutter time estimations */\n static constexpr uint32_t kFrameSkipCount = 6;\n \n-/* Maximum ISO value for analogue gain */\n-static constexpr uint32_t kMinISO = 100;\n-static constexpr uint32_t kMaxISO = 1500;\n-\n-/* Maximum analogue gain value\n- * \\todo grab it from a camera helper */\n-static constexpr uint32_t kMinGain = kMinISO / 100;\n-static constexpr uint32_t kMaxGain = kMaxISO / 100;\n+/*\n+ * Minimum analogue gain value\n+ * \\todo grab it from a camera helper\n+ */\n+static constexpr uint32_t kMinGain = 1;\n+/*\n+ * Maximum analogue gain value\n+ * \\todo grab it from a camera helper\n+ */\n+static constexpr uint32_t kMaxGain = 15;\n \n /* \\todo use calculated value based on sensor */\n static constexpr uint32_t kMinExposure = 1;\n@@ -45,6 +67,7 @@ static constexpr uint32_t kMaxExposure = 1976;\n \n /* Histogram constants */\n static constexpr uint32_t knumHistogramBins = 256;\n+/* Target value to reach for the top 2% of the histogram */\n static constexpr double kEvGainTarget = 0.5;\n \n Agc::Agc()\n@@ -54,10 +77,18 @@ Agc::Agc()\n {\n }\n \n+/**\n+ * \\brief Configure the AGC given a configInfo\n+ * \\param[in] context The shared IPA context\n+ * \\param[in] configInfo The IPA configuration data\n+ *\n+ * \\return 0\n+ */\n int Agc::configure(IPAContext &context, const IPAConfigInfo &configInfo)\n {\n \tstride_ = context.configuration.grid.stride;\n \n+\t/* \\todo use the configInfo fields and IPAContext to store the limits */\n \tlineDuration_ = configInfo.sensorInfo.lineLength * 1.0s\n \t\t / configInfo.sensorInfo.pixelRate;\n \tmaxExposureTime_ = kMaxExposure * lineDuration_;\n@@ -65,9 +96,15 @@ int Agc::configure(IPAContext &context, const IPAConfigInfo &configInfo)\n \treturn 0;\n }\n \n+/**\n+ * \\brief Estimate the mean quantile of the top 2% of the histogram\n+ * \\param[in] stats The statistics computed by the ImgU\n+ * \\param[in] grid The grid used to store the statistics in the IPU3\n+ */\n void Agc::processBrightness(const ipu3_uapi_stats_3a *stats,\n \t\t\t const ipu3_uapi_grid_config &grid)\n {\n+\t/* Initialise the histogram array */\n \tuint32_t hist[knumHistogramBins] = { 0 };\n \n \tfor (unsigned int cellY = 0; cellY < grid.height; cellY++) {\n@@ -83,6 +120,11 @@ void Agc::processBrightness(const ipu3_uapi_stats_3a *stats,\n \t\t\tif (cell->sat_ratio == 0) {\n \t\t\t\tuint8_t gr = cell->Gr_avg;\n \t\t\t\tuint8_t gb = cell->Gb_avg;\n+\t\t\t\t/*\n+\t\t\t\t * Store the average green value to estimate the\n+\t\t\t\t * brightness. Even the over exposed pixels are\n+\t\t\t\t * taken into account.\n+\t\t\t\t */\n \t\t\t\thist[(gr + gb) / 2]++;\n \t\t\t}\n \t\t}\n@@ -92,18 +134,24 @@ void Agc::processBrightness(const ipu3_uapi_stats_3a *stats,\n \tiqMean_ = Histogram(Span(hist)).interQuantileMean(0.98, 1.0);\n }\n \n+/**\n+ * \\brief Apply a filter on the exposure value to limit the speed of changes\n+ */\n void Agc::filterExposure()\n {\n \tdouble speed = 0.2;\n \tif (prevExposure_ == 0s) {\n-\t\t/* DG stands for digital gain.*/\n+\t\t/*\n+\t\t * DG stands for digital gain, which is always 1.0 for now as it\n+\t\t * is not implemented right now.\n+\t\t */\n \t\tprevExposure_ = currentExposure_;\n \t\tprevExposureNoDg_ = currentExposureNoDg_;\n \t} else {\n \t\t/*\n \t\t * If we are close to the desired result, go faster to avoid making\n \t\t * multiple micro-adjustments.\n-\t\t * \\ todo: Make this customisable?\n+\t\t * \\todo: Make this customisable?\n \t\t */\n \t\tif (prevExposure_ < 1.2 * currentExposure_ &&\n \t\t prevExposure_ > 0.8 * currentExposure_)\n@@ -114,10 +162,12 @@ void Agc::filterExposure()\n \t\tprevExposureNoDg_ = speed * currentExposureNoDg_ +\n \t\t\t\tprevExposureNoDg_ * (1.0 - speed);\n \t}\n+\n \t/*\n \t * We can't let the no_dg exposure deviate too far below the\n \t * total exposure, as there might not be enough digital gain available\n \t * in the ISP to hide it (which will cause nasty oscillation).\n+\t * \\todo implement digital gain setting\n \t */\n \tdouble fastReduceThreshold = 0.4;\n \tif (prevExposureNoDg_ <\n@@ -126,6 +176,11 @@ void Agc::filterExposure()\n \tLOG(IPU3Agc, Debug) << \"After filtering, total_exposure \" << prevExposure_;\n }\n \n+/**\n+ * \\brief Estimate the new exposure and gain values\n+ * \\param[inout] exposure The exposure value reference as a number of lines\n+ * \\param[inout] gain The gain reference to be updated\n+ */\n void Agc::lockExposureGain(uint32_t &exposure, double &gain)\n {\n \t/* Algorithm initialization should wait for first valid frames */\n@@ -138,22 +193,33 @@ void Agc::lockExposureGain(uint32_t &exposure, double &gain)\n \tif (std::abs(iqMean_ - kEvGainTarget * knumHistogramBins) <= 1) {\n \t\tLOG(IPU3Agc, Debug) << \"!!! Good exposure with iqMean = \" << iqMean_;\n \t} else {\n+\t\t/* Estimate the gain needed to have the proportion wanted */\n \t\tdouble newGain = kEvGainTarget * knumHistogramBins / iqMean_;\n \n \t\t/* extracted from Rpi::Agc::computeTargetExposure */\n+\t\t/* Calculate the shutter time in seconds */\n \t\tlibcamera::utils::Duration currentShutter = exposure * lineDuration_;\n+\n+\t\t/*\n+\t\t * Estimate the current exposure value for the scene as shutter\n+\t\t * time multiplicated by the analogue gain.\n+\t\t */\n \t\tcurrentExposureNoDg_ = currentShutter * gain;\n \t\tLOG(IPU3Agc, Debug) << \"Actual total exposure \" << currentExposureNoDg_\n \t\t\t\t << \" Shutter speed \" << currentShutter\n \t\t\t\t << \" Gain \" << gain;\n+\n+\t\t/* Apply the gain calculated to the current exposure value */\n \t\tcurrentExposure_ = currentExposureNoDg_ * newGain;\n+\n+\t\t/* Clamp the exposure value to the min and max authorized */\n \t\tlibcamera::utils::Duration maxTotalExposure = maxExposureTime_ * kMaxGain;\n \t\tcurrentExposure_ = std::min(currentExposure_, maxTotalExposure);\n \t\tLOG(IPU3Agc, Debug) << \"Target total exposure \" << currentExposure_;\n \n-\t\t/* \\todo: estimate if we need to desaturate */\n \t\tfilterExposure();\n \n+\t\t/* Divide the exposure value as new exposure and gain values */\n \t\tlibcamera::utils::Duration newExposure = 0.0s;\n \t\tif (currentShutter < maxExposureTime_) {\n \t\t\texposure = std::clamp(static_cast(exposure * currentExposure_ / currentExposureNoDg_), kMinExposure, kMaxExposure);\n@@ -169,10 +235,20 @@ void Agc::lockExposureGain(uint32_t &exposure, double &gain)\n \tlastFrame_ = frameCount_;\n }\n \n+/**\n+ * \\brief Process IPU3 statistics, and run AGC operations\n+ * \\param[in] context The shared IPA context\n+ * \\param[in] stats The IPU3 statistics and ISP results\n+ *\n+ * Identify the current image brightness, and use that to estimate the optimal\n+ * new exposure and gain for the scene.\n+ */\n void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats)\n {\n+\t/* Get the latest exposure and gain applied */\n \tuint32_t &exposure = context.frameContext.agc.exposure;\n \tdouble &gain = context.frameContext.agc.gain;\n+\n \tprocessBrightness(stats, context.configuration.grid.bdsGrid);\n \tlockExposureGain(exposure, gain);\n \tframeCount_++;\ndiff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h\nindex 64b71c65..c81ed6e7 100644\n--- a/src/ipa/ipu3/algorithms/agc.h\n+++ b/src/ipa/ipu3/algorithms/agc.h\n@@ -2,7 +2,7 @@\n /*\n * Copyright (C) 2021, Ideas On Board\n *\n- * agc.h - IPU3 AGC/AEC control algorithm\n+ * agc.h - IPU3 AGC/AEC mean-based control algorithm\n */\n #ifndef __LIBCAMERA_IPU3_ALGORITHMS_AGC_H__\n #define __LIBCAMERA_IPU3_ALGORITHMS_AGC_H__\n","prefixes":["libcamera-devel","06/18"]}