{"id":13837,"url":"https://patchwork.libcamera.org/api/1.1/patches/13837/?format=json","web_url":"https://patchwork.libcamera.org/patch/13837/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20210913145810.66515-10-jeanmichel.hautbois@ideasonboard.com>","date":"2021-09-13T14:58:08","name":"[libcamera-devel,09/11] ipa: ipu3: agc: Document AGC mean-based algorithm","commit_ref":null,"pull_url":null,"state":"changes-requested","archived":false,"hash":"49fb4cf5b956afe0f87e8b4807e6f63bfc4427a7","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/1.1/people/75/?format=json","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/13837/mbox/","series":[{"id":2523,"url":"https://patchwork.libcamera.org/api/1.1/series/2523/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=2523","date":"2021-09-13T14:57:59","name":"Document all the IPU3 IPA classes","version":1,"mbox":"https://patchwork.libcamera.org/series/2523/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/13837/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/13837/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 66CBABDB1D\n\tfor ;\n\tMon, 13 Sep 2021 14:58:25 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 2026A69199;\n\tMon, 13 Sep 2021 16:58:25 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C561B6918A\n\tfor ;\n\tMon, 13 Sep 2021 16:58:15 +0200 (CEST)","from tatooine.ideasonboard.com (unknown\n\t[IPv6:2a01:e0a:169:7140:edc5:688b:2ede:8b4b])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 7B332510;\n\tMon, 13 Sep 2021 16:58:15 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"ZVZQYLwB\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1631545095;\n\tbh=N4WiT6HCg2xzi0XLnQoniIH/xebceEZlDejew+etlxo=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=ZVZQYLwB9z+7LPOeh1OP2WuSX2y4Obg+Jxysp2KJZ6DxBFVNS3JTKatwzyDUgTt/X\n\t2BRVwqa/DSJwuIaSGPuXDmX07IDbsdeJ6wmizg5lnvGkvKH3faFt4zrQ233gmyQgyy\n\t/lF/6hNbMZmrW+pMgNV0+z86WWM5KcADgvWAnwpM=","From":"Jean-Michel Hautbois ","To":"libcamera-devel@lists.libcamera.org","Date":"Mon, 13 Sep 2021 16:58:08 +0200","Message-Id":"<20210913145810.66515-10-jeanmichel.hautbois@ideasonboard.com>","X-Mailer":"git-send-email 2.30.2","In-Reply-To":"<20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com>","References":"<20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH 09/11] ipa: ipu3: agc: Document AGC\n\tmean-based algorithm","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"The AGC class was not documented while developing. Extend that to\nreference the origins of the implementation, and improve the\ndescriptions on how the algorithm operates internally.\n\nSigned-off-by: Jean-Michel Hautbois \nSigned-off-by: Kieran Bingham \n---\n src/ipa/ipu3/algorithms/agc.cpp | 116 +++++++++++++++++++++++++++-----\n src/ipa/ipu3/algorithms/agc.h | 2 +-\n 2 files changed, 101 insertions(+), 17 deletions(-)","diff":"diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp\nindex 2ef998b7..9a55bb72 100644\n--- a/src/ipa/ipu3/algorithms/agc.cpp\n+++ b/src/ipa/ipu3/algorithms/agc.cpp\n@@ -2,7 +2,7 @@\n /*\n * Copyright (C) 2021, Ideas On Board\n *\n- * ipu3_agc.cpp - AGC/AEC control algorithm\n+ * ipu3_agc.cpp - AGC/AEC mean-based control algorithm\n */\n \n #include \"agc.h\"\n@@ -17,27 +17,49 @@\n \n #include \"libipa/histogram.h\"\n \n+/**\n+ * \\file agc.h\n+ */\n+\n namespace libcamera {\n \n using namespace std::literals::chrono_literals;\n \n namespace ipa::ipu3::algorithms {\n \n+/**\n+ * \\class Agc\n+ * \\brief A mean-based auto-exposure algorithm\n+ *\n+ * This algorithm calculates a shutter time and a gain so that the average value\n+ * of the green channel of the brightest 2% of pixels approaches 0.5. The AWB\n+ * gains are not used here, and all cells in the grid have the same weight, like\n+ * an average-metering case. In this metering mode, the camera uses light\n+ * information from the entire scene and creates an average for the final\n+ * exposure setting, giving no weighting to any particular portion of the\n+ * metered area.\n+ *\n+ * Reference: Battiato, Messina & Castorina. (2008). Exposure\n+ * Correction for Imaging Devices: An Overview. 10.1201/9781420054538.ch12.\n+ */\n+\n LOG_DEFINE_CATEGORY(IPU3Agc)\n \n /* Number of frames to wait before calculating stats on minimum exposure */\n static constexpr uint32_t kInitialFrameMinAECount = 4;\n-/* Number of frames to wait between new gain/exposure estimations */\n+/* Number of frames to wait between new gain/shutter time estimations */\n static constexpr uint32_t kFrameSkipCount = 6;\n \n-/* Maximum ISO value for analogue gain */\n-static constexpr uint32_t kMinISO = 100;\n-static constexpr uint32_t kMaxISO = 1500;\n-\n-/* Maximum analogue gain value\n- * \\todo grab it from a camera helper */\n-static constexpr uint32_t kMinGain = kMinISO / 100;\n-static constexpr uint32_t kMaxGain = kMaxISO / 100;\n+/*\n+ * Minimum analogue gain value\n+ * \\todo grab it from a camera helper\n+ */\n+static constexpr uint32_t kMinGain = 1;\n+/*\n+ * Maximum analogue gain value\n+ * \\todo grab it from a camera helper\n+ */\n+static constexpr uint32_t kMaxGain = 15;\n \n /* \\todo use calculated value based on sensor */\n static constexpr uint32_t kMinExposure = 1;\n@@ -45,6 +67,7 @@ static constexpr uint32_t kMaxExposure = 1976;\n \n /* Histogram constants */\n static constexpr uint32_t knumHistogramBins = 256;\n+/* Target value to reach for the top 2% of the histogram */\n static constexpr double kEvGainTarget = 0.5;\n \n /* A cell is 8 bytes and contains averages for RGB values and saturation ratio */\n@@ -57,9 +80,17 @@ Agc::Agc()\n {\n }\n \n+/**\n+ * \\brief Configure the AGC given a configInfo\n+ * \\param[in] context The shared IPA context\n+ * \\param[in] configInfo The IPA configuration data\n+ *\n+ * \\return 0\n+ */\n int Agc::configure([[maybe_unused]] IPAContext &context,\n-\t\t const IPAConfigInfo &configInfo)\n+\t\t const IPAConfigInfo &configInfo)\n {\n+\t/* \\todo use the configInfo fields and IPAContext to store the limits */\n \tlineDuration_ = configInfo.sensorInfo.lineLength * 1.0s\n \t\t / configInfo.sensorInfo.pixelRate;\n \tmaxExposureTime_ = kMaxExposure * lineDuration_;\n@@ -67,9 +98,22 @@ int Agc::configure([[maybe_unused]] IPAContext &context,\n \treturn 0;\n }\n \n+/**\n+ * \\brief Estimate the mean quantile of the top 2% of the histogram\n+ * \\param[in] stats The statistics computed by the ImgU\n+ * \\param[in] grid The grid used to store the statistics in the IPU3\n+ */\n void Agc::processBrightness(const ipu3_uapi_stats_3a *stats,\n \t\t\t const ipu3_uapi_grid_config &grid)\n {\n+\t/*\n+\t * Get the applied grid from the statistics buffer. When the kernel\n+\t * receives a grid from the parameters buffer, it will check and align\n+\t * all the values. For instance, it will automatically fill the x_end\n+\t * value based on x_start, grid width and log2 width.\n+\t * \\todo Use the grid calculated in configure as there is a bug in IPU3\n+\t * causing the width (maybe height) to be bit-shifted.\n+\t */\n \tconst struct ipu3_uapi_grid_config statsAeGrid = stats->stats_4a_config.awb_config.grid;\n \tRectangle aeRegion = { statsAeGrid.x_start,\n \t\t\t statsAeGrid.y_start,\n@@ -85,6 +129,7 @@ void Agc::processBrightness(const ipu3_uapi_stats_3a *stats,\n \tuint32_t endX = (startX + (aeRegion.size().width >> grid.block_width_log2)) << grid.block_width_log2;\n \tuint32_t i, j;\n \n+\t/* Initialise the histogram array */\n \tuint32_t hist[knumHistogramBins] = { 0 };\n \tfor (j = topleftY;\n \t j < topleftY + (aeRegion.size().height >> grid.block_height_log2);\n@@ -92,13 +137,18 @@ void Agc::processBrightness(const ipu3_uapi_stats_3a *stats,\n \t\tfor (i = startX + startY; i < endX + startY; i += kCellSize) {\n \t\t\t/*\n \t\t\t * The grid width (and maybe height) is not reliable.\n-\t\t\t * We observed a bit shift which makes the value 160 to be 32 in the stats grid.\n-\t\t\t * Use the one passed at init time.\n+\t\t\t * We observed a bit shift which makes the value 160 to\n+\t\t\t * be 32 in the stats grid. Use the one from configure.\n \t\t\t */\n \t\t\tconst ipu3_uapi_awb_set_item *currentCell = &stats->awb_raw_buffer.meta_data[i + j * grid.width];\n \t\t\tif (currentCell->sat_ratio == 0) {\n \t\t\t\tuint8_t Gr = currentCell->Gr_avg;\n \t\t\t\tuint8_t Gb = currentCell->Gb_avg;\n+\t\t\t\t/*\n+\t\t\t\t * Store the average green value to estimate the\n+\t\t\t\t * brightness. Even the over exposed pixels are\n+\t\t\t\t * taken into account.\n+\t\t\t\t */\n \t\t\t\thist[(Gr + Gb) / 2]++;\n \t\t\t}\n \t\t}\n@@ -108,18 +158,24 @@ void Agc::processBrightness(const ipu3_uapi_stats_3a *stats,\n \tiqMean_ = Histogram(Span(hist)).interQuantileMean(0.98, 1.0);\n }\n \n+/**\n+ * \\brief Apply a filter on the exposure value to limit the speed of changes\n+ */\n void Agc::filterExposure()\n {\n \tdouble speed = 0.2;\n \tif (prevExposure_ == 0s) {\n-\t\t/* DG stands for digital gain.*/\n+\t\t/*\n+\t\t * DG stands for digital gain, which is always 1.0 for now as it\n+\t\t * is not implemented right now.\n+\t\t */\n \t\tprevExposure_ = currentExposure_;\n \t\tprevExposureNoDg_ = currentExposureNoDg_;\n \t} else {\n \t\t/*\n \t\t * If we are close to the desired result, go faster to avoid making\n \t\t * multiple micro-adjustments.\n-\t\t * \\ todo: Make this customisable?\n+\t\t * \\todo: Make this customisable?\n \t\t */\n \t\tif (prevExposure_ < 1.2 * currentExposure_ &&\n \t\t prevExposure_ > 0.8 * currentExposure_)\n@@ -130,10 +186,12 @@ void Agc::filterExposure()\n \t\tprevExposureNoDg_ = speed * currentExposureNoDg_ +\n \t\t\t\tprevExposureNoDg_ * (1.0 - speed);\n \t}\n+\n \t/*\n \t * We can't let the no_dg exposure deviate too far below the\n \t * total exposure, as there might not be enough digital gain available\n \t * in the ISP to hide it (which will cause nasty oscillation).\n+\t * \\todo implement digital gain setting\n \t */\n \tdouble fastReduceThreshold = 0.4;\n \tif (prevExposureNoDg_ <\n@@ -142,6 +200,11 @@ void Agc::filterExposure()\n \tLOG(IPU3Agc, Debug) << \"After filtering, total_exposure \" << prevExposure_;\n }\n \n+/**\n+ * \\brief Estimate the new exposure and gain values\n+ * \\param[inout] exposure The exposure value reference as a number of lines\n+ * \\param[inout] gain The gain reference to be updated\n+ */\n void Agc::lockExposureGain(uint32_t &exposure, double &gain)\n {\n \t/* Algorithm initialization should wait for first valid frames */\n@@ -154,22 +217,33 @@ void Agc::lockExposureGain(uint32_t &exposure, double &gain)\n \tif (std::abs(iqMean_ - kEvGainTarget * knumHistogramBins) <= 1) {\n \t\tLOG(IPU3Agc, Debug) << \"!!! Good exposure with iqMean = \" << iqMean_;\n \t} else {\n+\t\t/* Estimate the gain needed to have the proportion wanted */\n \t\tdouble newGain = kEvGainTarget * knumHistogramBins / iqMean_;\n \n \t\t/* extracted from Rpi::Agc::computeTargetExposure */\n+\t\t/* Calculate the shutter time in seconds */\n \t\tlibcamera::utils::Duration currentShutter = exposure * lineDuration_;\n+\n+\t\t/*\n+\t\t * Estimate the current exposure value for the scene as shutter\n+\t\t * time multiplicated by the analogue gain.\n+\t\t */\n \t\tcurrentExposureNoDg_ = currentShutter * gain;\n \t\tLOG(IPU3Agc, Debug) << \"Actual total exposure \" << currentExposureNoDg_\n \t\t\t\t << \" Shutter speed \" << currentShutter\n \t\t\t\t << \" Gain \" << gain;\n+\n+\t\t/* Apply the gain calculated to the current exposure value */\n \t\tcurrentExposure_ = currentExposureNoDg_ * newGain;\n+\n+\t\t/* Clamp the exposure value to the min and max authorized */\n \t\tlibcamera::utils::Duration maxTotalExposure = maxExposureTime_ * kMaxGain;\n \t\tcurrentExposure_ = std::min(currentExposure_, maxTotalExposure);\n \t\tLOG(IPU3Agc, Debug) << \"Target total exposure \" << currentExposure_;\n \n-\t\t/* \\todo: estimate if we need to desaturate */\n \t\tfilterExposure();\n \n+\t\t/* Divide the exposure value as new exposure and gain values */\n \t\tlibcamera::utils::Duration newExposure = 0.0s;\n \t\tif (currentShutter < maxExposureTime_) {\n \t\t\texposure = std::clamp(static_cast(exposure * currentExposure_ / currentExposureNoDg_), kMinExposure, kMaxExposure);\n@@ -185,10 +259,20 @@ void Agc::lockExposureGain(uint32_t &exposure, double &gain)\n \tlastFrame_ = frameCount_;\n }\n \n+/**\n+ * \\brief Process IPU3 statistics, and run AGC operations\n+ * \\param[in] context The shared IPA context\n+ * \\param[in] stats The IPU3 statistics and ISP results\n+ *\n+ * Identify the current image brightness, and use that to estimate the optimal\n+ * new exposure and gain for the scene.\n+ */\n void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats)\n {\n+\t/* Get the latest exposure and gain applied */\n \tuint32_t &exposure = context.frameContext.agc.exposure;\n \tdouble &gain = context.frameContext.agc.gain;\n+\n \tprocessBrightness(stats, context.configuration.grid.bdsGrid);\n \tlockExposureGain(exposure, gain);\n \tframeCount_++;\ndiff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h\nindex e36797d3..9449ba48 100644\n--- a/src/ipa/ipu3/algorithms/agc.h\n+++ b/src/ipa/ipu3/algorithms/agc.h\n@@ -2,7 +2,7 @@\n /*\n * Copyright (C) 2021, Ideas On Board\n *\n- * agc.h - IPU3 AGC/AEC control algorithm\n+ * agc.h - IPU3 AGC/AEC mean-based control algorithm\n */\n #ifndef __LIBCAMERA_IPU3_ALGORITHMS_AGC_H__\n #define __LIBCAMERA_IPU3_ALGORITHMS_AGC_H__\n","prefixes":["libcamera-devel","09/11"]}