{"id":14476,"url":"https://patchwork.libcamera.org/api/1.1/patches/14476/?format=json","web_url":"https://patchwork.libcamera.org/patch/14476/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20211108131350.130665-8-jeanmichel.hautbois@ideasonboard.com>","date":"2021-11-08T13:13:35","name":"[libcamera-devel,07/22] ipa: ipu3: agc: Improve gain calculation","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"268dd7ba2fc16d58b06ef2283e5a39fe99b1fc43","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/1.1/people/75/?format=json","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/14476/mbox/","series":[{"id":2696,"url":"https://patchwork.libcamera.org/api/1.1/series/2696/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=2696","date":"2021-11-08T13:13:28","name":"IPA: IPU3: Introduce per-frame controls","version":1,"mbox":"https://patchwork.libcamera.org/series/2696/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/14476/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/14476/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C9FEDBDB1C\n\tfor ;\n\tMon, 8 Nov 2021 13:14:10 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id D44A76039C;\n\tMon, 8 Nov 2021 14:14:09 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id AF16360368\n\tfor ;\n\tMon, 8 Nov 2021 14:14:00 +0100 (CET)","from tatooine.ideasonboard.com (unknown\n\t[IPv6:2a01:e0a:169:7140:c2bb:76d0:68d7:a9a5])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 6EF71110E;\n\tMon, 8 Nov 2021 14:14:00 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"afwnBodz\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1636377240;\n\tbh=8ssxltMYMTz012AgiKESipgfnNQY25+vOCry7kbaFLQ=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=afwnBodzIkU8zNr+snHIN2+U3thd4qH8rRCh2Zf6bPYp96m91uOylp56ODpXFfG5z\n\tddf02XjaTk+hj+W6n/ebws+65h2lvDERUjj+NmsTuon5OZY5PqYygNWCW7F1DlpfMK\n\tQB7X2JDiuOiKxejBqz9xK/VbzLYVSDNnfL4721s8=","From":"Jean-Michel Hautbois ","To":"libcamera-devel@lists.libcamera.org","Date":"Mon, 8 Nov 2021 14:13:35 +0100","Message-Id":"<20211108131350.130665-8-jeanmichel.hautbois@ideasonboard.com>","X-Mailer":"git-send-email 2.32.0","In-Reply-To":"<20211108131350.130665-1-jeanmichel.hautbois@ideasonboard.com>","References":"<20211108131350.130665-1-jeanmichel.hautbois@ideasonboard.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH 07/22] ipa: ipu3: agc: Improve gain\n\tcalculation","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"When an image is partially saturated, its brightness is not increasing\nlinearly when the shutter time or gain increases. It is a big issue with\na backlight as the algorithm is fading to darkness right now.\n\nIntroduce a function to estimate the brightness of the frame, based on\nthe current exposure/gain and loop on it several times to estimate it\nagain and approach the non linear function.\n\nInspired-by: 7de5506c30b3 (\"libcamera: src: ipa: raspberrypi: agc: Improve gain update calculation for partly saturated images\")\nSigned-off-by: Jean-Michel Hautbois \n---\n src/ipa/ipu3/algorithms/agc.cpp | 77 ++++++++++++++++++++++++++++++++-\n src/ipa/ipu3/algorithms/agc.h | 6 ++-\n 2 files changed, 80 insertions(+), 3 deletions(-)","diff":"diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp\nindex 119a7938..e4048d40 100644\n--- a/src/ipa/ipu3/algorithms/agc.cpp\n+++ b/src/ipa/ipu3/algorithms/agc.cpp\n@@ -186,8 +186,9 @@ void Agc::filterExposure()\n * \\brief Estimate the new exposure and gain values\n * \\param[inout] exposure The exposure value reference as a number of lines\n * \\param[inout] gain The gain reference to be updated\n+ * \\param[in] currentYGain The gain calculated on the current brightness level\n */\n-void Agc::computeExposure(uint32_t &exposure, double &analogueGain)\n+void Agc::computeExposure(uint32_t &exposure, double &analogueGain, double currentYGain)\n {\n \t/* Estimate the gain needed to have the proportion wanted */\n \tdouble evGain = kEvGainTarget * knumHistogramBins / iqMean_;\n@@ -206,6 +207,14 @@ void Agc::computeExposure(uint32_t &exposure, double &analogueGain)\n \t\t\t << \" Gain \" << analogueGain\n \t\t\t << \" Needed ev gain \" << evGain;\n \n+\tif (evGain < currentYGain)\n+\t\tevGain = currentYGain;\n+\n+\tif (std::abs(evGain - 1.0) < 0.01) {\n+\t\tLOG(IPU3Agc, Debug) << \"We are well exposed (iqMean = \"\n+\t\t\t\t << iqMean_ << \")\";\n+\t}\n+\n \t/*\n \t * Calculate the current exposure value for the scene as the latest\n \t * exposure value applied multiplied by the new estimated gain.\n@@ -253,6 +262,48 @@ void Agc::computeExposure(uint32_t &exposure, double &analogueGain)\n \tprevExposureValue_ = shutterTime * analogueGain;\n }\n \n+/**\n+ * \\brief Estimate the average brightness of the frame\n+ * \\param[in] context The shared IPA context\n+ * \\param[in] grid The grid used to store the statistics in the IPU3\n+ * \\param[in] stats The IPU3 statistics and ISP results\n+ * \\param[in] currentYGain The gain calculated on the current brightness level\n+ */\n+double Agc::computeInitialY(IPAFrameContext &frameContext,\n+\t\t\t const ipu3_uapi_grid_config &grid,\n+\t\t\t const ipu3_uapi_stats_3a *stats,\n+\t\t\t double currentYGain)\n+{\n+\tdouble redSum = 0, greenSum = 0, blueSum = 0;\n+\n+\tfor (unsigned int cellY = 0; cellY < grid.height; cellY++) {\n+\t\tfor (unsigned int cellX = 0; cellX < grid.width; cellX++) {\n+\t\t\tuint32_t cellPosition = cellY * stride_ + cellX;\n+\n+\t\t\tconst ipu3_uapi_awb_set_item *cell =\n+\t\t\t\treinterpret_cast(\n+\t\t\t\t\t&stats->awb_raw_buffer.meta_data[cellPosition]\n+\t\t\t\t);\n+\n+\t\t\tredSum += cell->R_avg * currentYGain;\n+\t\t\tgreenSum += (cell->Gr_avg + cell->Gb_avg) / 2 * currentYGain;\n+\t\t\tblueSum += cell->B_avg * currentYGain;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Estimate the sum of the brightness values, weighted with the gains\n+\t * applied on the channels in AWB.\n+\t */\n+\tdouble Y_sum = redSum * frameContext.awb.gains.red * .299 +\n+\t\t greenSum * frameContext.awb.gains.green * .587 +\n+\t\t blueSum * frameContext.awb.gains.blue * .114;\n+\n+\t/* And return the average brightness */\n+\treturn Y_sum / (grid.height * grid.width);\n+}\n+\n+\n /**\n * \\brief Process IPU3 statistics, and run AGC operations\n * \\param[in] context The shared IPA context\n@@ -267,7 +318,29 @@ void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats)\n \tuint32_t &exposure = context.frameContext.agc.exposure;\n \tdouble &analogueGain = context.frameContext.agc.gain;\n \tmeasureBrightness(stats, context.configuration.grid.bdsGrid);\n-\tcomputeExposure(exposure, analogueGain);\n+\n+\tdouble currentYGain = 1.0;\n+\t/* \\todo: the target Y needs to be grabbed from a configuration */\n+\tdouble targetY = 60;\n+\t/*\n+\t * Do this calculation a few times as brightness increase can be\n+\t * non-linear when there are saturated regions.\n+\t */\n+\tfor (int i = 0; i < 8; i++) {\n+\t\tdouble initialY = computeInitialY(context.frameContext,\n+\t\t\t\t\t\t context.configuration.grid.bdsGrid,\n+\t\t\t\t\t\t stats, currentYGain);\n+\t\tdouble extra_gain = std::min(10.0, targetY / (initialY + .001));\n+\n+\t\tcurrentYGain *= extra_gain;\n+\t\tLOG(IPU3Agc, Debug) << \"Initial Y \" << initialY\n+\t\t\t\t << \" target \" << targetY\n+\t\t\t\t << \" gives gain \" << currentYGain;\n+\t\tif (extra_gain < 1.01)\n+\t\t\tbreak;\n+\t}\n+\n+\tcomputeExposure(exposure, analogueGain, currentYGain);\n \tframeCount_++;\n }\n \ndiff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h\nindex 69e0b831..0a9152a9 100644\n--- a/src/ipa/ipu3/algorithms/agc.h\n+++ b/src/ipa/ipu3/algorithms/agc.h\n@@ -34,7 +34,11 @@ private:\n \tvoid measureBrightness(const ipu3_uapi_stats_3a *stats,\n \t\t\t const ipu3_uapi_grid_config &grid);\n \tvoid filterExposure();\n-\tvoid computeExposure(uint32_t &exposure, double &gain);\n+\tvoid computeExposure(uint32_t &exposure, double &gain, double currentYGain);\n+\tdouble computeInitialY(IPAFrameContext &frameContext,\n+\t\t\t const ipu3_uapi_grid_config &grid,\n+\t\t\t const ipu3_uapi_stats_3a *stats,\n+\t\t\t double currentYGain);\n \n \tuint64_t frameCount_;\n \tuint64_t lastFrame_;\n","prefixes":["libcamera-devel","07/22"]}