{"id":11780,"url":"https://patchwork.libcamera.org/api/1.1/patches/11780/?format=json","web_url":"https://patchwork.libcamera.org/patch/11780/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20210329191826.77817-6-jeanmichel.hautbois@ideasonboard.com>","date":"2021-03-29T19:18:26","name":"[libcamera-devel,v3,5/5] ipa: ipu3: Add support for IPU3 AEC/AGC algorithm","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"b19e06ae96b7b9bfed1a4331e3930f6db8008555","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/1.1/people/75/?format=json","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/11780/mbox/","series":[{"id":1865,"url":"https://patchwork.libcamera.org/api/1.1/series/1865/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=1865","date":"2021-03-29T19:18:21","name":"Implement IPA algorithms and demo with IPU3","version":3,"mbox":"https://patchwork.libcamera.org/series/1865/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/11780/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/11780/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 245F6C32F0\n\tfor ;\n\tMon, 29 Mar 2021 19:18:43 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id C93266878F;\n\tMon, 29 Mar 2021 21:18:42 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 5221068788\n\tfor ;\n\tMon, 29 Mar 2021 21:18:38 +0200 (CEST)","from localhost.localdomain (unknown\n\t[IPv6:2a01:e0a:169:7140:9714:da21:50fd:f63f])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 5D385A49;\n\tMon, 29 Mar 2021 21:18:37 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"hFpu4ebp\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1617045517;\n\tbh=0rl1kk4O8LqjDsPc6RST9Cis5vPWbA/rCwpp+p9JlZM=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=hFpu4ebpszEcx80qHZuN9oiJopr0ARB2D1MH2+4ht2mhTg5mNmpqvWXFSm5Aye/Gi\n\tWz76noUQA2chnxvbZzugvT+/8bMXf2hLIFxK6JamGZaoFRs/zXPSV9RE0vlOlDc4QA\n\tuBef0XzbcCoQ2JjS6+Srkc0nLJOotsWPDu+NuT3w=","From":"Jean-Michel Hautbois ","To":"libcamera-devel@lists.libcamera.org","Date":"Mon, 29 Mar 2021 21:18:26 +0200","Message-Id":"<20210329191826.77817-6-jeanmichel.hautbois@ideasonboard.com>","X-Mailer":"git-send-email 2.27.0","In-Reply-To":"<20210329191826.77817-1-jeanmichel.hautbois@ideasonboard.com>","References":"<20210329191826.77817-1-jeanmichel.hautbois@ideasonboard.com>","MIME-Version":"1.0","Subject":"[libcamera-devel] [PATCH v3 5/5] ipa: ipu3: Add support for IPU3\n\tAEC/AGC algorithm","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Inherit from the Algorithm class to implement basic auto-exposure and\nauto-gain functions.\nExtract computeTargetExposure() and computeGain() and adapt those to the\nIPU3 structure.\nFiletering was added too as it avoids big steps when exposure changes a\nlot.\n\nSigned-off-by: Jean-Michel Hautbois \n---\n src/ipa/ipu3/ipu3.cpp | 12 +-\n src/ipa/ipu3/ipu3_agc.cpp | 229 ++++++++++++++++++++++++++++++++++++++\n src/ipa/ipu3/ipu3_agc.h | 67 +++++++++++\n src/ipa/ipu3/meson.build | 1 +\n 4 files changed, 307 insertions(+), 2 deletions(-)\n create mode 100644 src/ipa/ipu3/ipu3_agc.cpp\n create mode 100644 src/ipa/ipu3/ipu3_agc.h","diff":"diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp\nindex cc741d69..59ed09e6 100644\n--- a/src/ipa/ipu3/ipu3.cpp\n+++ b/src/ipa/ipu3/ipu3.cpp\n@@ -21,6 +21,7 @@\n #include \"libcamera/internal/buffer.h\"\n #include \"libcamera/internal/log.h\"\n \n+#include \"ipu3_agc.h\"\n #include \"ipu3_awb.h\"\n \n static const uint32_t kMaxCellWidthPerSet = 160;\n@@ -54,6 +55,7 @@ private:\n \t\t\t const ipu3_uapi_stats_3a *stats);\n \n \tvoid setControls(unsigned int frame);\n+\tvoid calculateBdsGrid(const Size &bdsOutputSize);\n \n \tstd::map buffers_;\n \n@@ -69,6 +71,8 @@ private:\n \n \t/* Interface to the AWB algorithm */\n \tstd::unique_ptr awbAlgo_;\n+\t/* Interface to the AEC/AGC algorithm */\n+\tstd::unique_ptr agcAlgo_;\n \t/* Local parameter storage */\n \tstruct ipu3_uapi_params params_;\n \n@@ -157,10 +161,13 @@ void IPAIPU3::configure(const std::map &entityControls\n \n \tparams_ = {};\n \n+\tcalculateBdsGrid(bdsOutputSize);\n+\n \tawbAlgo_ = std::make_unique();\n \tawbAlgo_->initialise(params_, bdsOutputSize, bdsGrid_);\n \n-\tsetControls(0);\n+\tagcAlgo_ = std::make_unique();\n+\tagcAlgo_->initialise(bdsGrid_);\n }\n \n void IPAIPU3::mapBuffers(const std::vector &buffers)\n@@ -232,7 +239,8 @@ void IPAIPU3::processControls([[maybe_unused]] unsigned int frame,\n \n void IPAIPU3::fillParams(unsigned int frame, ipu3_uapi_params *params)\n {\n-\tawbAlgo_->updateWbParameters(params_, 1.0);\n+\tif (agcAlgo_->updateControls())\n+\t\tawbAlgo_->updateWbParameters(params_, agcAlgo_->gamma());\n \n \t*params = params_;\n \ndiff --git a/src/ipa/ipu3/ipu3_agc.cpp b/src/ipa/ipu3/ipu3_agc.cpp\nnew file mode 100644\nindex 00000000..14baad9b\n--- /dev/null\n+++ b/src/ipa/ipu3/ipu3_agc.cpp\n@@ -0,0 +1,229 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2021, Ideas On Board\n+ *\n+ * ipu3_agc.cpp - AGC/AEC control algorithm\n+ */\n+\n+#include \"ipu3_agc.h\"\n+\n+#include \n+#include \n+#include \n+\n+#include \"libcamera/internal/log.h\"\n+\n+#include \"libipa/histogram.h\"\n+\n+namespace libcamera {\n+\n+namespace ipa {\n+\n+LOG_DEFINE_CATEGORY(IPU3Agc)\n+\n+/* Number of frames to wait before calculating stats on minimum exposure */\n+static const uint32_t kInitialFrameMinAECount = 4;\n+/* Number of frames to wait between new gain/exposure estimations */\n+static const uint32_t kFrameSkipCount = 6;\n+\n+/* Maximum ISO value for analogue gain */\n+static const uint32_t kMinISO = 100;\n+static const uint32_t kMaxISO = 1500;\n+/* Maximum analogue gain value\n+ * \\todo grab it from a camera helper */\n+static const uint32_t kMinGain = kMinISO / 100;\n+static const uint32_t kMaxGain = kMaxISO / 100;\n+/* \\todo use calculated value based on sensor */\n+static const uint32_t kMinExposure = 1;\n+static const uint32_t kMaxExposure = 1976;\n+/* \\todo those should be get from pipeline handler ! */\n+/* line duration in microseconds */\n+static const double kLineDuration = 16.8;\n+static const double kMaxExposureTime = kMaxExposure * kLineDuration;\n+/* Histogram constants */\n+static const uint32_t knumHistogramBins = 256;\n+static const double kEvGainTarget = 0.5;\n+\n+IPU3Agc::IPU3Agc()\n+\t: frameCount_(0), lastFrame_(0),\n+\t converged_(false), updateControls_(false)\n+{\n+\tiqMean_ = 0.0;\n+\tgamma_ = 1.0;\n+\thistLow_ = 0;\n+\thistHigh_ = 255;\n+\tprevTotalExposure_ = 0.0;\n+\tprevTotalExposureNoDg_ = 0.0;\n+\tcurrentTotalExposure_ = 0.0;\n+\tcurrentTotalExposureNoDg_ = 0.0;\n+}\n+\n+IPU3Agc::~IPU3Agc()\n+{\n+}\n+\n+void IPU3Agc::initialise(struct ipu3_uapi_grid_config &bdsGrid)\n+{\n+\taeGrid_ = bdsGrid;\n+}\n+void IPU3Agc::processBrightness(const ipu3_uapi_stats_3a *stats)\n+{\n+\tconst struct ipu3_uapi_grid_config statsAeGrid = stats->stats_4a_config.awb_config.grid;\n+\tRectangle aeRegion = { statsAeGrid.x_start,\n+\t\t\t statsAeGrid.y_start,\n+\t\t\t static_cast(statsAeGrid.x_end - statsAeGrid.x_start) + 1,\n+\t\t\t static_cast(statsAeGrid.y_end - statsAeGrid.y_start) + 1 };\n+\tPoint topleft = aeRegion.topLeft();\n+\tuint32_t startY = (topleft.y >> aeGrid_.block_height_log2) * aeGrid_.width << aeGrid_.block_width_log2;\n+\tuint32_t startX = (topleft.x >> aeGrid_.block_width_log2) << aeGrid_.block_width_log2;\n+\tuint32_t endX = (startX + (aeRegion.size().width >> aeGrid_.block_width_log2)) << aeGrid_.block_width_log2;\n+\tuint32_t i, j;\n+\tuint32_t count = 0;\n+\n+\tcellsBrightness_.clear();\n+\n+\tfor (j = (topleft.y >> aeGrid_.block_height_log2);\n+\t j < (topleft.y >> aeGrid_.block_height_log2) + (aeRegion.size().height >> aeGrid_.block_height_log2);\n+\t j++) {\n+\t\tfor (i = startX + startY; i < endX + startY; i += 8) {\n+\t\t\t/* grid width (and maybe height) is not reliable.\n+\t\t\t * We observed a bit shift which makes the value 160 to be 32 in the stats grid.\n+\t\t\t * Use the one passed at init time. */\n+\t\t\tif (stats->awb_raw_buffer.meta_data[i + 4 + j * aeGrid_.width] == 0) {\n+\t\t\t\tuint8_t Gr = stats->awb_raw_buffer.meta_data[i + j * aeGrid_.width];\n+\t\t\t\tuint8_t R = stats->awb_raw_buffer.meta_data[i + 1 + j * aeGrid_.width];\n+\t\t\t\tuint8_t B = stats->awb_raw_buffer.meta_data[i + 2 + j * aeGrid_.width];\n+\t\t\t\tuint8_t Gb = stats->awb_raw_buffer.meta_data[i + 3 + j * aeGrid_.width];\n+\n+\t\t\t\tcellsBrightness_.push_back(static_cast(0.2125 * R + 0.7154 * (Gr + Gb) / 2 + 0.0722 * B));\n+\t\t\t\tcount++;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tstd::vector::iterator maxIntensity = std::max_element(cellsBrightness_.begin(), cellsBrightness_.end());\n+\tLOG(IPU3Agc, Debug) << \"Most frequent intensity is \" << *maxIntensity << \" at \" << std::distance(cellsBrightness_.begin(), maxIntensity);\n+\n+\t/* \\todo create a class to generate histograms ! */\n+\tuint32_t hist[knumHistogramBins] = { 0 };\n+\tfor (uint32_t const &val : cellsBrightness_)\n+\t\thist[val]++;\n+\n+\tdouble mean = 0.0;\n+\tfor (i = 0; i < knumHistogramBins; i++) {\n+\t\tmean += hist[i] * i;\n+\t}\n+\tmean /= count;\n+\n+\tdouble variance = 0.0;\n+\tfor (i = 0; i < knumHistogramBins; i++) {\n+\t\tvariance += ((i - mean) * (i - mean)) * hist[i];\n+\t}\n+\tvariance /= count;\n+\tvariance = std::sqrt(variance);\n+\n+\tLOG(IPU3Agc, Debug) << \"mean value is: \" << mean << \" and variance is \" << variance;\n+\t/* Limit the gamma effect for now */\n+\tgamma_ = 1.1;\n+\n+\tconst auto [minBrightness, maxBrightness] = std::minmax_element(cellsBrightness_.begin(), cellsBrightness_.end());\n+\thistLow_ = *minBrightness;\n+\thistHigh_ = *maxBrightness;\n+\n+\tHistogram histogram(hist, knumHistogramBins);\n+\tiqMean_ = histogram.interQuantileMean(0.98, 1.0);\n+}\n+\n+void IPU3Agc::filterExposure(bool desaturate)\n+{\n+\tdouble speed = 0.2;\n+\tif (prevTotalExposure_ == 0.0) {\n+\t\tprevTotalExposure_ = currentTotalExposure_;\n+\t\tprevTotalExposureNoDg_ = currentTotalExposureNoDg_;\n+\t} else {\n+\t\t/* If close to the result go faster, to save making so many\n+\t\t * micro-adjustments on the way.\n+\t\t * \\ todo: Make this customisable? */\n+\t\tif (prevTotalExposure_ < 1.2 * currentTotalExposure_ &&\n+\t\t prevTotalExposure_ > 0.8 * currentTotalExposure_)\n+\t\t\tspeed = sqrt(speed);\n+\t\tprevTotalExposure_ = speed * currentTotalExposure_ +\n+\t\t\t\t prevTotalExposure_ * (1.0 - speed);\n+\t\t/* When desaturing, take a big jump down in exposure_no_dg,\n+\t\t * which we'll hide with digital gain. */\n+\t\tif (desaturate)\n+\t\t\tprevTotalExposureNoDg_ =\n+\t\t\t\tcurrentTotalExposureNoDg_;\n+\t\telse\n+\t\t\tprevTotalExposureNoDg_ =\n+\t\t\t\tspeed * currentTotalExposureNoDg_ +\n+\t\t\t\tprevTotalExposureNoDg_ * (1.0 - speed);\n+\t}\n+\t/* We can't let the no_dg exposure deviate too far below the\n+\t * total exposure, as there might not be enough digital gain available\n+\t * in the ISP to hide it (which will cause nasty oscillation). */\n+\tdouble fastReduceThreshold = 0.4;\n+\tif (prevTotalExposureNoDg_ <\n+\t prevTotalExposure_ * fastReduceThreshold)\n+\t\tprevTotalExposureNoDg_ = prevTotalExposure_ * fastReduceThreshold;\n+\tLOG(IPU3Agc, Debug) << \"After filtering, total_exposure \" << prevTotalExposure_;\n+}\n+\n+void IPU3Agc::lockExposureGain(uint32_t &exposure, uint32_t &gain)\n+{\n+\tupdateControls_ = false;\n+\n+\t/* Algorithm initialization wait for first valid frames */\n+\t/* \\todo - have a number of frames given by DelayedControls ?\n+\t * - implement a function for IIR */\n+\tif ((frameCount_ == kInitialFrameMinAECount) || (frameCount_ - lastFrame_ >= kFrameSkipCount)) {\n+\t\t/* Are we correctly exposed ? */\n+\t\tdouble newGain = kEvGainTarget * knumHistogramBins / iqMean_;\n+\n+\t\tif (std::abs(iqMean_ - kEvGainTarget * knumHistogramBins) <= 1) {\n+\t\t\tLOG(IPU3Agc, Debug) << \"!!! Good exposure with iqMean = \" << iqMean_;\n+\t\t\tconverged_ = true;\n+\t\t} else {\n+\t\t\t/* extracted from Rpi::Agc::computeTargetExposure */\n+\t\t\tdouble currentShutter = exposure * kLineDuration;\n+\t\t\tcurrentTotalExposureNoDg_ = currentShutter * gain;\n+\t\t\tLOG(IPU3Agc, Debug) << \"Actual total exposure \" << currentTotalExposureNoDg_\n+\t\t\t\t\t << \" Shutter speed \" << currentShutter\n+\t\t\t\t\t << \" Gain \" << gain;\n+\t\t\tcurrentTotalExposure_ = currentTotalExposureNoDg_ * newGain;\n+\t\t\tdouble maxTotalExposure = kMaxExposureTime * kMaxGain;\n+\t\t\tcurrentTotalExposure_ = std::min(currentTotalExposure_, maxTotalExposure);\n+\t\t\tLOG(IPU3Agc, Debug) << \"Target total exposure \" << currentTotalExposure_;\n+\n+\t\t\t/* \\todo: estimate if we need to desaturate */\n+\t\t\tfilterExposure(false);\n+\n+\t\t\tdouble newExposure = 0.0;\n+\t\t\tif (currentShutter < kMaxExposureTime) {\n+\t\t\t\texposure = std::clamp(static_cast(exposure * currentTotalExposure_ / currentTotalExposureNoDg_), kMinExposure, kMaxExposure);\n+\t\t\t\tnewExposure = currentTotalExposure_ / exposure;\n+\t\t\t\tgain = std::clamp(static_cast(gain * currentTotalExposure_ / newExposure), kMinGain, kMaxGain);\n+\t\t\t\tupdateControls_ = true;\n+\t\t\t} else if (currentShutter >= kMaxExposureTime) {\n+\t\t\t\tgain = std::clamp(static_cast(gain * currentTotalExposure_ / currentTotalExposureNoDg_), kMinGain, kMaxGain);\n+\t\t\t\tnewExposure = currentTotalExposure_ / gain;\n+\t\t\t\texposure = std::clamp(static_cast(exposure * currentTotalExposure_ / newExposure), kMinExposure, kMaxExposure);\n+\t\t\t\tupdateControls_ = true;\n+\t\t\t}\n+\t\t\tLOG(IPU3Agc, Debug) << \"Adjust exposure \" << exposure * kLineDuration << \" and gain \" << gain;\n+\t\t}\n+\t\tlastFrame_ = frameCount_;\n+\t} else {\n+\t\tupdateControls_ = false;\n+\t}\n+}\n+\n+void IPU3Agc::process(const ipu3_uapi_stats_3a *stats, uint32_t &exposure, uint32_t &gain)\n+{\n+\tprocessBrightness(stats);\n+\tlockExposureGain(exposure, gain);\n+\tframeCount_++;\n+}\n+\n+} /* namespace ipa */\n+\n+} /* namespace libcamera */\ndiff --git a/src/ipa/ipu3/ipu3_agc.h b/src/ipa/ipu3/ipu3_agc.h\nnew file mode 100644\nindex 00000000..d4657a81\n--- /dev/null\n+++ b/src/ipa/ipu3/ipu3_agc.h\n@@ -0,0 +1,67 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2021, Ideas On Board\n+ *\n+ * ipu3_agc.h - IPU3 AGC/AEC control algorithm\n+ */\n+#ifndef __LIBCAMERA_IPU3_AGC_H__\n+#define __LIBCAMERA_IPU3_AGC_H__\n+\n+#include \n+#include \n+\n+#include \n+\n+#include \n+\n+#include \"libipa/algorithm.h\"\n+\n+namespace libcamera {\n+\n+namespace ipa {\n+\n+class IPU3Agc : public Algorithm\n+{\n+public:\n+\tIPU3Agc();\n+\t~IPU3Agc();\n+\n+\tvoid initialise(struct ipu3_uapi_grid_config &bdsGrid);\n+\tvoid process(const ipu3_uapi_stats_3a *stats, uint32_t &exposure, uint32_t &gain);\n+\tbool converged() { return converged_; }\n+\tbool updateControls() { return updateControls_; }\n+\t/* \\todo Use a metadata exchange between IPAs */\n+\tdouble gamma() { return gamma_; }\n+\n+private:\n+\tvoid processBrightness(const ipu3_uapi_stats_3a *stats);\n+\tvoid filterExposure(bool desaturate);\n+\tvoid lockExposureGain(uint32_t &exposure, uint32_t &gain);\n+\n+\tstruct ipu3_uapi_grid_config aeGrid_;\n+\n+\tuint64_t frameCount_;\n+\tuint64_t lastFrame_;\n+\n+\t/* Vector of calculated brightness for each cell */\n+\tstd::vector cellsBrightness_;\n+\n+\tbool converged_;\n+\tbool updateControls_;\n+\n+\tdouble iqMean_;\n+\tdouble gamma_;\n+\tuint32_t histLow_;\n+\tuint32_t histHigh_;\n+\n+\tdouble prevTotalExposure_;\n+\tdouble prevTotalExposureNoDg_;\n+\tdouble currentTotalExposure_;\n+\tdouble currentTotalExposureNoDg_;\n+};\n+\n+} /* namespace ipa */\n+\n+} /* namespace libcamera */\n+\n+#endif /* __LIBCAMERA_IPU3_AGC_H__ */\ndiff --git a/src/ipa/ipu3/meson.build b/src/ipa/ipu3/meson.build\nindex 07a864c8..43ad0e0d 100644\n--- a/src/ipa/ipu3/meson.build\n+++ b/src/ipa/ipu3/meson.build\n@@ -4,6 +4,7 @@ ipa_name = 'ipa_ipu3'\n \n ipu3_ipa_sources = files([\n 'ipu3.cpp',\n+ 'ipu3_agc.cpp',\n 'ipu3_awb.cpp',\n ])\n \n","prefixes":["libcamera-devel","v3","5/5"]}