{"id":15568,"url":"https://patchwork.libcamera.org/api/patches/15568/?format=json","web_url":"https://patchwork.libcamera.org/patch/15568/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20220328092433.69856-6-jeanmichel.hautbois@ideasonboard.com>","date":"2022-03-28T09:24:33","name":"[libcamera-devel,v4,5/5] ipa: rkisp1: Introduce AWB","commit_ref":"4b2251509f1543d018fea577e7e2760bb913ee86","pull_url":null,"state":"accepted","archived":false,"hash":"30a97a9e38940c1e34fabea5b47273ab08af37b8","submitter":{"id":75,"url":"https://patchwork.libcamera.org/api/people/75/?format=json","name":"Jean-Michel Hautbois","email":"jeanmichel.hautbois@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/15568/mbox/","series":[{"id":3005,"url":"https://patchwork.libcamera.org/api/series/3005/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=3005","date":"2022-03-28T09:24:28","name":"IPA RkISP1 awb and misc improvements","version":4,"mbox":"https://patchwork.libcamera.org/series/3005/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/15568/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/15568/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 6E8CFC3267\n\tfor ;\n\tMon, 28 Mar 2022 09:24:47 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 1E16C65633;\n\tMon, 28 Mar 2022 11:24:47 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id DF3BF65634\n\tfor ;\n\tMon, 28 Mar 2022 11:24:42 +0200 (CEST)","from tatooine.ideasonboard.com (unknown\n\t[IPv6:2a01:e0a:169:7140:4f49:c672:4655:7dbb])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 5797CB9C;\n\tMon, 28 Mar 2022 11:24:42 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1648459487;\n\tbh=nQ0ceSe4JXN8Fh7yx01fAZsg5TftRn7EsPWbb1iZ0kM=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=xmeW3ZAyNa+w9knpaoW3zc5uMM6vc1aj5felZsvlawphiPLxbmOxZyZg9fObQLWU3\n\ttPS9YHYEEvzr9LxpsYhd+4wpfh/E7nQALT2Hf5D0Ma3GQyGBohVgYw9vjcPeUM1boK\n\t+qft7JwYc7SRRSLr3lGLCGBVUL/fOxYr1mba1lz33EBPV5DdRkpn46ugVjhoH38tuF\n\tuHsN5d6xlubsUfyNT/Mq+HwIs6crAHgczRIoR5cxlxkDDKHKbU7d5dgK15NBFg3opT\n\t67SOceQhiLN+ZsLifuLb7RGbtl4j/viFER3VV4SAzpoIs52z8g73VQn9eH9L/alzg+\n\tYAQtApjyoPMuw==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1648459482;\n\tbh=nQ0ceSe4JXN8Fh7yx01fAZsg5TftRn7EsPWbb1iZ0kM=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=qhpf/7d9KLBbqFkIax+JdMghk+Fx+P1C7+gJfXYtx+dD4zlNCRFDyJzmkDI53C/gO\n\tSFOuqV3HMwklyZkuxY8du5wIPvwg89zbZ6wr8OSKTEHJSrNPzwziyrEMYkIOx8iKeg\n\t0bPwRp/lkSlOwoYNlgc1ae7TOmM+KxESs0l7rVxg="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"qhpf/7d9\"; dkim-atps=neutral","To":"libcamera-devel@lists.libcamera.org","Date":"Mon, 28 Mar 2022 11:24:33 +0200","Message-Id":"<20220328092433.69856-6-jeanmichel.hautbois@ideasonboard.com>","X-Mailer":"git-send-email 2.32.0","In-Reply-To":"<20220328092433.69856-1-jeanmichel.hautbois@ideasonboard.com>","References":"<20220328092433.69856-1-jeanmichel.hautbois@ideasonboard.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH v4 5/5] ipa: rkisp1: Introduce AWB","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Jean-Michel Hautbois via libcamera-devel\n\t","Reply-To":"Jean-Michel Hautbois ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"The RkISP1 ISP calculates a mean value for Y, Cr and Cb at each frame.\nThere is a RGB mode which could theoretically give us the values for R,\nG and B directly, but it seems to be failing right now.\n\nConvert those values into R, G and B and estimate the gain to apply in a\ngrey world.\n\nSigned-off-by: Jean-Michel Hautbois \nTested-by: Peter Griffin \nReviewed-by: Laurent Pinchart \n---\nv4: - swapped blue and green :-(\n - saturate after filtering\n - introduce a speed variable for filtering\n---\n src/ipa/rkisp1/algorithms/awb.cpp | 178 ++++++++++++++++++++++++++\n src/ipa/rkisp1/algorithms/awb.h | 33 +++++\n src/ipa/rkisp1/algorithms/meson.build | 1 +\n src/ipa/rkisp1/ipa_context.cpp | 28 ++++\n src/ipa/rkisp1/ipa_context.h | 16 +++\n src/ipa/rkisp1/rkisp1.cpp | 2 +\n 6 files changed, 258 insertions(+)\n create mode 100644 src/ipa/rkisp1/algorithms/awb.cpp\n create mode 100644 src/ipa/rkisp1/algorithms/awb.h","diff":"diff --git a/src/ipa/rkisp1/algorithms/awb.cpp b/src/ipa/rkisp1/algorithms/awb.cpp\nnew file mode 100644\nindex 00000000..be4585c6\n--- /dev/null\n+++ b/src/ipa/rkisp1/algorithms/awb.cpp\n@@ -0,0 +1,178 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2021-2022, Ideas On Board\n+ *\n+ * awb.cpp - AWB control algorithm\n+ */\n+\n+#include \"awb.h\"\n+\n+#include \n+#include \n+\n+#include \n+\n+#include \n+\n+/**\n+ * \\file awb.h\n+ */\n+\n+namespace libcamera {\n+\n+namespace ipa::rkisp1::algorithms {\n+\n+/**\n+ * \\class Awb\n+ * \\brief A Grey world white balance correction algorithm\n+ */\n+\n+LOG_DEFINE_CATEGORY(RkISP1Awb)\n+\n+/**\n+ * \\copydoc libcamera::ipa::Algorithm::configure\n+ */\n+int Awb::configure(IPAContext &context,\n+\t\t const IPACameraSensorInfo &configInfo)\n+{\n+\tcontext.frameContext.awb.gains.red = 1.0;\n+\tcontext.frameContext.awb.gains.blue = 1.0;\n+\tcontext.frameContext.awb.gains.green = 1.0;\n+\n+\t/*\n+\t * Define the measurement window for AWB as a centered rectangle\n+\t * covering 3/4 of the image width and height.\n+\t */\n+\tcontext.configuration.awb.measureWindow.h_offs = configInfo.outputSize.width / 8;\n+\tcontext.configuration.awb.measureWindow.v_offs = configInfo.outputSize.height / 8;\n+\tcontext.configuration.awb.measureWindow.h_size = 3 * configInfo.outputSize.width / 4;\n+\tcontext.configuration.awb.measureWindow.v_size = 3 * configInfo.outputSize.height / 4;\n+\n+\treturn 0;\n+}\n+\n+uint32_t Awb::estimateCCT(double red, double green, double blue)\n+{\n+\t/* Convert the RGB values to CIE tristimulus values (XYZ) */\n+\tdouble X = (-0.14282) * (red) + (1.54924) * (green) + (-0.95641) * (blue);\n+\tdouble Y = (-0.32466) * (red) + (1.57837) * (green) + (-0.73191) * (blue);\n+\tdouble Z = (-0.68202) * (red) + (0.77073) * (green) + (0.56332) * (blue);\n+\n+\t/* Calculate the normalized chromaticity values */\n+\tdouble x = X / (X + Y + Z);\n+\tdouble y = Y / (X + Y + Z);\n+\n+\t/* Calculate CCT */\n+\tdouble n = (x - 0.3320) / (0.1858 - y);\n+\treturn 449 * n * n * n + 3525 * n * n + 6823.3 * n + 5520.33;\n+}\n+\n+/**\n+ * \\copydoc libcamera::ipa::Algorithm::prepare\n+ */\n+void Awb::prepare(IPAContext &context, rkisp1_params_cfg *params)\n+{\n+\tparams->others.awb_gain_config.gain_green_b = 256 * context.frameContext.awb.gains.green;\n+\tparams->others.awb_gain_config.gain_blue = 256 * context.frameContext.awb.gains.blue;\n+\tparams->others.awb_gain_config.gain_red = 256 * context.frameContext.awb.gains.red;\n+\tparams->others.awb_gain_config.gain_green_r = 256 * context.frameContext.awb.gains.green;\n+\n+\t/* Update the gains. */\n+\tparams->module_cfg_update |= RKISP1_CIF_ISP_MODULE_AWB_GAIN;\n+\n+\t/* If we already have configured the gains and window, return. */\n+\tif (context.frameContext.frameCount > 0)\n+\t\treturn;\n+\n+\t/* Configure the gains to apply. */\n+\tparams->module_en_update |= RKISP1_CIF_ISP_MODULE_AWB_GAIN;\n+\t/* Update the ISP to apply the gains configured. */\n+\tparams->module_ens |= RKISP1_CIF_ISP_MODULE_AWB_GAIN;\n+\n+\t/* Configure the measure window for AWB. */\n+\tparams->meas.awb_meas_config.awb_wnd = context.configuration.awb.measureWindow;\n+\t/*\n+\t * Measure Y, Cr and Cb means.\n+\t * \\todo RGB is not working, the kernel seems to not configure it ?\n+\t */\n+\tparams->meas.awb_meas_config.awb_mode = RKISP1_CIF_ISP_AWB_MODE_YCBCR;\n+\t/* Reference Cr and Cb. */\n+\tparams->meas.awb_meas_config.awb_ref_cb = 128;\n+\tparams->meas.awb_meas_config.awb_ref_cr = 128;\n+\t/* Y values to include are between min_y and max_y only. */\n+\tparams->meas.awb_meas_config.min_y = 16;\n+\tparams->meas.awb_meas_config.max_y = 250;\n+\t/* Maximum Cr+Cb value to take into account for awb. */\n+\tparams->meas.awb_meas_config.max_csum = 250;\n+\t/* Minimum Cr and Cb values to take into account. */\n+\tparams->meas.awb_meas_config.min_c = 16;\n+\t/* Number of frames to use to estimate the mean (0 means 1 frame). */\n+\tparams->meas.awb_meas_config.frames = 0;\n+\n+\t/* Update AWB measurement unit configuration. */\n+\tparams->module_cfg_update |= RKISP1_CIF_ISP_MODULE_AWB;\n+\t/* Make sure the ISP is measuring the means for the next frame. */\n+\tparams->module_en_update |= RKISP1_CIF_ISP_MODULE_AWB;\n+\tparams->module_ens |= RKISP1_CIF_ISP_MODULE_AWB;\n+}\n+\n+/**\n+ * \\copydoc libcamera::ipa::Algorithm::process\n+ */\n+void Awb::process([[maybe_unused]] IPAContext &context, const rkisp1_stat_buffer *stats)\n+{\n+\tconst rkisp1_cif_isp_stat *params = &stats->params;\n+\tconst rkisp1_cif_isp_awb_stat *awb = ¶ms->awb;\n+\tIPAFrameContext &frameContext = context.frameContext;\n+\n+\t/* Get the YCbCr mean values */\n+\tdouble yMean = awb->awb_mean[0].mean_y_or_g;\n+\tdouble crMean = awb->awb_mean[0].mean_cr_or_r;\n+\tdouble cbMean = awb->awb_mean[0].mean_cb_or_b;\n+\n+\t/*\n+\t * Convert from YCbCr to RGB.\n+\t * The hardware uses the following formulas:\n+\t * Y = 16 + 0.2500 R + 0.5000 G + 0.1094 B\n+\t * Cb = 128 - 0.1406 R - 0.2969 G + 0.4375 B\n+\t * Cr = 128 + 0.4375 R - 0.3750 G - 0.0625 B\n+\t *\n+\t * The inverse matrix is thus:\n+\t * [[1,1636, -0,0623, 1,6008]\n+\t * [1,1636, -0,4045, -0,7949]\n+\t * [1,1636, 1,9912, -0,0250]]\n+\t */\n+\tyMean -= 16;\n+\tcbMean -= 128;\n+\tcrMean -= 128;\n+\tdouble redMean = 1.1636 * yMean - 0.0623 * cbMean + 1.6008 * crMean;\n+\tdouble greenMean = 1.1636 * yMean - 0.4045 * cbMean - 0.7949 * crMean;\n+\tdouble blueMean = 1.1636 * yMean + 1.9912 * cbMean - 0.0250 * crMean;\n+\n+\t/* Estimate the red and blue gains to apply in a grey world. */\n+\tdouble redGain = greenMean / (redMean + 1);\n+\tdouble blueGain = greenMean / (blueMean + 1);\n+\n+\t/* Filter the values to avoid oscillations. */\n+\tdouble speed = 0.2;\n+\tredGain = speed * redGain + (1 - speed) * frameContext.awb.gains.red;\n+\tblueGain = speed * blueGain + (1 - speed) * frameContext.awb.gains.blue;\n+\n+\t/*\n+\t * Gain values are unsigned integer value, range 0 to 4 with 8 bit\n+\t * fractional part.\n+\t */\n+\tframeContext.awb.gains.red = std::clamp(redGain, 0.0, 1023.0 / 256);\n+\tframeContext.awb.gains.blue = std::clamp(blueGain, 0.0, 1023.0 / 256);\n+\t/* Hardcode the green gain to 1.0. */\n+\tframeContext.awb.gains.green = 1.0;\n+\n+\tframeContext.awb.temperatureK = estimateCCT(redMean, greenMean, blueMean);\n+\n+\tLOG(RkISP1Awb, Debug) << \"Gain found for red: \" << context.frameContext.awb.gains.red\n+\t\t\t << \" and for blue: \" << context.frameContext.awb.gains.blue;\n+}\n+\n+} /* namespace ipa::rkisp1::algorithms */\n+\n+} /* namespace libcamera */\ndiff --git a/src/ipa/rkisp1/algorithms/awb.h b/src/ipa/rkisp1/algorithms/awb.h\nnew file mode 100644\nindex 00000000..11946643\n--- /dev/null\n+++ b/src/ipa/rkisp1/algorithms/awb.h\n@@ -0,0 +1,33 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2021-2022, Ideas On Board\n+ *\n+ * awb.h - AWB control algorithm\n+ */\n+\n+#pragma once\n+\n+#include \n+\n+#include \"algorithm.h\"\n+\n+namespace libcamera {\n+\n+namespace ipa::rkisp1::algorithms {\n+\n+class Awb : public Algorithm\n+{\n+public:\n+\tAwb() = default;\n+\t~Awb() = default;\n+\n+\tint configure(IPAContext &context, const IPACameraSensorInfo &configInfo) override;\n+\tvoid prepare(IPAContext &context, rkisp1_params_cfg *params) override;\n+\tvoid process(IPAContext &context, const rkisp1_stat_buffer *stats) override;\n+\n+private:\n+\tuint32_t estimateCCT(double red, double green, double blue);\n+};\n+\n+} /* namespace ipa::rkisp1::algorithms */\n+} /* namespace libcamera */\ndiff --git a/src/ipa/rkisp1/algorithms/meson.build b/src/ipa/rkisp1/algorithms/meson.build\nindex 27c97731..7ec53d89 100644\n--- a/src/ipa/rkisp1/algorithms/meson.build\n+++ b/src/ipa/rkisp1/algorithms/meson.build\n@@ -2,5 +2,6 @@\n \n rkisp1_ipa_algorithms = files([\n 'agc.cpp',\n+ 'awb.cpp',\n 'blc.cpp',\n ])\ndiff --git a/src/ipa/rkisp1/ipa_context.cpp b/src/ipa/rkisp1/ipa_context.cpp\nindex 4a1ab058..1559d3ff 100644\n--- a/src/ipa/rkisp1/ipa_context.cpp\n+++ b/src/ipa/rkisp1/ipa_context.cpp\n@@ -81,6 +81,14 @@ namespace libcamera::ipa::rkisp1 {\n * \\brief Hardware revision of the ISP\n */\n \n+/**\n+ * \\var IPASessionConfiguration::awb\n+ * \\brief AWB parameters configuration of the IPA\n+ *\n+ * \\var IPASessionConfiguration::awb.measureWindow\n+ * \\brief AWB measure window\n+ */\n+\n /**\n * \\var IPASessionConfiguration::sensor\n * \\brief Sensor-specific configuration of the IPA\n@@ -105,6 +113,26 @@ namespace libcamera::ipa::rkisp1 {\n * The gain should be adapted to the sensor specific gain code before applying.\n */\n \n+/**\n+ * \\var IPAFrameContext::awb\n+ * \\brief Context for the Automatic White Balance algorithm\n+ *\n+ * \\struct IPAFrameContext::awb.gains\n+ * \\brief White balance gains\n+ *\n+ * \\var IPAFrameContext::awb.gains.red\n+ * \\brief White balance gain for R channel\n+ *\n+ * \\var IPAFrameContext::awb.gains.green\n+ * \\brief White balance gain for G channel\n+ *\n+ * \\var IPAFrameContext::awb.gains.blue\n+ * \\brief White balance gain for B channel\n+ *\n+ * \\var IPAFrameContext::awb.temperatureK\n+ * \\brief Estimated color temperature\n+ */\n+\n /**\n * \\var IPAFrameContext::sensor\n * \\brief Effective sensor values\ndiff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h\nindex 35e9b8e5..f387cace 100644\n--- a/src/ipa/rkisp1/ipa_context.h\n+++ b/src/ipa/rkisp1/ipa_context.h\n@@ -12,6 +12,8 @@\n \n #include \n \n+#include \n+\n namespace libcamera {\n \n namespace ipa::rkisp1 {\n@@ -25,6 +27,10 @@ struct IPASessionConfiguration {\n \t\tstruct rkisp1_cif_isp_window measureWindow;\n \t} agc;\n \n+\tstruct {\n+\t\tstruct rkisp1_cif_isp_window measureWindow;\n+\t} awb;\n+\n \tstruct {\n \t\tutils::Duration lineDuration;\n \t} sensor;\n@@ -40,6 +46,16 @@ struct IPAFrameContext {\n \t\tdouble gain;\n \t} agc;\n \n+\tstruct {\n+\t\tstruct {\n+\t\t\tdouble red;\n+\t\t\tdouble green;\n+\t\t\tdouble blue;\n+\t\t} gains;\n+\n+\t\tdouble temperatureK;\n+\t} awb;\n+\n \tstruct {\n \t\tuint32_t exposure;\n \t\tdouble gain;\ndiff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp\nindex 83ed21be..13ab1cdc 100644\n--- a/src/ipa/rkisp1/rkisp1.cpp\n+++ b/src/ipa/rkisp1/rkisp1.cpp\n@@ -27,6 +27,7 @@\n \n #include \"algorithms/agc.h\"\n #include \"algorithms/algorithm.h\"\n+#include \"algorithms/awb.h\"\n #include \"algorithms/blc.h\"\n #include \"libipa/camera_sensor_helper.h\"\n \n@@ -121,6 +122,7 @@ int IPARkISP1::init(const IPASettings &settings, unsigned int hwRevision)\n \n \t/* Construct our Algorithms */\n \talgorithms_.push_back(std::make_unique());\n+\talgorithms_.push_back(std::make_unique());\n \talgorithms_.push_back(std::make_unique());\n \n \treturn 0;\n","prefixes":["libcamera-devel","v4","5/5"]}