[{"id":20458,"web_url":"https://patchwork.libcamera.org/comment/20458/","msgid":"<163519274237.1095920.17406685107666631503@Monstersaurus>","date":"2021-10-25T20:12:22","subject":"Re: [libcamera-devel] [PATCH v3 05/19] ipa: ipu3: awb: Add AWB\n\tclass documentation","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Quoting Jean-Michel Hautbois (2021-10-22 16:12:04)\n> The AWB algorithm is based on the Grey world algorithm and uses the\n> statistics generated by the ImgU for that. Explain how it uses those,\n> and reference the original algorithm at the same time.\n> \n> Signed-off-by: Jean-Michel Hautbois \n> \n> ---\n> v3:\n> - move the diagram from Accumulator to AWB\n> - add a word to say green gains are always set to 1\n> ---\n> src/ipa/ipu3/algorithms/awb.cpp | 108 ++++++++++++++++++++------------\n> 1 file changed, 67 insertions(+), 41 deletions(-)\n> \n> diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp\n> index 91364a04..7e10460f 100644\n> --- a/src/ipa/ipu3/algorithms/awb.cpp\n> +++ b/src/ipa/ipu3/algorithms/awb.cpp\n> @@ -11,6 +11,10 @@\n> \n> #include \n> \n> +/**\n> + * \\file awb.h\n> + */\n> +\n> namespace libcamera {\n> \n> namespace ipa::ipu3::algorithms {\n> @@ -42,47 +46,6 @@ static constexpr uint32_t kMinCellsPerZoneRatio = 255 * 90 / 100;\n> * \\struct Accumulator\n> * \\brief RGB statistics for a given zone\n> *\n> - * - Cells are defined in Pixels\n> - * - Zones are defined in Cells\n> - *\n> - * 80 cells\n> - * /───────────── 1280 pixels ───────────\\\n> - * 16 zones\n> - * 16\n> - * ┌────┬────┬────┬────┬────┬─ ──────┬────┐ \\\n> - * │Cell│ │ │ │ │ | │ │ │\n> - * 16 │ px │ │ │ │ │ | │ │ │\n> - * ├────┼────┼────┼────┼────┼─ ──────┼────┤ │\n> - * │ │ │ │ │ │ | │ │\n> - * │ │ │ │ │ │ | │ │ 7\n> - * │ ── │ ── │ ── │ ── │ ── │ ── ── ─┤ ── │ 1 2 4\n> - * │ │ │ │ │ │ | │ │ 2 0 5\n> - *\n> - * │ │ │ │ │ │ | │ │ z p c\n> - * ├────┼────┼────┼────┼────┼─ ──────┼────┤ o i e\n> - * │ │ │ │ │ │ | │ │ n x l\n> - * │ │ | │ │ e e l\n> - * ├─── ───┼─ ──────┼────┤ s l s\n> - * │ │ | │ │ s\n> - * │ │ | │ │\n> - * ├─── Zone of Cells ───┼─ ──────┼────┤ │\n> - * │ (5 x 4) │ | │ │ │\n> - * │ │ | │ │ │\n> - * ├── ───┼─ ──────┼────┤ │\n> - * │ │ │ | │ │ │\n> - * │ │ │ │ │ │ | │ │ │\n> - * └────┴────┴────┴────┴────┴─ ──────┴────┘ /\n> - *\n> - *\n> - * The algorithm works with a fixed number of zones \\a kAwbStatsSizeX x\n> - * \\a kAwbStatsSizeY. For example, a frame of 1280x720 is divided into 80x45\n> - * cells of [16x16] pixels. In the case of \\a kAwbStatsSizeX=16 and\n> - * \\a kAwbStatsSizeY=12 the zones are made of [5x4] cells. The cells are\n> - * left-aligned and calculated by IPAIPU3::calculateBdsGrid().\n> - *\n> - * Each statistics cell represents the average value of the pixels in that cell\n> - * split by colour components.\n> - *\n> * The Accumulator structure stores the sum of the average of each cell in a\n> * zone of the image, as well as the number of cells which were unsaturated and\n> * therefore included in the average.\n> @@ -152,6 +115,69 @@ static const struct ipu3_uapi_ccm_mat_config imguCssCcmDefault = {\n> 0, 0, 8191, 0\n> };\n> \n> +/**\n> + * \\class Awb\n> + * \\brief A Grey world white balance correction algorithm\n> + *\n> + * The Grey World algorithm assumes that the scene, in average, is neutral grey.\n> + * Reference: Lam, Edmund & Fung, George. (2008). Automatic White Balancing in\n> + * Digital Photography. 10.1201/9781420054538.ch10.\n> + *\n> + * The IPU3 generates statistics from the Bayer Down Scaler output into a grid\n> + * defined in the ipu3_uapi_awb_config_s structure.\n> + *\n> + * - Cells are defined in Pixels\n> + * - Zones are defined in Cells\n> + *\n> + * 80 cells\n> + * /───────────── 1280 pixels ───────────\\\n> + * 16 zones\n> + * 16\n> + * ┌────┬────┬────┬────┬────┬─ ──────┬────┐ \\\n> + * │Cell│ │ │ │ │ | │ │ │\n> + * 16 │ px │ │ │ │ │ | │ │ │\n> + * ├────┼────┼────┼────┼────┼─ ──────┼────┤ │\n> + * │ │ │ │ │ │ | │ │\n> + * │ │ │ │ │ │ | │ │ 7\n> + * │ ── │ ── │ ── │ ── │ ── │ ── ── ─┤ ── │ 1 2 4\n> + * │ │ │ │ │ │ | │ │ 2 0 5\n> + *\n> + * │ │ │ │ │ │ | │ │ z p c\n> + * ├────┼────┼────┼────┼────┼─ ──────┼────┤ o i e\n> + * │ │ │ │ │ │ | │ │ n x l\n> + * │ │ | │ │ e e l\n> + * ├─── ───┼─ ──────┼────┤ s l s\n> + * │ │ | │ │ s\n> + * │ │ | │ │\n> + * ├─── Zone of Cells ───┼─ ──────┼────┤ │\n> + * │ (5 x 4) │ | │ │ │\n> + * │ │ | │ │ │\n> + * ├── ───┼─ ──────┼────┤ │\n> + * │ │ │ | │ │ │\n> + * │ │ │ │ │ │ | │ │ │\n> + * └────┴────┴────┴────┴────┴─ ──────┴────┘ /\n> + *\n> + *\n> + * The algorithm works with a fixed number of zones \\a kAwbStatsSizeX x\n> + * \\a kAwbStatsSizeY. For example, a frame of 1280x720 is divided into 80x45\n> + * cells of [16x16] pixels. In the case of \\a kAwbStatsSizeX=16 and\n> + * \\a kAwbStatsSizeY=12 the zones are made of [5x4] cells. The cells are\n> + * left-aligned and calculated by IPAIPU3::calculateBdsGrid().\n> + *\n> + * Each statistics cell represents the average value of the pixels in that cell\n> + * split by colour components.\n> + *\n> + * Before calculating the gains, we will convert the statistics from the BDS\n> + * grid to an internal grid configuration in generateAwbStats.\n> + * As part of converting the statistics to an internal grid, the saturation\n> + * flag from the originating grid cell is used to decide if the zone contains\n> + * saturated pixels or not, making the zone relevant or not.\n> + * A saturated zone will be excluded from the calculation.\n> + *\n> + * The Grey World algorithm will then estimate the red and blue gains to apply, and\n> + * store the results in the metadata. The green gain is always set to 1.\n\nI might have said \"always set to 1, so that the Red and Blue are\nadjusted in relation to the current green levels\" or something to\nexplain 'why' they're always one. But that might not be a very good\nexplanation, and it's probably fine without it.\n\n\nReviewed-by: Kieran Bingham \n\n> + */\n> +\n> Awb::Awb()\n> : Algorithm()\n> {\n> -- \n> 2.32.0\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id DC9C2BDB1C\n\tfor ;\n\tMon, 25 Oct 2021 20:12:26 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 543E16486E;\n\tMon, 25 Oct 2021 22:12:26 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 1255660125\n\tfor ;\n\tMon, 25 Oct 2021 22:12:25 +0200 (CEST)","from pendragon.ideasonboard.com\n\t(cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id A0CCCE0A;\n\tMon, 25 Oct 2021 22:12:24 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"hdqe2q08\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1635192744;\n\tbh=6FtKITIcC/FXj3+bQhfiu42KyDPIZH4NEZCDno17NlY=;\n\th=In-Reply-To:References:Subject:From:To:Date:From;\n\tb=hdqe2q08h8AnCEAaDPw+RaWB24uYe9k4+TDOc/Km2reiI/IUCB5qGywyta3dfWgiC\n\tC2R1G3F3/C25V23PYEd1SMcg+H5VyBBgL4L8rua1DLab2LQ122CSyn9OwDqNd9qo1b\n\tvQ/Yf7TiSYHotrZyccakI7RigT2uvuyCa4GREHBQ=","Content-Type":"text/plain; charset=\"utf-8\"","MIME-Version":"1.0","Content-Transfer-Encoding":"quoted-printable","In-Reply-To":"<20211022151218.111966-6-jeanmichel.hautbois@ideasonboard.com>","References":"<20211022151218.111966-1-jeanmichel.hautbois@ideasonboard.com>\n\t<20211022151218.111966-6-jeanmichel.hautbois@ideasonboard.com>","From":"Kieran Bingham ","To":"Jean-Michel Hautbois ,\n\tlibcamera-devel@lists.libcamera.org","Date":"Mon, 25 Oct 2021 21:12:22 +0100","Message-ID":"<163519274237.1095920.17406685107666631503@Monstersaurus>","User-Agent":"alot/0.9.1","Subject":"Re: [libcamera-devel] [PATCH v3 05/19] ipa: ipu3: awb: Add AWB\n\tclass documentation","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":20460,"web_url":"https://patchwork.libcamera.org/comment/20460/","msgid":"","date":"2021-10-25T20:33:31","subject":"Re: [libcamera-devel] [PATCH v3 05/19] ipa: ipu3: awb: Add AWB\n\tclass documentation","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Jean-Michel,\n\nThank you for the patch.\n\nOn Fri, Oct 22, 2021 at 05:12:04PM +0200, Jean-Michel Hautbois wrote:\n> The AWB algorithm is based on the Grey world algorithm and uses the\n> statistics generated by the ImgU for that. Explain how it uses those,\n> and reference the original algorithm at the same time.\n> \n> Signed-off-by: Jean-Michel Hautbois \n> \n> ---\n> v3:\n> - move the diagram from Accumulator to AWB\n> - add a word to say green gains are always set to 1\n> ---\n> src/ipa/ipu3/algorithms/awb.cpp | 108 ++++++++++++++++++++------------\n> 1 file changed, 67 insertions(+), 41 deletions(-)\n> \n> diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp\n> index 91364a04..7e10460f 100644\n> --- a/src/ipa/ipu3/algorithms/awb.cpp\n> +++ b/src/ipa/ipu3/algorithms/awb.cpp\n> @@ -11,6 +11,10 @@\n> \n> #include \n> \n> +/**\n> + * \\file awb.h\n> + */\n> +\n> namespace libcamera {\n> \n> namespace ipa::ipu3::algorithms {\n> @@ -42,47 +46,6 @@ static constexpr uint32_t kMinCellsPerZoneRatio = 255 * 90 / 100;\n> * \\struct Accumulator\n> * \\brief RGB statistics for a given zone\n> *\n> - * - Cells are defined in Pixels\n> - * - Zones are defined in Cells\n> - *\n> - * 80 cells\n> - * /───────────── 1280 pixels ───────────\\\n> - * 16 zones\n> - * 16\n> - * ┌────┬────┬────┬────┬────┬─ ──────┬────┐ \\\n> - * │Cell│ │ │ │ │ | │ │ │\n> - * 16 │ px │ │ │ │ │ | │ │ │\n> - * ├────┼────┼────┼────┼────┼─ ──────┼────┤ │\n> - * │ │ │ │ │ │ | │ │\n> - * │ │ │ │ │ │ | │ │ 7\n> - * │ ── │ ── │ ── │ ── │ ── │ ── ── ─┤ ── │ 1 2 4\n> - * │ │ │ │ │ │ | │ │ 2 0 5\n> - *\n> - * │ │ │ │ │ │ | │ │ z p c\n> - * ├────┼────┼────┼────┼────┼─ ──────┼────┤ o i e\n> - * │ │ │ │ │ │ | │ │ n x l\n> - * │ │ | │ │ e e l\n> - * ├─── ───┼─ ──────┼────┤ s l s\n> - * │ │ | │ │ s\n> - * │ │ | │ │\n> - * ├─── Zone of Cells ───┼─ ──────┼────┤ │\n> - * │ (5 x 4) │ | │ │ │\n> - * │ │ | │ │ │\n> - * ├── ───┼─ ──────┼────┤ │\n> - * │ │ │ | │ │ │\n> - * │ │ │ │ │ │ | │ │ │\n> - * └────┴────┴────┴────┴────┴─ ──────┴────┘ /\n> - *\n> - *\n> - * The algorithm works with a fixed number of zones \\a kAwbStatsSizeX x\n> - * \\a kAwbStatsSizeY. For example, a frame of 1280x720 is divided into 80x45\n> - * cells of [16x16] pixels. In the case of \\a kAwbStatsSizeX=16 and\n> - * \\a kAwbStatsSizeY=12 the zones are made of [5x4] cells. The cells are\n> - * left-aligned and calculated by IPAIPU3::calculateBdsGrid().\n> - *\n> - * Each statistics cell represents the average value of the pixels in that cell\n> - * split by colour components.\n> - *\n> * The Accumulator structure stores the sum of the average of each cell in a\n> * zone of the image, as well as the number of cells which were unsaturated and\n> * therefore included in the average.\n> @@ -152,6 +115,69 @@ static const struct ipu3_uapi_ccm_mat_config imguCssCcmDefault = {\n> \t0, 0, 8191, 0\n> };\n> \n> +/**\n> + * \\class Awb\n> + * \\brief A Grey world white balance correction algorithm\n> + *\n> + * The Grey World algorithm assumes that the scene, in average, is neutral grey.\n> + * Reference: Lam, Edmund & Fung, George. (2008). Automatic White Balancing in\n> + * Digital Photography. 10.1201/9781420054538.ch10.\n> + *\n> + * The IPU3 generates statistics from the Bayer Down Scaler output into a grid\n> + * defined in the ipu3_uapi_awb_config_s structure.\n> + *\n> + * - Cells are defined in Pixels\n> + * - Zones are defined in Cells\n> + *\n> + * 80 cells\n> + * /───────────── 1280 pixels ───────────\\\n> + * 16 zones\n> + * 16\n> + * ┌────┬────┬────┬────┬────┬─ ──────┬────┐ \\\n> + * │Cell│ │ │ │ │ | │ │ │\n> + * 16 │ px │ │ │ │ │ | │ │ │\n> + * ├────┼────┼────┼────┼────┼─ ──────┼────┤ │\n> + * │ │ │ │ │ │ | │ │\n> + * │ │ │ │ │ │ | │ │ 7\n> + * │ ── │ ── │ ── │ ── │ ── │ ── ── ─┤ ── │ 1 2 4\n> + * │ │ │ │ │ │ | │ │ 2 0 5\n> + *\n> + * │ │ │ │ │ │ | │ │ z p c\n> + * ├────┼────┼────┼────┼────┼─ ──────┼────┤ o i e\n> + * │ │ │ │ │ │ | │ │ n x l\n> + * │ │ | │ │ e e l\n> + * ├─── ───┼─ ──────┼────┤ s l s\n> + * │ │ | │ │ s\n> + * │ │ | │ │\n> + * ├─── Zone of Cells ───┼─ ──────┼────┤ │\n> + * │ (5 x 4) │ | │ │ │\n> + * │ │ | │ │ │\n> + * ├── ───┼─ ──────┼────┤ │\n> + * │ │ │ | │ │ │\n> + * │ │ │ │ │ │ | │ │ │\n> + * └────┴────┴────┴────┴────┴─ ──────┴────┘ /\n> + *\n> + *\n> + * The algorithm works with a fixed number of zones \\a kAwbStatsSizeX x\n> + * \\a kAwbStatsSizeY. For example, a frame of 1280x720 is divided into 80x45\n> + * cells of [16x16] pixels. In the case of \\a kAwbStatsSizeX=16 and\n> + * \\a kAwbStatsSizeY=12 the zones are made of [5x4] cells. The cells are\n> + * left-aligned and calculated by IPAIPU3::calculateBdsGrid().\n> + *\n> + * Each statistics cell represents the average value of the pixels in that cell\n> + * split by colour components.\n> + *\n> + * Before calculating the gains, we will convert the statistics from the BDS\n> + * grid to an internal grid configuration in generateAwbStats.\n> + * As part of converting the statistics to an internal grid, the saturation\n> + * flag from the originating grid cell is used to decide if the zone contains\n> + * saturated pixels or not, making the zone relevant or not.\n> + * A saturated zone will be excluded from the calculation.\n\n * In each cell, the ImgU computes for each colour component the average of all\n * unsaturated pixels (below a programmable threshold). It also provides the\n * ratio of saturated pixels in the cell.\n *\n * The AWB algorithm operates on a coarser grid, made by grouping cells from the\n * hardware grid into zones. The number of zones is fixed to \\a kAwbStatsSizeX x\n * \\a kAwbStatsSizeY. For example, a frame of 1280x720 is divided into 80x45\n * cells of [16x16] pixels and 16x12 zones of [5x4] cells each\n * (\\a kAwbStatsSizeX=16 and \\a kAwbStatsSizeY=12). If the number of cells isn't\n * an exact multiple of the number of zones, the right-most and bottom-most\n * cells are ignored. The grid configuration is computed by\n * IPAIPU3::calculateBdsGrid().\n *\n * Before calculating the gains, the algorithm aggregates the cell averages for\n * each zone in generateAwbStats(). Cells that have a too high ratio of\n * saturated pixels are ignored, and only zones that contain enough\n * non-saturated cells are then used by the algorithm.\n\nReviewed-by: Laurent Pinchart \n\n> + *\n> + * The Grey World algorithm will then estimate the red and blue gains to apply, and\n> + * store the results in the metadata. The green gain is always set to 1.\n> + */\n> +\n> Awb::Awb()\n> \t: Algorithm()\n> {","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 5EA40BDB1C\n\tfor ;\n\tMon, 25 Oct 2021 20:33:56 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id A6E4E6487A;\n\tMon, 25 Oct 2021 22:33:55 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 3196460125\n\tfor ;\n\tMon, 25 Oct 2021 22:33:54 +0200 (CEST)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id A67A1E0A;\n\tMon, 25 Oct 2021 22:33:53 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"rkuzlRky\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1635194033;\n\tbh=sReuP4xypgui90CLSgqRxxzFSC8IQm1Cn8ODHiTsPd0=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=rkuzlRkyz/hvxG5IJm4SKJPvRUeQMCu0od9cxhTwtW7EIHjVz1txI3NSBXPS8/mUV\n\tiG5vn3JctlgOv+HVorwhUjI3ArjLB3+nNd4QWqp2odU0GR3goXgoJk3fmcJ2vG6uQf\n\tlqbz13yQE2D5vk0qN2rEyArd6pMpuiV5bp78ikF4=","Date":"Mon, 25 Oct 2021 23:33:31 +0300","From":"Laurent Pinchart ","To":"Jean-Michel Hautbois ","Message-ID":"","References":"<20211022151218.111966-1-jeanmichel.hautbois@ideasonboard.com>\n\t<20211022151218.111966-6-jeanmichel.hautbois@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<20211022151218.111966-6-jeanmichel.hautbois@ideasonboard.com>","Subject":"Re: [libcamera-devel] [PATCH v3 05/19] ipa: ipu3: awb: Add AWB\n\tclass documentation","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]