[{"id":31662,"web_url":"https://patchwork.libcamera.org/comment/31662/","msgid":"<172849082102.532453.11177311064491427651@ping.linuxembedded.co.uk>","date":"2024-10-09T16:20:21","subject":"Re: [PATCH v2 08/10] ipa: mali-c55: Add AWB Algorithm","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Quoting Daniel Scally (2024-07-09 15:49:48)\n> Add a simple grey-world auto white balance algorithm to the mali-c55\n> IPA.\n> \n> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com>\n> Co-developed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>\n> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>\n> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com>\n> ---\n> Changes in v2:\n> \n>         - Use the union rather than reinterpret_cast<>() to abstract the block\n> \n>  src/ipa/mali-c55/algorithms/awb.cpp     | 227 ++++++++++++++++++++++++\n>  src/ipa/mali-c55/algorithms/awb.h       |  40 +++++\n>  src/ipa/mali-c55/algorithms/meson.build |   1 +\n>  src/ipa/mali-c55/ipa_context.h          |  10 ++\n>  4 files changed, 278 insertions(+)\n>  create mode 100644 src/ipa/mali-c55/algorithms/awb.cpp\n>  create mode 100644 src/ipa/mali-c55/algorithms/awb.h\n> \n> diff --git a/src/ipa/mali-c55/algorithms/awb.cpp b/src/ipa/mali-c55/algorithms/awb.cpp\n> new file mode 100644\n> index 00000000..b7b74992\n> --- /dev/null\n> +++ b/src/ipa/mali-c55/algorithms/awb.cpp\n> @@ -0,0 +1,227 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2024, Ideas On Board Oy\n> + *\n> + * awb.cpp - Mali C55 grey world auto white balance algorithm\n> + */\n> +\n> +#include \"awb.h\"\n> +\n> +#include <cmath>\n> +\n> +#include <libcamera/base/log.h>\n> +\n> +#include <libcamera/control_ids.h>\n> +\n> +namespace libcamera {\n> +\n> +namespace ipa::mali_c55::algorithms {\n> +\n> +LOG_DEFINE_CATEGORY(MaliC55Awb)\n> +\n> +/* Number of frames at which we should run AWB at full speed */\n> +static constexpr uint32_t kNumStartupFrames = 4;\n> +\n> +Awb::Awb()\n> +{\n> +}\n> +\n> +int Awb::configure([[maybe_unused]] IPAContext &context,\n> +                  [[maybe_unused]] const IPACameraSensorInfo &configInfo)\n> +{\n> +       /*\n> +        * Initially we have no idea what the colour balance will be like, so\n> +        * for the first frame we will make no assumptions and leave the R/B\n> +        * channels unmodified.\n> +        */\n> +       context.activeState.awb.rGain = 1.0;\n> +       context.activeState.awb.bGain = 1.0;\n> +\n> +       return 0;\n> +}\n> +\n> +size_t Awb::fillGainsParamBlock(mali_c55_params_block block, IPAContext &context,\n> +                               IPAFrameContext &frameContext)\n> +{\n> +       block.header->type = MALI_C55_PARAM_BLOCK_AWB_GAINS;\n> +       block.header->enabled = true;\n> +       block.header->size = sizeof(struct mali_c55_params_awb_gains);\n> +\n> +       double rGain = context.activeState.awb.rGain;\n> +       double bGain = context.activeState.awb.bGain;\n> +\n> +       /*\n> +        * The gains here map as follows:\n> +        *      gain00 = R\n> +        *      gain01 = Gr\n> +        *      gain10 = Gb\n> +        *      gain11 = B\n> +        *\n> +        * This holds true regardless of the bayer order of the input data, as\n> +        * the mapping is done internally in the ISP.\n> +        */\n> +       block.awb_gains->gain00 = int(rGain * pow(2, 8));\n> +       block.awb_gains->gain01 = 256; /* Otherwise known as 1.0 */\n> +       block.awb_gains->gain10 = 256;\n> +       block.awb_gains->gain11 = int(bGain * pow(2, 8));\n\nC++ casts?\n\nOr a small lambda or helper that scales or converts the gain? Then\ngain01 and gain10 could also be assigned a more descriptive\n\n\tblock.awb_gains->gain00\t= scaledGain(rGain);\n\tblock.awb_gains->gain01 = scaledGain(1.0);\n\tblock.awb_gains->gain10 = scaledGain(1.0);\n\tblock.awb_gains->gain11 = scaledGain(bGain);\n?\n\n(scaledGain could be something named more appropriate I'm sure)\n\nAha, I think I can infer below that 256/pow(2,8) is just converting to\nthe Q4.8 format ?\n\n> +\n> +       frameContext.awb.rGain = rGain;\n> +       frameContext.awb.bGain = bGain;\n> +\n> +       return sizeof(struct mali_c55_params_awb_gains);\n> +}\n> +\n> +size_t Awb::fillConfigParamBlock(mali_c55_params_block block)\n> +{\n> +       block.header->type = MALI_C55_PARAM_BLOCK_AWB_CONFIG;\n> +       block.header->enabled = true;\n> +       block.header->size = sizeof(struct mali_c55_params_awb_config);\n> +\n> +       /* Tap the stats after the purple fringe block */\n> +       block.awb_config->tap_point = MALI_C55_AWB_STATS_TAP_PF;\n> +\n> +       /* Get R/G and B/G ratios as statistics */\n> +       block.awb_config->stats_mode = MALI_C55_AWB_MODE_RGBG;\n> +\n> +       /* Default white level */\n> +       block.awb_config->white_level = 1023;\n> +\n> +       /* Default black level */\n> +       block.awb_config->black_level = 0;\n> +\n> +       /*\n> +        * By default pixels are included who's colour ratios are bounded in a\n> +        * region (on a cr ratio x cb ratio graph) defined by four points:\n> +        *      (0.25, 0.25)\n> +        *      (0.25, 1.99609375)\n> +        *      (1.99609375, 1.99609375)\n> +        *      (1.99609375, 0.25)\n> +        *\n> +        * The ratios themselves are stored in Q4.8 format.\n> +        *\n> +        * \\todo should these perhaps be tunable?\n> +        */\n> +       block.awb_config->cr_max = 511;\n> +       block.awb_config->cr_min = 64;\n> +       block.awb_config->cb_max = 511;\n> +       block.awb_config->cb_min = 64;\n\nWe really need some better Q(a.b) helpers across libcamera. Are these\nvalues just encodings of the above coordinates ?\n\nAnyway, this is just implementation detail and can be continued as we\ndevelop.\n\n\nReviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>\n\n> +\n> +       /* We use the full 15x15 zoning scheme */\n> +       block.awb_config->nodes_used_horiz = 15;\n> +       block.awb_config->nodes_used_vert = 15;\n> +\n> +       /*\n> +        * We set the trimming boundaries equivalent to the main boundaries. In\n> +        * other words; no trimming.\n> +        */\n> +       block.awb_config->cr_high = 511;\n> +       block.awb_config->cr_low = 64;\n> +       block.awb_config->cb_high = 511;\n> +       block.awb_config->cb_low = 64;\n> +\n> +       return sizeof(struct mali_c55_params_awb_config);\n> +}\n> +\n> +void Awb::prepare(IPAContext &context, const uint32_t frame,\n> +                 IPAFrameContext &frameContext, mali_c55_params_buffer *params)\n> +{\n> +       mali_c55_params_block block;\n> +       block.data = &params->data[params->total_size];\n> +\n> +       params->total_size += fillGainsParamBlock(block, context, frameContext);\n> +\n> +       if (frame > 0)\n> +               return;\n> +\n> +       block.data = &params->data[params->total_size];\n> +       params->total_size += fillConfigParamBlock(block);\n> +}\n> +\n> +void Awb::process(IPAContext &context, const uint32_t frame,\n> +                 IPAFrameContext &frameContext, const mali_c55_stats_buffer *stats,\n> +                 [[maybe_unused]] ControlList &metadata)\n> +{\n> +       const struct mali_c55_awb_average_ratios *awb_ratios = stats->awb_ratios;\n> +\n> +       /*\n> +        * The ISP produces average R:G and B:G ratios for zones. We take the\n> +        * average of all the zones with data and simply invert them to provide\n> +        * gain figures that we can apply to approximate a grey world.\n> +        */\n> +       unsigned int counted_zones = 0;\n> +       double rgSum = 0, bgSum = 0;\n> +\n> +       for (unsigned int i = 0; i < 225; i++) {\n> +               if (!awb_ratios[i].num_pixels)\n> +                       continue;\n> +\n> +               /*\n> +                * The statistics are in Q4.8 format, so we convert to double\n> +                * here.\n> +                */\n> +               rgSum += (awb_ratios[i].avg_rg_gr * pow(2, -8));\n> +               bgSum += (awb_ratios[i].avg_bg_br * pow(2, -8));\n> +               counted_zones++;\n> +       }\n> +\n> +       /*\n> +        * Sometimes the first frame's statistics have no valid pixels, in which\n> +        * case we'll just assume a grey world until they say otherwise.\n> +        */\n> +       double rgAvg, bgAvg;\n> +       if (!counted_zones) {\n> +               rgAvg = 1.0;\n> +               bgAvg = 1.0;\n> +       } else {\n> +               rgAvg = rgSum / counted_zones;\n> +               bgAvg = bgSum / counted_zones;\n> +       }\n> +\n> +       /*\n> +        * The statistics are generated _after_ white balancing is performed in\n> +        * the ISP. To get the true ratio we therefore have to adjust the stats\n> +        * figure by the gains that were applied when the statistics for this\n> +        * frame were generated.\n> +        */\n> +       double rRatio = rgAvg / frameContext.awb.rGain;\n> +       double bRatio = bgAvg / frameContext.awb.bGain;\n> +\n> +       /*\n> +        * And then we can simply invert the ratio to find the gain we should\n> +        * apply.\n> +        */\n> +       double rGain = 1 / rRatio;\n> +       double bGain = 1 / bRatio;\n> +\n> +       /*\n> +        * Running at full speed, this algorithm results in oscillations in the\n> +        * colour balance. To remove those we dampen the speed at which it makes\n> +        * changes in gain, unless we're in the startup phase in which case we\n> +        * want to fix the miscolouring as quickly as possible.\n> +        */\n> +       double speed = frame < kNumStartupFrames ? 1.0 : 0.2;\n> +       rGain = speed * rGain + context.activeState.awb.rGain * (1.0 - speed);\n> +       bGain = speed * bGain + context.activeState.awb.bGain * (1.0 - speed);\n> +\n> +       context.activeState.awb.rGain = rGain;\n> +       context.activeState.awb.bGain = bGain;\n> +\n> +       metadata.set(controls::ColourGains, {\n> +               static_cast<float>(frameContext.awb.rGain),\n> +               static_cast<float>(frameContext.awb.bGain),\n> +       });\n> +\n> +       LOG(MaliC55Awb, Debug) << \"For frame number \" << frame << \": \"\n> +               << \"Average R/G Ratio: \" << rgAvg\n> +               << \", Average B/G Ratio: \" << bgAvg\n> +               << \"\\nrGain applied to this frame: \" << frameContext.awb.rGain\n> +               << \", bGain applied to this frame: \" << frameContext.awb.bGain\n> +               << \"\\nrGain to apply: \" << context.activeState.awb.rGain\n> +               << \", bGain to apply: \" << context.activeState.awb.bGain;\n> +}\n> +\n> +REGISTER_IPA_ALGORITHM(Awb, \"Awb\")\n> +\n> +} /* namespace ipa::mali_c55::algorithms */\n> +\n> +} /* namespace libcamera */\n> diff --git a/src/ipa/mali-c55/algorithms/awb.h b/src/ipa/mali-c55/algorithms/awb.h\n> new file mode 100644\n> index 00000000..800c2e83\n> --- /dev/null\n> +++ b/src/ipa/mali-c55/algorithms/awb.h\n> @@ -0,0 +1,40 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2024, Ideas on Board Oy\n> + *\n> + * awb.h - Mali C55 grey world auto white balance algorithm\n> + */\n> +\n> +#include \"algorithm.h\"\n> +#include \"ipa_context.h\"\n> +\n> +namespace libcamera {\n> +\n> +namespace ipa::mali_c55::algorithms {\n> +\n> +class Awb : public Algorithm\n> +{\n> +public:\n> +       Awb();\n> +       ~Awb() = default;\n> +\n> +       int configure(IPAContext &context,\n> +                     const IPACameraSensorInfo &configInfo) override;\n> +       void prepare(IPAContext &context, const uint32_t frame,\n> +                    IPAFrameContext &frameContext,\n> +                    mali_c55_params_buffer *params) override;\n> +       void process(IPAContext &context, const uint32_t frame,\n> +                    IPAFrameContext &frameContext,\n> +                    const mali_c55_stats_buffer *stats,\n> +                    ControlList &metadata) override;\n> +\n> +private:\n> +       size_t fillGainsParamBlock(mali_c55_params_block block,\n> +                                  IPAContext &context,\n> +                                  IPAFrameContext &frameContext);\n> +       size_t fillConfigParamBlock(mali_c55_params_block block);\n> +};\n> +\n> +} /* namespace ipa::mali_c55::algorithms */\n> +\n> +} /* namespace libcamera */\n> diff --git a/src/ipa/mali-c55/algorithms/meson.build b/src/ipa/mali-c55/algorithms/meson.build\n> index 96808431..f11791aa 100644\n> --- a/src/ipa/mali-c55/algorithms/meson.build\n> +++ b/src/ipa/mali-c55/algorithms/meson.build\n> @@ -2,5 +2,6 @@\n>  \n>  mali_c55_ipa_algorithms = files([\n>      'agc.cpp',\n> +    'awb.cpp',\n>      'blc.cpp',\n>  ])\n> diff --git a/src/ipa/mali-c55/ipa_context.h b/src/ipa/mali-c55/ipa_context.h\n> index 73a7cd78..105e5776 100644\n> --- a/src/ipa/mali-c55/ipa_context.h\n> +++ b/src/ipa/mali-c55/ipa_context.h\n> @@ -50,6 +50,11 @@ struct IPAActiveState {\n>                 uint32_t exposureMode;\n>                 uint32_t temperatureK;\n>         } agc;\n> +\n> +       struct {\n> +               double rGain;\n> +               double bGain;\n> +       } awb;\n>  };\n>  \n>  struct IPAFrameContext : public FrameContext {\n> @@ -58,6 +63,11 @@ struct IPAFrameContext : public FrameContext {\n>                 double sensorGain;\n>                 double ispGain;\n>         } agc;\n> +\n> +       struct {\n> +               double rGain;\n> +               double bGain;\n> +       } awb;\n>  };\n>  \n>  struct IPAContext {\n> -- \n> 2.34.1\n>","headers":{"Return-Path":"<libcamera-devel-bounces@lists.libcamera.org>","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id EFD11C32DE\n\tfor <parsemail@patchwork.libcamera.org>;\n\tWed,  9 Oct 2024 16:20:27 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id AED0B63538;\n\tWed,  9 Oct 2024 18:20:26 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 2AB37618C5\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tWed,  9 Oct 2024 18:20:24 +0200 (CEST)","from pendragon.ideasonboard.com\n\t(cpc89244-aztw30-2-0-cust6594.18-1.cable.virginm.net [86.31.185.195])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 611C82EC;\n\tWed,  9 Oct 2024 18:18:46 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"aCfn+dYK\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1728490726;\n\tbh=dU8Gx3gC6yE4yK+fYhaqpXLqB/w2LomaDKCe0W+6Ie0=;\n\th=In-Reply-To:References:Subject:From:Cc:To:Date:From;\n\tb=aCfn+dYKHbomlv8Nv3EZXrwj4Fy5rl1622ynZJKEw4TZam3AdAMGptd0JwLxgJPzS\n\tN1qKj1wHD19/DLigIt9jGkRHNY59criqnbCu5df4wlyGP+zyJLl7W5dRxrsRsWiiTu\n\tWiLxuCRdj5+Uh8WobnOsGVryO4zPqDMEUDU/e97E=","Content-Type":"text/plain; charset=\"utf-8\"","MIME-Version":"1.0","Content-Transfer-Encoding":"quoted-printable","In-Reply-To":"<20240709144950.3277837-9-dan.scally@ideasonboard.com>","References":"<20240709144950.3277837-1-dan.scally@ideasonboard.com>\n\t<20240709144950.3277837-9-dan.scally@ideasonboard.com>","Subject":"Re: [PATCH v2 08/10] ipa: mali-c55: Add AWB Algorithm","From":"Kieran Bingham <kieran.bingham@ideasonboard.com>","Cc":"Daniel Scally <dan.scally@ideasonboard.com>,\n\tNayden Kanchev <nayden.kanchev@arm.com>,\n\tJacopo Mondi <jacopo.mondi@ideasonboard.com>","To":"Daniel Scally <dan.scally@ideasonboard.com>,\n\tlibcamera-devel@lists.libcamera.org","Date":"Wed, 09 Oct 2024 17:20:21 +0100","Message-ID":"<172849082102.532453.11177311064491427651@ping.linuxembedded.co.uk>","User-Agent":"alot/0.10","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"<libcamera-devel.lists.libcamera.org>","List-Unsubscribe":"<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>","List-Archive":"<https://lists.libcamera.org/pipermail/libcamera-devel/>","List-Post":"<mailto:libcamera-devel@lists.libcamera.org>","List-Help":"<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>","List-Subscribe":"<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" <libcamera-devel-bounces@lists.libcamera.org>"}}]