[{"id":18928,"web_url":"https://patchwork.libcamera.org/comment/18928/","msgid":"","date":"2021-08-18T20:48:26","subject":"Re: [libcamera-devel] [PATCH v3 4/9] ipa: ipu3: Introduce modular\n\talgorithm","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Jean-Michel,\n\nThank you for the patch.\n\nOn Wed, Aug 18, 2021 at 05:53:58PM +0200, Jean-Michel Hautbois wrote:\n> Implement a new modular framework for algorithms with a common context\n> structure that is passed to each algorithm through a common API.\n> \n> This patch:\n> - removes all the local references from IPAIPU3 and uses IPAContext\n> - implements the list of pointers and the loop at configure call on each\n> algorithm\n> - loops in fillParams on each prepare() call on the algorithm list\n> - loops in prepareStats on each process() call on the algorithm list\n> \n> Signed-off-by: Jean-Michel Hautbois \n> ---\n> src/ipa/ipu3/ipa_context.h | 27 +++--------\n> src/ipa/ipu3/ipu3.cpp | 94 +++++++++++++++++++++++++++++++++-----\n> 2 files changed, 89 insertions(+), 32 deletions(-)\n> \n> diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h\n> index 76534f7f..46291d9e 100644\n> --- a/src/ipa/ipu3/ipa_context.h\n> +++ b/src/ipa/ipu3/ipa_context.h\n> @@ -10,37 +10,22 @@\n> \n> #include \n> \n> +#include \n> +\n> namespace libcamera {\n> \n> namespace ipa::ipu3 {\n> \n> -/**\n> - * Fixed configuration of the IPA\n> - *\n> - * This structure contains data set at the configure state of the IPA.\n> - * It can't be modified while streaming.\n> - */\n\nCould you please move the documentation to ipu3.cpp in 2/9 already ? It\nwill make review easier.\n\n> struct IPAConfiguration {\n> +\tstruct Grid {\n> +\t\tipu3_uapi_grid_config bdsGrid;\n> +\t\tSize bdsOutputSize;\n> +\t} grid;\n> };\n> \n> -/**\n> - * Context of a frame for each algorithms\n> - *\n> - * This may be stored in a way that is associated with a given request\n> - * lifetime, though for now a single instance is used.\n> - * It contains informations updated while streaming by the algorithms.\n> - */\n> struct IPAFrameContext {\n> };\n> \n> -/**\n> - * Context information shared between the algorithms\n> - *\n> - * The global context structure contains:\n> - * - a configuration structure set while the IPA is configured, and not\n> - * modified when streaming\n> - * - a frameContext structure updated by the algorithms while streaming\n> - */\n> struct IPAContext {\n> \tIPAConfiguration configuration;\n> \tIPAFrameContext frameContext;\n> diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp\n> index c34fa460..dc6f0735 100644\n> --- a/src/ipa/ipu3/ipu3.cpp\n> +++ b/src/ipa/ipu3/ipu3.cpp\n> @@ -29,6 +29,7 @@\n> \n> #include \"libcamera/internal/mapped_framebuffer.h\"\n> \n> +#include \"algorithms/algorithm.h\"\n> #include \"ipu3_agc.h\"\n> #include \"ipu3_awb.h\"\n> #include \"libipa/camera_sensor_helper.h\"\n> @@ -36,6 +37,56 @@\n> static constexpr uint32_t kMaxCellWidthPerSet = 160;\n> static constexpr uint32_t kMaxCellHeightPerSet = 56;\n> \n> +/**\n> + * \\file ipa_context.h\n> + * \\brief Context information shared between the algorithms\n> + */\n> +\n> +/**\n> + * \\struct IPAConfiguration\n> + * \\brief Fixed configuration of the IPA\n> + *\n> + * This structure contains data set at the configure state of the IPA.\n> + * It can't be modified while streaming.\n> + */\n> +\n> +/**\n> + * \\struct IPAConfiguration::Grid\n\nThis should instead document \"\\var IPAConfiguration::grid\". We never use\nthe IPAConfiguration::Grid structure, but we use the\nIPAConfiguration::grid field. That's why I recommended dropping the\nstructure name, to avoid double documentation.\n\n> + * \\brief Grid configuration of the IPA\n> + *\n> + * This structure contains all the parameters needed to use the statistics.\n\nDoesn't seem to match the structure.\n\n> + *\n\nExtra blank line.\n\n> + */\n> +\n> +/**\n> + * \\var IPAConfiguration::Grid::bdsGrid\n> + * \\brief Bayer Down Scaler grid plane config used by the kernel\n> + */\n> +\n> +/**\n> + * \\var IPAConfiguration::Grid::bdsOutputSize\n> + * \\brief BDS output size configured by the pipeline handler\n> + */\n> +\n> +/**\n> + * \\struct IPAFrameContext\n> + * \\brief Context of a frame for each algorithms\n> + *\n> + * This may be stored in a way that is associated with a given request\n> + * lifetime, though for now a single instance is used.\n> + * It contains informations updated while streaming by the algorithms.\n> + */\n> +\n> +/**\n> + * \\struct IPAContext\n> + * \\brief Context information shared between the algorithms\n> + *\n> + * The global context structure contains:\n> + * - a configuration structure set while the IPA is configured, and not\n> + * modified when streaming\n> + * - a frameContext structure updated by the algorithms while streaming\n> + */\n> +\n> namespace libcamera {\n> \n> LOG_DEFINE_CATEGORY(IPAIPU3)\n> @@ -91,10 +142,12 @@ private:\n> \t/* Interface to the Camera Helper */\n> \tstd::unique_ptr camHelper_;\n> \n> +\t/* Maintain the algorithms used by the IPA */\n> +\tstd::list> algorithms_;\n> +\n> \t/* Local parameter storage */\n> +\tstruct IPAContext context_;\n> \tstruct ipu3_uapi_params params_;\n> -\n> -\tstruct ipu3_uapi_grid_config bdsGrid_;\n> };\n> \n> /**\n> @@ -191,7 +244,13 @@ void IPAIPU3::calculateBdsGrid(const Size &bdsOutputSize)\n> \tuint32_t minError = std::numeric_limits::max();\n> \tSize best;\n> \tSize bestLog2;\n> -\tbdsGrid_ = {};\n> +\tipu3_uapi_grid_config &bdsGrid = context_.configuration.grid.bdsGrid;\n> +\n> +\t/* Set the BDS output size in the IPAConfiguration structure */\n> +\tcontext_.configuration.grid.bdsOutputSize = bdsOutputSize;\n> +\n> +\tbdsGrid.x_start = 0;\n> +\tbdsGrid.y_start = 0;\n\nI'd reorder the code to avoid mixing the bdsGrid and bdsOutputSize\nconfiguration. Those two lines, as well as the local bdsGrid variable,\nwould then move below a the beginning of the next hunk.\n\n> \n> \tfor (uint32_t widthShift = 3; widthShift <= 7; ++widthShift) {\n> \t\tuint32_t width = std::min(kMaxCellWidthPerSet,\n> @@ -215,14 +274,14 @@ void IPAIPU3::calculateBdsGrid(const Size &bdsOutputSize)\n> \t\t}\n> \t}\n> \n> -\tbdsGrid_.width = best.width >> bestLog2.width;\n> -\tbdsGrid_.block_width_log2 = bestLog2.width;\n> -\tbdsGrid_.height = best.height >> bestLog2.height;\n> -\tbdsGrid_.block_height_log2 = bestLog2.height;\n> +\tbdsGrid.width = best.width >> bestLog2.width;\n> +\tbdsGrid.block_width_log2 = bestLog2.width;\n> +\tbdsGrid.height = best.height >> bestLog2.height;\n> +\tbdsGrid.block_height_log2 = bestLog2.height;\n> \n> \tLOG(IPAIPU3, Debug) << \"Best grid found is: (\"\n> -\t\t\t << (int)bdsGrid_.width << \" << \" << (int)bdsGrid_.block_width_log2 << \") x (\"\n> -\t\t\t << (int)bdsGrid_.height << \" << \" << (int)bdsGrid_.block_height_log2 << \")\";\n> +\t\t\t << (int)bdsGrid.width << \" << \" << (int)bdsGrid.block_width_log2 << \") x (\"\n> +\t\t\t << (int)bdsGrid.height << \" << \" << (int)bdsGrid.block_height_log2 << \")\";\n> }\n> \n> int IPAIPU3::configure(const IPAConfigInfo &configInfo)\n> @@ -264,15 +323,22 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo)\n> \n> \tdefVBlank_ = itVBlank->second.def().get();\n> \n> +\t/* Clean context and IPU3 parameters at configuration */\n> \tparams_ = {};\n> +\tcontext_ = {};\n> \n> \tcalculateBdsGrid(configInfo.bdsOutputSize);\n\nBlank line.\n\n> +\tfor (auto const &algo : algorithms_) {\n> +\t\tint ret = algo->configure(context_, configInfo);\n> +\t\tif (ret)\n> +\t\t\treturn ret;\n> +\t}\n> \n> \tawbAlgo_ = std::make_unique();\n> -\tawbAlgo_->initialise(params_, configInfo.bdsOutputSize, bdsGrid_);\n> +\tawbAlgo_->initialise(params_, context_.configuration.grid.bdsOutputSize, context_.configuration.grid.bdsGrid);\n\nLine wrap.\n\n> \n> \tagcAlgo_ = std::make_unique();\n> -\tagcAlgo_->initialise(bdsGrid_, sensorInfo_);\n> +\tagcAlgo_->initialise(context_.configuration.grid.bdsGrid, sensorInfo_);\n> \n> \treturn 0;\n> }\n> @@ -346,6 +412,9 @@ void IPAIPU3::processControls([[maybe_unused]] unsigned int frame,\n> \n> void IPAIPU3::fillParams(unsigned int frame, ipu3_uapi_params *params)\n> {\n> +\tfor (auto const &algo : algorithms_)\n> +\t\talgo->prepare(context_, params_);\n> +\n> \tif (agcAlgo_->updateControls())\n> \t\tawbAlgo_->updateWbParameters(params_, agcAlgo_->gamma());\n> \n> @@ -363,6 +432,9 @@ void IPAIPU3::parseStatistics(unsigned int frame,\n> {\n> \tControlList ctrls(controls::controls);\n> \n> +\tfor (auto const &algo : algorithms_)\n> +\t\talgo->process(context_, stats);\n> +\n> \tdouble gain = camHelper_->gain(gain_);\n> \tagcAlgo_->process(stats, exposure_, gain);\n> \tgain_ = camHelper_->gainCode(gain);","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 0E870BD87C\n\tfor ;\n\tWed, 18 Aug 2021 20:48:37 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 709BE68895;\n\tWed, 18 Aug 2021 22:48:36 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 0BC116888A\n\tfor ;\n\tWed, 18 Aug 2021 22:48:35 +0200 (CEST)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 79F59EE;\n\tWed, 18 Aug 2021 22:48:34 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"qGQak9Le\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1629319714;\n\tbh=GArHD7PNRo9qWgi5MJrB3FQnI+rBPorURYbZd/+pvZw=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=qGQak9Le1pPlcs3ikoSrstY9e4CwcaxaE8dBM92b/2gowQsRrL1d5Vj1WlvZyvqlo\n\tgTTrb/GY3WkkHBJOd48rGyR3ufki25Kmy3Jxb4oGAuIVbf0yL3qfjCok81uHd31uNc\n\tWHOMCwuqHdXk87aMJCjoVQPWVgLMxopuDlyBi0Zk=","Date":"Wed, 18 Aug 2021 23:48:26 +0300","From":"Laurent Pinchart ","To":"Jean-Michel Hautbois ","Message-ID":"","References":"<20210818155403.268694-1-jeanmichel.hautbois@ideasonboard.com>\n\t<20210818155403.268694-5-jeanmichel.hautbois@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20210818155403.268694-5-jeanmichel.hautbois@ideasonboard.com>","Subject":"Re: [libcamera-devel] [PATCH v3 4/9] ipa: ipu3: Introduce modular\n\talgorithm","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]