[{"id":29541,"web_url":"https://patchwork.libcamera.org/comment/29541/","msgid":"<20240515092056.tty4yzie42tyhl4l@jasper>","date":"2024-05-15T09:20:56","subject":"Re: [PATCH v3 2/3] ipa: libipa: Add MatrixInterpolator class","submitter":{"id":184,"url":"https://patchwork.libcamera.org/api/people/184/","name":"Stefan Klug","email":"stefan.klug@ideasonboard.com"},"content":"On Tue, May 07, 2024 at 03:10:15PM +0900, Paul Elder wrote:\n> Add a class to encapsulate the functionality of fetching a matrix based\n> on an integer key, and interpolating if there is no exact match. This is\n> expected to be used by both color correction matrices / crosstalk\n> correction as well as lens shading correction.\n> \n> Signed-off-by: Paul Elder \n> \n> ---\n> Changes in v3:\n> - add a constructor that takes a map of unsigned int -> matrix\n> - s/unit/identity\n> - clear matrices on reset and when reading from yaml\n> - add assert at get()\n> \n> Changes in v2:\n> - initialize to identity matrix\n> - add a function to reset to identity matrix\n> - other minor fixes\n> ---\n> src/ipa/libipa/matrix_interpolator.cpp | 54 ++++++++++++\n> src/ipa/libipa/matrix_interpolator.h | 116 +++++++++++++++++++++++++\n> src/ipa/libipa/meson.build | 2 +\n> 3 files changed, 172 insertions(+)\n> create mode 100644 src/ipa/libipa/matrix_interpolator.cpp\n> create mode 100644 src/ipa/libipa/matrix_interpolator.h\n> \n> diff --git a/src/ipa/libipa/matrix_interpolator.cpp b/src/ipa/libipa/matrix_interpolator.cpp\n> new file mode 100644\n> index 00000000..fc1837e7\n> --- /dev/null\n> +++ b/src/ipa/libipa/matrix_interpolator.cpp\n> @@ -0,0 +1,54 @@\n> +/* SPDX-License-Identifier: BSD-2-Clause */\n> +/*\n> + * Copyright (C) 2019, Raspberry Pi Ltd\n> + * Copyright (C) 2024, Paul Elder \n> + *\n> + * matrix_interpolator.cpp - Helper class for interpolating maps of matrices\n> + */\n> +#include \"matrix_interpolator.h\"\n> +\n> +#include \n> +#include \n> +\n> +#include \n> +\n> +#include \"libcamera/internal/yaml_parser.h\"\n> +\n> +#include \"matrix.h\"\n> +\n> +/**\n> + * \\file ccm.h\n> + * \\brief Helper class for interpolating maps of matrices\n> + */\n> +\n> +namespace libcamera {\n> +\n> +LOG_DEFINE_CATEGORY(MatrixInterpolator)\n> +\n> +namespace ipa {\n> +\n> +/**\n> + * \\class MatrixInterpolator\n> + * \\brief Class for storing, retrieving, and interpolating matrices\n> + */\n> +\n> +/**\n> + * \\fn int MatrixInterpolator::readYaml(const libcamera::YamlObject &yaml)\n> + * \\brief Initialize an MatrixInterpolator instance from yaml\n> + * \\tparam T Type of data stored in the matrices\n> + * \\tparam R Number of rows of the matrices\n> + * \\tparam C Number of columns of the matrices\n> + * \\param[in] yaml The yaml object that contains the map of unsigned integers to matrices\n> + * \\return Zero on success, negative error code otherwise\n> + */\n> +\n> +/**\n> + * \\fn Matrix MatrixInterpolator::get(unsigned int key)\n> + * \\brief Retrieve a matrix from the list of matrices, interpolating if necessary\n> + * \\param[in] key The unsigned integer key of the matrix to retrieve\n> + * \\return The matrix corresponding to the color temperature\n> + */\n> +\n> +} /* namespace ipa */\n> +\n> +} /* namespace libcamera */\n> diff --git a/src/ipa/libipa/matrix_interpolator.h b/src/ipa/libipa/matrix_interpolator.h\n> new file mode 100644\n> index 00000000..2c6263cb\n> --- /dev/null\n> +++ b/src/ipa/libipa/matrix_interpolator.h\n> @@ -0,0 +1,116 @@\n> +/* SPDX-License-Identifier: BSD-2-Clause */\n> +/*\n> + * Copyright (C) 2019, Raspberry Pi Ltd\n> + * Copyright (C) 2024, Paul Elder \n> + *\n> + * matrix_interpolator.h - Helper class for interpolating maps of matrices\n> + */\n> +\n> +#pragma once\n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +\n> +#include \"libcamera/internal/yaml_parser.h\"\n> +\n> +#include \"matrix.h\"\n> +\n> +namespace libcamera {\n> +\n> +LOG_DECLARE_CATEGORY(MatrixInterpolator)\n> +\n> +namespace ipa {\n> +\n> +template +\t std::enable_if_t> * = nullptr>\n> +class MatrixInterpolator\n> +{\n> +public:\n> +\tMatrixInterpolator()\n> +\t{\n> +\t\treset();\n> +\t};\n> +\n> +\tMatrixInterpolator(const std::map> &matrices)\n> +\t{\n> +\t\tfor (const auto &pair : matrices)\n> +\t\t\tmatrices_[pair.first] = pair.second;\n> +\t};\n> +\n> +\t~MatrixInterpolator(){};\n> +\n> +\tvoid reset()\n> +\t{\n> +\t\tMatrix identity;\n> +\t\tmatrices_.clear();\n> +\t\tmatrices_[0] = identity;\n> +\t}\n> +\n> +\tint readYaml(const libcamera::YamlObject &yaml)\n> +\t{\n> +\t\tint ret;\n> +\n> +\t\tmatrices_.clear();\n> +\n> +\t\tif (!yaml.isDictionary()) {\n> +\t\t\tLOG(MatrixInterpolator, Error) << \"yaml object must be a dictionary\";\n> +\t\t\treturn -EINVAL;\n> +\t\t}\n> +\n\nI have the feeling that this might not be flexible enough. At the moment\nthis expects a yaml representation of\n\n2000: [ matrix values... ]\n3000: [ ... ]\n\nWe might need to add more infos for a given color temperature. In my\nprototypes I used a list like this:\n\n- ct: 2000\n ccm: [matrix values]\n- ct: 3000\n ccm: [ matrix values ]\n\n\nThis adds the flexibility to e.g. add the offset vector.\nMy diff is:\n-\tint readYaml(const libcamera::YamlObject &yaml)\n+\tint readYaml(const libcamera::YamlObject &yaml, const std::string& key_property, const std::string& value_property)\n \t{\n \t\tint ret;\n\t\t\n\t\tmatrices_.clear(); \n\n-\t\tif (!yaml.isDictionary()) {\n-\t\t\tLOG(MatrixInterpolator, Error) << \"yaml object must be a dictionary\";\n+\t\tif (!yaml.isList()) {\n+\t\t\tLOG(MatrixInterpolator, Error) << \"yaml object must be a list\";\n \t\t\treturn -EINVAL;\n \t\t}\n \n-\t\tfor (const auto &[ctStr, value] : yaml.asDict()) {\n-\t\t\tunsigned int ct = std::stoul(ctStr);\n+\t\tfor (const auto &value : yaml.asList()) {\n+\t\t\tunsigned int ct = std::stoul(value[key_property].get(\"\"));\n \t\t\tMatrix matrix;\n-\t\t\tif ((ret = matrix.readYaml(value)) < 0) {\n+\t\t\tif ((ret = matrix.readYaml(value[value_property])) < 0) {\n \t\t\t\tLOG(MatrixInterpolator, Error) << \"Failed to read matrix\";\n \t\t\t\treturn ret;\n \t\t\t}\n \n\n\nI'm not yet sure where we will end up. This largely depends on how the\ntuning file format evolves. What are your thoughts about that?\n\n> +\t\tfor (const auto &[ctStr, value] : yaml.asDict()) {\n> +\t\t\tunsigned int ct = std::stoul(ctStr);\n> +\t\t\tMatrix matrix;\n> +\t\t\tif ((ret = matrix.readYaml(value)) < 0) {\n> +\t\t\t\tLOG(MatrixInterpolator, Error) << \"Failed to read matrix\";\n> +\t\t\t\treturn ret;\n> +\t\t\t}\n> +\n> +\t\t\tmatrices_[ct] = matrix;\n> +\n> +\t\t\tLOG(MatrixInterpolator, Debug)\n> +\t\t\t\t<< \"Read matrix for key \" << ct << \": \"\n> +\t\t\t\t<< matrices_[ct].toString();\n> +\t\t}\n> +\n> +\t\tif (matrices_.size() < 1) {\n> +\t\t\tLOG(MatrixInterpolator, Error) << \"Need at least one matrix\";\n> +\t\t\treturn -EINVAL;\n> +\t\t}\n> +\n> +\t\treturn 0;\n> +\t}\n> +\n> +\tMatrix get(unsigned int ct)\n> +\t{\n> +\t\tASSERT(matrices_.size() > 0);\n\nI'm a bit concerned here. This check only runs in debug mode. But in\ncase the readYaml gets an invalid object, it leaves the interpolator\nwith matrices_.size() == 0 and we would crash here. I think we should at\nleast catch that in release mode and maybe return a identity matrix (and\nthen there is need to inject the identity in the constructor anymore)\n\nCheers,\nStefan\n\n> +\n> +\t\tif (matrices_.size() == 1 ||\n> +\t\t ct <= matrices_.begin()->first)\n> +\t\t\treturn matrices_.begin()->second;\n> +\n> +\t\tif (ct >= matrices_.rbegin()->first)\n> +\t\t\treturn matrices_.rbegin()->second;\n> +\n> +\t\tif (matrices_.count(ct))\n> +\t\t\treturn matrices_[ct];\n> +\n> +\t\t/* The above four guarantee that this will succeed */\n> +\t\tauto iter = matrices_.upper_bound(ct);\n> +\t\tunsigned int ctUpper = iter->first;\n> +\t\tunsigned int ctLower = (--iter)->first;\n> +\n> +\t\tdouble lambda = (ct - ctLower) / static_cast(ctUpper - ctLower);\n> +\t\treturn lambda * matrices_[ctUpper] + (1.0 - lambda) * matrices_[ctLower];\n> +\t}\n> +\n> +private:\n> +\tstd::map> matrices_;\n> +};\n> +\n> +} /* namespace ipa */\n> +\n> +} /* namespace libcamera */\n> diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build\n> index 1e34355f..305acf64 100644\n> --- a/src/ipa/libipa/meson.build\n> +++ b/src/ipa/libipa/meson.build\n> @@ -8,6 +8,7 @@ libipa_headers = files([\n> 'fc_queue.h',\n> 'histogram.h',\n> 'matrix.h',\n> + 'matrix_interpolator.h',\n> 'module.h',\n> 'pwl.h',\n> ])\n> @@ -20,6 +21,7 @@ libipa_sources = files([\n> 'fc_queue.cpp',\n> 'histogram.cpp',\n> 'matrix.cpp',\n> + 'matrix_interpolator.cpp',\n> 'module.cpp',\n> 'pwl.cpp'\n> ])\n> -- \n> 2.39.2\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 52292BD78E\n\tfor ;\n\tWed, 15 May 2024 09:21:02 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 7D2226347E;\n\tWed, 15 May 2024 11:21:01 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 505D263466\n\tfor ;\n\tWed, 15 May 2024 11:20:59 +0200 (CEST)","from ideasonboard.com (unknown\n\t[IPv6:2a00:6020:448c:6c00:f83e:cbfd:31ad:3642])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 74BE6512;\n\tWed, 15 May 2024 11:20:51 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"R1ij9+AY\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1715764851;\n\tbh=47MQHAX41m8oETBGv4JxFHEDyDFyiSVLFOMSyHEd32o=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=R1ij9+AYrKee5R6R6Wmq4eVSILZDVJCU2VcOxn3s697zJp6H5r17Cgo0vwAHJEvCN\n\tZqU/5tqjE+cBql0U075ehWP25ACC9mfAU+P8mNbXz7uDaN8PHnohSoRnV28IxhgDfw\n\tRBU9z23JBro8W0ieMaAUe4KEkSWUoWkZN3PL/u1k=","Date":"Wed, 15 May 2024 11:20:56 +0200","From":"Stefan Klug ","To":"Paul Elder ","Cc":"libcamera-devel@lists.libcamera.org","Subject":"Re: [PATCH v3 2/3] ipa: libipa: Add MatrixInterpolator class","Message-ID":"<20240515092056.tty4yzie42tyhl4l@jasper>","References":"<20240507061016.1716450-1-paul.elder@ideasonboard.com>\n\t<20240507061016.1716450-3-paul.elder@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20240507061016.1716450-3-paul.elder@ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":29545,"web_url":"https://patchwork.libcamera.org/comment/29545/","msgid":"","date":"2024-05-17T03:50:50","subject":"Re: [PATCH v3 2/3] ipa: libipa: Add MatrixInterpolator class","submitter":{"id":17,"url":"https://patchwork.libcamera.org/api/people/17/","name":"Paul Elder","email":"paul.elder@ideasonboard.com"},"content":"On Wed, May 15, 2024 at 11:20:56AM +0200, Stefan Klug wrote:\n> On Tue, May 07, 2024 at 03:10:15PM +0900, Paul Elder wrote:\n> > Add a class to encapsulate the functionality of fetching a matrix based\n> > on an integer key, and interpolating if there is no exact match. This is\n> > expected to be used by both color correction matrices / crosstalk\n> > correction as well as lens shading correction.\n> > \n> > Signed-off-by: Paul Elder \n> > \n> > ---\n> > Changes in v3:\n> > - add a constructor that takes a map of unsigned int -> matrix\n> > - s/unit/identity\n> > - clear matrices on reset and when reading from yaml\n> > - add assert at get()\n> > \n> > Changes in v2:\n> > - initialize to identity matrix\n> > - add a function to reset to identity matrix\n> > - other minor fixes\n> > ---\n> > src/ipa/libipa/matrix_interpolator.cpp | 54 ++++++++++++\n> > src/ipa/libipa/matrix_interpolator.h | 116 +++++++++++++++++++++++++\n> > src/ipa/libipa/meson.build | 2 +\n> > 3 files changed, 172 insertions(+)\n> > create mode 100644 src/ipa/libipa/matrix_interpolator.cpp\n> > create mode 100644 src/ipa/libipa/matrix_interpolator.h\n> > \n> > diff --git a/src/ipa/libipa/matrix_interpolator.cpp b/src/ipa/libipa/matrix_interpolator.cpp\n> > new file mode 100644\n> > index 00000000..fc1837e7\n> > --- /dev/null\n> > +++ b/src/ipa/libipa/matrix_interpolator.cpp\n> > @@ -0,0 +1,54 @@\n> > +/* SPDX-License-Identifier: BSD-2-Clause */\n> > +/*\n> > + * Copyright (C) 2019, Raspberry Pi Ltd\n> > + * Copyright (C) 2024, Paul Elder \n> > + *\n> > + * matrix_interpolator.cpp - Helper class for interpolating maps of matrices\n> > + */\n> > +#include \"matrix_interpolator.h\"\n> > +\n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +\n> > +#include \"libcamera/internal/yaml_parser.h\"\n> > +\n> > +#include \"matrix.h\"\n> > +\n> > +/**\n> > + * \\file ccm.h\n> > + * \\brief Helper class for interpolating maps of matrices\n> > + */\n> > +\n> > +namespace libcamera {\n> > +\n> > +LOG_DEFINE_CATEGORY(MatrixInterpolator)\n> > +\n> > +namespace ipa {\n> > +\n> > +/**\n> > + * \\class MatrixInterpolator\n> > + * \\brief Class for storing, retrieving, and interpolating matrices\n> > + */\n> > +\n> > +/**\n> > + * \\fn int MatrixInterpolator::readYaml(const libcamera::YamlObject &yaml)\n> > + * \\brief Initialize an MatrixInterpolator instance from yaml\n> > + * \\tparam T Type of data stored in the matrices\n> > + * \\tparam R Number of rows of the matrices\n> > + * \\tparam C Number of columns of the matrices\n> > + * \\param[in] yaml The yaml object that contains the map of unsigned integers to matrices\n> > + * \\return Zero on success, negative error code otherwise\n> > + */\n> > +\n> > +/**\n> > + * \\fn Matrix MatrixInterpolator::get(unsigned int key)\n> > + * \\brief Retrieve a matrix from the list of matrices, interpolating if necessary\n> > + * \\param[in] key The unsigned integer key of the matrix to retrieve\n> > + * \\return The matrix corresponding to the color temperature\n> > + */\n> > +\n> > +} /* namespace ipa */\n> > +\n> > +} /* namespace libcamera */\n> > diff --git a/src/ipa/libipa/matrix_interpolator.h b/src/ipa/libipa/matrix_interpolator.h\n> > new file mode 100644\n> > index 00000000..2c6263cb\n> > --- /dev/null\n> > +++ b/src/ipa/libipa/matrix_interpolator.h\n> > @@ -0,0 +1,116 @@\n> > +/* SPDX-License-Identifier: BSD-2-Clause */\n> > +/*\n> > + * Copyright (C) 2019, Raspberry Pi Ltd\n> > + * Copyright (C) 2024, Paul Elder \n> > + *\n> > + * matrix_interpolator.h - Helper class for interpolating maps of matrices\n> > + */\n> > +\n> > +#pragma once\n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +\n> > +#include \"libcamera/internal/yaml_parser.h\"\n> > +\n> > +#include \"matrix.h\"\n> > +\n> > +namespace libcamera {\n> > +\n> > +LOG_DECLARE_CATEGORY(MatrixInterpolator)\n> > +\n> > +namespace ipa {\n> > +\n> > +template > +\t std::enable_if_t> * = nullptr>\n> > +class MatrixInterpolator\n> > +{\n> > +public:\n> > +\tMatrixInterpolator()\n> > +\t{\n> > +\t\treset();\n> > +\t};\n> > +\n> > +\tMatrixInterpolator(const std::map> &matrices)\n> > +\t{\n> > +\t\tfor (const auto &pair : matrices)\n> > +\t\t\tmatrices_[pair.first] = pair.second;\n> > +\t};\n> > +\n> > +\t~MatrixInterpolator(){};\n> > +\n> > +\tvoid reset()\n> > +\t{\n> > +\t\tMatrix identity;\n> > +\t\tmatrices_.clear();\n> > +\t\tmatrices_[0] = identity;\n> > +\t}\n> > +\n> > +\tint readYaml(const libcamera::YamlObject &yaml)\n> > +\t{\n> > +\t\tint ret;\n> > +\n> > +\t\tmatrices_.clear();\n> > +\n> > +\t\tif (!yaml.isDictionary()) {\n> > +\t\t\tLOG(MatrixInterpolator, Error) << \"yaml object must be a dictionary\";\n> > +\t\t\treturn -EINVAL;\n> > +\t\t}\n> > +\n> \n> I have the feeling that this might not be flexible enough. At the moment\n> this expects a yaml representation of\n> \n> 2000: [ matrix values... ]\n> 3000: [ ... ]\n> \n> We might need to add more infos for a given color temperature. In my\n> prototypes I used a list like this:\n> \n> - ct: 2000\n> ccm: [matrix values]\n> - ct: 3000\n> ccm: [ matrix values ]\n\nYes, I think that will be necessary, also to match the layout with lsc\nso that we can use this for lsc in the future.\n\n> \n> \n> This adds the flexibility to e.g. add the offset vector.\n> My diff is:\n> -\tint readYaml(const libcamera::YamlObject &yaml)\n> +\tint readYaml(const libcamera::YamlObject &yaml, const std::string& key_property, const std::string& value_property)\n\nAh, good idea. I was trying to figure out how to support different size\nmatrices in the same map...\n\n> \t{\n> \t\tint ret;\n> \t\t\n> \t\tmatrices_.clear(); \n> \n> -\t\tif (!yaml.isDictionary()) {\n> -\t\t\tLOG(MatrixInterpolator, Error) << \"yaml object must be a dictionary\";\n> +\t\tif (!yaml.isList()) {\n> +\t\t\tLOG(MatrixInterpolator, Error) << \"yaml object must be a list\";\n> \t\t\treturn -EINVAL;\n> \t\t}\n> \n> -\t\tfor (const auto &[ctStr, value] : yaml.asDict()) {\n> -\t\t\tunsigned int ct = std::stoul(ctStr);\n> +\t\tfor (const auto &value : yaml.asList()) {\n> +\t\t\tunsigned int ct = std::stoul(value[key_property].get(\"\"));\n> \t\t\tMatrix matrix;\n> -\t\t\tif ((ret = matrix.readYaml(value)) < 0) {\n> +\t\t\tif ((ret = matrix.readYaml(value[value_property])) < 0) {\n> \t\t\t\tLOG(MatrixInterpolator, Error) << \"Failed to read matrix\";\n> \t\t\t\treturn ret;\n> \t\t\t}\n> \n> \n> \n> I'm not yet sure where we will end up. This largely depends on how the\n> tuning file format evolves. What are your thoughts about that?\n> \n> > +\t\tfor (const auto &[ctStr, value] : yaml.asDict()) {\n> > +\t\t\tunsigned int ct = std::stoul(ctStr);\n> > +\t\t\tMatrix matrix;\n> > +\t\t\tif ((ret = matrix.readYaml(value)) < 0) {\n> > +\t\t\t\tLOG(MatrixInterpolator, Error) << \"Failed to read matrix\";\n> > +\t\t\t\treturn ret;\n> > +\t\t\t}\n> > +\n> > +\t\t\tmatrices_[ct] = matrix;\n> > +\n> > +\t\t\tLOG(MatrixInterpolator, Debug)\n> > +\t\t\t\t<< \"Read matrix for key \" << ct << \": \"\n> > +\t\t\t\t<< matrices_[ct].toString();\n> > +\t\t}\n> > +\n> > +\t\tif (matrices_.size() < 1) {\n> > +\t\t\tLOG(MatrixInterpolator, Error) << \"Need at least one matrix\";\n> > +\t\t\treturn -EINVAL;\n> > +\t\t}\n> > +\n> > +\t\treturn 0;\n> > +\t}\n> > +\n> > +\tMatrix get(unsigned int ct)\n> > +\t{\n> > +\t\tASSERT(matrices_.size() > 0);\n> \n> I'm a bit concerned here. This check only runs in debug mode. But in\n> case the readYaml gets an invalid object, it leaves the interpolator\n> with matrices_.size() == 0 and we would crash here. I think we should at\n> least catch that in release mode and maybe return a identity matrix (and\n> then there is need to inject the identity in the constructor anymore)\n\ntbh I think it should be fine because it'll be caught in readYaml and we\nreturn -EINVAL. Is that not sufficient?\n\n\nThanks,\n\nPaul\n\n> \n> > +\n> > +\t\tif (matrices_.size() == 1 ||\n> > +\t\t ct <= matrices_.begin()->first)\n> > +\t\t\treturn matrices_.begin()->second;\n> > +\n> > +\t\tif (ct >= matrices_.rbegin()->first)\n> > +\t\t\treturn matrices_.rbegin()->second;\n> > +\n> > +\t\tif (matrices_.count(ct))\n> > +\t\t\treturn matrices_[ct];\n> > +\n> > +\t\t/* The above four guarantee that this will succeed */\n> > +\t\tauto iter = matrices_.upper_bound(ct);\n> > +\t\tunsigned int ctUpper = iter->first;\n> > +\t\tunsigned int ctLower = (--iter)->first;\n> > +\n> > +\t\tdouble lambda = (ct - ctLower) / static_cast(ctUpper - ctLower);\n> > +\t\treturn lambda * matrices_[ctUpper] + (1.0 - lambda) * matrices_[ctLower];\n> > +\t}\n> > +\n> > +private:\n> > +\tstd::map> matrices_;\n> > +};\n> > +\n> > +} /* namespace ipa */\n> > +\n> > +} /* namespace libcamera */\n> > diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build\n> > index 1e34355f..305acf64 100644\n> > --- a/src/ipa/libipa/meson.build\n> > +++ b/src/ipa/libipa/meson.build\n> > @@ -8,6 +8,7 @@ libipa_headers = files([\n> > 'fc_queue.h',\n> > 'histogram.h',\n> > 'matrix.h',\n> > + 'matrix_interpolator.h',\n> > 'module.h',\n> > 'pwl.h',\n> > ])\n> > @@ -20,6 +21,7 @@ libipa_sources = files([\n> > 'fc_queue.cpp',\n> > 'histogram.cpp',\n> > 'matrix.cpp',\n> > + 'matrix_interpolator.cpp',\n> > 'module.cpp',\n> > 'pwl.cpp'\n> > ])\n> > -- \n> > 2.39.2\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 98D89BDE6B\n\tfor ;\n\tFri, 17 May 2024 03:51:02 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 884E76347E;\n\tFri, 17 May 2024 05:51:01 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 1DF7661A51\n\tfor ;\n\tFri, 17 May 2024 05:50:59 +0200 (CEST)","from pyrite.rasen.tech (h175-177-049-156.catv02.itscom.jp\n\t[175.177.49.156])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id A09AAD7E;\n\tFri, 17 May 2024 05:50:48 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"a/zowxca\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1715917849;\n\tbh=pV8Qy6GADjbqbx9dDzuXM5lIchuh5RhWo24ZRAzfEc4=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=a/zowxcaxD1tTyojAONovcpphE9shJUzBTr8W9UIZBAOmIKYXzJhIZm5TOnYrEJfh\n\tMbVPTHNOBIZlagGRHfrPjUGjNBcTOSDz39G6Amx0NRy6Y83GaD8wm6dhGKKdduWd2d\n\tT5v6V02bmOyYJXvx2ZC5K2DG07sds+rDNOpYufiw=","Date":"Fri, 17 May 2024 12:50:50 +0900","From":"Paul Elder ","To":"Stefan Klug ","Cc":"libcamera-devel@lists.libcamera.org","Subject":"Re: [PATCH v3 2/3] ipa: libipa: Add MatrixInterpolator class","Message-ID":"","References":"<20240507061016.1716450-1-paul.elder@ideasonboard.com>\n\t<20240507061016.1716450-3-paul.elder@ideasonboard.com>\n\t<20240515092056.tty4yzie42tyhl4l@jasper>","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"<20240515092056.tty4yzie42tyhl4l@jasper>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":29547,"web_url":"https://patchwork.libcamera.org/comment/29547/","msgid":"<20240517075658.cs3x2j4wv7wxrpyg@jasper>","date":"2024-05-17T07:56:58","subject":"Re: [PATCH v3 2/3] ipa: libipa: Add MatrixInterpolator class","submitter":{"id":184,"url":"https://patchwork.libcamera.org/api/people/184/","name":"Stefan Klug","email":"stefan.klug@ideasonboard.com"},"content":"Hi Paul,\n\nOn Fri, May 17, 2024 at 12:50:50PM +0900, Paul Elder wrote:\n> On Wed, May 15, 2024 at 11:20:56AM +0200, Stefan Klug wrote:\n> > On Tue, May 07, 2024 at 03:10:15PM +0900, Paul Elder wrote:\n> > > Add a class to encapsulate the functionality of fetching a matrix based\n> > > on an integer key, and interpolating if there is no exact match. This is\n> > > expected to be used by both color correction matrices / crosstalk\n> > > correction as well as lens shading correction.\n> > > \n> > > Signed-off-by: Paul Elder \n> > > \n> > > ---\n> > > Changes in v3:\n> > > - add a constructor that takes a map of unsigned int -> matrix\n> > > - s/unit/identity\n> > > - clear matrices on reset and when reading from yaml\n> > > - add assert at get()\n> > > \n> > > Changes in v2:\n> > > - initialize to identity matrix\n> > > - add a function to reset to identity matrix\n> > > - other minor fixes\n> > > ---\n> > > src/ipa/libipa/matrix_interpolator.cpp | 54 ++++++++++++\n> > > src/ipa/libipa/matrix_interpolator.h | 116 +++++++++++++++++++++++++\n> > > src/ipa/libipa/meson.build | 2 +\n> > > 3 files changed, 172 insertions(+)\n> > > create mode 100644 src/ipa/libipa/matrix_interpolator.cpp\n> > > create mode 100644 src/ipa/libipa/matrix_interpolator.h\n> > > \n> > > diff --git a/src/ipa/libipa/matrix_interpolator.cpp b/src/ipa/libipa/matrix_interpolator.cpp\n> > > new file mode 100644\n> > > index 00000000..fc1837e7\n> > > --- /dev/null\n> > > +++ b/src/ipa/libipa/matrix_interpolator.cpp\n> > > @@ -0,0 +1,54 @@\n> > > +/* SPDX-License-Identifier: BSD-2-Clause */\n> > > +/*\n> > > + * Copyright (C) 2019, Raspberry Pi Ltd\n> > > + * Copyright (C) 2024, Paul Elder \n> > > + *\n> > > + * matrix_interpolator.cpp - Helper class for interpolating maps of matrices\n> > > + */\n> > > +#include \"matrix_interpolator.h\"\n> > > +\n> > > +#include \n> > > +#include \n> > > +\n> > > +#include \n> > > +\n> > > +#include \"libcamera/internal/yaml_parser.h\"\n> > > +\n> > > +#include \"matrix.h\"\n> > > +\n> > > +/**\n> > > + * \\file ccm.h\n> > > + * \\brief Helper class for interpolating maps of matrices\n> > > + */\n> > > +\n> > > +namespace libcamera {\n> > > +\n> > > +LOG_DEFINE_CATEGORY(MatrixInterpolator)\n> > > +\n> > > +namespace ipa {\n> > > +\n> > > +/**\n> > > + * \\class MatrixInterpolator\n> > > + * \\brief Class for storing, retrieving, and interpolating matrices\n> > > + */\n> > > +\n> > > +/**\n> > > + * \\fn int MatrixInterpolator::readYaml(const libcamera::YamlObject &yaml)\n> > > + * \\brief Initialize an MatrixInterpolator instance from yaml\n> > > + * \\tparam T Type of data stored in the matrices\n> > > + * \\tparam R Number of rows of the matrices\n> > > + * \\tparam C Number of columns of the matrices\n> > > + * \\param[in] yaml The yaml object that contains the map of unsigned integers to matrices\n> > > + * \\return Zero on success, negative error code otherwise\n> > > + */\n> > > +\n> > > +/**\n> > > + * \\fn Matrix MatrixInterpolator::get(unsigned int key)\n> > > + * \\brief Retrieve a matrix from the list of matrices, interpolating if necessary\n> > > + * \\param[in] key The unsigned integer key of the matrix to retrieve\n> > > + * \\return The matrix corresponding to the color temperature\n> > > + */\n> > > +\n> > > +} /* namespace ipa */\n> > > +\n> > > +} /* namespace libcamera */\n> > > diff --git a/src/ipa/libipa/matrix_interpolator.h b/src/ipa/libipa/matrix_interpolator.h\n> > > new file mode 100644\n> > > index 00000000..2c6263cb\n> > > --- /dev/null\n> > > +++ b/src/ipa/libipa/matrix_interpolator.h\n> > > @@ -0,0 +1,116 @@\n> > > +/* SPDX-License-Identifier: BSD-2-Clause */\n> > > +/*\n> > > + * Copyright (C) 2019, Raspberry Pi Ltd\n> > > + * Copyright (C) 2024, Paul Elder \n> > > + *\n> > > + * matrix_interpolator.h - Helper class for interpolating maps of matrices\n> > > + */\n> > > +\n> > > +#pragma once\n> > > +\n> > > +#include \n> > > +#include \n> > > +#include \n> > > +#include \n> > > +\n> > > +#include \n> > > +\n> > > +#include \"libcamera/internal/yaml_parser.h\"\n> > > +\n> > > +#include \"matrix.h\"\n> > > +\n> > > +namespace libcamera {\n> > > +\n> > > +LOG_DECLARE_CATEGORY(MatrixInterpolator)\n> > > +\n> > > +namespace ipa {\n> > > +\n> > > +template > > +\t std::enable_if_t> * = nullptr>\n> > > +class MatrixInterpolator\n> > > +{\n> > > +public:\n> > > +\tMatrixInterpolator()\n> > > +\t{\n> > > +\t\treset();\n> > > +\t};\n> > > +\n> > > +\tMatrixInterpolator(const std::map> &matrices)\n> > > +\t{\n> > > +\t\tfor (const auto &pair : matrices)\n> > > +\t\t\tmatrices_[pair.first] = pair.second;\n> > > +\t};\n> > > +\n> > > +\t~MatrixInterpolator(){};\n> > > +\n> > > +\tvoid reset()\n> > > +\t{\n> > > +\t\tMatrix identity;\n> > > +\t\tmatrices_.clear();\n> > > +\t\tmatrices_[0] = identity;\n> > > +\t}\n> > > +\n> > > +\tint readYaml(const libcamera::YamlObject &yaml)\n> > > +\t{\n> > > +\t\tint ret;\n> > > +\n> > > +\t\tmatrices_.clear();\n> > > +\n> > > +\t\tif (!yaml.isDictionary()) {\n> > > +\t\t\tLOG(MatrixInterpolator, Error) << \"yaml object must be a dictionary\";\n> > > +\t\t\treturn -EINVAL;\n> > > +\t\t}\n> > > +\n> > \n> > I have the feeling that this might not be flexible enough. At the moment\n> > this expects a yaml representation of\n> > \n> > 2000: [ matrix values... ]\n> > 3000: [ ... ]\n> > \n> > We might need to add more infos for a given color temperature. In my\n> > prototypes I used a list like this:\n> > \n> > - ct: 2000\n> > ccm: [matrix values]\n> > - ct: 3000\n> > ccm: [ matrix values ]\n> \n> Yes, I think that will be necessary, also to match the layout with lsc\n> so that we can use this for lsc in the future.\n> \n> > \n> > \n> > This adds the flexibility to e.g. add the offset vector.\n> > My diff is:\n> > -\tint readYaml(const libcamera::YamlObject &yaml)\n> > +\tint readYaml(const libcamera::YamlObject &yaml, const std::string& key_property, const std::string& value_property)\n> \n> Ah, good idea. I was trying to figure out how to support different size\n> matrices in the same map...\n> \n> > \t{\n> > \t\tint ret;\n> > \t\t\n> > \t\tmatrices_.clear(); \n> > \n> > -\t\tif (!yaml.isDictionary()) {\n> > -\t\t\tLOG(MatrixInterpolator, Error) << \"yaml object must be a dictionary\";\n> > +\t\tif (!yaml.isList()) {\n> > +\t\t\tLOG(MatrixInterpolator, Error) << \"yaml object must be a list\";\n> > \t\t\treturn -EINVAL;\n> > \t\t}\n> > \n> > -\t\tfor (const auto &[ctStr, value] : yaml.asDict()) {\n> > -\t\t\tunsigned int ct = std::stoul(ctStr);\n> > +\t\tfor (const auto &value : yaml.asList()) {\n> > +\t\t\tunsigned int ct = std::stoul(value[key_property].get(\"\"));\n> > \t\t\tMatrix matrix;\n> > -\t\t\tif ((ret = matrix.readYaml(value)) < 0) {\n> > +\t\t\tif ((ret = matrix.readYaml(value[value_property])) < 0) {\n> > \t\t\t\tLOG(MatrixInterpolator, Error) << \"Failed to read matrix\";\n> > \t\t\t\treturn ret;\n> > \t\t\t}\n> > \n> > \n> > \n> > I'm not yet sure where we will end up. This largely depends on how the\n> > tuning file format evolves. What are your thoughts about that?\n> > \n> > > +\t\tfor (const auto &[ctStr, value] : yaml.asDict()) {\n> > > +\t\t\tunsigned int ct = std::stoul(ctStr);\n> > > +\t\t\tMatrix matrix;\n> > > +\t\t\tif ((ret = matrix.readYaml(value)) < 0) {\n> > > +\t\t\t\tLOG(MatrixInterpolator, Error) << \"Failed to read matrix\";\n> > > +\t\t\t\treturn ret;\n> > > +\t\t\t}\n> > > +\n> > > +\t\t\tmatrices_[ct] = matrix;\n> > > +\n> > > +\t\t\tLOG(MatrixInterpolator, Debug)\n> > > +\t\t\t\t<< \"Read matrix for key \" << ct << \": \"\n> > > +\t\t\t\t<< matrices_[ct].toString();\n> > > +\t\t}\n> > > +\n> > > +\t\tif (matrices_.size() < 1) {\n> > > +\t\t\tLOG(MatrixInterpolator, Error) << \"Need at least one matrix\";\n> > > +\t\t\treturn -EINVAL;\n> > > +\t\t}\n> > > +\n> > > +\t\treturn 0;\n> > > +\t}\n> > > +\n> > > +\tMatrix get(unsigned int ct)\n> > > +\t{\n> > > +\t\tASSERT(matrices_.size() > 0);\n> > \n> > I'm a bit concerned here. This check only runs in debug mode. But in\n> > case the readYaml gets an invalid object, it leaves the interpolator\n> > with matrices_.size() == 0 and we would crash here. I think we should at\n> > least catch that in release mode and maybe return a identity matrix (and\n> > then there is need to inject the identity in the constructor anymore)\n> \n> tbh I think it should be fine because it'll be caught in readYaml and we\n> return -EINVAL. Is that not sufficient?\n\nYes, that's right. The error case should be handled in the layers above.\nAnd after the -EINVAL this function *should* not be called. So I think\nit's just personal preference (personally I would throw here, which is\nequally drastic as getting a possible segfault but more deterministic :-). \nSo it's ok for me.\n\nCheers,\nStefan\n\n> \n> \n> Thanks,\n> \n> Paul\n> \n> > \n> > > +\n> > > +\t\tif (matrices_.size() == 1 ||\n> > > +\t\t ct <= matrices_.begin()->first)\n> > > +\t\t\treturn matrices_.begin()->second;\n> > > +\n> > > +\t\tif (ct >= matrices_.rbegin()->first)\n> > > +\t\t\treturn matrices_.rbegin()->second;\n> > > +\n> > > +\t\tif (matrices_.count(ct))\n> > > +\t\t\treturn matrices_[ct];\n> > > +\n> > > +\t\t/* The above four guarantee that this will succeed */\n> > > +\t\tauto iter = matrices_.upper_bound(ct);\n> > > +\t\tunsigned int ctUpper = iter->first;\n> > > +\t\tunsigned int ctLower = (--iter)->first;\n> > > +\n> > > +\t\tdouble lambda = (ct - ctLower) / static_cast(ctUpper - ctLower);\n> > > +\t\treturn lambda * matrices_[ctUpper] + (1.0 - lambda) * matrices_[ctLower];\n> > > +\t}\n> > > +\n> > > +private:\n> > > +\tstd::map> matrices_;\n> > > +};\n> > > +\n> > > +} /* namespace ipa */\n> > > +\n> > > +} /* namespace libcamera */\n> > > diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build\n> > > index 1e34355f..305acf64 100644\n> > > --- a/src/ipa/libipa/meson.build\n> > > +++ b/src/ipa/libipa/meson.build\n> > > @@ -8,6 +8,7 @@ libipa_headers = files([\n> > > 'fc_queue.h',\n> > > 'histogram.h',\n> > > 'matrix.h',\n> > > + 'matrix_interpolator.h',\n> > > 'module.h',\n> > > 'pwl.h',\n> > > ])\n> > > @@ -20,6 +21,7 @@ libipa_sources = files([\n> > > 'fc_queue.cpp',\n> > > 'histogram.cpp',\n> > > 'matrix.cpp',\n> > > + 'matrix_interpolator.cpp',\n> > > 'module.cpp',\n> > > 'pwl.cpp'\n> > > ])\n> > > -- \n> > > 2.39.2\n> > >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 2E6E0BD78E\n\tfor ;\n\tFri, 17 May 2024 07:57:04 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 3D8CD6347E;\n\tFri, 17 May 2024 09:57:03 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id A4F8561A5A\n\tfor ;\n\tFri, 17 May 2024 09:57:01 +0200 (CEST)","from ideasonboard.com (unknown\n\t[IPv6:2a00:6020:448c:6c00:d572:8aa2:3e8e:5b99])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 6340C475;\n\tFri, 17 May 2024 09:56:52 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"AHHlBJ0P\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1715932612;\n\tbh=dORiHOvZKXwER7zBcIgJnlWCQJ9H1nDdDX/RJiPgORc=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=AHHlBJ0PbNBUFgDJnaB8FoUH5OIJwY03My2exlVU3SXRYPOm9dTfmjlAIkmeEzQIK\n\tipV2y8EeQo7ozmcoBHTyMsKzpGiWPyD+srjNLjwQ0qbJwA1hVebORglSo2R09RoLbu\n\tUWLCfHAE0tRH7jX+pfssK5Aupw0eBeVdtMlawmQ0=","Date":"Fri, 17 May 2024 09:56:58 +0200","From":"Stefan Klug ","To":"Paul Elder ","Cc":"libcamera-devel@lists.libcamera.org","Subject":"Re: [PATCH v3 2/3] ipa: libipa: Add MatrixInterpolator class","Message-ID":"<20240517075658.cs3x2j4wv7wxrpyg@jasper>","References":"<20240507061016.1716450-1-paul.elder@ideasonboard.com>\n\t<20240507061016.1716450-3-paul.elder@ideasonboard.com>\n\t<20240515092056.tty4yzie42tyhl4l@jasper>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]