From patchwork Sun Jun 16 17:45:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 20342 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4A9DEC3293 for ; Sun, 16 Jun 2024 17:46:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 7CF5E65493; Sun, 16 Jun 2024 19:46:08 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="v5ZZFtMe"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 12CBE65456 for ; Sun, 16 Jun 2024 19:46:05 +0200 (CEST) Received: from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi [81.175.209.231]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id E7BB02D5; Sun, 16 Jun 2024 19:45:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1718559948; bh=P0nSlFwrhwM8Ai3dKZjNHH17pbH7p/amKaROnHiC2JE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=v5ZZFtMeqDgLh0dwsz1h73LkRXr9ynSjuS3+D0iQsKzKrlU92kL6BKBhx8dT3meN/ zx7MKf/5Kn88Ba8Tt3yzMFg38iMbpMgQBctoYk4ml5NzGCQJXKjAYoqvuPiBA9lM1x UaJVerX//xls4ilod57PUK8eKTf0N6lSFVbvzoS8= From: Laurent Pinchart To: libcamera-devel@lists.libcamera.org Cc: Paul Elder Subject: [PATCH v10 1/4] ipa: libipa: Add Matrix class Date: Sun, 16 Jun 2024 20:45:38 +0300 Message-ID: <20240616174541.10708-2-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240616174541.10708-1-laurent.pinchart@ideasonboard.com> References: <20240616174541.10708-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Paul Elder Add a class to represent a Matrix object and operations for adding matrices, multipling a matrix by a scalar, and multiplying two matrices. Signed-off-by: Paul Elder Reviewed-by: Stefan Klug Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart Signed-off-by: Laurent Pinchart --- Changes since v9: - Add missing curly braces - Drop unneeded __DOXYGEN__ guard - Replace R and C with Rows and Cols where missing - Add missing () in \fn - Minor docs improvements No change in v9 Changes in v8: - remove readYaml and replace with yaml get - which makes this patch depend on "ipa: libipa: Vector and Pwl improvements" - make default constructor make a zero matrix, and add an identity() static function - add operator*= - add operator* for reverse order of scalar-matrix product - s/\/Rows/g, s/\/Cols/g Changes in v7: - fix copyright and license Changes in v6: - fix doxygen Changes in v5: - add documentation Changes in v4: - remove stray semicolons - add operator<< - clean up/optimize constructor - replace get() and set() with operator[] (and a second [] can be used as operator[] returns a Span) Changes in v3: - fix template parameters of operator* to allow different types for the scalar multiplier and the matrix's number type - clear data in constructors - fix assert in constructor Changes v2: - make rows and columns into template arguments - initialize to identity matrix on construction - add getter and setter - change from struct to class - fix matrix multiplication - clean up unused includes - avoid dereferencing an absent std::optional --- src/ipa/libipa/matrix.cpp | 149 +++++++++++++++++++++++++++ src/ipa/libipa/matrix.h | 204 +++++++++++++++++++++++++++++++++++++ src/ipa/libipa/meson.build | 2 + 3 files changed, 355 insertions(+) create mode 100644 src/ipa/libipa/matrix.cpp create mode 100644 src/ipa/libipa/matrix.h diff --git a/src/ipa/libipa/matrix.cpp b/src/ipa/libipa/matrix.cpp new file mode 100644 index 000000000000..7f000382d33b --- /dev/null +++ b/src/ipa/libipa/matrix.cpp @@ -0,0 +1,149 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Paul Elder + * + * Matrix and related operations + */ + +#include "matrix.h" + +#include + +/** + * \file matrix.h + * \brief Matrix class + */ + +namespace libcamera { + +LOG_DEFINE_CATEGORY(Matrix) + +namespace ipa { + +/** + * \class Matrix + * \brief Matrix class + * \tparam T Type of numerical values to be stored in the matrix + * \tparam Rows Number of rows in the matrix + * \tparam Cols Number of columns in the matrix + */ + +/** + * \fn Matrix::Matrix() + * \brief Construct a zero matrix + */ + +/** + * \fn Matrix::Matrix(const std::vector &data) + * \brief Construct a matrix from supplied data + * \param[in] data Data from which to construct a matrix + * + * \a data is a one-dimensional vector and will be turned into a matrix in + * row-major order. The size of \a data must be equal to the product of the + * number of rows and columns of the matrix (Rows x Cols). + */ + +/** + * \fn Matrix::identity() + * \brief Construct an identity matrix + */ + +/** + * \fn Matrix::toString() + * \brief Assemble and return a string describing the matrix + * \return A string describing the matrix + */ + +/** + * \fn Span Matrix::operator[](size_t i) const + * \brief Index to a row in the matrix + * \param[in] i Index of row to retrieve + * + * This operator[] returns a Span, which can then be indexed into again with + * another operator[], allowing a convenient m[i][j] to access elements of the + * matrix. Note that the lifetime of the Span returned by this first-level + * operator[] is bound to that of the Matrix itself, so it is not recommended + * to save the Span that is the result of this operator[]. + * + * \return Row \a i from the matrix, as a Span + */ + +/** + * \fn Matrix::operator[](size_t i) + * \copydoc Matrix::operator[](size_t i) const + */ + +/** + * \fn Matrix &Matrix::operator*=(U d) + * \brief Multiply the matrix by a scalar in-place + * \tparam U Type of the numerical scalar value + * \param d The scalar multiplier + * \return Product of this matrix and scalar \a d + */ + +/** + * \fn Matrix::Matrix operator*(T d, const Matrix &m) + * \brief Multiply the matrix by a scalar + * \tparam T Type of the numerical scalar value + * \tparam U Type of numerical values in the matrix + * \tparam Rows Number of rows in the matrix + * \tparam Cols Number of columns in the matrix + * \param d The scalar multiplier + * \param m The matrix + * \return Product of scalar \a d and matrix \a m + */ + +/** + * \fn Matrix::Matrix operator*(const Matrix &m, T d) + * \copydoc operator*(T d, const Matrix &m) + */ + +/** + * \fn Matrix operator*(const Matrix &m1, const Matrix &m2) + * \brief Matrix multiplication + * \tparam T Type of numerical values in the matrices + * \tparam R1 Number of rows in the first matrix + * \tparam C1 Number of columns in the first matrix + * \tparam R2 Number of rows in the second matrix + * \tparam C2 Number of columns in the second matrix + * \param m1 Multiplicand matrix + * \param m2 Multiplier matrix + * \return Matrix product of matrices \a m1 and \a m2 + */ + +/** + * \fn Matrix operator+(const Matrix &m1, const Matrix &m2) + * \brief Matrix addition + * \tparam T Type of numerical values in the matrices + * \tparam Rows Number of rows in the matrices + * \tparam Cols Number of columns in the matrices + * \param m1 Summand matrix + * \param m2 Summand matrix + * \return Matrix sum of matrices \a m1 and \a m2 + */ + +} /* namespace ipa */ + +#ifndef __DOXYGEN__ +/* + * The YAML data shall be a list of numerical values. Its size shall be equal + * to the product of the number of rows and columns of the matrix (Rows x + * Cols). The values shall be stored in row-major order. + */ +bool matrixValidateYaml(const YamlObject &obj, unsigned int size) +{ + if (!obj.isList()) + return false; + + if (obj.size() != size) { + LOG(Matrix, Error) + << "Wrong number of values in matrix: expected " + << size << ", got " << obj.size(); + return false; + } + + return true; +} +#endif /* __DOXYGEN__ */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/matrix.h b/src/ipa/libipa/matrix.h new file mode 100644 index 000000000000..8aa8f3438caa --- /dev/null +++ b/src/ipa/libipa/matrix.h @@ -0,0 +1,204 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Paul Elder + * + * Matrix and related operations + */ +#pragma once + +#include +#include +#include +#include + +#include +#include + +#include "libcamera/internal/yaml_parser.h" + +namespace libcamera { + +LOG_DECLARE_CATEGORY(Matrix) + +namespace ipa { + +#ifndef __DOXYGEN__ +template> * = nullptr> +#else +template +#endif /* __DOXYGEN__ */ +class Matrix +{ +public: + Matrix() + { + data_.fill(static_cast(0)); + } + + Matrix(const std::vector &data) + { + std::copy(data.begin(), data.end(), data_.begin()); + } + + static Matrix identity() + { + Matrix ret; + for (size_t i = 0; i < std::min(Rows, Cols); i++) + ret[i][i] = static_cast(1); + return ret; + } + + ~Matrix() = default; + + const std::string toString() const + { + std::stringstream out; + + out << "Matrix { "; + for (unsigned int i = 0; i < Rows; i++) { + out << "[ "; + for (unsigned int j = 0; j < Cols; j++) { + out << (*this)[i][j]; + out << ((j + 1 < Cols) ? ", " : " "); + } + out << ((i + 1 < Rows) ? "], " : "]"); + } + out << " }"; + + return out.str(); + } + + Span operator[](size_t i) const + { + return Span{ &data_.data()[i * Cols], Cols }; + } + + Span operator[](size_t i) + { + return Span{ &data_.data()[i * Cols], Cols }; + } + +#ifndef __DOXYGEN__ + template>> +#else + template +#endif /* __DOXYGEN__ */ + Matrix &operator*=(U d) + { + for (unsigned int i = 0; i < Rows * Cols; i++) + data_[i] *= d; + return *this; + } + +private: + std::array data_; +}; + +#ifndef __DOXYGEN__ +template> * = nullptr> +#else +template +#endif /* __DOXYGEN__ */ +Matrix operator*(T d, const Matrix &m) +{ + Matrix result; + + for (unsigned int i = 0; i < Rows; i++) { + for (unsigned int j = 0; j < Cols; j++) + result[i][j] = d * m[i][j]; + } + + return result; +} + +#ifndef __DOXYGEN__ +template> * = nullptr> +#else +template +#endif /* __DOXYGEN__ */ +Matrix operator*(const Matrix &m, T d) +{ + return d * m; +} + +#ifndef __DOXYGEN__ +template * = nullptr> +#else +template +#endif /* __DOXYGEN__ */ +Matrix operator*(const Matrix &m1, const Matrix &m2) +{ + Matrix result; + + for (unsigned int i = 0; i < R1; i++) { + for (unsigned int j = 0; j < C2; j++) { + T sum = 0; + + for (unsigned int k = 0; k < C1; k++) + sum += m1[i][k] * m2[k][j]; + + result[i][j] = sum; + } + } + + return result; +} + +template +Matrix operator+(const Matrix &m1, const Matrix &m2) +{ + Matrix result; + + for (unsigned int i = 0; i < Rows; i++) { + for (unsigned int j = 0; j < Cols; j++) + result[i][j] = m1[i][j] + m2[i][j]; + } + + return result; +} + +#ifndef __DOXYGEN__ +bool matrixValidateYaml(const YamlObject &obj, unsigned int size); +#endif /* __DOXYGEN__ */ + +} /* namespace ipa */ + +#ifndef __DOXYGEN__ +template +std::ostream &operator<<(std::ostream &out, const ipa::Matrix &m) +{ + out << m.toString(); + return out; +} + +template +struct YamlObject::Getter> { + std::optional> get(const YamlObject &obj) const + { + if (!ipa::matrixValidateYaml(obj, Rows * Cols)) + return std::nullopt; + + ipa::Matrix matrix; + T *data = &matrix[0][0]; + + unsigned int i = 0; + for (const YamlObject &entry : obj.asList()) { + const auto value = entry.get(); + if (!value) + return std::nullopt; + + data[i++] = *value; + } + + return matrix; + } +}; +#endif /* __DOXYGEN__ */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index 496084233de7..1952dccf6236 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -7,6 +7,7 @@ libipa_headers = files([ 'exposure_mode_helper.h', 'fc_queue.h', 'histogram.h', + 'matrix.h', 'module.h', 'pwl.h', 'vector.h', @@ -19,6 +20,7 @@ libipa_sources = files([ 'exposure_mode_helper.cpp', 'fc_queue.cpp', 'histogram.cpp', + 'matrix.cpp', 'module.cpp', 'pwl.cpp', 'vector.cpp', From patchwork Sun Jun 16 17:45:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 20343 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 66FF3C3237 for ; Sun, 16 Jun 2024 17:46:11 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 380E56548D; Sun, 16 Jun 2024 19:46:10 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="dBnod4+s"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7E6D06548A for ; Sun, 16 Jun 2024 19:46:06 +0200 (CEST) Received: from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi [81.175.209.231]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 30BB22D5; Sun, 16 Jun 2024 19:45:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1718559950; bh=2IsjnC/zzXE8iO7osj0HgCO2mBWqado2ooJPy/BtrAg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dBnod4+st39w+yhb8AdFpiaL7uuGTvOAhZ7OAya96d65xd/Hgb8F7aHZKFvPNLg8M qHEoxgOAETu/v/B7xBIKOJWhYK0hvWw98+TBNjl+JU2EZIB8ztrAeaF2EK+YC9rwAL CNjnbGHY0Zm/v2hpacU51/dUsh7JZAh38iPjzkt4= From: Laurent Pinchart To: libcamera-devel@lists.libcamera.org Cc: Paul Elder Subject: [PATCH v10 2/4] ipa: libipa: Add MatrixInterpolator class Date: Sun, 16 Jun 2024 20:45:39 +0300 Message-ID: <20240616174541.10708-3-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240616174541.10708-1-laurent.pinchart@ideasonboard.com> References: <20240616174541.10708-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Paul Elder Add a class to encapsulate the functionality of fetching a matrix based on an integer key, and interpolating if there is no exact match. This is expected to be used by both color correction matrices / crosstalk correction as well as lens shading correction. A cache is included only for exact matches of the key. The caller is expected to decide the tolererance for rounding. Signed-off-by: Paul Elder Reviewed-by: Stefan Klug Reviewed-by: Kieran Bingham Signed-off-by: Laurent Pinchart --- No change in v9 Changes in v8: - use yaml get function - use Matrix::identity() (instead of the default constructor, now that it makes a zero matrix instead of an identity matrix) Changes in v7: - remove cache - fix copyright and license Changes in v6: - fix doxygen Changes in v5: - improve documentation - replace count() with find() != end() (in get()) - add cache Changes in v4: - remove stray semicolons - read from the new yaml layout (which mirrors what we have at the moment for lsc), and add keys to make it more flexible Changes in v3: - add a constructor that takes a map of unsigned int -> matrix - s/unit/identity - clear matrices on reset and when reading from yaml - add assert at get() Changes in v2: - initialize to identity matrix - add a function to reset to identity matrix - other minor fixes --- src/ipa/libipa/matrix_interpolator.cpp | 110 ++++++++++++++++++++++ src/ipa/libipa/matrix_interpolator.h | 122 +++++++++++++++++++++++++ src/ipa/libipa/meson.build | 2 + 3 files changed, 234 insertions(+) create mode 100644 src/ipa/libipa/matrix_interpolator.cpp create mode 100644 src/ipa/libipa/matrix_interpolator.h diff --git a/src/ipa/libipa/matrix_interpolator.cpp b/src/ipa/libipa/matrix_interpolator.cpp new file mode 100644 index 000000000000..04ca177f72c6 --- /dev/null +++ b/src/ipa/libipa/matrix_interpolator.cpp @@ -0,0 +1,110 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Paul Elder + * + * Helper class for interpolating maps of matrices + */ +#include "matrix_interpolator.h" + +#include +#include + +#include + +#include "libcamera/internal/yaml_parser.h" + +#include "matrix.h" + +/** + * \file matrix_interpolator.h + * \brief Helper class for interpolating maps of matrices + */ + +namespace libcamera { + +LOG_DEFINE_CATEGORY(MatrixInterpolator) + +namespace ipa { + +/** + * \class MatrixInterpolator + * \brief Class for storing, retrieving, and interpolating matrices + * \tparam T Type of numerical values to be stored in the matrices + * \tparam R Number of rows in the matrices + * \tparam C Number of columns in the matrices + * + * The main use case is to pass a map from color temperatures to corresponding + * matrices (eg. color correction), and then requesting a matrix for a specific + * color temperature. This class will abstract away the interpolation portion. + */ + +/** + * \fn MatrixInterpolator::MatrixInterpolator(const std::map> &matrices) + * \brief Construct a matrix interpolator from a map of matrices + * \param matrices Map from which to construct the matrix interpolator + */ + +/** + * \fn MatrixInterpolator::reset() + * \brief Reset the matrix interpolator content to a single identity matrix + */ + +/** + * \fn int MatrixInterpolator::readYaml() + * \brief Initialize an MatrixInterpolator instance from yaml + * \tparam T Type of data stored in the matrices + * \tparam R Number of rows of the matrices + * \tparam C Number of columns of the matrices + * \param[in] yaml The yaml object that contains the map of unsigned integers to matrices + * \param[in] key_name The name of the key in the yaml object + * \param[in] matrix_name The name of the matrix in the yaml object + * + * The yaml object is expected to be a list of maps. Each map has two or more + * pairs: one of \a key_name to the key value (usually color temperature), and + * one or more of \a matrix_name to the matrix. This is a bit difficult to + * explain, so here is an example (in python, as it is easier to parse than + * yaml): + * [ + * { + * 'ct': 2860, + * 'ccm': [ 2.12089, -0.52461, -0.59629, + * -0.85342, 2.80445, -0.95103, + * -0.26897, -1.14788, 2.41685 ], + * 'offsets': [ 0, 0, 0 ] + * }, + * + * { + * 'ct': 2960, + * 'ccm': [ 2.26962, -0.54174, -0.72789, + * -0.77008, 2.60271, -0.83262, + * -0.26036, -1.51254, 2.77289 ], + * 'offsets': [ 0, 0, 0 ] + * }, + * + * { + * 'ct': 3603, + * 'ccm': [ 2.18644, -0.66148, -0.52496, + * -0.77828, 2.69474, -0.91645, + * -0.25239, -0.83059, 2.08298 ], + * 'offsets': [ 0, 0, 0 ] + * }, + * ] + * + * In this case, \a key_name would be 'ct', and \a matrix_name can be either + * 'ccm' or 'offsets'. This way multiple matrix interpolators can be defined in + * one set of color temperature ranges in the tuning file, and they can be + * retrieved separately with the \a matrix_name parameter. + * + * \return Zero on success, negative error code otherwise + */ + +/** + * \fn Matrix MatrixInterpolator::get(unsigned int key) + * \brief Retrieve a matrix from the list of matrices, interpolating if necessary + * \param[in] key The unsigned integer key of the matrix to retrieve + * \return The matrix corresponding to the color temperature + */ + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/matrix_interpolator.h b/src/ipa/libipa/matrix_interpolator.h new file mode 100644 index 000000000000..087c4fd14e4f --- /dev/null +++ b/src/ipa/libipa/matrix_interpolator.h @@ -0,0 +1,122 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Paul Elder + * + * Helper class for interpolating maps of matrices + */ + +#pragma once + +#include +#include +#include +#include + +#include + +#include "libcamera/internal/yaml_parser.h" + +#include "matrix.h" + +namespace libcamera { + +LOG_DECLARE_CATEGORY(MatrixInterpolator) + +namespace ipa { + +#ifndef __DOXYGEN__ +template> * = nullptr> +#else +template +#endif /* __DOXYGEN__ */ +class MatrixInterpolator +{ +public: + MatrixInterpolator() + { + reset(); + } + + MatrixInterpolator(const std::map> &matrices) + { + for (const auto &pair : matrices) + matrices_[pair.first] = pair.second; + } + + ~MatrixInterpolator() {} + + void reset() + { + matrices_.clear(); + matrices_[0] = Matrix::identity(); + } + + int readYaml(const libcamera::YamlObject &yaml, + const std::string &key_name, + const std::string &matrix_name) + { + matrices_.clear(); + + if (!yaml.isList()) { + LOG(MatrixInterpolator, Error) << "yaml object must be a list"; + return -EINVAL; + } + + for (const auto &value : yaml.asList()) { + unsigned int ct = std::stoul(value[key_name].get("")); + std::optional> matrix = + value[matrix_name].get>(); + if (!matrix) { + LOG(MatrixInterpolator, Error) << "Failed to read matrix"; + return -EINVAL; + } + + matrices_[ct] = *matrix; + + LOG(MatrixInterpolator, Debug) + << "Read matrix '" << matrix_name << "' for key '" + << key_name << "' " << ct << ": " + << matrices_[ct].toString(); + } + + if (matrices_.size() < 1) { + LOG(MatrixInterpolator, Error) << "Need at least one matrix"; + return -EINVAL; + } + + return 0; + } + + Matrix get(unsigned int ct) + { + ASSERT(matrices_.size() > 0); + + if (matrices_.size() == 1 || + ct <= matrices_.begin()->first) + return matrices_.begin()->second; + + if (ct >= matrices_.rbegin()->first) + return matrices_.rbegin()->second; + + if (matrices_.find(ct) != matrices_.end()) + return matrices_[ct]; + + /* The above four guarantee that this will succeed */ + auto iter = matrices_.upper_bound(ct); + unsigned int ctUpper = iter->first; + unsigned int ctLower = (--iter)->first; + + double lambda = (ct - ctLower) / static_cast(ctUpper - ctLower); + Matrix ret = + lambda * matrices_[ctUpper] + (1.0 - lambda) * matrices_[ctLower]; + return ret; + } + +private: + std::map> matrices_; +}; + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index 1952dccf6236..eff8ce2660f1 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -8,6 +8,7 @@ libipa_headers = files([ 'fc_queue.h', 'histogram.h', 'matrix.h', + 'matrix_interpolator.h', 'module.h', 'pwl.h', 'vector.h', @@ -21,6 +22,7 @@ libipa_sources = files([ 'fc_queue.cpp', 'histogram.cpp', 'matrix.cpp', + 'matrix_interpolator.cpp', 'module.cpp', 'pwl.cpp', 'vector.cpp', From patchwork Sun Jun 16 17:45:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 20344 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id DB894C32CF for ; Sun, 16 Jun 2024 17:46:12 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 288D265497; Sun, 16 Jun 2024 19:46:12 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="J34AT7mg"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D04CC65490 for ; Sun, 16 Jun 2024 19:46:07 +0200 (CEST) Received: from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi [81.175.209.231]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 70CC82D5; Sun, 16 Jun 2024 19:45:51 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1718559951; bh=mTzbNY1TvRpZMGhbf3kYd120tG4UA0q0g0J3ACMt5O4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=J34AT7mgP22vtpYxSaFVHXAonwbErdFtv8XkBIKUvvxoFnsLWZ0lmA6DE8xjzW5Zv 7PS5SoKtiBdvKSk0+BQ9OSyGkq/1P9v/X9CGX4ecHmpuhgVfgJMfiGNldxO57LxJ0m ieWSsxibl7vryTORgHwZ/cJFOucraV6jpaqD74q4= From: Laurent Pinchart To: libcamera-devel@lists.libcamera.org Cc: Paul Elder Subject: [PATCH v10 3/4] ipa: rkisp1: algorithms: Add crosstalk algorithm Date: Sun, 16 Jun 2024 20:45:40 +0300 Message-ID: <20240616174541.10708-4-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240616174541.10708-1-laurent.pinchart@ideasonboard.com> References: <20240616174541.10708-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Paul Elder Add an algorithm module to the rkisp1 IPA for crosstalk correction. Signed-off-by: Paul Elder Reviewed-by: Stefan Klug Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart Signed-off-by: Laurent Pinchart --- Changes since v9: - Add \todo comment about colour temperature filtering Changes in v9: - compilation fixes Changes in v8: - cleanups Changes in v7: - make offsets_ default to zero-matrices as opposed to identity matrices - checkstyle - populate metadata - add ccm to IPAFrameContext - don't update the ccm if the color temperature didn't change No change in v6 Changes in v5: - clean up documentation - coalesce parseYaml into init Changes in v4: - remove stray semicolons - use the new matrix interpolator readYaml - use the new matrix operator[] getter Changes in v3: - read ccm offsets from tuning data, and write these offsets to the parameters buffer - make parseYaml return void, as it should fill in default data if unable to read, thus never failing Changes in v2: - rename ctk to ccm - reset the matrix interpolator to identity matrix if failed to read from tuning file --- src/ipa/rkisp1/algorithms/ccm.cpp | 147 ++++++++++++++++++++++++++ src/ipa/rkisp1/algorithms/ccm.h | 49 +++++++++ src/ipa/rkisp1/algorithms/meson.build | 1 + src/ipa/rkisp1/ipa_context.h | 5 + 4 files changed, 202 insertions(+) create mode 100644 src/ipa/rkisp1/algorithms/ccm.cpp create mode 100644 src/ipa/rkisp1/algorithms/ccm.h diff --git a/src/ipa/rkisp1/algorithms/ccm.cpp b/src/ipa/rkisp1/algorithms/ccm.cpp new file mode 100644 index 000000000000..c1f5403a8ab2 --- /dev/null +++ b/src/ipa/rkisp1/algorithms/ccm.cpp @@ -0,0 +1,147 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Ideas On Board + * + * RkISP1 Color Correction Matrix control algorithm + */ + +#include "ccm.h" + +#include +#include +#include +#include +#include + +#include +#include + +#include + +#include + +#include "libcamera/internal/yaml_parser.h" + +#include "../utils.h" +#include "libipa/matrix_interpolator.h" + +/** + * \file ccm.h + */ + +namespace libcamera { + +namespace ipa::rkisp1::algorithms { + +/** + * \class Ccm + * \brief A color correction matrix algorithm + */ + +LOG_DEFINE_CATEGORY(RkISP1Ccm) + +/** + * \copydoc libcamera::ipa::Algorithm::init + */ +int Ccm::init([[maybe_unused]] IPAContext &context, const YamlObject &tuningData) +{ + int ret = ccm_.readYaml(tuningData["ccms"], "ct", "ccm"); + if (ret < 0) { + LOG(RkISP1Ccm, Warning) + << "Failed to parse 'ccm' " + << "parameter from tuning file; falling back to unit matrix"; + ccm_.reset(); + } + + ret = offsets_.readYaml(tuningData["ccms"], "ct", "offsets"); + if (ret < 0) { + LOG(RkISP1Ccm, Warning) + << "Failed to parse 'offsets' " + << "parameter from tuning file; falling back to zero offsets"; + /* + * MatrixInterpolator::reset() resets to identity matrices + * while here we need zero matrices so we need to construct it + * ourselves. + */ + Matrix m({ 0, 0, 0 }); + std::map> matrices = { { 0, m } }; + offsets_ = MatrixInterpolator(matrices); + } + + return 0; +} + +void Ccm::setParameters(rkisp1_params_cfg *params, + const Matrix &matrix, + const Matrix &offsets) +{ + struct rkisp1_cif_isp_ctk_config &config = params->others.ctk_config; + + /* + * 4 bit integer and 7 bit fractional, ranging from -8 (0x400) to + * +7.992 (0x3ff) + */ + for (unsigned int i = 0; i < 3; i++) { + for (unsigned int j = 0; j < 3; j++) + config.coeff[i][j] = + utils::floatingToFixedPoint<4, 7, uint16_t, double>(matrix[i][j]); + } + + for (unsigned int i = 0; i < 3; i++) + config.ct_offset[i] = offsets[i][0] & 0xfff; + + LOG(RkISP1Ccm, Debug) << "Setting matrix " << matrix; + LOG(RkISP1Ccm, Debug) << "Setting offsets " << offsets; + + params->module_en_update |= RKISP1_CIF_ISP_MODULE_CTK; + params->module_ens |= RKISP1_CIF_ISP_MODULE_CTK; + params->module_cfg_update |= RKISP1_CIF_ISP_MODULE_CTK; +} + +/** + * \copydoc libcamera::ipa::Algorithm::prepare + */ +void Ccm::prepare(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + rkisp1_params_cfg *params) +{ + uint32_t ct = context.activeState.awb.temperatureK; + + /* + * \todo The colour temperature will likely be noisy, add filtering to + * avoid updating the CCM matrix all the time. + */ + if (frame > 0 && ct == ct_) + return; + + ct_ = ct; + Matrix ccm = ccm_.get(ct); + Matrix offsets = offsets_.get(ct); + + frameContext.ccm.ccm = ccm; + + setParameters(params, ccm, offsets); +} + +/** + * \copydoc libcamera::ipa::Algorithm::process + */ +void Ccm::process([[maybe_unused]] IPAContext &context, + [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, + [[maybe_unused]] const rkisp1_stat_buffer *stats, + ControlList &metadata) +{ + float m[9]; + for (unsigned int i = 0; i < 3; i++) { + for (unsigned int j = 0; j < 3; j++) + m[i] = frameContext.ccm.ccm[i][j]; + } + metadata.set(controls::ColourCorrectionMatrix, m); +} + +REGISTER_IPA_ALGORITHM(Ccm, "Ccm") + +} /* namespace ipa::rkisp1::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/rkisp1/algorithms/ccm.h b/src/ipa/rkisp1/algorithms/ccm.h new file mode 100644 index 000000000000..30cb882180cc --- /dev/null +++ b/src/ipa/rkisp1/algorithms/ccm.h @@ -0,0 +1,49 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Ideas On Board + * + * RkISP1 Color Correction Matrix control algorithm + */ + +#pragma once + +#include + +#include "libipa/matrix.h" +#include "libipa/matrix_interpolator.h" + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::rkisp1::algorithms { + +class Ccm : public Algorithm +{ +public: + Ccm() {} + ~Ccm() = default; + + int init(IPAContext &context, const YamlObject &tuningData) override; + void prepare(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + rkisp1_params_cfg *params) override; + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const rkisp1_stat_buffer *stats, + ControlList &metadata) override; + +private: + void parseYaml(const YamlObject &tuningData); + void setParameters(rkisp1_params_cfg *params, + const Matrix &matrix, + const Matrix &offsets); + + unsigned int ct_; + MatrixInterpolator ccm_; + MatrixInterpolator offsets_; +}; + +} /* namespace ipa::rkisp1::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/rkisp1/algorithms/meson.build b/src/ipa/rkisp1/algorithms/meson.build index 6ee71a9b5da3..1734a6675f78 100644 --- a/src/ipa/rkisp1/algorithms/meson.build +++ b/src/ipa/rkisp1/algorithms/meson.build @@ -4,6 +4,7 @@ rkisp1_ipa_algorithms = files([ 'agc.cpp', 'awb.cpp', 'blc.cpp', + 'ccm.cpp', 'cproc.cpp', 'dpcc.cpp', 'dpf.cpp', diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index 6022480d4fd2..4f01a7f8ed33 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -17,6 +17,7 @@ #include #include +#include namespace libcamera { @@ -163,6 +164,10 @@ struct IPAFrameContext : public FrameContext { uint32_t exposure; double gain; } sensor; + + struct { + Matrix ccm; + } ccm; }; struct IPAContext { From patchwork Sun Jun 16 17:45:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 20345 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 66DE9C32D0 for ; Sun, 16 Jun 2024 17:46:15 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BDC8F65493; Sun, 16 Jun 2024 19:46:14 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="LT53VQdh"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4157F65489 for ; Sun, 16 Jun 2024 19:46:09 +0200 (CEST) Received: from pendragon.ideasonboard.com (81-175-209-231.bb.dnainternet.fi [81.175.209.231]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id BF4652D5; Sun, 16 Jun 2024 19:45:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1718559952; bh=NdUij6PJHSPgcDitQ/3pZgzqjn82tJfGLWD7hV0NMHM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LT53VQdhlzQR5PdvAnH9KorP0g6u0dkaBLV+xP7cVFe3r+p0HkPgvI++VaHqu6xQO G7/6aEiofEmtT/BQ4aD/HvUrXn/+MS4T0riHIt2SMAs7lqzq0Ixs2xOTvgYmbH6qaX cg0eWw5EOhdCcAMGrZzQMA1buKsop71HjW+BDvYc= From: Laurent Pinchart To: libcamera-devel@lists.libcamera.org Cc: Paul Elder Subject: [PATCH v10 4/4] ipa: libipa: vector: Add matrix-vector multiplication Date: Sun, 16 Jun 2024 20:45:41 +0300 Message-ID: <20240616174541.10708-5-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240616174541.10708-1-laurent.pinchart@ideasonboard.com> References: <20240616174541.10708-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Paul Elder Add an operation for multiplying a matrix with a vector. Signed-off-by: Paul Elder Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart Signed-off-by: Laurent Pinchart --- Changes since v3: - Drop unneeded __DOXYGEN__ guard Changes in v3: - make SFINAE more concise Changes in v2: - s/D/Rows/ in Vector class --- src/ipa/libipa/vector.cpp | 11 +++++++++++ src/ipa/libipa/vector.h | 17 +++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/src/ipa/libipa/vector.cpp b/src/ipa/libipa/vector.cpp index b071b261b9c4..bd00b01961d5 100644 --- a/src/ipa/libipa/vector.cpp +++ b/src/ipa/libipa/vector.cpp @@ -123,6 +123,17 @@ namespace ipa { * \return The length of the vector */ +/** + * \fn Vector operator*(const Matrix &m, const Vector &v) + * \brief Multiply a matrix by a vector + * \tparam T Numerical type of the contents of the matrix and vector + * \tparam Rows The number of rows in the matrix + * \tparam Cols The number of columns in the matrix (= rows in the vector) + * \param m The matrix + * \param v The vector + * \return Product of matrix \a m and vector \a v + */ + /** * \fn bool operator==(const Vector &lhs, const Vector &rhs) * \brief Compare vectors for equality diff --git a/src/ipa/libipa/vector.h b/src/ipa/libipa/vector.h index 2a2906202ce4..556e0967c869 100644 --- a/src/ipa/libipa/vector.h +++ b/src/ipa/libipa/vector.h @@ -16,6 +16,8 @@ #include "libcamera/internal/yaml_parser.h" +#include "matrix.h" + namespace libcamera { LOG_DECLARE_CATEGORY(Vector) @@ -140,6 +142,21 @@ private: std::array data_; }; +template +Vector operator*(const Matrix &m, const Vector &v) +{ + Vector result; + + for (unsigned int i = 0; i < Rows; i++) { + T sum = 0; + for (unsigned int j = 0; j < Cols; j++) + sum += m[i][j] * v[j]; + result[i] = sum; + } + + return result; +} + template bool operator==(const Vector &lhs, const Vector &rhs) {