From patchwork Thu Jan 9 11:53:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 22488 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 072FEC32EA for ; Thu, 9 Jan 2025 11:55:13 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 6613C68503; Thu, 9 Jan 2025 12:55:13 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="uEfE0ZQU"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 65CFB61891 for ; Thu, 9 Jan 2025 12:55:08 +0100 (CET) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:93b9:eca8:897d:eae6]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 5A6AE5B3; Thu, 9 Jan 2025 12:54:14 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1736423654; bh=DAOhp6TPjYaC2DQfshUa3EWKpVkGcmlGrP5+8xVtt0Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uEfE0ZQUVVVsQ/qDlUo3plFMONCWxBwecq3lr7S3kUbnROFn+S3m1XM4sHkG6oTJT coSFQbm/71s0gxK3UQt7PteQNsmLcudY9PkDrbzT5tE8F3Qn6AymJqbXj637GJs4fs DdKb5Wk+KTfZ7m1x6XXUwXuLmhgS2NpO+UzmYyww= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v1 01/11] libipa: interpolator: Add accessor to internal data Date: Thu, 9 Jan 2025 12:53:52 +0100 Message-ID: <20250109115412.356768-2-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250109115412.356768-1-stefan.klug@ideasonboard.com> References: <20250109115412.356768-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The only way to access the internal data of an Interpolator is through the getInterpolated() method. Sometimes it is necessary to to access the internal data directly to iterate over it. Add an accessor for that. While at it, remove a line break from the doxygen documentation for interpolate() so that doxygen is able to correctly match the function. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder Reviewed-by: Daniel Scally --- src/ipa/libipa/interpolator.cpp | 10 ++++++++-- src/ipa/libipa/interpolator.h | 5 +++++ 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/src/ipa/libipa/interpolator.cpp b/src/ipa/libipa/interpolator.cpp index 73e8d3b7de14..f901a86e4c74 100644 --- a/src/ipa/libipa/interpolator.cpp +++ b/src/ipa/libipa/interpolator.cpp @@ -125,6 +125,13 @@ namespace ipa { * Overwrites the internal map using move semantics. */ +/** + * \fn std::map &Interpolator::data() const + * \brief Access the internal map + * + * \return The internal map + */ + /** * \fn const T& Interpolator::getInterpolated() * \brief Retrieve an interpolated value for the given key @@ -136,8 +143,7 @@ namespace ipa { */ /** - * \fn void Interpolator::interpolate(const T &a, const T &b, T &dest, double - * lambda) + * \fn void Interpolator::interpolate(const T &a, const T &b, T &dest, double lambda) * \brief Interpolate between two instances of T * \param a The first value to interpolate * \param b The second value to interpolate diff --git a/src/ipa/libipa/interpolator.h b/src/ipa/libipa/interpolator.h index fffce21465fe..7880db6976d1 100644 --- a/src/ipa/libipa/interpolator.h +++ b/src/ipa/libipa/interpolator.h @@ -81,6 +81,11 @@ public: lastInterpolatedKey_.reset(); } + const std::map &data() const + { + return data_; + } + const T &getInterpolated(unsigned int key, unsigned int *quantizedKey = nullptr) { ASSERT(data_.size() > 0); From patchwork Thu Jan 9 11:53:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 22489 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 61548C32EA for ; Thu, 9 Jan 2025 11:55:16 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9498F684EA; Thu, 9 Jan 2025 12:55:15 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="IpMUMXep"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6C4BA684E0 for ; Thu, 9 Jan 2025 12:55:10 +0100 (CET) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:93b9:eca8:897d:eae6]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id E36E46F3; Thu, 9 Jan 2025 12:54:16 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1736423657; bh=4wCwvb9rwqa8z7IgKnIk4/c9q6lnKUjzpR7afccj5/g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IpMUMXepxp2KdPXwudx9Ehz7jaSq1XyGciCaugJJWpiCGvZh+cnCY/pXrV5an787L 9x/FcarK5+TcNssCfD9WlXMMLZt4V1q8V8H3t6nWCyRu9cWZ/UHUQ36dBIJDNdngzs U1HIFQHP5tEAG6BR4Ehvja4Rh2wS/F4kZVt+fCbI= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v1 02/11] libipa: pwl: Add clear() function Date: Thu, 9 Jan 2025 12:53:53 +0100 Message-ID: <20250109115412.356768-3-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250109115412.356768-1-stefan.klug@ideasonboard.com> References: <20250109115412.356768-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Sometimes it is necessary to clear a pwl. Add a function for that. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder Reviewed-by: Daniel Scally --- src/ipa/libipa/pwl.cpp | 5 +++++ src/ipa/libipa/pwl.h | 1 + 2 files changed, 6 insertions(+) diff --git a/src/ipa/libipa/pwl.cpp b/src/ipa/libipa/pwl.cpp index 88fe2022d66d..3fa005ba92df 100644 --- a/src/ipa/libipa/pwl.cpp +++ b/src/ipa/libipa/pwl.cpp @@ -159,6 +159,11 @@ void Pwl::prepend(double x, double y, const double eps) * \return True if there are no points in the function, false otherwise */ +/** + * \fn Pwl::clear() + * \brief Clear the piecewise linear function + */ + /** * \fn Pwl::size() const * \brief Retrieve the number of points in the piecewise linear function diff --git a/src/ipa/libipa/pwl.h b/src/ipa/libipa/pwl.h index d4ec9f4f18fb..93ced1a39745 100644 --- a/src/ipa/libipa/pwl.h +++ b/src/ipa/libipa/pwl.h @@ -49,6 +49,7 @@ public: void append(double x, double y, double eps = 1e-6); bool empty() const { return points_.empty(); } + void clear() { points_.clear(); } size_t size() const { return points_.size(); } Interval domain() const; From patchwork Thu Jan 9 11:53:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 22490 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7945BC32EA for ; Thu, 9 Jan 2025 11:55:18 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D7A8168516; Thu, 9 Jan 2025 12:55:17 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="LJoO8apu"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 29D17684E2 for ; Thu, 9 Jan 2025 12:55:13 +0100 (CET) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:93b9:eca8:897d:eae6]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id A4D076F3; Thu, 9 Jan 2025 12:54:19 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1736423659; bh=PxYbVII4EarCn618bzTqWAT5fn7Dx+QR10wHtw3ZamU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LJoO8apu+EWcaRXDDQU1AMcX6GhoCps7mpeT1a1cIM6FjjHzZO+6npWxM/kzsQymU ff908SDqCsRv2ViGJmfTAr973v7Rfb2XvIVM+a73ToAj8gK9T3nBvwLf2YOjXYb+7Y sOsaKr4dGMxuto2Tj994L5uamyLgz3LPKU3DxOvQ= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v1 03/11] libipa: Add AWB algorithm base class Date: Thu, 9 Jan 2025 12:53:54 +0100 Message-ID: <20250109115412.356768-4-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250109115412.356768-1-stefan.klug@ideasonboard.com> References: <20250109115412.356768-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a class to provide a generic interface for auto white balance algorithms. Concrete AWB algorithms are expected to subclass the AwbAlgorithm class to implement their functionality. Pipeline handlers are expected to subclass the AwbStats class and implement the necessary function to give the algorithm access to the hardware specific statistics data. Signed-off-by: Stefan Klug Reviewed-by: Daniel Scally --- src/ipa/libipa/awb.cpp | 137 +++++++++++++++++++++++++++++++++++++ src/ipa/libipa/awb.h | 51 ++++++++++++++ src/ipa/libipa/meson.build | 2 + 3 files changed, 190 insertions(+) create mode 100644 src/ipa/libipa/awb.cpp create mode 100644 src/ipa/libipa/awb.h diff --git a/src/ipa/libipa/awb.cpp b/src/ipa/libipa/awb.cpp new file mode 100644 index 000000000000..74e88d513b27 --- /dev/null +++ b/src/ipa/libipa/awb.cpp @@ -0,0 +1,137 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Ideas on Board Oy + * + * Generic AWB algorithms + */ + +#include "awb.h" + +#include + +/** + * \file awb.h + * \brief Base classes for AWB algorithms + */ + +namespace libcamera { + +LOG_DEFINE_CATEGORY(Awb) + +namespace ipa { + +/** + * \class AwbResult + * \brief The result of an awb calculation + * + * This class holds the result of an auto white balance calculation. + */ + +/** + * \var AwbResult::gains + * \brief The calculated white balance gains + */ + +/** + * \var AwbResult::colourTemperature + * \brief The calculated colour temperature in Kelvin + */ + +/** + * \class AwbStats + * \brief A abstract class wrapping system specific AWB statistics + * + * Pipline handlers using an AWB algorithm based on the AwbAlgorithm class need + * to implement this class to give the algorithm access to the hardware specific + * statics data. + */ + +/** + * \fn AwbStats::computeColourError + * \brief Compute an error value for when the given gains would be applied + * \param[in] gains The gains to apply + * + * Compute an error value (non-greyness) assuming the given \a gains would be + * applied. To keep the actual implementations computationally inexpensive, + * the squared colour error shall be returned. + * + * If the awb statistics provide multiple zones, the sum over all zones needs to + * calculated. + * + * \return The computed error value + */ + +/** + * \fn AwbStats::getRGBMeans + * \brief Get RGB means of the statistics + * + * Fetch the RGB means from the statistics. The values of each channel are + * dimensionless and only the ratios are used for further calculations. This is + * used by the simple gray world model to calculate the gains to apply. + * + * \return The RGB means + */ + +/** + * \class AwbAlgorithm + * \brief A base class for auto white balance algorithms + * + * This class is a base class for auto white balance algorithms. It provides an + * interface for the algorithms to implement, and is used by the pipeline + * handler to interact with the concrete implementation. + */ + +/** + * \fn AwbAlgorithm::init + * \brief Initialize the algorithm with the given tuning data + * \param[in] tuningData The tuning data to use for the algorithm + * + * \return 0 on success, a negative error code otherwise + */ + +/** + * \fn AwbAlgorithm::calculateAwb + * \brief Calculate awb data from the given statistics + * \param[in] stats The statistics to use for the calculation + * \param[in] lux The lux value of the scene + * + * Calculate a AwbResult object from the given statistics and lux value. A \a + * lux value of 0 means it is unknown or invalid and the algorithm shall ignore + * it. + * + * \return The awb result + */ + +/** + * \fn AwbAlgorithm::gainsFromColourTemperature + * \brief Compute white balance gains from a colour temperature + * \param[in] colourTemperature The colour temperature in Kelvin + * + * Compute the white balance gains from a \a colourTemperature. This function + * does not take any statistics into account. It is used to compute the colour + * gains when the user manually specifies a colour temperature. + * + * \return The colour gains + */ + +/** + * \fn AwbAlgorithm::controls + * \brief Get the controls info map for this algorithm + * + * \return The controls info map + */ + +/** + * \fn AwbAlgorithm::handleControls + * \param[in] controls The controls to handle + * \brief Handle the controls supplied in a request + */ + +/** + * \var AwbAlgorithm::controls_ + * \brief Controls info map for the controls provided by the algorithm + */ + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/awb.h b/src/ipa/libipa/awb.h new file mode 100644 index 000000000000..2dd471606ec4 --- /dev/null +++ b/src/ipa/libipa/awb.h @@ -0,0 +1,51 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Ideas on Board Oy + * + * Generic AWB algorithms + */ + +#pragma once + +#include +#include "libcamera/internal/yaml_parser.h" + +#include "vector.h" + +namespace libcamera { + +namespace ipa { + +struct AwbResult { + RGB gains; + double colourTemperature; +}; + +struct AwbStats { + virtual double computeColourError(const RGB &gains) const = 0; + virtual RGB getRGBMeans() const = 0; +}; + +class AwbAlgorithm +{ +public: + virtual ~AwbAlgorithm() = default; + + virtual int init(const YamlObject &tuningData) = 0; + virtual AwbResult calculateAwb(const AwbStats &stats, int lux) = 0; + virtual RGB gainsFromColourTemperature(double colourTemperature) = 0; + + const ControlInfoMap::Map &controls() const + { + return controls_; + } + + virtual void handleControls([[maybe_unused]] const ControlList &controls) {} + +protected: + ControlInfoMap::Map controls_; +}; + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index f2b2f4be50db..03e879c5834f 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -3,6 +3,7 @@ libipa_headers = files([ 'agc_mean_luminance.h', 'algorithm.h', + 'awb.h', 'camera_sensor_helper.h', 'colours.h', 'exposure_mode_helper.h', @@ -20,6 +21,7 @@ libipa_headers = files([ libipa_sources = files([ 'agc_mean_luminance.cpp', 'algorithm.cpp', + 'awb.cpp', 'camera_sensor_helper.cpp', 'colours.cpp', 'exposure_mode_helper.cpp', From patchwork Thu Jan 9 11:53:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 22491 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9B558C32EA for ; Thu, 9 Jan 2025 11:55:22 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3C1726851D; Thu, 9 Jan 2025 12:55:22 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="SZhJzbFU"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 13560684E7 for ; Thu, 9 Jan 2025 12:55:16 +0100 (CET) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:93b9:eca8:897d:eae6]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 8A9446F3; Thu, 9 Jan 2025 12:54:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1736423662; bh=y/72oxfBbHJsaPv/liZH+fW/XZcS/CceluJaJDIB9Y8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SZhJzbFUe3Q7YaR+v6PvTKDlKCl9EopBg6gkwuUue5wjJ1daD0vHE14ZN1w1CIVwi hO9RZkhKlFm3eQ4s/LE6N5CY0zSmxLAC0ccnZflUTndgQzh/M4xngG8e8OGO3lzVDm jIDJ2onxsZojW38DiK0FpwbLcnEN6PDA9qqbNmJs= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v1 04/11] libipa: awb: Add helper functions for AWB mode support Date: Thu, 9 Jan 2025 12:53:55 +0100 Message-ID: <20250109115412.356768-5-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250109115412.356768-1-stefan.klug@ideasonboard.com> References: <20250109115412.356768-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The AWB modes are specified in the libcamera core controls. It is therefore quite likely that every AWB algorithm will implement them. Add helper functions for parsing and storing the configured modes and the currently selected mode to the AwbAlgorithm base class. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder Reviewed-by: Daniel Scally --- src/ipa/libipa/awb.cpp | 100 +++++++++++++++++++++++++++++++++++++++++ src/ipa/libipa/awb.h | 12 +++++ 2 files changed, 112 insertions(+) diff --git a/src/ipa/libipa/awb.cpp b/src/ipa/libipa/awb.cpp index 74e88d513b27..07502da66f73 100644 --- a/src/ipa/libipa/awb.cpp +++ b/src/ipa/libipa/awb.cpp @@ -9,6 +9,8 @@ #include +#include + /** * \file awb.h * \brief Base classes for AWB algorithms @@ -132,6 +134,104 @@ namespace ipa { * \brief Controls info map for the controls provided by the algorithm */ +/** + * \var AwbAlgorithm::modes_ + * \brief Map of all configured modes + * \sa AwbAlgorithm::parseModeConfigs + */ + +/** + * \class AwbAlgorithm::ModeConfig + * \brief Holds the configuration of a single AWB mode + * + * Awb modes limit the regulation of the AWB algorithm to a specific range of + * colour temperatures. + */ + +/** + * \var AwbAlgorithm::ModeConfig::ctLo + * \brief The lowest valid colour temperature of that mode + */ + +/** + * \var AwbAlgorithm::ModeConfig::ctHi + * \brief The highest valid colour temperature of that mode + */ + +/** + * \brief Parse the mode configurations from the tuning data + * \param[in] tuningData the YamlObject representing the tuning data + * + * Utility function to parse the tuning data for a AwbMode entry and read all + * provided modes. It populetes AwbAlgorithm::controls_, AwbAlgorithm::modes_ + * and sets the current mode to AwbAuto. + * + * \return Zero on success, negative error code otherwise + */ +int AwbAlgorithm::parseModeConfigs(const YamlObject &tuningData) +{ + std::vector availableModes; + + const YamlObject &yamlModes = tuningData[controls::AwbMode.name()]; + if (!yamlModes.isDictionary()) { + LOG(Awb, Error) + << "AwbModes must be a dictionary."; + return -EINVAL; + } + + for (const auto &[modeName, modeDict] : yamlModes.asDict()) { + if (controls::AwbModeNameValueMap.find(modeName) == + controls::AwbModeNameValueMap.end()) { + LOG(Awb, Warning) + << "Skipping unknown awb mode '" + << modeName << "'"; + continue; + } + + if (!modeDict.isDictionary()) { + LOG(Awb, Error) + << "Invalid awb mode '" << modeName << "'"; + return -EINVAL; + } + + const auto &modeValue = static_cast( + controls::AwbModeNameValueMap.at(modeName)); + + auto &config = modes_[modeValue]; + auto hi = modeDict["hi"].get(); + if (!hi) { + LOG(Awb, Error) << "Failed to read hi param of mode " + << modeName; + return -EINVAL; + } + config.ctHi = *hi; + + auto lo = modeDict["lo"].get(); + if (!lo) { + LOG(Awb, Error) << "Failed to read low param of mode " + << modeName; + return -EINVAL; + } + config.ctLo = *lo; + + availableModes.push_back(modeValue); + } + + if (modes_.empty()) { + LOG(Awb, Error) << "No AWB modes configured"; + return -EINVAL; + } + + if (modes_.find(controls::AwbAuto) == modes_.end()) { + LOG(Awb, Error) << "AwbAuto mode is missing in the configuration."; + return -EINVAL; + } + + controls_[&controls::AwbMode] = ControlInfo(availableModes); + + return 0; +} + } /* namespace ipa */ } /* namespace libcamera */ diff --git a/src/ipa/libipa/awb.h b/src/ipa/libipa/awb.h index 2dd471606ec4..426fdd6dae77 100644 --- a/src/ipa/libipa/awb.h +++ b/src/ipa/libipa/awb.h @@ -7,7 +7,11 @@ #pragma once +#include + +#include #include + #include "libcamera/internal/yaml_parser.h" #include "vector.h" @@ -43,7 +47,15 @@ public: virtual void handleControls([[maybe_unused]] const ControlList &controls) {} protected: + int parseModeConfigs(const YamlObject &tuningData); + + struct ModeConfig { + double ctHi; + double ctLo; + }; + ControlInfoMap::Map controls_; + std::map modes_; }; } /* namespace ipa */ From patchwork Thu Jan 9 11:53:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 22492 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 216C9C32EA for ; Thu, 9 Jan 2025 11:55:26 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 912B668509; Thu, 9 Jan 2025 12:55:25 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="XOR++Wqq"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id AB17E684E2 for ; Thu, 9 Jan 2025 12:55:18 +0100 (CET) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:93b9:eca8:897d:eae6]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 2B3375B3; Thu, 9 Jan 2025 12:54:25 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1736423665; bh=PZMYoREg2rnkTargG1Vpo/vWcRdyKXgVX3J/qaM6rZw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XOR++Wqq2fxlIDlILb3+LM+ZFPuqhThTEgbrBI3lVjTSXGWjM1dHT0MT+RSF+djIv pA3TdZhPscNEmMkHxOn/d/2qdnqcYzMDAKFxsKkuBgkVqgvIWrgAxT5+XkxIMzoZV/ hTMDo7EoGPSck0G20Fkdz2Su/HtBX38O8wKB+/Fk= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v1 05/11] libipa: Add grey world AWB algorithm Date: Thu, 9 Jan 2025 12:53:56 +0100 Message-ID: <20250109115412.356768-6-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250109115412.356768-1-stefan.klug@ideasonboard.com> References: <20250109115412.356768-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add the grey world algorithm that is currently used in rkisp1 to libipa. No changes in functionality were made. Signed-off-by: Stefan Klug Reviewed-by: Daniel Scally --- src/ipa/libipa/awb_grey.cpp | 114 ++++++++++++++++++++++++++++++++++++ src/ipa/libipa/awb_grey.h | 35 +++++++++++ src/ipa/libipa/meson.build | 2 + 3 files changed, 151 insertions(+) create mode 100644 src/ipa/libipa/awb_grey.cpp create mode 100644 src/ipa/libipa/awb_grey.h diff --git a/src/ipa/libipa/awb_grey.cpp b/src/ipa/libipa/awb_grey.cpp new file mode 100644 index 000000000000..192a7cf3834a --- /dev/null +++ b/src/ipa/libipa/awb_grey.cpp @@ -0,0 +1,114 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Ideas on Board Oy + * + * Base class for bayesian AWB algorithm + */ + +#include "awb_grey.h" + +#include + +#include +#include + +#include "colours.h" + +using namespace libcamera::controls; + +/** + * \file awb_grey.h + * \brief Implementation of a grey world AWB algorithm + */ + +namespace libcamera { + +LOG_DECLARE_CATEGORY(Awb) +namespace ipa { + +/** + * \class AwbGrey + * \brief A Grey world auto white balance algorithm + */ + +/** + * \brief Initialize the algorithm with the given tuning data + * \param[in] tuningData The tuning data for the algorithm + * + * Load the colour temperature curve from the tuning data. If there is no tuning + * data available, continue with a warning. Manual colour temperature will not + * work in that case. + * + * \return 0 on success, a negative error code otherwise + */ +int AwbGrey::init(const YamlObject &tuningData) +{ + Interpolator> gains; + int ret = gains.readYaml(tuningData["colourGains"], "ct", "gains"); + if (ret < 0) + LOG(Awb, Warning) + << "Failed to parse 'colourGains' " + << "parameter from tuning file; " + << "manual colour temperature will not work properly"; + else + colourGainCurve_ = gains; + + return 0; +} + +/** + * \brief Calculate awb data from the given statistics + * \param[in] stats The statistics to use for the calculation + * \param[in] lux The lux value of the scene + * + * Estimates the colour temperature based on the coulours::estimateCCT function. + * The gains are calculated purely based on the RGB means provided by the \a + * stats. The colour temperature is not taken into account when calculating the + * gains. + * + * The \a lux parameter is not used in this algorithm. + * + * \return The awb result + */ +AwbResult AwbGrey::calculateAwb(const AwbStats &stats, [[maybe_unused]] int lux) +{ + AwbResult result; + auto means = stats.getRGBMeans(); + result.colourTemperature = estimateCCT(means); + + /* + * Estimate the red and blue gains to apply in a grey world. The green + * gain is hardcoded to 1.0. Avoid divisions by zero by clamping the + * divisor to a minimum value of 1.0. + */ + result.gains.r() = means.g() / std::max(means.r(), 1.0); + result.gains.g() = 1.0; + result.gains.b() = means.g() / std::max(means.b(), 1.0); + return result; +} + +/** + * \brief Compute white balance gains from a colour temperature + * \param[in] colourTemperature The colour temperature in Kelvin + * + * Compute the white balance gains from a \a colourTemperature. This function + * does not take any statistics into account. It simply interpolates the colour + * gains configured in the colour temperature curve. + * + * \return The colour gains if a colour temperature curve is available, + * [1, 1, 1] otherwise. + */ +RGB AwbGrey::gainsFromColourTemperature(double colourTemperature) +{ + if (!colourGainCurve_) { + LOG(Awb, Error) << "No gains defined"; + return RGB({ 1.0, 1.0, 1.0 }); + } + + auto gains = colourGainCurve_->getInterpolated(colourTemperature); + return { { gains[0], 1.0, gains[0] } }; +} + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/awb_grey.h b/src/ipa/libipa/awb_grey.h new file mode 100644 index 000000000000..6eda8e5498fb --- /dev/null +++ b/src/ipa/libipa/awb_grey.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Ideas on Board Oy + * + * AWB grey world algorithm + */ + +#pragma once + +#include "libcamera/internal/yaml_parser.h" + +#include "awb.h" +#include "interpolator.h" +#include "vector.h" + +namespace libcamera { + +namespace ipa { + +class AwbGrey : public AwbAlgorithm +{ +public: + AwbGrey() = default; + + int init(const YamlObject &tuningData) override; + AwbResult calculateAwb(const AwbStats &stats, int lux) override; + RGB gainsFromColourTemperature(double coulourTemperature) override; + +private: + std::optional>> colourGainCurve_; +}; + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index 03e879c5834f..c550a6eb45b6 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -3,6 +3,7 @@ libipa_headers = files([ 'agc_mean_luminance.h', 'algorithm.h', + 'awb_grey.h', 'awb.h', 'camera_sensor_helper.h', 'colours.h', @@ -21,6 +22,7 @@ libipa_headers = files([ libipa_sources = files([ 'agc_mean_luminance.cpp', 'algorithm.cpp', + 'awb_grey.cpp', 'awb.cpp', 'camera_sensor_helper.cpp', 'colours.cpp', From patchwork Thu Jan 9 11:53:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 22493 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 3E376C32EA for ; Thu, 9 Jan 2025 11:55:29 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 1AFDD68509; Thu, 9 Jan 2025 12:55:28 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="eSDVKRbM"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 31A2F68500 for ; Thu, 9 Jan 2025 12:55:21 +0100 (CET) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:93b9:eca8:897d:eae6]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id ADE286F3; Thu, 9 Jan 2025 12:54:27 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1736423667; bh=J5zO0Xh+JnHKDvThY9tQgugnjJ2fLntcAVol9EqDm/A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eSDVKRbMm0tdpk0RtAsHmX+HtQQy1HzR+3HSreyhIgheuCuljGx69/+hbnirT2cvJ 8aRPNJaBZEb3+j3F5YkPoq3iv6o3Z7aSSPaGW+54Vt/hhiNkfgTqqDQew3Nd7KGnw8 4zD/ljIcTBTsVVPX2/Bjf+FvKwlVld0RVoyrByQ4= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v1 06/11] ipa: rkisp1: Move calculation of RGB means into own function Date: Thu, 9 Jan 2025 12:53:57 +0100 Message-ID: <20250109115412.356768-7-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250109115412.356768-1-stefan.klug@ideasonboard.com> References: <20250109115412.356768-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Move the calculation of the RGB means into an own function for better code clarity. This commit doesn't contain any functional changes. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder Reviewed-by: Daniel Scally --- src/ipa/rkisp1/algorithms/awb.cpp | 90 +++++++++++++++++-------------- src/ipa/rkisp1/algorithms/awb.h | 3 ++ 2 files changed, 52 insertions(+), 41 deletions(-) diff --git a/src/ipa/rkisp1/algorithms/awb.cpp b/src/ipa/rkisp1/algorithms/awb.cpp index cffaa06a22c1..f6449565834b 100644 --- a/src/ipa/rkisp1/algorithms/awb.cpp +++ b/src/ipa/rkisp1/algorithms/awb.cpp @@ -229,7 +229,7 @@ void Awb::process(IPAContext &context, const rkisp1_cif_isp_stat *params = &stats->params; const rkisp1_cif_isp_awb_stat *awb = ¶ms->awb; IPAActiveState &activeState = context.activeState; - RGB rgbMeans; + RGB rgbMeans = calculateRgbMeans(frameContext, awb); metadata.set(controls::AwbEnable, frameContext.awb.autoEnabled); metadata.set(controls::ColourGains, { @@ -243,6 +243,53 @@ void Awb::process(IPAContext &context, return; } + /* + * If the means are too small we don't have enough information to + * meaningfully calculate gains. Freeze the algorithm in that case. + */ + if (rgbMeans.r() < kMeanMinThreshold && rgbMeans.g() < kMeanMinThreshold && + rgbMeans.b() < kMeanMinThreshold) + return; + + activeState.awb.temperatureK = estimateCCT(rgbMeans); + + /* Metadata shall contain the up to date measurement */ + metadata.set(controls::ColourTemperature, activeState.awb.temperatureK); + + /* + * Estimate the red and blue gains to apply in a grey world. The green + * gain is hardcoded to 1.0. Avoid divisions by zero by clamping the + * divisor to a minimum value of 1.0. + */ + RGB gains({ rgbMeans.g() / std::max(rgbMeans.r(), 1.0), + 1.0, + rgbMeans.g() / std::max(rgbMeans.b(), 1.0) }); + + /* + * Clamp the gain values to the hardware, which expresses gains as Q2.8 + * unsigned integer values. Set the minimum just above zero to avoid + * divisions by zero when computing the raw means in subsequent + * iterations. + */ + gains = gains.max(1.0 / 256).min(1023.0 / 256); + + /* Filter the values to avoid oscillations. */ + double speed = 0.2; + gains = gains * speed + activeState.awb.gains.automatic * (1 - speed); + + activeState.awb.gains.automatic = gains; + + LOG(RkISP1Awb, Debug) + << std::showpoint + << "Means " << rgbMeans << ", gains " + << activeState.awb.gains.automatic << ", temp " + << activeState.awb.temperatureK << "K"; +} + +RGB Awb::calculateRgbMeans(const IPAFrameContext &frameContext, const rkisp1_cif_isp_awb_stat *awb) const +{ + Vector rgbMeans; + if (rgbMode_) { rgbMeans = {{ static_cast(awb->awb_mean[0].mean_y_or_g), @@ -301,46 +348,7 @@ void Awb::process(IPAContext &context, */ rgbMeans /= frameContext.awb.gains; - /* - * If the means are too small we don't have enough information to - * meaningfully calculate gains. Freeze the algorithm in that case. - */ - if (rgbMeans.r() < kMeanMinThreshold && rgbMeans.g() < kMeanMinThreshold && - rgbMeans.b() < kMeanMinThreshold) - return; - - activeState.awb.temperatureK = estimateCCT(rgbMeans); - - /* - * Estimate the red and blue gains to apply in a grey world. The green - * gain is hardcoded to 1.0. Avoid divisions by zero by clamping the - * divisor to a minimum value of 1.0. - */ - RGB gains({ - rgbMeans.g() / std::max(rgbMeans.r(), 1.0), - 1.0, - rgbMeans.g() / std::max(rgbMeans.b(), 1.0) - }); - - /* - * Clamp the gain values to the hardware, which expresses gains as Q2.8 - * unsigned integer values. Set the minimum just above zero to avoid - * divisions by zero when computing the raw means in subsequent - * iterations. - */ - gains = gains.max(1.0 / 256).min(1023.0 / 256); - - /* Filter the values to avoid oscillations. */ - double speed = 0.2; - gains = gains * speed + activeState.awb.gains.automatic * (1 - speed); - - activeState.awb.gains.automatic = gains; - - LOG(RkISP1Awb, Debug) - << std::showpoint - << "Means " << rgbMeans << ", gains " - << activeState.awb.gains.automatic << ", temp " - << activeState.awb.temperatureK << "K"; + return rgbMeans; } REGISTER_IPA_ALGORITHM(Awb, "Awb") diff --git a/src/ipa/rkisp1/algorithms/awb.h b/src/ipa/rkisp1/algorithms/awb.h index e424804861a6..2a64cb60d5f4 100644 --- a/src/ipa/rkisp1/algorithms/awb.h +++ b/src/ipa/rkisp1/algorithms/awb.h @@ -38,6 +38,9 @@ public: ControlList &metadata) override; private: + RGB calculateRgbMeans(const IPAFrameContext &frameContext, + const rkisp1_cif_isp_awb_stat *awb) const; + std::optional>> colourGainCurve_; bool rgbMode_; }; From patchwork Thu Jan 9 11:53:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 22494 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 54644C32EA for ; Thu, 9 Jan 2025 11:55:31 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B966268521; Thu, 9 Jan 2025 12:55:30 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="F7zlJZm2"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4036C684E7 for ; Thu, 9 Jan 2025 12:55:24 +0100 (CET) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:93b9:eca8:897d:eae6]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id B237C6F3; Thu, 9 Jan 2025 12:54:30 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1736423670; bh=ghfQ3/PBL5n2umd5HmRqnXrUw/6zbhgzVqNusjNPIYA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=F7zlJZm2mnwdDN3QIu2B0Qd24up6QM4YFOBJhkrsWoFCvU6xH2ONysf/cv3sNr8Xp FlSc0d5/8zlSAdLd0Yt40SIX7KGFz2ucm4NFLmqVRWbWJXEl4ZvNsujUFJLCzaGm84 bRV/Pq6wOu1MXOIo+dTKC4jwXXK4Homs74z2EHx4= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v1 07/11] ipa: rkisp1: Use grey world algorithm from libipa Date: Thu, 9 Jan 2025 12:53:58 +0100 Message-ID: <20250109115412.356768-8-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250109115412.356768-1-stefan.klug@ideasonboard.com> References: <20250109115412.356768-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that libipa contains a grey world algorithm, use that. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder Reviewed-by: Daniel Scally --- src/ipa/rkisp1/algorithms/awb.cpp | 72 ++++++++++++++++++++----------- src/ipa/rkisp1/algorithms/awb.h | 4 +- 2 files changed, 49 insertions(+), 27 deletions(-) diff --git a/src/ipa/rkisp1/algorithms/awb.cpp b/src/ipa/rkisp1/algorithms/awb.cpp index f6449565834b..42a4784998bc 100644 --- a/src/ipa/rkisp1/algorithms/awb.cpp +++ b/src/ipa/rkisp1/algorithms/awb.cpp @@ -16,6 +16,7 @@ #include +#include "libipa/awb_grey.h" #include "libipa/colours.h" /** @@ -40,6 +41,28 @@ constexpr int32_t kDefaultColourTemperature = 5000; /* Minimum mean value below which AWB can't operate. */ constexpr double kMeanMinThreshold = 2.0; +class RkISP1AwbStats : public AwbStats +{ +public: + RkISP1AwbStats(const RGB &rgbMeans) + : rgbMeans_(rgbMeans) {} + + double computeColourError([[maybe_unused]] const RGB &gains) const override + { + LOG(RkISP1Awb, Error) + << "RkISP1AwbStats::computeColourError is not implemented"; + return 0.0; + } + + RGB getRGBMeans() const override + { + return rgbMeans_; + }; + +private: + RGB rgbMeans_; +}; + Awb::Awb() : rgbMode_(false) { @@ -55,15 +78,12 @@ int Awb::init(IPAContext &context, const YamlObject &tuningData) kMaxColourTemperature, kDefaultColourTemperature); - Interpolator> gainCurve; - int ret = gainCurve.readYaml(tuningData["colourGains"], "ct", "gains"); - if (ret < 0) - LOG(RkISP1Awb, Warning) - << "Failed to parse 'colourGains' " - << "parameter from tuning file; " - << "manual colour temperature will not work properly"; - else - colourGainCurve_ = gainCurve; + awbAlgo_ = std::make_unique(); + int ret = awbAlgo_->init(tuningData); + if (ret) { + LOG(RkISP1Awb, Error) << "Failed to init awb algorithm"; + return ret; + } return 0; } @@ -128,10 +148,10 @@ void Awb::queueRequest(IPAContext &context, * This will be fixed with the bayes AWB algorithm. */ update = true; - } else if (colourTemperature && colourGainCurve_) { - const auto &gains = colourGainCurve_->getInterpolated(*colourTemperature); - awb.gains.manual.r() = gains[0]; - awb.gains.manual.b() = gains[1]; + } else if (colourTemperature) { + const auto &gains = awbAlgo_->gainsFromColourTemperature(*colourTemperature); + awb.gains.manual.r() = gains.r(); + awb.gains.manual.b() = gains.b(); awb.temperatureK = *colourTemperature; update = true; } @@ -251,33 +271,33 @@ void Awb::process(IPAContext &context, rgbMeans.b() < kMeanMinThreshold) return; - activeState.awb.temperatureK = estimateCCT(rgbMeans); + /* + * \Todo: Hardcode lux to a fixed value, until an estimation is + * implemented. + */ + int lux = 1000; + + RkISP1AwbStats awbStats{ rgbMeans }; + AwbResult r = awbAlgo_->calculateAwb(awbStats, lux); + + activeState.awb.temperatureK = r.colourTemperature; /* Metadata shall contain the up to date measurement */ metadata.set(controls::ColourTemperature, activeState.awb.temperatureK); - /* - * Estimate the red and blue gains to apply in a grey world. The green - * gain is hardcoded to 1.0. Avoid divisions by zero by clamping the - * divisor to a minimum value of 1.0. - */ - RGB gains({ rgbMeans.g() / std::max(rgbMeans.r(), 1.0), - 1.0, - rgbMeans.g() / std::max(rgbMeans.b(), 1.0) }); - /* * Clamp the gain values to the hardware, which expresses gains as Q2.8 * unsigned integer values. Set the minimum just above zero to avoid * divisions by zero when computing the raw means in subsequent * iterations. */ - gains = gains.max(1.0 / 256).min(1023.0 / 256); + r.gains = r.gains.max(1.0 / 256).min(1023.0 / 256); /* Filter the values to avoid oscillations. */ double speed = 0.2; - gains = gains * speed + activeState.awb.gains.automatic * (1 - speed); + r.gains = r.gains * speed + activeState.awb.gains.automatic * (1 - speed); - activeState.awb.gains.automatic = gains; + activeState.awb.gains.automatic = r.gains; LOG(RkISP1Awb, Debug) << std::showpoint diff --git a/src/ipa/rkisp1/algorithms/awb.h b/src/ipa/rkisp1/algorithms/awb.h index 2a64cb60d5f4..3382b2178567 100644 --- a/src/ipa/rkisp1/algorithms/awb.h +++ b/src/ipa/rkisp1/algorithms/awb.h @@ -9,6 +9,7 @@ #include +#include "libipa/awb.h" #include "libipa/interpolator.h" #include "libipa/vector.h" @@ -41,7 +42,8 @@ private: RGB calculateRgbMeans(const IPAFrameContext &frameContext, const rkisp1_cif_isp_awb_stat *awb) const; - std::optional>> colourGainCurve_; + std::unique_ptr awbAlgo_; + bool rgbMode_; }; From patchwork Thu Jan 9 11:53:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 22495 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B367DC32EA for ; Thu, 9 Jan 2025 11:55:33 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2FA5768536; Thu, 9 Jan 2025 12:55:33 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="R4FHUePN"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 864D668516 for ; Thu, 9 Jan 2025 12:55:27 +0100 (CET) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:93b9:eca8:897d:eae6]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id D594F6F3; Thu, 9 Jan 2025 12:54:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1736423673; bh=TJbTuznBSn4VRnnE9XyebP476AFHHF98ZfIO1m1aLWY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=R4FHUePNiQbngTC1yzUTkj7iEOGUA563OBr75THxjHmnI/U7LMtLt1IUwOSgnAxQ0 YFLRdReKw+5g4O3vkTJqa/KDTsbbUirxQ50Kv5kWPUTFN3NmtxlzjPGHEDSho513z4 ZFq3vs5ACeYoUv3qZ+Kd2Xq4EgE71qMzS2DEiR3c= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v1 08/11] libtuning: module: awb: Add bayes AWB support Date: Thu, 9 Jan 2025 12:53:59 +0100 Message-ID: <20250109115412.356768-9-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250109115412.356768-1-stefan.klug@ideasonboard.com> References: <20250109115412.356768-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" To support the bayesian AWB algorithm in libtuning, the necessary data needs to be collected and written to the tuning file. Prior probabilities and AwbModes are defined by the user and therefore added to the example config file. Extend the output to also contain the necessary data for the bayesian AWB algorithm. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder --- utils/tuning/config-example.yaml | 34 +++++++++++++++++++- utils/tuning/libtuning/modules/awb/awb.py | 16 +++++---- utils/tuning/libtuning/modules/awb/rkisp1.py | 21 ++++++++---- 3 files changed, 58 insertions(+), 13 deletions(-) diff --git a/utils/tuning/config-example.yaml b/utils/tuning/config-example.yaml index 1b7f52cd2fff..30e88341df01 100644 --- a/utils/tuning/config-example.yaml +++ b/utils/tuning/config-example.yaml @@ -5,7 +5,39 @@ general: do_alsc_colour: 1 luminance_strength: 0.5 awb: - greyworld: 0 + # Algorithm can be either 'grey' or 'bayes' + algorithm: bayes + # priors is only used for the bayes algorithm + priors: + - lux: 0 + ct: [ 2000, 13000] + probability: [ 1.0, 1.0 ] + AwbMode: + AwbAuto: + lo: 2500 + hi: 8000 + AwbIncandescent: + lo: 2500 + hi: 3000 + AwbTungsten: + lo: 3000 + hi: 3500 + AwbFluorescent: + lo: 4000 + hi: 4700 + AwbIndoor: + lo: 3000 + hi: 5000 + AwbDaylight: + lo: 5500 + hi: 6500 + AwbCloudy: + lo: 6500 + hi: 8000 + # One custom mode can be defined if needed + #AwbCustom: + # lo: 2000 + # hi: 1300 macbeth: small: 1 show: 0 diff --git a/utils/tuning/libtuning/modules/awb/awb.py b/utils/tuning/libtuning/modules/awb/awb.py index c154cf3b8609..0dc4f59dcb26 100644 --- a/utils/tuning/libtuning/modules/awb/awb.py +++ b/utils/tuning/libtuning/modules/awb/awb.py @@ -27,10 +27,14 @@ class AWB(Module): imgs = [img for img in images if img.macbeth is not None] - gains, _, _ = awb(imgs, None, None, False) - gains = np.reshape(gains, (-1, 3)) + ct_curve, transverse_pos, transverse_neg = awb(imgs, None, None, False) + ct_curve = np.reshape(ct_curve, (-1, 3)) + gains = [{ + 'ct': int(v[0]), + 'gains': [float(1.0 / v[1]), float(1.0 / v[2])] + } for v in ct_curve] + + return {'colourGains': gains, + 'transversePos': transverse_pos, + 'transverseNeg': transverse_neg} - return [{ - 'ct': int(v[0]), - 'gains': [float(1.0 / v[1]), float(1.0 / v[2])] - } for v in gains] diff --git a/utils/tuning/libtuning/modules/awb/rkisp1.py b/utils/tuning/libtuning/modules/awb/rkisp1.py index 0c95843b83d3..d562d26eb8cc 100644 --- a/utils/tuning/libtuning/modules/awb/rkisp1.py +++ b/utils/tuning/libtuning/modules/awb/rkisp1.py @@ -6,9 +6,6 @@ from .awb import AWB -import libtuning as lt - - class AWBRkISP1(AWB): hr_name = 'AWB (RkISP1)' out_name = 'Awb' @@ -20,8 +17,20 @@ class AWBRkISP1(AWB): return True def process(self, config: dict, images: list, outputs: dict) -> dict: - output = {} - - output['colourGains'] = self.do_calculation(images) + if not 'awb' in config['general']: + raise ValueError('AWB configuration missing') + awb_config = config['general']['awb'] + algorithm = awb_config['algorithm'] + + output = {'algorithm': algorithm} + data = self.do_calculation(images) + if algorithm == 'grey': + output['colourGains'] = data['colourGains'] + elif algorithm == 'bayes': + output['AwbMode'] = awb_config['AwbMode'] + output['priors'] = awb_config['priors'] + output.update(data) + else: + raise ValueError(f"Unknown AWB algorithm {output['algorithm']}") return output From patchwork Thu Jan 9 11:54:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 22496 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9ADEAC32EA for ; Thu, 9 Jan 2025 11:55:35 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 6E47E68543; Thu, 9 Jan 2025 12:55:34 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="rbpSD3AH"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9F6FE68543 for ; Thu, 9 Jan 2025 12:55:29 +0100 (CET) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:93b9:eca8:897d:eae6]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 0F1036F3; Thu, 9 Jan 2025 12:54:36 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1736423676; bh=RXjL+83YbrnyIi8mdVx0Qlsf/GxksgPYpwzss/VtojQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rbpSD3AHlnvRjknDk8F5iT/pKBMmgn1vKOFzYTeJqGghZlcC94gLRvorgt4tpOqsC InQD5IV86PQJBJHajKIbynTR1u6csiasYOdCsgfpF0kW32Qwe1bIHlZ3Czo12hHipc 4Je9VtULqwJSkESG67d4iPZelfSNmzKyH5008LQY= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v1 09/11] libipa: Add bayesian AWB algorithm Date: Thu, 9 Jan 2025 12:54:00 +0100 Message-ID: <20250109115412.356768-10-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250109115412.356768-1-stefan.klug@ideasonboard.com> References: <20250109115412.356768-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The bayesian AWB algorithm is an AWB algorithm that takes prior probabilities for a given light source dependent on the current lux level into account. The biggest improvement compared to the grey world model comes from the search of the ideal white point on the CT curve. The algorithm walks the CT curve to minimize the colour error for a given statistics. After the minimium is found it additionally tries to search the area around that spot and also off the curve. So even without defined prior probabilities this algorithm provides much better results than the grey world algorithm. The logic for this code was taken from the RaspberryPi implementation. The logic was only minimally adjusted for usage with the rkisp1 and a few things were left out (see doxygen doc for the AwbBayes class). The code is refactored to better fit the libcamera code style and to make use of the syntactic sugar provided by the Interpolator and Vector classes. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder Reviewed-by: Daniel Scally --- src/ipa/libipa/awb_bayes.cpp | 457 +++++++++++++++++++++++++++++++++++ src/ipa/libipa/awb_bayes.h | 67 +++++ src/ipa/libipa/meson.build | 2 + 3 files changed, 526 insertions(+) create mode 100644 src/ipa/libipa/awb_bayes.cpp create mode 100644 src/ipa/libipa/awb_bayes.h diff --git a/src/ipa/libipa/awb_bayes.cpp b/src/ipa/libipa/awb_bayes.cpp new file mode 100644 index 000000000000..1e69ecd3e3f3 --- /dev/null +++ b/src/ipa/libipa/awb_bayes.cpp @@ -0,0 +1,457 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2019, Raspberry Pi Ltd + * Copyright (C) 2024 Ideas on Board Oy + * + * Implementation of a bayesian AWB algorithm + */ + +#include "awb_bayes.h" + +#include + +#include +#include + +#include "colours.h" + +/** + * \file awb_bayes.h + * \brief Implementation of bayesian auto white balance algorithm + * + * This implementation is based on the initial implementation done by + * RaspberryPi. + * \todo: Documentation + * + * \todo As the statistics module of the rkisp1 provides less data than the one + * from the RaspberryPi (vc4). The RaspberryPi statistics measure a grid of + * zones while the rkisp1 ony measures a single area. Therefore this algorithm + * doesn't contain all the features implemented by RaspberryPi. + * The following parts are not implemented: + * + * - min_pixels: minimum proportion of pixels counted within AWB region for it + * to be "useful" + * - min_g: minimum G value of those pixels, to be regarded a "useful" + * - min_regions: number of AWB regions that must be "useful" in order to do the + * AWB calculation + * - deltaLimit: clamp on colour error term (so as not to penalize non-grey + * excessively) + * - bias_proportion: The biasProportion parameter adds a small proportion of + * the counted pixels to a region biased to the biasCT colour temperature. + * A typical value for biasProportion would be between 0.05 to 0.1. + * - bias_ct: CT target for the search bias + * - sensitivityR: red sensitivity ratio (set to canonical sensor's R/G divided + * by this sensor's R/G) + * - sensitivityB: blue sensitivity ratio (set to canonical sensor's B/G divided + * by this sensor's B/G) + */ + +namespace libcamera { + +LOG_DECLARE_CATEGORY(Awb) + +namespace ipa { + +/** + * \brief Step size control for CT search + */ +constexpr double kSearchStep = 0.2; + +/** + * \copydoc Interpolator::interpolate() + */ +template<> +void Interpolator::interpolate(const Pwl &a, const Pwl &b, Pwl &dest, double lambda) +{ + dest = Pwl::combine(a, b, + [=](double /*x*/, double y0, double y1) -> double { + return y0 * (1.0 - lambda) + y1 * lambda; + }); +} + +/** + * \class AwbBayes + * \brief Implementation of a bayesian auto white balance algorithm + * + * In a bayesian AWB algorithm the auto white balance estimation is improved by + * taking the likelihood of a given lightsource based on the estimated lux level + * into account. E.g. If it is very bright we can assume that we are outside and + * that colour temperatures around 6500 are preferred. + * + * The second part of this algorithm is the search for the most likely colour + * temperature. It is implemented in AwbBayes::coarseSearch() and in + * AwbBayes::fineSearch(). The search works very well without prior likelihoods + * and therefore the algorithm itself provides very good results even without + * prior likelihoods. + */ + +/** + * \var AwbBayes::transversePos_ + * \brief How far to wander off CT curve towards "more purple" + */ + +/** + * \var AwbBayes::transverseNeg_ + * \brief How far to wander off CT curve towards "more green" + */ + +/** + * \var AwbBayes::currentMode_ + * \brief The currently selected mode + */ + +int AwbBayes::init(const YamlObject &tuningData) +{ + int ret = colourGainCurve_.readYaml(tuningData["colourGains"], "ct", "gains"); + if (ret) { + LOG(Awb, Error) + << "Failed to parse 'colourGains' " + << "parameter from tuning file"; + return ret; + } + + ctR_.clear(); + ctB_.clear(); + for (const auto &[ct, g] : colourGainCurve_.data()) { + ctR_.append(ct, 1.0 / g[0]); + ctB_.append(ct, 1.0 / g[1]); + } + + /* We will want the inverse functions of these too. */ + ctRInverse_ = ctR_.inverse().first; + ctBInverse_ = ctB_.inverse().first; + + ret = readPriors(tuningData); + if (ret) { + LOG(Awb, Error) << "Failed to read priors"; + return ret; + } + + ret = parseModeConfigs(tuningData); + if (ret) { + LOG(Awb, Error) + << "Failed to parse mode parameter from tuning file"; + return ret; + } + currentMode_ = &modes_[controls::AwbAuto]; + + transversePos_ = tuningData["transversePos"].get(0.01); + transverseNeg_ = tuningData["transverseNeg"].get(0.01); + if (transversePos_ <= 0 || transverseNeg_ <= 0) { + LOG(Awb, Error) << "AwbConfig: transversePos/Neg must be > 0"; + return -EINVAL; + } + + return 0; +} + +int AwbBayes::readPriors(const YamlObject &tuningData) +{ + const auto &priorsList = tuningData["priors"]; + std::map priors; + + if (!priorsList) { + LOG(Awb, Error) << "No priors specified"; + return -EINVAL; + } + + for (const auto &p : priorsList.asList()) { + if (!p.contains("lux")) { + LOG(Awb, Error) << "Missing lux value"; + return -EINVAL; + } + + uint32_t lux = p["lux"].get(0); + if (priors.count(lux)) { + LOG(Awb, Error) << "Duplicate prior for lux value " << lux; + return -EINVAL; + } + + std::vector temperatures = + p["ct"].getList().value_or(std::vector{}); + std::vector probabilites = + p["probability"].getList().value_or(std::vector{}); + + if (temperatures.size() != probabilites.size()) { + LOG(Awb, Error) + << "Ct and probability array sizes are unequal"; + return -EINVAL; + } + + if (temperatures.empty()) { + LOG(Awb, Error) + << "Ct and probability arrays are empty"; + return -EINVAL; + } + + std::map ctToProbability; + for (unsigned int i = 0; i < temperatures.size(); i++) { + int t = temperatures[i]; + if (ctToProbability.count(t)) { + LOG(Awb, Error) << "Duplicate ct value"; + return -EINVAL; + } + + ctToProbability[t] = probabilites[i]; + } + + auto &pwl = priors[lux]; + for (const auto &[ct, prob] : ctToProbability) { + pwl.append(ct, prob); + } + } + + if (priors.empty()) { + LOG(Awb, Error) << "No priors specified"; + ; + return -EINVAL; + } + + priors_.setData(std::move(priors)); + + return 0; +} + +void AwbBayes::handleControls(const ControlList &controls) +{ + auto mode = controls.get(controls::AwbMode); + if (mode) { + auto it = modes_.find(static_cast(*mode)); + if (it != modes_.end()) + currentMode_ = &it->second; + else + LOG(Awb, Error) << "Unknown AWB mode"; + } +} + +RGB AwbBayes::gainsFromColourTemperature(double colourTemperature) +{ + /* + * \todo: In the RaspberryPi code, the ct curve was interpolated in + * the white point space (1/x) not in gains space. This feels counter + * intuitive, as the gains are in linear space. But I can't prove it. + */ + const auto &gains = colourGainCurve_.getInterpolated(colourTemperature); + return { { gains[0], 1.0, gains[1] } }; +} + +AwbResult AwbBayes::calculateAwb(const AwbStats &stats, int lux) +{ + ipa::Pwl prior; + if (lux > 0) { + prior = priors_.getInterpolated(lux); + prior.map([](double x, double y) { + LOG(Awb, Debug) << "(" << x << "," << y << ")"; + }); + } else { + prior.append(0, 1.0); + } + + double t = coarseSearch(prior, stats); + double r = ctR_.eval(t); + double b = ctB_.eval(t); + LOG(Awb, Debug) + << "After coarse search: r " << r << " b " << b << " (gains r " + << 1 / r << " b " << 1 / b << ")"; + + /* + * Original comment from RaspberryPi: + * Not entirely sure how to handle the fine search yet. Mostly the + * estimated CT is already good enough, but the fine search allows us to + * wander transversely off the CT curve. Under some illuminants, where + * there may be more or less green light, this may prove beneficial, + * though I probably need more real datasets before deciding exactly how + * this should be controlled and tuned. + */ + fineSearch(t, r, b, prior, stats); + LOG(Awb, Debug) + << "After fine search: r " << r << " b " << b << " (gains r " + << 1 / r << " b " << 1 / b << ")"; + + return { { { 1.0 / r, 1.0, 1.0 / b } }, t }; +} + +double AwbBayes::coarseSearch(const ipa::Pwl &prior, const AwbStats &stats) const +{ + std::vector points; + size_t bestPoint = 0; + double t = currentMode_->ctLo; + int spanR = -1; + int spanB = -1; + + /* Step down the CT curve evaluating log likelihood. */ + while (true) { + double r = ctR_.eval(t, &spanR); + double b = ctB_.eval(t, &spanB); + RGB gains({ 1 / r, 1.0, 1 / b }); + double delta2Sum = stats.computeColourError(gains); + double priorLogLikelihood = prior.eval(prior.domain().clamp(t)); + double finalLogLikelihood = delta2Sum - priorLogLikelihood; + + LOG(Awb, Debug) << "Coarse search t: " << t + << " gains: " << gains + << " error: " << delta2Sum + << " prior: " << priorLogLikelihood + << " likelihood: " << finalLogLikelihood; + + points.push_back({ { t, finalLogLikelihood } }); + if (points.back().y() < points[bestPoint].y()) + bestPoint = points.size() - 1; + + if (t == currentMode_->ctHi) + break; + + /* + * Ensure even steps along the r/b curve by scaling them by the + * current t. + */ + t = std::min(t + t / 10 * kSearchStep, currentMode_->ctHi); + } + + t = points[bestPoint].x(); + LOG(Awb, Debug) << "Coarse search found CT " << t; + + /* + * We have the best point of the search, but refine it with a quadratic + * interpolation around its neighbors. + */ + if (points.size() > 2) { + bestPoint = std::clamp(bestPoint, 1ul, points.size() - 2); + t = interpolateQuadratic(points[bestPoint - 1], + points[bestPoint], + points[bestPoint + 1]); + LOG(Awb, Debug) + << "After quadratic refinement, coarse search has CT " + << t; + } + + return t; +} + +void AwbBayes::fineSearch(double &t, double &r, double &b, ipa::Pwl const &prior, const AwbStats &stats) const +{ + int spanR = -1; + int spanB = -1; + double step = t / 10 * kSearchStep * 0.1; + int nsteps = 5; + ctR_.eval(t, &spanR); + ctB_.eval(t, &spanB); + double rDiff = ctR_.eval(t + nsteps * step, &spanR) - + ctR_.eval(t - nsteps * step, &spanR); + double bDiff = ctB_.eval(t + nsteps * step, &spanB) - + ctB_.eval(t - nsteps * step, &spanB); + Pwl::Point transverse({ bDiff, -rDiff }); + if (transverse.length2() < 1e-6) + return; + /* + * transverse is a unit vector orthogonal to the b vs. r function + * (pointing outwards with r and b increasing) + */ + transverse = transverse / transverse.length(); + double bestLogLikelihood = 0; + double bestT = 0; + Pwl::Point bestRB; + double transverseRange = transverseNeg_ + transversePos_; + const int maxNumDeltas = 12; + + /* a transverse step approximately every 0.01 r/b units */ + int numDeltas = floor(transverseRange * 100 + 0.5) + 1; + numDeltas = std::clamp(numDeltas, 3, maxNumDeltas); + + /* + * Step down CT curve. March a bit further if the transverse range is + * large. + */ + nsteps += numDeltas; + for (int i = -nsteps; i <= nsteps; i++) { + double tTest = t + i * step; + double priorLogLikelihood = + prior.eval(prior.domain().clamp(tTest)); + Pwl::Point rbStart{ { ctR_.eval(tTest, &spanR), + ctB_.eval(tTest, &spanB) } }; + Pwl::Point samples[maxNumDeltas]; + int bestPoint = 0; + + /* + * Sample numDeltas points transversely *off* the CT curve + * in the range [-transverseNeg, transversePos]. + * The x values of a sample contains the distance and the y + * value contains the corresponding log likelihood. + */ + double transverseStep = transverseRange / (numDeltas - 1); + for (int j = 0; j < numDeltas; j++) { + auto &p = samples[j]; + p.x() = -transverseNeg_ + transverseStep * j; + Pwl::Point rbTest = rbStart + transverse * p.x(); + RGB gains({ 1 / rbTest[0], 1.0, 1 / rbTest[1] }); + double delta2Sum = stats.computeColourError(gains); + p.y() = delta2Sum - priorLogLikelihood; + + if (p.y() < samples[bestPoint].y()) + bestPoint = j; + } + + /* + * We have all samples transversely across the CT curve, + * now let's do a quadratic interpolation for the best result. + */ + bestPoint = std::clamp(bestPoint, 1, numDeltas - 2); + double bestOffset = interpolateQuadratic(samples[bestPoint - 1], + samples[bestPoint], + samples[bestPoint + 1]); + Pwl::Point rbTest = rbStart + transverse * bestOffset; + RGB gains({ 1 / rbTest[0], 1.0, 1 / rbTest[1] }); + double delta2Sum = stats.computeColourError(gains); + double finalLogLikelihood = delta2Sum - priorLogLikelihood; + LOG(Awb, Debug) + << "Fine search t: " << tTest + << " r: " << rbTest[0] + << " b: " << rbTest[1] + << " offset: " << bestOffset + << " likelihood: " << finalLogLikelihood + << (finalLogLikelihood < bestLogLikelihood ? " NEW BEST" : ""); + + if (bestT == 0 || finalLogLikelihood < bestLogLikelihood) { + bestLogLikelihood = finalLogLikelihood; + bestT = tTest; + bestRB = rbTest; + } + } + + t = bestT; + r = bestRB[0]; + b = bestRB[1]; + LOG(Awb, Debug) + << "Fine search found t " << t << " r " << r << " b " << b; +} + +/** + * \brief Find extremum of function + * \param[in] a First point + * \param[in] b Second point + * \param[in] c Third point + * + * Given 3 points on a curve, find the extremum of the function in that interval + * by fitting a quadratic. + * + * \return The x value of the extremum clamped to the interval [a.x(), c.x()] + */ +double AwbBayes::interpolateQuadratic(Pwl::Point const &a, Pwl::Point const &b, + Pwl::Point const &c) const +{ + const double eps = 1e-3; + Pwl::Point ca = c - a; + Pwl::Point ba = b - a; + double denominator = 2 * (ba.y() * ca.x() - ca.y() * ba.x()); + if (std::abs(denominator) > eps) { + double numerator = ba.y() * ca.x() * ca.x() - ca.y() * ba.x() * ba.x(); + double result = numerator / denominator + a.x(); + return std::max(a.x(), std::min(c.x(), result)); + } + /* has degenerated to straight line segment */ + return a.y() < c.y() - eps ? a.x() : (c.y() < a.y() - eps ? c.x() : b.x()); +} + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/awb_bayes.h b/src/ipa/libipa/awb_bayes.h new file mode 100644 index 000000000000..5284f0ca4cc0 --- /dev/null +++ b/src/ipa/libipa/awb_bayes.h @@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Ideas on Board Oy + * + * Base class for bayes AWB algorithms + */ + +#pragma once + +#include +#include +#include +#include + +#include + +#include +#include + +#include "libcamera/internal/yaml_parser.h" + +#include "awb.h" +#include "interpolator.h" +#include "pwl.h" +#include "vector.h" + +namespace libcamera { + +namespace ipa { + +class AwbBayes : public AwbAlgorithm +{ +public: + AwbBayes() = default; + + int init(const YamlObject &tuningData) override; + AwbResult calculateAwb(const AwbStats &stats, int lux) override; + RGB gainsFromColourTemperature(double temperatureK) override; + void handleControls(const ControlList &controls) override; + +private: + int readPriors(const YamlObject &tuningData); + + void fineSearch(double &t, double &r, double &b, ipa::Pwl const &prior, + const AwbStats &stats) const; + double coarseSearch(const ipa::Pwl &prior, const AwbStats &stats) const; + double interpolateQuadratic(ipa::Pwl::Point const &a, + ipa::Pwl::Point const &b, + ipa::Pwl::Point const &c) const; + + Interpolator priors_; + Interpolator> colourGainCurve_; + + ipa::Pwl ctR_; + ipa::Pwl ctB_; + ipa::Pwl ctRInverse_; + ipa::Pwl ctBInverse_; + + double transversePos_; + double transverseNeg_; + + ModeConfig *currentMode_ = nullptr; +}; + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index c550a6eb45b6..b519c8146d7c 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -3,6 +3,7 @@ libipa_headers = files([ 'agc_mean_luminance.h', 'algorithm.h', + 'awb_bayes.h', 'awb_grey.h', 'awb.h', 'camera_sensor_helper.h', @@ -22,6 +23,7 @@ libipa_headers = files([ libipa_sources = files([ 'agc_mean_luminance.cpp', 'algorithm.cpp', + 'awb_bayes.cpp', 'awb_grey.cpp', 'awb.cpp', 'camera_sensor_helper.cpp', From patchwork Thu Jan 9 11:54:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 22497 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 3E4E8C32EA for ; Thu, 9 Jan 2025 11:55:39 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id CCB136854E; Thu, 9 Jan 2025 12:55:38 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ksFaXMYv"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2C0C36854B for ; Thu, 9 Jan 2025 12:55:32 +0100 (CET) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:93b9:eca8:897d:eae6]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 898A65B3; Thu, 9 Jan 2025 12:54:38 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1736423678; bh=/DuFEHPaZsM3vdI7ESRF992+IXEEYo800VfIGt5f5Ug=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ksFaXMYvqm15PIQLKVNSiSsKkb4Uir5+1Z1Bp2JnPV7RV8v1x6BSc6INn5A56WtFr uvHGa3VpgfI19t0yui2aCWkMkkRIO34Lww6XJPEFdSbZOf4XoEEUVSvYczo+8xdxMT /R4zqrdks5+Eg4hGWOnivd2K0fcIs0tQTB6aV+OM= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v1 10/11] ipa: rkisp1: Add support for bayes AWB algorithm from libipa Date: Thu, 9 Jan 2025 12:54:01 +0100 Message-ID: <20250109115412.356768-11-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250109115412.356768-1-stefan.klug@ideasonboard.com> References: <20250109115412.356768-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that libipa contains a bayes AWB algorithm, add it as supported algorithm to the rkisp1 ipa. The decision between the grey world algorithm and the bayesian is done based on the "algorithm" property of the "Awb" algorithm in the tuning file. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder Reviewed-by: Daniel Scally --- Todo: The lux level is currently hardcoded to 1000. The result from the lux estimation needs to be used here. --- src/ipa/rkisp1/algorithms/awb.cpp | 44 ++++++++++++++++++++++++++----- 1 file changed, 38 insertions(+), 6 deletions(-) diff --git a/src/ipa/rkisp1/algorithms/awb.cpp b/src/ipa/rkisp1/algorithms/awb.cpp index 42a4784998bc..39a2c0589943 100644 --- a/src/ipa/rkisp1/algorithms/awb.cpp +++ b/src/ipa/rkisp1/algorithms/awb.cpp @@ -16,6 +16,7 @@ #include +#include "libipa/awb_bayes.h" #include "libipa/awb_grey.h" #include "libipa/colours.h" @@ -45,13 +46,23 @@ class RkISP1AwbStats : public AwbStats { public: RkISP1AwbStats(const RGB &rgbMeans) - : rgbMeans_(rgbMeans) {} + : rgbMeans_(rgbMeans) + { + rg_ = rgbMeans_.r() / rgbMeans_.g(); + bg_ = rgbMeans_.b() / rgbMeans_.g(); + } - double computeColourError([[maybe_unused]] const RGB &gains) const override + double computeColourError(const RGB &gains) const override { - LOG(RkISP1Awb, Error) - << "RkISP1AwbStats::computeColourError is not implemented"; - return 0.0; + /* + * Compute the sum of the squared colour error (non-greyness) as it + * appears in the log likelihood equation. + */ + double deltaR = gains.r() * rg_ - 1.0; + double deltaB = gains.b() * bg_ - 1.0; + double delta2 = deltaR * deltaR + deltaB * deltaB; + + return delta2; } RGB getRGBMeans() const override @@ -61,6 +72,8 @@ public: private: RGB rgbMeans_; + double rg_; + double bg_; }; Awb::Awb() @@ -78,13 +91,30 @@ int Awb::init(IPAContext &context, const YamlObject &tuningData) kMaxColourTemperature, kDefaultColourTemperature); - awbAlgo_ = std::make_unique(); + if (!tuningData.contains("algorithm")) + LOG(RkISP1Awb, Info) << "No awb algorithm specified." + << " Default to grey world"; + + auto mode = tuningData["algorithm"].get("grey"); + if (mode == "grey") { + awbAlgo_ = std::make_unique(); + } else if (mode == "bayes") { + awbAlgo_ = std::make_unique(); + } else { + LOG(RkISP1Awb, Error) << "Unknown awb algorithm: " << mode; + return -EINVAL; + } + LOG(RkISP1Awb, Debug) << "Using awb algorithm: " << mode; + int ret = awbAlgo_->init(tuningData); if (ret) { LOG(RkISP1Awb, Error) << "Failed to init awb algorithm"; return ret; } + const auto &src = awbAlgo_->controls(); + cmap.insert(src.begin(), src.end()); + return 0; } @@ -131,6 +161,8 @@ void Awb::queueRequest(IPAContext &context, << (*awbEnable ? "Enabling" : "Disabling") << " AWB"; } + awbAlgo_->handleControls(controls); + frameContext.awb.autoEnabled = awb.autoEnabled; if (awb.autoEnabled) From patchwork Thu Jan 9 11:54:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 22498 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 789CDC32EA for ; Thu, 9 Jan 2025 11:55:42 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 1984868541; Thu, 9 Jan 2025 12:55:42 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="p8eWFrIh"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DD31D68549 for ; Thu, 9 Jan 2025 12:55:34 +0100 (CET) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:93b9:eca8:897d:eae6]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 535676F3; Thu, 9 Jan 2025 12:54:41 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1736423681; bh=GhhoTJtyX+AEcCErZ2QM5LTWxhyVrJ4mPx3umBjy400=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=p8eWFrIh/qjxrgIvvkz5ol4wVYuJlhxHlEqi8rIG83xJzMTwDsPoVnR1JhLI6WV+F 6WXW4/t5/4FFq06SDnvQTNgpPCrHhDeJbSs8FXymfJflVG4j2fLtoJDpbu0L/YQamV 6fpEemmKVLK9NqgxBO4DxSeF1XK87vwWLeIDrTSU= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v1 11/11] ipa: rkisp1: awb: Apply gains based on default colour temperature on start Date: Thu, 9 Jan 2025 12:54:02 +0100 Message-ID: <20250109115412.356768-12-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250109115412.356768-1-stefan.klug@ideasonboard.com> References: <20250109115412.356768-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The colour gains are initialized with a default value of 1. Improve that by querying the auto white balance algorithm for the gains for a default colour temperature. This is still not based on measurements, but it is still better than the current implementation. If the algorithm doesn't implement mapping from colour temperature to gains, it will internally fallback to 1.0. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder Reviewed-by: Daniel Scally --- src/ipa/rkisp1/algorithms/awb.cpp | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/ipa/rkisp1/algorithms/awb.cpp b/src/ipa/rkisp1/algorithms/awb.cpp index 39a2c0589943..1761185b1181 100644 --- a/src/ipa/rkisp1/algorithms/awb.cpp +++ b/src/ipa/rkisp1/algorithms/awb.cpp @@ -125,7 +125,8 @@ int Awb::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) { context.activeState.awb.gains.manual = RGB{ 1.0 }; - context.activeState.awb.gains.automatic = RGB{ 1.0 }; + context.activeState.awb.gains.automatic = + awbAlgo_->gainsFromColourTemperature(kDefaultColourTemperature); context.activeState.awb.autoEnabled = true; context.activeState.awb.temperatureK = kDefaultColourTemperature;