From patchwork Wed Dec 18 07:46:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 22389 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2D6A4C3305 for ; Wed, 18 Dec 2024 07:46:25 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BE58C6807F; Wed, 18 Dec 2024 08:46:24 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="r4LZpHOM"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 95C1968085 for ; Wed, 18 Dec 2024 08:46:22 +0100 (CET) Received: from neptunite.hamster-moth.ts.net (unknown [IPv6:2404:7a81:160:2100:6d4:c4ff:fe49:f0b3]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 4459759D; Wed, 18 Dec 2024 08:45:43 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1734507944; bh=ywYKR4I7Ki0ObB5BaSmuAO7d86grQy4lNS3Pll7wmto=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=r4LZpHOMcMYQPVEadJ7y6J7UoRFYP8kPdFr7pOg8uKdC83uThlwXRPPPOw/OxxZ6f +cQqohUlSLiQQ5Qbw0uXXGV9ww9Rg2jadRS9ETYPHjPXwSdc69WhOVO/dEf9m4ywGc l2L4bE4GC9E0Fyxw3iauu4jEg5uWB86O39FSlS3Y= From: Paul Elder To: libcamera-devel@lists.libcamera.org, laurent.pinchart@ideasonboard.com, stefan.klug@ideasonboard.com Cc: Paul Elder Subject: [PATCH v3 1/2] ipa: libipa: Add Lux helper Date: Wed, 18 Dec 2024 16:46:00 +0900 Message-Id: <20241218074601.3552093-2-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241218074601.3552093-1-paul.elder@ideasonboard.com> References: <20241218074601.3552093-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a Lux helper to libipa that does the estimation of the lux level given gain, exposure, and luminance histogram. The helper also handles reading the reference values from the tuning file. These are expected to be common operations of lux algorithm modules in IPAs, and is modeled/copied from Raspberry Pi. Signed-off-by: Paul Elder Reviewed-by: Laurent Pinchart Reviewed-by: Kieran Bingham Reviewed-by: Isaac Scott --- Changes in v3: - improve documentation - minor docs formatting fixes - replace setBinSize() with constructor parameter - s/readYaml/parseTuningData/ - remove unnecessary includes Changes in v2: - improve documentation - add binSize member variable and corresponding setter - remove aperture - split gain into analogue and digital - change tuning file names into camel case --- src/ipa/libipa/lux.cpp | 181 +++++++++++++++++++++++++++++++++++++ src/ipa/libipa/lux.h | 42 +++++++++ src/ipa/libipa/meson.build | 2 + 3 files changed, 225 insertions(+) create mode 100644 src/ipa/libipa/lux.cpp create mode 100644 src/ipa/libipa/lux.h diff --git a/src/ipa/libipa/lux.cpp b/src/ipa/libipa/lux.cpp new file mode 100644 index 000000000000..b60060e21803 --- /dev/null +++ b/src/ipa/libipa/lux.cpp @@ -0,0 +1,181 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2019, Raspberry Pi Ltd + * Copyright (C) 2024, Paul Elder + * + * Helper class that implements lux estimation + */ +#include "lux.h" + +#include +#include + +#include + +#include "libcamera/internal/yaml_parser.h" + +#include "histogram.h" + +/** + * \file lux.h + * \brief Helper class that implements lux estimation + * + * Estimating the lux level of an image is a common operation that can for + * instance be used to adjust the target Y value in AGC or for Bayesian AWB + * estimation. + */ + +namespace libcamera { + +using namespace std::literals::chrono_literals; + +LOG_DEFINE_CATEGORY(Lux) + +namespace ipa { + +/** + * \class Lux + * \brief Class that implements lux estimation + * + * IPAs that wish to use lux esimation should create a Lux algorithm module + * that lightly wraps this module by providing the platform-specific luminance + * histogram. The Lux entry in the tuning file must then precede the algorithms + * that depend on the estimated lux value. + */ + +/** + * \var Lux::binSize_ + * \brief The maximum count of each bin + */ + +/** + * \var Lux::referenceExposureTime_ + * \brief The exposure time of the reference image, in microseconds + */ + +/** + * \var Lux::referenceAnalogueGain_ + * \brief The analogue gain of the reference image + */ + +/** + * \var Lux::referenceDigitalGain_ + * \brief The analogue gain of the reference image + */ + +/** + * \var Lux::referenceY_ + * \brief The measured luminance of the reference image, out of the bin size + * + * \sa binSize_ + */ + +/** + * \var Lux::referenceLux_ + * \brief The estimated lux level of the reference image + */ + +/** + * \brief Construct the Lux helper module + * \param[in] binSize The maximum count of each bin + */ +Lux::Lux(unsigned int binSize) + : binSize_(binSize) +{ +} + +/** + * \brief Parse tuning data + * \param[in] tuningData The YamlObject representing the tuning data + * + * This function parses yaml tuning data for the common Lux module. It requires + * reference exposure time, analogue gain, digital gain, and lux values. + * + * \code{.unparsed} + * algorithms: + * - Lux: + * referenceExposureTime: 10000 + * referenceAnalogueGain: 4.0 + * referenceDigitalGain: 1.0 + * referenceY: 12000 + * referenceLux: 1000 + * \endcode + * + * \return 0 on success or a negative error code + */ +int Lux::parseTuningData(const YamlObject &tuningData) +{ + auto value = tuningData["referenceExposureTime"].get(); + if (!value) { + LOG(Lux, Error) << "Missing tuning parameter: " + << "'referenceExposureTime'"; + return -EINVAL; + } + referenceExposureTime_ = *value * 1.0us; + + value = tuningData["referenceAnalogueGain"].get(); + if (!value) { + LOG(Lux, Error) << "Missing tuning parameter: " + << "'referenceAnalogueGain'"; + return -EINVAL; + } + referenceAnalogueGain_ = *value; + + value = tuningData["referenceDigitalGain"].get(); + if (!value) { + LOG(Lux, Error) << "Missing tuning parameter: " + << "'referenceDigitalGain'"; + return -EINVAL; + } + referenceDigitalGain_ = *value; + + value = tuningData["referenceY"].get(); + if (!value) { + LOG(Lux, Error) << "Missing tuning parameter: " + << "'referenceY'"; + return -EINVAL; + } + referenceY_ = *value; + + value = tuningData["referenceLux"].get(); + if (!value) { + LOG(Lux, Error) << "Missing tuning parameter: " + << "'referenceLux'"; + return -EINVAL; + } + referenceLux_ = *value; + + return 0; +} + +/** + * \brief Estimate lux given runtime values + * \param[in] exposureTime Exposure time applied to the frame + * \param[in] aGain Analogue gain applied to the frame + * \param[in] dGain Digital gain applied to the frame + * \param[in] yHist Histogram from the ISP statistics + * + * Estimate the lux given the exposure time, gain, and histogram. + * + * \return Estimated lux value + */ +double Lux::estimateLux(utils::Duration exposureTime, + double aGain, double dGain, + const Histogram &yHist) const +{ + double currentY = yHist.interQuantileMean(0, 1); + double exposureTimeRatio = referenceExposureTime_ / exposureTime; + double aGainRatio = referenceAnalogueGain_ / aGain; + double dGainRatio = referenceDigitalGain_ / dGain; + double yRatio = currentY * (binSize_ / yHist.bins()) / referenceY_; + + double estimatedLux = exposureTimeRatio * aGainRatio * dGainRatio * + yRatio * referenceLux_; + + LOG(Lux, Debug) << "Estimated lux " << estimatedLux; + return estimatedLux; +} + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/lux.h b/src/ipa/libipa/lux.h new file mode 100644 index 000000000000..93ca64795803 --- /dev/null +++ b/src/ipa/libipa/lux.h @@ -0,0 +1,42 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2019, Raspberry Pi Ltd + * Copyright (C) 2024, Paul Elder + * + * Helper class that implements lux estimation + */ + +#pragma once + +#include + +namespace libcamera { + +class YamlObject; + +namespace ipa { + +class Histogram; + +class Lux +{ +public: + Lux(unsigned int binSize); + + int parseTuningData(const YamlObject &tuningData); + double estimateLux(utils::Duration exposureTime, + double aGain, double dGain, + const Histogram &yHist) const; + +private: + unsigned int binSize_; + utils::Duration referenceExposureTime_; + double referenceAnalogueGain_; + double referenceDigitalGain_; + double referenceY_; + double referenceLux_; +}; + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index a7f16ff63079..f2b2f4be50db 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -11,6 +11,7 @@ libipa_headers = files([ 'histogram.h', 'interpolator.h', 'lsc_polynomial.h', + 'lux.h', 'module.h', 'pwl.h', 'vector.h', @@ -27,6 +28,7 @@ libipa_sources = files([ 'histogram.cpp', 'interpolator.cpp', 'lsc_polynomial.cpp', + 'lux.cpp', 'module.cpp', 'pwl.cpp', 'vector.cpp', From patchwork Wed Dec 18 07:46:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 22390 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A3FF3C3301 for ; Wed, 18 Dec 2024 07:46:29 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 563DF6808F; Wed, 18 Dec 2024 08:46:29 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="pysREpFw"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id BF25868087 for ; Wed, 18 Dec 2024 08:46:24 +0100 (CET) Received: from neptunite.hamster-moth.ts.net (unknown [IPv6:2404:7a81:160:2100:6d4:c4ff:fe49:f0b3]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 5C22534D; Wed, 18 Dec 2024 08:45:45 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1734507947; bh=2oaId2rbD8SZH/CsaKO7kqww85mU9sprEf53Ie62oyc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pysREpFwtwvn5oqSWdqyU3yT7FCeJI0voKNnqS9DnPYPiLBavYlsZOjiMKVymmRob OuQX2qJew7HtDYUT5Rj86YuBA6RvwjMoonazobw+HzpLUWiIzVa5AbceS40doh0BjP b4/jIrS+k0LL+5XGkIPCqQB+cQXVLVque9BtJgu8= From: Paul Elder To: libcamera-devel@lists.libcamera.org, laurent.pinchart@ideasonboard.com, stefan.klug@ideasonboard.com Cc: Paul Elder , Kieran Bingham Subject: [PATCH v3 2/2] ipa: rkisp1: Add Lux algorithm module Date: Wed, 18 Dec 2024 16:46:01 +0900 Message-Id: <20241218074601.3552093-3-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241218074601.3552093-1-paul.elder@ideasonboard.com> References: <20241218074601.3552093-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a lux algorithm module to rkisp1 IPA for estimating the lux level of an image. This is reported in metadata, as well as saved in the frame context so that other algorithms (mainly AGC) can use its value. It does not set any controls. Signed-off-by: Paul Elder Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart Reviewed-by: Isaac Scott --- Changes in v3: - remove unnecessary includes - replace ipa::Lux::bitSize() with ipa::Lux::Lux() - move frameContext.agc.lux to frameContext.lux.lux Changes in v2: - fix bitrot - fixes corresponding to changes in the previous patch --- src/ipa/rkisp1/algorithms/lux.cpp | 80 +++++++++++++++++++++++++++ src/ipa/rkisp1/algorithms/lux.h | 36 ++++++++++++ src/ipa/rkisp1/algorithms/meson.build | 1 + src/ipa/rkisp1/ipa_context.h | 4 ++ 4 files changed, 121 insertions(+) create mode 100644 src/ipa/rkisp1/algorithms/lux.cpp create mode 100644 src/ipa/rkisp1/algorithms/lux.h diff --git a/src/ipa/rkisp1/algorithms/lux.cpp b/src/ipa/rkisp1/algorithms/lux.cpp new file mode 100644 index 000000000000..a1e3f36b75cb --- /dev/null +++ b/src/ipa/rkisp1/algorithms/lux.cpp @@ -0,0 +1,80 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Ideas On Board + * + * lux.cpp - RkISP1 Lux control + */ + +#include "lux.h" + +#include + +#include + +#include "libipa/histogram.h" +#include "libipa/lux.h" + +/** + * \file lux.h + */ + +namespace libcamera { + +namespace ipa::rkisp1::algorithms { + +/** + * \class Lux + * \brief RkISP1 Lux control + * + * The Lux algorithm is responsible for estimating the lux level of the image. + * It doesn't take or generate any controls, but it provides a lux level for + * other algorithms (such as AGC) to use. + */ + +/** + * \brief Construct an rkisp1 Lux algo module + * + * The Lux helper is initialized to 65535 as that is the max bin count on the + * rkisp1. + */ +Lux::Lux() + : lux_(65535) +{ +} + +/** + * \copydoc libcamera::ipa::Algorithm::init + */ +int Lux::init([[maybe_unused]] IPAContext &context, const YamlObject &tuningData) +{ + return lux_.parseTuningData(tuningData); +} + +/** + * \copydoc libcamera::ipa::Algorithm::process + */ +void Lux::process(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, + const rkisp1_stat_buffer *stats, + ControlList &metadata) +{ + utils::Duration exposureTime = context.configuration.sensor.lineDuration + * frameContext.sensor.exposure; + double gain = frameContext.sensor.gain; + + /* \todo Deduplicate the histogram calculation from AGC */ + const rkisp1_cif_isp_stat *params = &stats->params; + Histogram yHist({ params->hist.hist_bins, context.hw->numHistogramBins }, + [](uint32_t x) { return x >> 4; }); + + double lux = lux_.estimateLux(exposureTime, gain, 1.0, yHist); + frameContext.lux.lux = lux; + metadata.set(controls::Lux, lux); +} + +REGISTER_IPA_ALGORITHM(Lux, "Lux") + +} /* namespace ipa::rkisp1::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/rkisp1/algorithms/lux.h b/src/ipa/rkisp1/algorithms/lux.h new file mode 100644 index 000000000000..8a90de55b8ff --- /dev/null +++ b/src/ipa/rkisp1/algorithms/lux.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Ideas On Board + * + * lux.h - RkISP1 Lux control + */ + +#pragma once + +#include + +#include "libipa/lux.h" + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::rkisp1::algorithms { + +class Lux : public Algorithm +{ +public: + Lux(); + + int init(IPAContext &context, const YamlObject &tuningData) override; + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const rkisp1_stat_buffer *stats, + ControlList &metadata) override; + +private: + ipa::Lux lux_; +}; + +} /* namespace ipa::rkisp1::algorithms */ +} /* namespace libcamera */ diff --git a/src/ipa/rkisp1/algorithms/meson.build b/src/ipa/rkisp1/algorithms/meson.build index 1734a6675f78..c66b0b70b82f 100644 --- a/src/ipa/rkisp1/algorithms/meson.build +++ b/src/ipa/rkisp1/algorithms/meson.build @@ -12,4 +12,5 @@ rkisp1_ipa_algorithms = files([ 'goc.cpp', 'gsl.cpp', 'lsc.cpp', + 'lux.cpp', ]) diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index deb8c196f1b8..65ba45151611 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -168,6 +168,10 @@ struct IPAFrameContext : public FrameContext { struct { Matrix ccm; } ccm; + + struct { + double lux; + } lux; }; struct IPAContext {