From patchwork Fri Apr 12 09:16:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 19865 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7FDF2C3285 for ; Fri, 12 Apr 2024 09:17:17 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B295D63365; Fri, 12 Apr 2024 11:17:16 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="Rg8rbsXO"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E80816334D for ; Fri, 12 Apr 2024 11:17:14 +0200 (CEST) Received: from pyrite.hamster-moth.ts.net (h175-177-049-156.catv02.itscom.jp [175.177.49.156]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 7496EA68; Fri, 12 Apr 2024 11:16:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1712913391; bh=D439QK52+qfT/ZkMkznJtkTLpW05cf4Otgc9lSuDJdE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Rg8rbsXOXgWzFnhg3FO6GeP6MRrwT0KyoEl1YZhjjMU6SVRLFG0NaUqMMGectLNFY pZsZ2vEY17iFRfjdsk1o/7nnVyaXjJNjhb4tk5eiCH7ro4UUQVSllSFCbX5p2QVoxr sIVuDHxNF9ZcKLPJdLTZdX2frB4dWi16JEyeB3ag= From: Paul Elder To: libcamera-devel@lists.libcamera.org Cc: Paul Elder Subject: [PATCH 1/3] ipa: libipa: Add Lux helper Date: Fri, 12 Apr 2024 18:16:58 +0900 Message-Id: <20240412091700.1817754-2-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240412091700.1817754-1-paul.elder@ideasonboard.com> References: <20240412091700.1817754-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a Lux helper to libipa that does the estimation of the lux level given gain, exposure, aperture, and luminance histogram. The helper also handles reading the reference values from the tuning file. These are expected to be common operations of lux algorithm modules in IPAs, and is modeled/copied from Raspberry Pi. Signed-off-by: Paul Elder --- src/ipa/libipa/lux.cpp | 119 +++++++++++++++++++++++++++++++++++++ src/ipa/libipa/lux.h | 45 ++++++++++++++ src/ipa/libipa/meson.build | 2 + 3 files changed, 166 insertions(+) create mode 100644 src/ipa/libipa/lux.cpp create mode 100644 src/ipa/libipa/lux.h diff --git a/src/ipa/libipa/lux.cpp b/src/ipa/libipa/lux.cpp new file mode 100644 index 00000000..756cd3c4 --- /dev/null +++ b/src/ipa/libipa/lux.cpp @@ -0,0 +1,119 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2019, Raspberry Pi Ltd + * Copyright (C) 2024, Paul Elder + * + * lux.h - Helper class that implements lux estimation + */ +#include "lux.h" + +#include +#include + +#include + +#include "libcamera/internal/yaml_parser.h" + +#include "histogram.h" + +/** + * \file lux.h + * \brief Helper class that implements lux estimation + * + * As estimating the lux level of an image is expected to be a common + * operation, it is implemented in a helper in libipa. + */ + +namespace libcamera { + +using namespace std::literals::chrono_literals; + +LOG_DEFINE_CATEGORY(Lux) + +namespace ipa { + +/** + * \class Lux + * \brief Class that implements lux estimation + */ + +/** + * \var Lux::referenceExposureTime_ + * \brief The exposure time of the reference image, in microseconds. + */ + +/** + * \var Lux::referenceGain_ + * \brief The analogue gain of the reference image. + */ + +/** + * \var Lux::referenceAperture_ + * \brief The aperture of the reference image in units of 1/f. + */ + +/** + * \var Lux::referenceY_ + * \brief The measured luminance of the reference image, out of 65536. + */ + +/** + * \var Lux::referenceLux_ + * \brief The estimated lux level of the reference image. + */ + +int Lux::readYaml(const YamlObject &tuningData) +{ + auto value = tuningData["reference_exposure_time"].get(); + if (!value) { + LOG(Lux, Error) << "Missing tuning parameter: 'reference_exposure_time'"; + return -EINVAL; + } + referenceExposureTime_ = *value * 1.0us; + + value = tuningData["reference_gain"].get(); + if (!value) { + LOG(Lux, Error) << "Missing tuning parameter: 'reference_gain'"; + return -EINVAL; + } + referenceGain_ = *value; + + referenceAperture_ = tuningData["reference_aperture"].get(1.0); + + value = tuningData["reference_Y"].get(); + if (!value) { + LOG(Lux, Error) << "Missing tuning parameter: 'reference_Y'"; + return -EINVAL; + } + referenceY_ = *value; + + value = tuningData["reference_lux"].get(); + if (!value) { + LOG(Lux, Error) << "Missing tuning parameter: 'reference_lux'"; + return -EINVAL; + } + referenceLux_ = *value; + + return 0; +} + +double Lux::process(double gain, utils::Duration exposureTime, double aperture, + const Histogram &yHist) const +{ + double currentY = yHist.interQuantileMean(0, 1); + double gainRatio = referenceGain_ / gain; + double exposureTimeRatio = referenceExposureTime_ / exposureTime; + double apertureRatio = referenceAperture_ / aperture; + double yRatio = currentY * (65536 / yHist.bins()) / referenceY_; + + double estimatedLux = exposureTimeRatio * gainRatio * + apertureRatio * apertureRatio * + yRatio * referenceLux_; + + LOG(Lux, Debug) << "Estimated lux " << estimatedLux; + return estimatedLux; +} + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/lux.h b/src/ipa/libipa/lux.h new file mode 100644 index 00000000..6bc9cf9f --- /dev/null +++ b/src/ipa/libipa/lux.h @@ -0,0 +1,45 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2019, Raspberry Pi Ltd + * Copyright (C) 2024, Paul Elder + * + * lux.h - Helper class that implements lux estimation + */ + +#pragma once + +#include +#include +#include + +#include + +#include "libcamera/internal/yaml_parser.h" + +#include "histogram.h" + +namespace libcamera { + +namespace ipa { + +class Lux +{ +public: + Lux() = default; + ~Lux() = default; + + int readYaml(const YamlObject &tuningData); + double process(double gain, utils::Duration exposureTime, + double aperture, const Histogram &yHist) const; + +private: + utils::Duration referenceExposureTime_; + double referenceGain_; + double referenceAperture_; + double referenceY_; + double referenceLux_; +}; + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index 0796982e..b6d58900 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -8,6 +8,7 @@ libipa_headers = files([ 'exposure_mode_helper.h', 'fc_queue.h', 'histogram.h', + 'lux.h', 'matrix.h', 'module.h', 'pwl.h', @@ -21,6 +22,7 @@ libipa_sources = files([ 'exposure_mode_helper.cpp', 'fc_queue.cpp', 'histogram.cpp', + 'lux.cpp', 'matrix.cpp', 'module.cpp', 'pwl.cpp' From patchwork Fri Apr 12 09:16:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 19866 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id CCDAAC3213 for ; Fri, 12 Apr 2024 09:17:20 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 70ABA6335E; Fri, 12 Apr 2024 11:17:20 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="hRvz9zmI"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id CC8AA6334D for ; Fri, 12 Apr 2024 11:17:16 +0200 (CEST) Received: from pyrite.hamster-moth.ts.net (h175-177-049-156.catv02.itscom.jp [175.177.49.156]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 4F22CA68; Fri, 12 Apr 2024 11:16:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1712913393; bh=9oRn1DvK3RvEz+XI3Wc4no9+0oyEO0kiQ89GCF2UKfk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hRvz9zmIBNtsr/EH4xnuaSsiEmS4ylRaRTncFfM0DuoOwwDcBQbaSm6+o5TVR3QSg IAmC7TiTxTLmeX8WjNPi/hkWVjpywoO5HQK23Y9C4MzSQN1k/QV3p1cMQ55GF7JkVy DzSiI2EbyG64J9X1I+lcLuSIlIaSY54Kx60BzlIw= From: Paul Elder To: libcamera-devel@lists.libcamera.org Cc: Paul Elder Subject: [PATCH 2/3] ipa: rkisp1: Add Lux algorithm module Date: Fri, 12 Apr 2024 18:16:59 +0900 Message-Id: <20240412091700.1817754-3-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240412091700.1817754-1-paul.elder@ideasonboard.com> References: <20240412091700.1817754-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a lux algorithm module to rkisp1 IPA for estimating the lux level of an image. This is reported in metadata, as well as saved in the frame context so that other algorithms (mainly AGC) can use its value. It does not set any controls. Signed-off-by: Paul Elder --- src/ipa/rkisp1/algorithms/lux.cpp | 76 +++++++++++++++++++++++++++ src/ipa/rkisp1/algorithms/lux.h | 39 ++++++++++++++ src/ipa/rkisp1/algorithms/meson.build | 1 + src/ipa/rkisp1/ipa_context.h | 1 + 4 files changed, 117 insertions(+) create mode 100644 src/ipa/rkisp1/algorithms/lux.cpp create mode 100644 src/ipa/rkisp1/algorithms/lux.h diff --git a/src/ipa/rkisp1/algorithms/lux.cpp b/src/ipa/rkisp1/algorithms/lux.cpp new file mode 100644 index 00000000..1816ddb2 --- /dev/null +++ b/src/ipa/rkisp1/algorithms/lux.cpp @@ -0,0 +1,76 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Ideas On Board + * + * lux.cpp - RkISP1 Lux control + */ + +#include "lux.h" + +#include +#include + +#include + +#include + +#include "libcamera/internal/yaml_parser.h" + +#include "libipa/histogram.h" +#include "libipa/lux.h" + +/** + * \file lux.h + */ + +namespace libcamera { + +namespace ipa::rkisp1::algorithms { + +/** + * \class Lux + * \brief RkISP1 Lux control + * + * The Lux algorithm is responsible for estimating the lux level of the image. + * It doesn't take or generate any controls, but it provides a lux level for + * other algorithms (such as AGC) to use. + */ + +LOG_DEFINE_CATEGORY(RkISP1Lux) + +/** + * \copydoc libcamera::ipa::Algorithm::init + */ +int Lux::init([[maybe_unused]] IPAContext &context, const YamlObject &tuningData) +{ + return lux_.readYaml(tuningData); +} + +/** + * \copydoc libcamera::ipa::Algorithm::process + */ +void Lux::process(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, + const rkisp1_stat_buffer *stats, + ControlList &metadata) +{ + utils::Duration exposureTime = context.configuration.sensor.lineDuration + * frameContext.sensor.exposure; + double gain = frameContext.sensor.gain; + + const rkisp1_cif_isp_stat *params = &stats->params; + Histogram yHist = Histogram(Span(params->hist.hist_bins, + context.hw->numHistogramBins), 4); + + /* todo Update this when we support aperture */ + double lux = lux_.process(gain, exposureTime, 1.0, yHist); + frameContext.agc.lux = lux; + metadata.set(controls::Lux, lux); +} + +REGISTER_IPA_ALGORITHM(Lux, "Lux") + +} /* namespace ipa::rkisp1::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/rkisp1/algorithms/lux.h b/src/ipa/rkisp1/algorithms/lux.h new file mode 100644 index 00000000..ea98c291 --- /dev/null +++ b/src/ipa/rkisp1/algorithms/lux.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Ideas On Board + * + * lux.h - RkISP1 Lux control + */ + +#pragma once + +#include + +#include "libcamera/internal/yaml_parser.h" + +#include "libipa/lux.h" + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::rkisp1::algorithms { + +class Lux : public Algorithm +{ +public: + Lux() = default; + ~Lux() = default; + + int init(IPAContext &context, const YamlObject &tuningData) override; + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const rkisp1_stat_buffer *stats, + ControlList &metadata) override; + +private: + ipa::Lux lux_; +}; + +} /* namespace ipa::rkisp1::algorithms */ +} /* namespace libcamera */ diff --git a/src/ipa/rkisp1/algorithms/meson.build b/src/ipa/rkisp1/algorithms/meson.build index c9891e87..b0381d5f 100644 --- a/src/ipa/rkisp1/algorithms/meson.build +++ b/src/ipa/rkisp1/algorithms/meson.build @@ -11,4 +11,5 @@ rkisp1_ipa_algorithms = files([ 'filter.cpp', 'gsl.cpp', 'lsc.cpp', + 'lux.cpp', ]) diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index dc876da0..c39e0e9b 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -118,6 +118,7 @@ struct IPAFrameContext : public FrameContext { int32_t exposureMode; int32_t constraintMode; utils::Duration maxShutterSpeed; + double lux; } agc; struct { From patchwork Fri Apr 12 09:17:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 19867 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5609DC3285 for ; Fri, 12 Apr 2024 09:17:21 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id F15E663360; Fri, 12 Apr 2024 11:17:20 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="bmxVlSm2"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A52976334D for ; Fri, 12 Apr 2024 11:17:18 +0200 (CEST) Received: from pyrite.hamster-moth.ts.net (h175-177-049-156.catv02.itscom.jp [175.177.49.156]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 27EC6D7E; Fri, 12 Apr 2024 11:16:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1712913395; bh=F4b1sisvNS2PUQCIik4PqmaTBBn9n/VFZAshk+gxS4M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bmxVlSm2pjn0AoSHUq2IMmoBszUXWBHbE0k8Wo2/w4qtMMQ5DniTVBXqWrT78YzYe 5jtCRgBT1Whrui0bfDqCUoooun4NBksCCI56lL4hMKVDJTPRhf5XuAtgfb9zZz7kI6 cA4Jqn/i37oCOzKLSD2poKl+3wawLoEHI5tHvWFQ= From: Paul Elder To: libcamera-devel@lists.libcamera.org Cc: Paul Elder Subject: [PATCH 3/3] ipa: rkisp1: agc: Plumb lux Date: Fri, 12 Apr 2024 18:17:00 +0900 Message-Id: <20240412091700.1817754-4-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240412091700.1817754-1-paul.elder@ideasonboard.com> References: <20240412091700.1817754-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that the rkisp1 IPA has a lux algorithm module, use the lux value that comes from that as opposed to hardcoding it. Signed-off-by: Paul Elder --- src/ipa/rkisp1/algorithms/agc.cpp | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index ed4d6330..5371ae48 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -424,8 +424,7 @@ void Agc::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, double analogueGain = frameContext.sensor.gain; utils::Duration effectiveExposureValue = exposureTime * analogueGain; - /* \todo Plumb in the lux value. Requires a lux algo + tuning */ - double lux = 400; + double lux = frameContext.agc.lux; utils::Duration shutterTime; double aGain, dGain;