From patchwork Tue Mar 19 12:35:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19766 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C0993C3272 for ; Tue, 19 Mar 2024 12:37:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 58FBD62DD1; Tue, 19 Mar 2024 13:37:24 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="bUfl3D5z"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C61AC62D37 for ; Tue, 19 Mar 2024 13:37:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851841; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KDlz08+dcNRxNN1chjBqyd09aH8j/whCCEHQRUTGoKs=; b=bUfl3D5zyhM9sBzhWvPGRPrqdgmKfGElR93TYh/NDVI8FPcTbbv1mhsqUGkbfNgihE3Pb2 3HQSoekDnkfLdFzkEUydzgiL3xc1dxvBaNonGqAUalcPTjw7pSrkALx80a0BsRoYxyaMex OXisUQfey2CqWGBlR7DRRIUCSev1TyM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-590-nVsfXiVLOs6qbMvIj8vrZA-1; Tue, 19 Mar 2024 08:37:18 -0400 X-MC-Unique: nVsfXiVLOs6qbMvIj8vrZA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 84815101A588; Tue, 19 Mar 2024 12:37:17 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2C8F21C060A4; Tue, 19 Mar 2024 12:37:15 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Andrey Konovalov , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Hans de Goede , Kieran Bingham , Dennis Bonke , Marttico , Toon Langendam Subject: [PATCH v6 09/18] libcamera: ipa: add Soft IPA Date: Tue, 19 Mar 2024 13:35:56 +0100 Message-ID: <20240319123622.675599-10-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov Define the Soft IPA main and event interfaces, add the Soft IPA implementation. The current src/ipa/meson.build assumes the IPA name to match the pipeline name. For this reason "-Dipas=simple" is used for the Soft IPA module. Auto exposure/gain and AWB implementation by Dennis, Toon and Martti. Auto exposure/gain targets a Mean Sample Value of 2.5 following the MSV calculation algorithm from: https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Signed-off-by: Andrey Konovalov Co-developed-by: Dennis Bonke Signed-off-by: Dennis Bonke Co-developed-by: Marttico Signed-off-by: Marttico Co-developed-by: Toon Langendam Signed-off-by: Toon Langendam Signed-off-by: Hans de Goede --- Documentation/Doxyfile.in | 1 + include/libcamera/ipa/meson.build | 1 + include/libcamera/ipa/soft.mojom | 28 +++ meson_options.txt | 2 +- src/ipa/simple/data/meson.build | 9 + src/ipa/simple/data/soft.conf | 3 + src/ipa/simple/meson.build | 25 +++ src/ipa/simple/soft_simple.cpp | 326 ++++++++++++++++++++++++++++++ 8 files changed, 394 insertions(+), 1 deletion(-) create mode 100644 include/libcamera/ipa/soft.mojom create mode 100644 src/ipa/simple/data/meson.build create mode 100644 src/ipa/simple/data/soft.conf create mode 100644 src/ipa/simple/meson.build create mode 100644 src/ipa/simple/soft_simple.cpp diff --git a/Documentation/Doxyfile.in b/Documentation/Doxyfile.in index a86ea6c1..2be8d47b 100644 --- a/Documentation/Doxyfile.in +++ b/Documentation/Doxyfile.in @@ -44,6 +44,7 @@ EXCLUDE = @TOP_SRCDIR@/include/libcamera/base/span.h \ @TOP_SRCDIR@/src/libcamera/pipeline/ \ @TOP_SRCDIR@/src/libcamera/tracepoints.cpp \ @TOP_BUILDDIR@/include/libcamera/internal/tracepoints.h \ + @TOP_BUILDDIR@/include/libcamera/ipa/soft_ipa_interface.h \ @TOP_BUILDDIR@/src/libcamera/proxy/ EXCLUDE_PATTERNS = @TOP_BUILDDIR@/include/libcamera/ipa/*_serializer.h \ diff --git a/include/libcamera/ipa/meson.build b/include/libcamera/ipa/meson.build index f3b4881c..3352d08f 100644 --- a/include/libcamera/ipa/meson.build +++ b/include/libcamera/ipa/meson.build @@ -65,6 +65,7 @@ pipeline_ipa_mojom_mapping = { 'ipu3': 'ipu3.mojom', 'rkisp1': 'rkisp1.mojom', 'rpi/vc4': 'raspberrypi.mojom', + 'simple': 'soft.mojom', 'vimc': 'vimc.mojom', } diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom new file mode 100644 index 00000000..c249bd75 --- /dev/null +++ b/include/libcamera/ipa/soft.mojom @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ + +/* + * \todo Document the interface and remove the related EXCLUDE_PATTERNS entry. + */ + +module ipa.soft; + +import "include/libcamera/ipa/core.mojom"; + +interface IPASoftInterface { + init(libcamera.IPASettings settings, + libcamera.SharedFD fdStats, + libcamera.SharedFD fdParams, + libcamera.ControlInfoMap sensorCtrlInfoMap) + => (int32 ret); + start() => (int32 ret); + stop(); + configure(libcamera.ControlInfoMap sensorCtrlInfoMap) + => (int32 ret); + + [async] processStats(libcamera.ControlList sensorControls); +}; + +interface IPASoftEventInterface { + setSensorControls(libcamera.ControlList sensorControls); + setIspParams(int32 dummy); +}; diff --git a/meson_options.txt b/meson_options.txt index 99dab96d..2644bef0 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -27,7 +27,7 @@ option('gstreamer', option('ipas', type : 'array', - choices : ['ipu3', 'rkisp1', 'rpi/vc4', 'vimc'], + choices : ['ipu3', 'rkisp1', 'rpi/vc4', 'simple', 'vimc'], description : 'Select which IPA modules to build') option('lc-compliance', diff --git a/src/ipa/simple/data/meson.build b/src/ipa/simple/data/meson.build new file mode 100644 index 00000000..33548cc6 --- /dev/null +++ b/src/ipa/simple/data/meson.build @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: CC0-1.0 + +conf_files = files([ + 'soft.conf', +]) + +install_data(conf_files, + install_dir : ipa_data_dir / 'soft', + install_tag : 'runtime') diff --git a/src/ipa/simple/data/soft.conf b/src/ipa/simple/data/soft.conf new file mode 100644 index 00000000..0c70e7c0 --- /dev/null +++ b/src/ipa/simple/data/soft.conf @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: LGPL-2.1-or-later +# +# Dummy configuration file for the soft IPA. diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build new file mode 100644 index 00000000..3e863db7 --- /dev/null +++ b/src/ipa/simple/meson.build @@ -0,0 +1,25 @@ +# SPDX-License-Identifier: CC0-1.0 + +ipa_name = 'ipa_soft_simple' + +mod = shared_module(ipa_name, + ['soft_simple.cpp', libcamera_generated_ipa_headers], + name_prefix : '', + include_directories : [ipa_includes, libipa_includes], + dependencies : libcamera_private, + link_with : libipa, + install : true, + install_dir : ipa_install_dir) + +if ipa_sign_module + custom_target(ipa_name + '.so.sign', + input : mod, + output : ipa_name + '.so.sign', + command : [ipa_sign, ipa_priv_key, '@INPUT@', '@OUTPUT@'], + install : false, + build_by_default : true) +endif + +subdir('data') + +ipa_names += ipa_name diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp new file mode 100644 index 00000000..312df4ba --- /dev/null +++ b/src/ipa/simple/soft_simple.cpp @@ -0,0 +1,326 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * + * soft_simple.cpp - Simple Software Image Processing Algorithm module + */ + +#include + +#include +#include +#include + +#include +#include + +#include +#include +#include + +#include "libcamera/internal/camera_sensor.h" +#include "libcamera/internal/software_isp/debayer_params.h" +#include "libcamera/internal/software_isp/swisp_stats.h" + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoft) + +namespace ipa::soft { + +class IPASoftSimple : public ipa::soft::IPASoftInterface +{ +public: + IPASoftSimple() + : params_(static_cast(MAP_FAILED)), + stats_(static_cast(MAP_FAILED)), ignore_updates_(0) + { + } + + ~IPASoftSimple() + { + if (stats_ != MAP_FAILED) + munmap(stats_, sizeof(SwIspStats)); + if (params_ != MAP_FAILED) + munmap(params_, sizeof(DebayerParams)); + } + + int init(const IPASettings &settings, + const SharedFD &fdStats, + const SharedFD &fdParams, + const ControlInfoMap &sensorInfoMap) override; + int configure(const ControlInfoMap &sensorInfoMap) override; + + int start() override; + void stop() override; + + void processStats(const ControlList &sensorControls) override; + +private: + void updateExposure(double exposureMSV); + + SharedFD fdStats_; + SharedFD fdParams_; + DebayerParams *params_; + SwIspStats *stats_; + + int32_t exposure_min_, exposure_max_; + int32_t again_min_, again_max_; + int32_t again_, exposure_; + unsigned int ignore_updates_; +}; + +int IPASoftSimple::init([[maybe_unused]] const IPASettings &settings, + const SharedFD &fdStats, + const SharedFD &fdParams, + const ControlInfoMap &sensorInfoMap) +{ + fdStats_ = fdStats; + if (!fdStats_.isValid()) { + LOG(IPASoft, Error) << "Invalid Statistics handle"; + return -ENODEV; + } + + fdParams_ = fdParams; + if (!fdParams_.isValid()) { + LOG(IPASoft, Error) << "Invalid Parameters handle"; + return -ENODEV; + } + + params_ = static_cast(mmap(nullptr, sizeof(DebayerParams), + PROT_WRITE, MAP_SHARED, + fdParams_.get(), 0)); + if (params_ == MAP_FAILED) { + LOG(IPASoft, Error) << "Unable to map Parameters"; + return -errno; + } + + stats_ = static_cast(mmap(nullptr, sizeof(SwIspStats), + PROT_READ, MAP_SHARED, + fdStats_.get(), 0)); + if (stats_ == MAP_FAILED) { + LOG(IPASoft, Error) << "Unable to map Statistics"; + return -errno; + } + + if (sensorInfoMap.find(V4L2_CID_EXPOSURE) == sensorInfoMap.end()) { + LOG(IPASoft, Error) << "Don't have exposure control"; + return -EINVAL; + } + + if (sensorInfoMap.find(V4L2_CID_ANALOGUE_GAIN) == sensorInfoMap.end()) { + LOG(IPASoft, Error) << "Don't have gain control"; + return -EINVAL; + } + + return 0; +} + +int IPASoftSimple::configure(const ControlInfoMap &sensorInfoMap) +{ + const ControlInfo &exposure_info = sensorInfoMap.find(V4L2_CID_EXPOSURE)->second; + const ControlInfo &gain_info = sensorInfoMap.find(V4L2_CID_ANALOGUE_GAIN)->second; + + exposure_min_ = exposure_info.min().get(); + exposure_max_ = exposure_info.max().get(); + if (!exposure_min_) { + LOG(IPASoft, Warning) << "Minimum exposure is zero, that can't be linear"; + exposure_min_ = 1; + } + + again_min_ = gain_info.min().get(); + again_max_ = gain_info.max().get(); + /* + * The camera sensor gain (g) is usually not equal to the value written + * into the gain register (x). But the way how the AGC algorithm changes + * the gain value to make the total exposure closer to the optimum assumes + * that g(x) is not too far from linear function. If the minimal gain is 0, + * the g(x) is likely to be far from the linear, like g(x) = a / (b * x + c). + * To avoid unexpected changes to the gain by the AGC algorithm (abrupt near + * one edge, and very small near the other) we limit the range of the gain + * values used. + */ + if (!again_min_) { + LOG(IPASoft, Warning) << "Minimum gain is zero, that can't be linear"; + again_min_ = std::min(100, again_min_ / 2 + again_max_ / 2); + } + + LOG(IPASoft, Info) << "Exposure " << exposure_min_ << "-" << exposure_max_ + << ", gain " << again_min_ << "-" << again_max_; + + return 0; +} + +int IPASoftSimple::start() +{ + return 0; +} + +void IPASoftSimple::stop() +{ +} + +/* + * The number of bins to use for the optimal exposure calculations. + */ +static constexpr unsigned int kExposureBinsCount = 5; +/* + * The exposure is optimal when the mean sample value of the histogram is + * in the middle of the range. + */ +static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; +/* + * The below value implements the hysteresis for the exposure adjustment. + * It is small enough to have the exposure close to the optimal, and is big + * enough to prevent the exposure from wobbling around the optimal value. + */ +static constexpr float kExposureSatisfactory = 0.2; + +void IPASoftSimple::processStats(const ControlList &sensorControls) +{ + /* + * Calculate red and blue gains for AWB. + * Clamp max gain at 4.0, this also avoids 0 division. + */ + if (stats_->sumR_ <= stats_->sumG_ / 4) + params_->gainR = 1024; + else + params_->gainR = 256 * stats_->sumG_ / stats_->sumR_; + + if (stats_->sumB_ <= stats_->sumG_ / 4) + params_->gainB = 1024; + else + params_->gainB = 256 * stats_->sumG_ / stats_->sumB_; + + /* Green gain and gamma values are fixed */ + params_->gainG = 256; + params_->gamma = 0.5; + + setIspParams.emit(0); + + /* + * AE / AGC, use 2 frames delay to make sure that the exposure and + * the gain set have applied to the camera sensor. + */ + if (ignore_updates_ > 0) { + --ignore_updates_; + return; + } + + /* + * Calculate Mean Sample Value (MSV) according to formula from: + * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf + */ + constexpr unsigned int yHistValsPerBin = + SwIspStats::kYHistogramSize / kExposureBinsCount; + constexpr unsigned int yHistValsPerBinMod = + SwIspStats::kYHistogramSize / + (SwIspStats::kYHistogramSize % kExposureBinsCount + 1); + int ExposureBins[kExposureBinsCount] = {}; + unsigned int denom = 0; + unsigned int num = 0; + + for (unsigned int i = 0; i < SwIspStats::kYHistogramSize; i++) { + unsigned int idx = (i - (i / yHistValsPerBinMod)) / yHistValsPerBin; + ExposureBins[idx] += stats_->yHistogram[i]; + } + + for (unsigned int i = 0; i < kExposureBinsCount; i++) { + LOG(IPASoft, Debug) << i << ": " << ExposureBins[i]; + denom += ExposureBins[i]; + num += ExposureBins[i] * (i + 1); + } + + float exposureMSV = (float)num / denom; + + /* sanity check */ + if (!sensorControls.contains(V4L2_CID_EXPOSURE) || + !sensorControls.contains(V4L2_CID_ANALOGUE_GAIN)) { + LOG(IPASoft, Error) << "Control(s) missing"; + return; + } + + ControlList ctrls(sensorControls); + + exposure_ = ctrls.get(V4L2_CID_EXPOSURE).get(); + again_ = ctrls.get(V4L2_CID_ANALOGUE_GAIN).get(); + + updateExposure(exposureMSV); + + ctrls.set(V4L2_CID_EXPOSURE, exposure_); + ctrls.set(V4L2_CID_ANALOGUE_GAIN, again_); + + ignore_updates_ = 2; + + setSensorControls.emit(ctrls); + + LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV + << " exp " << exposure_ << " again " << again_ + << " gain R/B " << params_->gainR << "/" << params_->gainB; +} + +void IPASoftSimple::updateExposure(double exposureMSV) +{ + /* DENOMINATOR of 10 gives ~10% increment/decrement; DENOMINATOR of 5 - about ~20% */ + static constexpr uint8_t kExpDenominator = 10; + static constexpr uint8_t kExpNumeratorUp = kExpDenominator + 1; + static constexpr uint8_t kExpNumeratorDown = kExpDenominator - 1; + + int next; + + if (exposureMSV < kExposureOptimal - kExposureSatisfactory) { + next = exposure_ * kExpNumeratorUp / kExpDenominator; + if (next - exposure_ < 1) + exposure_ += 1; + else + exposure_ = next; + if (exposure_ >= exposure_max_) { + next = again_ * kExpNumeratorUp / kExpDenominator; + if (next - again_ < 1) + again_ += 1; + else + again_ = next; + } + } + + if (exposureMSV > kExposureOptimal + kExposureSatisfactory) { + if (exposure_ == exposure_max_ && again_ != again_min_) { + next = again_ * kExpNumeratorDown / kExpDenominator; + if (again_ - next < 1) + again_ -= 1; + else + again_ = next; + } else { + next = exposure_ * kExpNumeratorDown / kExpDenominator; + if (exposure_ - next < 1) + exposure_ -= 1; + else + exposure_ = next; + } + } + + exposure_ = std::clamp(exposure_, exposure_min_, exposure_max_); + again_ = std::clamp(again_, again_min_, again_max_); +} + +} /* namespace ipa::soft */ + +/* + * External IPA module interface + */ +extern "C" { +const struct IPAModuleInfo ipaModuleInfo = { + IPA_MODULE_API_VERSION, + 0, + "SimplePipelineHandler", + "simple", +}; + +IPAInterface *ipaCreate() +{ + return new ipa::soft::IPASoftSimple(); +} + +} /* extern "C" */ + +} /* namespace libcamera */