From patchwork Wed Sep 4 07:44:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21161 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 99F0DC324C for ; Wed, 4 Sep 2024 07:45:17 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 42065634E9; Wed, 4 Sep 2024 09:45:17 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="KKizoD0i"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 48550634CB for ; Wed, 4 Sep 2024 09:45:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725435913; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xe9EiWkpEvv3s1Xx1vmcLBWhu8z0mKiMuU/c2VecRKs=; b=KKizoD0i6FgAskeQkvCPcit9lWyKR3DB5bJdJHmXz8MDrsB6ner45gYbEswi+BCQlcENmF ryuGx4/LRgHmbQJ+J75Ed+GX0kgKVSl+QTVN4846rGXZ0eF3h468b0rTBjS+W8b8Cwyvmb 8ILX7qmPrZL/+JZ8fTi6Qe82J4vfkIw= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-70-s5wBryNwNO67VyqUo3tr9Q-1; Wed, 04 Sep 2024 03:45:11 -0400 X-MC-Unique: s5wBryNwNO67VyqUo3tr9Q-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id C57191955E9D for ; Wed, 4 Sep 2024 07:45:10 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.152]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id A95F91956088; Wed, 4 Sep 2024 07:45:09 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 1/2] libcamera: software_isp: Add contrast algorithm Date: Wed, 4 Sep 2024 09:44:58 +0200 Message-ID: <20240904074500.106019-2-mzamazal@redhat.com> In-Reply-To: <20240904074500.106019-1-mzamazal@redhat.com> References: <20240904074500.106019-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 127.0.0.2 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Software ISP is currently fully automatic and doesn't allow image modifications by explicitly set control values. The user has no means to make the image looking better. This patch introduces contrast control algorithm, which can improve e.g. a flat looking image. Based on the provided contrast value with a range 0..infinity and 1.0 being the normal value, it applies a simple S-curve modification to the image. The contrast algorithm just handles the provided values, while the S-curve is applied in the gamma algorithm on the computed gamma curve whenever the contrast value changes. Since the algorithm is applied only on the lookup table already present, its overhead is negligible. This is a preparation patch without actually activating the contrast algorithm, which will be done in the following patch. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/contrast.cpp | 45 ++++++++++++++++++++++++++ src/ipa/simple/algorithms/contrast.h | 37 +++++++++++++++++++++ src/ipa/simple/algorithms/gamma.cpp | 22 ++++++++++--- src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/ipa_context.h | 4 +++ 5 files changed, 104 insertions(+), 5 deletions(-) create mode 100644 src/ipa/simple/algorithms/contrast.cpp create mode 100644 src/ipa/simple/algorithms/contrast.h diff --git a/src/ipa/simple/algorithms/contrast.cpp b/src/ipa/simple/algorithms/contrast.cpp new file mode 100644 index 00000000..75bf37ad --- /dev/null +++ b/src/ipa/simple/algorithms/contrast.cpp @@ -0,0 +1,45 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Contrast adjustment + */ + +#include "contrast.h" + +#include + +#include + +#include "control_ids.h" + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoftContrast) + +namespace ipa::soft::algorithms { + +int Contrast::configure(typename Module::Context &context, + [[maybe_unused]] const typename Module::Config &configInfo) +{ + context.activeState.knobs.contrast = std::optional(); + return 0; +} + +void Contrast::queueRequest(typename Module::Context &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] typename Module::FrameContext &frameContext, + const ControlList &controls) +{ + const auto &contrast = controls.get(controls::Contrast); + if (contrast.has_value()) { + context.activeState.knobs.contrast = contrast; + LOG(IPASoftContrast, Debug) << "Setting contrast to" << contrast.value(); + } +} + +REGISTER_IPA_ALGORITHM(Contrast, "Contrast") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/contrast.h b/src/ipa/simple/algorithms/contrast.h new file mode 100644 index 00000000..0b393309 --- /dev/null +++ b/src/ipa/simple/algorithms/contrast.h @@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Contrast adjustment + */ + +#pragma once + +#include + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class Contrast : public Algorithm +{ +public: + Contrast() = default; + ~Contrast() = default; + + int configure(typename Module::Context &context, + const typename Module::Config &configInfo) + override; + + void queueRequest(typename Module::Context &context, + const uint32_t frame, + typename Module::FrameContext &frameContext, + const ControlList &controls) + override; +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/gamma.cpp b/src/ipa/simple/algorithms/gamma.cpp index b03dff25..fe59ca9f 100644 --- a/src/ipa/simple/algorithms/gamma.cpp +++ b/src/ipa/simple/algorithms/gamma.cpp @@ -32,15 +32,26 @@ int Gamma::init(IPAContext &context, void Gamma::updateGammaTable(IPAContext &context) { auto &gammaTable = context.activeState.gamma.gammaTable; - auto blackLevel = context.activeState.black.level; + const auto blackLevel = context.activeState.black.level; const unsigned int blackIndex = blackLevel * IPAActiveState::kGammaLookupSize; + const auto contrast = context.activeState.knobs.contrast.value_or(1.0); + std::fill(gammaTable.begin(), gammaTable.begin() + blackIndex, 0); const float divisor = kGammaLookupSize - blackIndex - 1.0; - for (unsigned int i = blackIndex; i < kGammaLookupSize; i++) - gammaTable[i] = UINT8_MAX * powf((i - blackIndex) / divisor, - context.configuration.gamma); + for (unsigned int i = blackIndex; i < kGammaLookupSize; i++) { + double normalized = (i - blackIndex) / divisor; + /* Apply simple S-curve */ + if (normalized < 0.5) + normalized = 0.5 * std::pow(normalized / 0.5, contrast); + else + normalized = 1.0 - 0.5 * std::pow((1.0 - normalized) / 0.5, contrast); + gammaTable[i] = UINT8_MAX * + std::pow(normalized, context.configuration.gamma); + } + context.activeState.gamma.blackLevel = blackLevel; + context.activeState.gamma.contrast = contrast; } void Gamma::prepare(IPAContext &context, @@ -53,7 +64,8 @@ void Gamma::prepare(IPAContext &context, * since the black level gets updated only if a lower value is observed, * it's not permanently prone to minor fluctuations or rounding errors. */ - if (context.activeState.gamma.blackLevel != context.activeState.black.level) + if (context.activeState.gamma.blackLevel != context.activeState.black.level || + context.activeState.gamma.contrast != context.activeState.knobs.contrast) updateGammaTable(context); } diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index ec2638c2..73dc077e 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -4,6 +4,7 @@ soft_simple_ipa_algorithms = files([ 'awb.cpp', 'agc.cpp', 'blc.cpp', + 'contrast.cpp', 'gamma.cpp', 'lut.cpp', ]) diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index 08f965f4..6655baa9 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -46,7 +46,11 @@ struct IPAActiveState { struct { std::array gammaTable; double blackLevel; + double contrast; } gamma; + struct { + std::optional contrast; // 0..inf, 1 = neutral + } knobs; }; struct IPAFrameContext : public FrameContext { From patchwork Wed Sep 4 07:44:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21162 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9C54FC3259 for ; Wed, 4 Sep 2024 07:45:19 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 09E34634E8; Wed, 4 Sep 2024 09:45:18 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="FLb8WV6X"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DEC35634E0 for ; Wed, 4 Sep 2024 09:45:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725435914; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DB6a3dkmx0CJJGhia06QHcCu9v3Uh2fcKoQn/pYQ2EU=; b=FLb8WV6XItf4s66nUxe89gbHFlE7KMEIFDl3HaYf6SiZJbatJ7p9wJ4BkOfLP0Jj8EedCN fQq9MoHvaEp2dHcO6YGBvrILjpNMZ1Hkp4TRl3M2TZO3wcmrYgIntifLq3xw9/30Lt+1Sh fRaLzIZR0CY0bqs7H7qLtFyt/OulIfA= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-672-FvTbrNpZPayo4M4PvdZV8g-1; Wed, 04 Sep 2024 03:45:13 -0400 X-MC-Unique: FvTbrNpZPayo4M4PvdZV8g-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 86BB51953956 for ; Wed, 4 Sep 2024 07:45:12 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.152]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 68CD41956086; Wed, 4 Sep 2024 07:45:11 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 2/2] libcamera: software_isp: Add contrast control Date: Wed, 4 Sep 2024 09:44:59 +0200 Message-ID: <20240904074500.106019-3-mzamazal@redhat.com> In-Reply-To: <20240904074500.106019-1-mzamazal@redhat.com> References: <20240904074500.106019-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 127.0.0.2 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch introduces support for applying runtime controls to software ISP. It enables the contrast algorithm as the first control that can be used. Signed-off-by: Milan Zamazal --- .../libcamera/internal/software_isp/software_isp.h | 3 ++- include/libcamera/ipa/soft.mojom | 2 +- src/ipa/simple/data/uncalibrated.yaml | 1 + src/ipa/simple/soft_simple.cpp | 13 +++++++++++-- src/libcamera/pipeline/simple/simple.cpp | 2 +- src/libcamera/software_isp/software_isp.cpp | 8 ++++++-- 6 files changed, 22 insertions(+), 7 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index a3e3a9da..d51b03fd 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -46,7 +46,8 @@ LOG_DECLARE_CATEGORY(SoftwareIsp) class SoftwareIsp { public: - SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor); + SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor, + ControlInfoMap *ipaControls); ~SoftwareIsp(); int loadConfiguration([[maybe_unused]] const std::string &filename) { return 0; } diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index bc70d4e4..3b8a72a2 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -17,7 +17,7 @@ interface IPASoftInterface { libcamera.SharedFD fdStats, libcamera.SharedFD fdParams, libcamera.ControlInfoMap sensorCtrlInfoMap) - => (int32 ret); + => (int32 ret, libcamera.ControlInfoMap ipaControls); start() => (int32 ret); stop(); configure(IPAConfigInfo configInfo) diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index 324f98f4..2ed24e8c 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -4,6 +4,7 @@ version: 1 algorithms: - BlackLevel: + - Contrast: - Gamma: - Awb: - Lut: diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index ac7a22b7..d6c7ff7f 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -34,6 +34,10 @@ LOG_DEFINE_CATEGORY(IPASoft) namespace ipa::soft { +const ControlInfoMap::Map swispControls{ + { &controls::Contrast, ControlInfo(0.0f, 10.0f, 1.0f) }, +}; + /* Maximum number of frame contexts to be held */ static constexpr uint32_t kMaxFrameContexts = 16; @@ -50,7 +54,8 @@ public: int init(const IPASettings &settings, const SharedFD &fdStats, const SharedFD &fdParams, - const ControlInfoMap &sensorInfoMap) override; + const ControlInfoMap &sensorInfoMap, + ControlInfoMap *ipaControls) override; int configure(const IPAConfigInfo &configInfo) override; int start() override; @@ -87,7 +92,8 @@ IPASoftSimple::~IPASoftSimple() int IPASoftSimple::init(const IPASettings &settings, const SharedFD &fdStats, const SharedFD &fdParams, - const ControlInfoMap &sensorInfoMap) + const ControlInfoMap &sensorInfoMap, + ControlInfoMap *ipaControls) { camHelper_ = CameraSensorHelperFactoryBase::create(settings.sensorModel); if (!camHelper_) { @@ -158,6 +164,9 @@ int IPASoftSimple::init(const IPASettings &settings, stats_ = static_cast(mem); } + ControlInfoMap::Map ctrlMap = swispControls; + *ipaControls = ControlInfoMap(std::move(ctrlMap), controls::controls); + /* * Check if the sensor driver supports the controls required by the * Soft IPA. diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index fc80e665..86c4447e 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -531,7 +531,7 @@ int SimpleCameraData::init() * Instantiate Soft ISP if this is enabled for the given driver and no converter is used. */ if (!converter_ && pipe->swIspEnabled()) { - swIsp_ = std::make_unique(pipe, sensor_.get()); + swIsp_ = std::make_unique(pipe, sensor_.get(), &controlInfo_); if (!swIsp_->isValid()) { LOG(SimplePipeline, Warning) << "Failed to create software ISP, disabling software debayering"; diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index dbf27f31..78b78bab 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -13,6 +13,7 @@ #include #include +#include #include #include #include @@ -61,9 +62,11 @@ LOG_DEFINE_CATEGORY(SoftwareIsp) * \brief Constructs SoftwareIsp object * \param[in] pipe The pipeline handler in use * \param[in] sensor Pointer to the CameraSensor instance owned by the pipeline + * \param[out] ipaControls The IPA controls to update * handler */ -SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor) +SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor, + ControlInfoMap *ipaControls) : dmaHeap_(DmaBufAllocator::DmaBufAllocatorFlag::CmaHeap | DmaBufAllocator::DmaBufAllocatorFlag::SystemHeap | DmaBufAllocator::DmaBufAllocatorFlag::UDmaBuf) @@ -125,7 +128,8 @@ SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor) int ret = ipa_->init(IPASettings{ ipaTuningFile, sensor->model() }, debayer_->getStatsFD(), sharedParams_.fd(), - sensor->controls()); + sensor->controls(), + ipaControls); if (ret) { LOG(SoftwareIsp, Error) << "IPA init failed"; debayer_.reset();