From patchwork Wed Oct 9 19:33:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21569 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 94C9FC32DE for ; Wed, 9 Oct 2024 19:33:39 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4331D65370; Wed, 9 Oct 2024 21:33:39 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="WSQgfGqe"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5033163538 for ; Wed, 9 Oct 2024 21:33:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1728502416; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4s/TcOik+viXMR7VDGt05/Gtrx4cZxUaxgVi26AI9U4=; b=WSQgfGqeFiK4jFdSCIEn+XPxm9dRyIaqq7I3ycnzwXxOFDRSoQwsUn+WDClLg6+KD6LIrj IKbA1Zs0/TiGWabpcqupWLDhfto4BBhMzrGK5oH7Q06TV9bSep4E4Rr5T8uQtY5dNhsncP t8RkKpQN+4XCsQECw1CLYCLPqecQxh4= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-140-4RhMX9aCNpyD-Pr3Q_07Rw-1; Wed, 09 Oct 2024 15:33:32 -0400 X-MC-Unique: 4RhMX9aCNpyD-Pr3Q_07Rw-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EF95F19560A1; Wed, 9 Oct 2024 19:33:31 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.78]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 651EC300018D; Wed, 9 Oct 2024 19:33:30 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Kieran Bingham Subject: [PATCH v2 1/2] libcamera: software_isp: Add contrast algorithm Date: Wed, 9 Oct 2024 21:33:16 +0200 Message-ID: <20241009193318.2394762-2-mzamazal@redhat.com> In-Reply-To: <20241009193318.2394762-1-mzamazal@redhat.com> References: <20241009193318.2394762-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Software ISP is currently fully automatic and doesn't allow image modifications by explicitly set control values. The user has no means to make the image looking better. This patch introduces contrast control algorithm, which can improve e.g. a flat looking image. Based on the provided contrast value with a range 0..infinity and 1.0 being the normal value, it applies a simple S-curve modification to the image. The contrast algorithm just handles the provided values, while the S-curve is applied in the gamma algorithm on the computed gamma curve whenever the contrast value changes. Since the algorithm is applied only on the lookup table already present, its overhead is negligible. This is a preparation patch without actually activating the contrast algorithm, which will be done in the following patch. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/contrast.cpp | 45 ++++++++++++++++++++++++++ src/ipa/simple/algorithms/contrast.h | 37 +++++++++++++++++++++ src/ipa/simple/algorithms/lut.cpp | 20 +++++++++--- src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/ipa_context.h | 7 ++++ 5 files changed, 105 insertions(+), 5 deletions(-) create mode 100644 src/ipa/simple/algorithms/contrast.cpp create mode 100644 src/ipa/simple/algorithms/contrast.h diff --git a/src/ipa/simple/algorithms/contrast.cpp b/src/ipa/simple/algorithms/contrast.cpp new file mode 100644 index 000000000..1a2c14cf9 --- /dev/null +++ b/src/ipa/simple/algorithms/contrast.cpp @@ -0,0 +1,45 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Contrast adjustment + */ + +#include "contrast.h" + +#include + +#include + +#include "control_ids.h" + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoftContrast) + +namespace ipa::soft::algorithms { + +int Contrast::configure(typename Module::Context &context, + [[maybe_unused]] const typename Module::Config &configInfo) +{ + context.activeState.knobs.contrast = std::optional(); + return 0; +} + +void Contrast::queueRequest(typename Module::Context &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] typename Module::FrameContext &frameContext, + const ControlList &controls) +{ + const auto &contrast = controls.get(controls::Contrast); + if (contrast.has_value()) { + context.activeState.knobs.contrast = contrast; + LOG(IPASoftContrast, Debug) << "Setting contrast to " << contrast.value(); + } +} + +REGISTER_IPA_ALGORITHM(Contrast, "Contrast") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/contrast.h b/src/ipa/simple/algorithms/contrast.h new file mode 100644 index 000000000..0b3933099 --- /dev/null +++ b/src/ipa/simple/algorithms/contrast.h @@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Contrast adjustment + */ + +#pragma once + +#include + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class Contrast : public Algorithm +{ +public: + Contrast() = default; + ~Contrast() = default; + + int configure(typename Module::Context &context, + const typename Module::Config &configInfo) + override; + + void queueRequest(typename Module::Context &context, + const uint32_t frame, + typename Module::FrameContext &frameContext, + const ControlList &controls) + override; +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/lut.cpp b/src/ipa/simple/algorithms/lut.cpp index 9744e773a..8ebc824ce 100644 --- a/src/ipa/simple/algorithms/lut.cpp +++ b/src/ipa/simple/algorithms/lut.cpp @@ -32,16 +32,25 @@ int Lut::configure(IPAContext &context, void Lut::updateGammaTable(IPAContext &context) { auto &gammaTable = context.activeState.gamma.gammaTable; - auto blackLevel = context.activeState.blc.level; + const auto blackLevel = context.activeState.blc.level; const unsigned int blackIndex = blackLevel * gammaTable.size() / 256; + const auto contrast = context.activeState.knobs.contrast.value_or(1.0); std::fill(gammaTable.begin(), gammaTable.begin() + blackIndex, 0); const float divisor = gammaTable.size() - blackIndex - 1.0; - for (unsigned int i = blackIndex; i < gammaTable.size(); i++) - gammaTable[i] = UINT8_MAX * std::pow((i - blackIndex) / divisor, - context.configuration.gamma); + for (unsigned int i = blackIndex; i < gammaTable.size(); i++) { + double normalized = (i - blackIndex) / divisor; + /* Apply simple S-curve */ + if (normalized < 0.5) + normalized = 0.5 * std::pow(normalized / 0.5, contrast); + else + normalized = 1.0 - 0.5 * std::pow((1.0 - normalized) / 0.5, contrast); + gammaTable[i] = UINT8_MAX * + std::pow(normalized, context.configuration.gamma); + } context.activeState.gamma.blackLevel = blackLevel; + context.activeState.gamma.contrast = contrast; } void Lut::prepare(IPAContext &context, @@ -55,7 +64,8 @@ void Lut::prepare(IPAContext &context, * observed, it's not permanently prone to minor fluctuations or * rounding errors. */ - if (context.activeState.gamma.blackLevel != context.activeState.blc.level) + if (context.activeState.gamma.blackLevel != context.activeState.blc.level || + context.activeState.gamma.contrast != context.activeState.knobs.contrast) updateGammaTable(context); auto &gains = context.activeState.gains; diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index 37a2eb534..d75a7b2a1 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -4,5 +4,6 @@ soft_simple_ipa_algorithms = files([ 'awb.cpp', 'agc.cpp', 'blc.cpp', + 'contrast.cpp', 'lut.cpp', ]) diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index 3519f20f6..5118d6abf 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -8,8 +8,11 @@ #pragma once #include +#include #include +#include + #include namespace libcamera { @@ -44,7 +47,11 @@ struct IPAActiveState { struct { std::array gammaTable; uint8_t blackLevel; + double contrast; } gamma; + struct { + std::optional contrast; // 0..inf, 1 = neutral + } knobs; }; struct IPAFrameContext : public FrameContext { From patchwork Wed Oct 9 19:33:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21570 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D7732C32DE for ; Wed, 9 Oct 2024 19:33:42 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 80BA365370; Wed, 9 Oct 2024 21:33:42 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="ImCYfx/2"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0813A65369 for ; Wed, 9 Oct 2024 21:33:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1728502418; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Wj8NCScrvNGUQPcZoBrVbd/ZyBzALfuB6ydcxuhkP6M=; b=ImCYfx/2pvOvglisgeYBM8EUGq+BR0dlZTbbRKmKG6de189QeV/MHuBkSXoLnYT5DT77Mb a2wEF1dKv4Nh+PTAVLx1s7KtpCPovrRigZ8B1Vw8Ltgk4T7K+zK20tfJCA4tmivjtvQ2eU PYW0AR+lBuyo1mLGaofRyTORcr2HdZ0= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-301-zt_a0tSlNV26mbWJ1aUnow-1; Wed, 09 Oct 2024 15:33:34 -0400 X-MC-Unique: zt_a0tSlNV26mbWJ1aUnow-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id E1EB31955F41; Wed, 9 Oct 2024 19:33:33 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.78]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 78D07300018D; Wed, 9 Oct 2024 19:33:32 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Kieran Bingham Subject: [PATCH v2 2/2] libcamera: software_isp: Add contrast control Date: Wed, 9 Oct 2024 21:33:17 +0200 Message-ID: <20241009193318.2394762-3-mzamazal@redhat.com> In-Reply-To: <20241009193318.2394762-1-mzamazal@redhat.com> References: <20241009193318.2394762-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch introduces support for applying runtime controls to software ISP. It enables the contrast algorithm as the first control that can be used. Signed-off-by: Milan Zamazal Reviewed-by: Kieran Bingham --- .../libcamera/internal/software_isp/software_isp.h | 3 ++- include/libcamera/ipa/soft.mojom | 2 +- src/ipa/simple/algorithms/contrast.cpp | 7 +++++++ src/ipa/simple/algorithms/contrast.h | 2 ++ src/ipa/simple/data/uncalibrated.yaml | 1 + src/ipa/simple/ipa_context.h | 1 + src/ipa/simple/soft_simple.cpp | 11 ++++++++--- src/libcamera/pipeline/simple/simple.cpp | 2 +- src/libcamera/software_isp/software_isp.cpp | 8 ++++++-- 9 files changed, 29 insertions(+), 8 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index a3e3a9da4..d51b03fd6 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -46,7 +46,8 @@ LOG_DECLARE_CATEGORY(SoftwareIsp) class SoftwareIsp { public: - SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor); + SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor, + ControlInfoMap *ipaControls); ~SoftwareIsp(); int loadConfiguration([[maybe_unused]] const std::string &filename) { return 0; } diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index 347fd69b4..16e6a7071 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -17,7 +17,7 @@ interface IPASoftInterface { libcamera.SharedFD fdStats, libcamera.SharedFD fdParams, libcamera.ControlInfoMap sensorCtrlInfoMap) - => (int32 ret); + => (int32 ret, libcamera.ControlInfoMap ipaControls); start() => (int32 ret); stop(); configure(IPAConfigInfo configInfo) diff --git a/src/ipa/simple/algorithms/contrast.cpp b/src/ipa/simple/algorithms/contrast.cpp index 1a2c14cf9..8d9997d81 100644 --- a/src/ipa/simple/algorithms/contrast.cpp +++ b/src/ipa/simple/algorithms/contrast.cpp @@ -19,6 +19,13 @@ LOG_DEFINE_CATEGORY(IPASoftContrast) namespace ipa::soft::algorithms { +int Contrast::init(IPAContext &context, + [[maybe_unused]] const YamlObject &tuningData) +{ + context.ctrlMap[&controls::Contrast] = ControlInfo(0.0f, 10.0f, 1.0f); + return 0; +} + int Contrast::configure(typename Module::Context &context, [[maybe_unused]] const typename Module::Config &configInfo) { diff --git a/src/ipa/simple/algorithms/contrast.h b/src/ipa/simple/algorithms/contrast.h index 0b3933099..405345acc 100644 --- a/src/ipa/simple/algorithms/contrast.h +++ b/src/ipa/simple/algorithms/contrast.h @@ -21,6 +21,8 @@ public: Contrast() = default; ~Contrast() = default; + int init(IPAContext &context, const YamlObject &tuningData) override; + int configure(typename Module::Context &context, const typename Module::Config &configInfo) override; diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index 3f1471121..c091e30fd 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -5,6 +5,7 @@ version: 1 algorithms: - BlackLevel: - Awb: + - Contrast: - Lut: - Agc: ... diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index 5118d6abf..095c83800 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -65,6 +65,7 @@ struct IPAContext { IPASessionConfiguration configuration; IPAActiveState activeState; FCQueue frameContexts; + ControlInfoMap::Map ctrlMap; }; } /* namespace ipa::soft */ diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index b28c7039f..eb027d754 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -41,7 +41,7 @@ class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() - : context_({ {}, {}, { kMaxFrameContexts } }) + : context_({ {}, {}, { kMaxFrameContexts }, {} }) { } @@ -50,7 +50,8 @@ public: int init(const IPASettings &settings, const SharedFD &fdStats, const SharedFD &fdParams, - const ControlInfoMap &sensorInfoMap) override; + const ControlInfoMap &sensorInfoMap, + ControlInfoMap *ipaControls) override; int configure(const IPAConfigInfo &configInfo) override; int start() override; @@ -87,7 +88,8 @@ IPASoftSimple::~IPASoftSimple() int IPASoftSimple::init(const IPASettings &settings, const SharedFD &fdStats, const SharedFD &fdParams, - const ControlInfoMap &sensorInfoMap) + const ControlInfoMap &sensorInfoMap, + ControlInfoMap *ipaControls) { camHelper_ = CameraSensorHelperFactoryBase::create(settings.sensorModel); if (!camHelper_) { @@ -158,6 +160,9 @@ int IPASoftSimple::init(const IPASettings &settings, stats_ = static_cast(mem); } + ControlInfoMap::Map ctrlMap = context_.ctrlMap; + *ipaControls = ControlInfoMap(std::move(ctrlMap), controls::controls); + /* * Check if the sensor driver supports the controls required by the * Soft IPA. diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 3ddce71d3..38868df66 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -530,7 +530,7 @@ int SimpleCameraData::init() * Instantiate Soft ISP if this is enabled for the given driver and no converter is used. */ if (!converter_ && pipe->swIspEnabled()) { - swIsp_ = std::make_unique(pipe, sensor_.get()); + swIsp_ = std::make_unique(pipe, sensor_.get(), &controlInfo_); if (!swIsp_->isValid()) { LOG(SimplePipeline, Warning) << "Failed to create software ISP, disabling software debayering"; diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 47677784d..3d3f956cb 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -13,6 +13,7 @@ #include #include +#include #include #include @@ -60,9 +61,11 @@ LOG_DEFINE_CATEGORY(SoftwareIsp) * \brief Constructs SoftwareIsp object * \param[in] pipe The pipeline handler in use * \param[in] sensor Pointer to the CameraSensor instance owned by the pipeline + * \param[out] ipaControls The IPA controls to update * handler */ -SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor) +SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor, + ControlInfoMap *ipaControls) : dmaHeap_(DmaBufAllocator::DmaBufAllocatorFlag::CmaHeap | DmaBufAllocator::DmaBufAllocatorFlag::SystemHeap | DmaBufAllocator::DmaBufAllocatorFlag::UDmaBuf) @@ -124,7 +127,8 @@ SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor) int ret = ipa_->init(IPASettings{ ipaTuningFile, sensor->model() }, debayer_->getStatsFD(), sharedParams_.fd(), - sensor->controls()); + sensor->controls(), + ipaControls); if (ret) { LOG(SoftwareIsp, Error) << "IPA init failed"; debayer_.reset();