From patchwork Wed Feb 5 21:27:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 22736 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id CEF22C32F4 for ; Wed, 5 Feb 2025 21:28:00 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 7E325685E5; Wed, 5 Feb 2025 22:28:00 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="FFQDFFPg"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C77F5685DD for ; Wed, 5 Feb 2025 22:27:58 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738790877; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gu9W2L/K/kPTCkTbmOYYJlWjwmYzDXzV5tME2ZuVYrw=; b=FFQDFFPgstetESL+sVmsDtdqwenX1D79IE3ARpHQI/uVYqQNcdnsetMf2+05jd8o2E3dO1 bpfHeP+R5S3y7OaVj4eRkult8kUoQLEtjHhbPmJKKcG5grJjaIOrreA008lfwWe60cxvTL UPgEbAD626nc79SHJFQolRPKu3qY0MQ= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-617-YhmlzwdCPYy7-weSSpMlfA-1; Wed, 05 Feb 2025 16:27:56 -0500 X-MC-Unique: YhmlzwdCPYy7-weSSpMlfA-1 X-Mimecast-MFC-AGG-ID: YhmlzwdCPYy7-weSSpMlfA Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 6AFD91800878; Wed, 5 Feb 2025 21:27:55 +0000 (UTC) Received: from mzamazal-thinkpadp1gen3.tpbc.com (unknown [10.45.224.28]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 412221800872; Wed, 5 Feb 2025 21:27:53 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Robert Mader , Hans de Goede , Laurent Pinchart , Kieran Bingham Subject: [PATCH v6 07/10] libcamera: software_isp: Add CCM algorithm Date: Wed, 5 Feb 2025 22:27:20 +0100 Message-ID: <20250205212725.138055-9-mzamazal@redhat.com> In-Reply-To: <20250205212725.138055-1-mzamazal@redhat.com> References: <20250205212725.138055-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Q7sGjOiKb-3E-TiN5yuwCUXijBU0HuDf7OGLuQsaGZY_1738790875 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds color correction matrix (CCM) algorithm to software ISP. It is based on the corresponding algorithm in rkisp1. The primary difference against hardware pipelines is that applying the CCM is optional. Applying CCM causes a significant slowdown, time needed to process a frame raises by 40-90% on tested platforms. If CCM is really needed, it can be applied, if not, it's better to stick without it. This can be configured by presence or omission of Ccm algorithm in the tuning file. CCM is changed only if the determined temperature changes by at least 100 K (an arbitrarily selected value), to avoid recomputing the matrices and lookup tables all the time. Since the CCM is float, rather than double, to use the same type as in the rkisp1 pipeline, the type of color gains is changed from double to float. The outputs of the algorithm are not used yet, they will be enabled in followup patches. Signed-off-by: Milan Zamazal Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/ipa/simple/algorithms/awb.cpp | 4 +- src/ipa/simple/algorithms/ccm.cpp | 74 +++++++++++++++++++++++++++ src/ipa/simple/algorithms/ccm.h | 43 ++++++++++++++++ src/ipa/simple/algorithms/lut.cpp | 2 +- src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/ipa_context.h | 14 ++++- 6 files changed, 134 insertions(+), 4 deletions(-) create mode 100644 src/ipa/simple/algorithms/ccm.cpp create mode 100644 src/ipa/simple/algorithms/ccm.h diff --git a/src/ipa/simple/algorithms/awb.cpp b/src/ipa/simple/algorithms/awb.cpp index 4e31e386..ec77c6e5 100644 --- a/src/ipa/simple/algorithms/awb.cpp +++ b/src/ipa/simple/algorithms/awb.cpp @@ -59,9 +59,9 @@ void Awb::process(IPAContext &context, */ auto &gains = context.activeState.awb.gains; gains = { { - sumR <= sumG / 4 ? 4.0 : static_cast(sumG) / sumR, + sumR <= sumG / 4 ? 4.0f : static_cast(sumG) / sumR, 1.0, - sumB <= sumG / 4 ? 4.0 : static_cast(sumG) / sumB, + sumB <= sumG / 4 ? 4.0f : static_cast(sumG) / sumB, } }; RGB rgbGains{ { 1 / gains.r(), 1 / gains.g(), 1 / gains.b() } }; diff --git a/src/ipa/simple/algorithms/ccm.cpp b/src/ipa/simple/algorithms/ccm.cpp new file mode 100644 index 00000000..86e0395c --- /dev/null +++ b/src/ipa/simple/algorithms/ccm.cpp @@ -0,0 +1,74 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Ideas On Board + * Copyright (C) 2024-2025, Red Hat Inc. + * + * Color correction matrix + */ + +#include "ccm.h" + +#include +#include + +#include + +namespace { + +constexpr unsigned int kTemperatureThreshold = 100; + +} + +namespace libcamera { + +namespace ipa::soft::algorithms { + +LOG_DEFINE_CATEGORY(IPASoftCcm) + +int Ccm::init([[maybe_unused]] IPAContext &context, const YamlObject &tuningData) +{ + int ret = ccm_.readYaml(tuningData["ccms"], "ct", "ccm"); + if (ret < 0) { + LOG(IPASoftCcm, Error) + << "Failed to parse 'ccm' parameter from tuning file."; + return ret; + } + + return 0; +} + +void Ccm::prepare(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, [[maybe_unused]] DebayerParams *params) +{ + const unsigned int ct = context.activeState.awb.temperatureK; + + /* Change CCM only on bigger temperature changes. */ + if (frame > 0 && + utils::abs_diff(ct, lastCt_) < kTemperatureThreshold) { + frameContext.ccm.ccm = context.activeState.ccm.ccm; + context.activeState.ccm.changed = false; + return; + } + + lastCt_ = ct; + Matrix ccm = ccm_.getInterpolated(ct); + + context.activeState.ccm.ccm = ccm; + frameContext.ccm.ccm = ccm; + context.activeState.ccm.changed = true; +} + +void Ccm::process([[maybe_unused]] IPAContext &context, + [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, + [[maybe_unused]] const SwIspStats *stats, + ControlList &metadata) +{ + metadata.set(controls::ColourCorrectionMatrix, frameContext.ccm.ccm.data()); +} + +REGISTER_IPA_ALGORITHM(Ccm, "Ccm") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/ccm.h b/src/ipa/simple/algorithms/ccm.h new file mode 100644 index 00000000..f4e2b85b --- /dev/null +++ b/src/ipa/simple/algorithms/ccm.h @@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024-2025, Red Hat Inc. + * + * Color correction matrix + */ + +#pragma once + +#include "libcamera/internal/matrix.h" + +#include + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class Ccm : public Algorithm +{ +public: + Ccm() = default; + ~Ccm() = default; + + int init(IPAContext &context, const YamlObject &tuningData) override; + void prepare(IPAContext &context, + const uint32_t frame, + IPAFrameContext &frameContext, + DebayerParams *params) override; + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const SwIspStats *stats, + ControlList &metadata) override; + +private: + unsigned int lastCt_; + Interpolator> ccm_; +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/lut.cpp b/src/ipa/simple/algorithms/lut.cpp index 60dd0624..352bbf57 100644 --- a/src/ipa/simple/algorithms/lut.cpp +++ b/src/ipa/simple/algorithms/lut.cpp @@ -103,7 +103,7 @@ void Lut::prepare(IPAContext &context, const double div = static_cast(DebayerParams::kRGBLookupSize) / gammaTableSize; /* Apply gamma after gain! */ - const RGB lutGains = (gains * i / div).min(gammaTableSize - 1); + const RGB lutGains = (gains * i / div).min(gammaTableSize - 1); params->red[i] = gammaTable[static_cast(lutGains.r())]; params->green[i] = gammaTable[static_cast(lutGains.g())]; params->blue[i] = gammaTable[static_cast(lutGains.b())]; diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index 37a2eb53..2d0adb05 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -4,5 +4,6 @@ soft_simple_ipa_algorithms = files([ 'awb.cpp', 'agc.cpp', 'blc.cpp', + 'ccm.cpp', 'lut.cpp', ]) diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index df0552db..57883218 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -13,6 +13,8 @@ #include +#include "libcamera/internal/matrix.h" + #include #include @@ -37,7 +39,7 @@ struct IPAActiveState { } blc; struct { - RGB gains; + RGB gains; unsigned int temperatureK; } awb; @@ -47,6 +49,12 @@ struct IPAActiveState { uint8_t blackLevel; double contrast; } gamma; + + struct { + Matrix ccm; + bool changed; + } ccm; + struct { /* 0..2 range, 1.0 = normal */ std::optional contrast; @@ -54,6 +62,10 @@ struct IPAActiveState { }; struct IPAFrameContext : public FrameContext { + struct { + Matrix ccm; + } ccm; + struct { int32_t exposure; double gain;