From patchwork Wed Nov 20 18:01:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 22046 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 54B52C330B for ; Wed, 20 Nov 2024 18:01:31 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A50AF65F93; Wed, 20 Nov 2024 19:01:30 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="ZostFo0Y"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5BCCF65F83 for ; Wed, 20 Nov 2024 19:01:27 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1732125686; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=luCyc9hNUm4p3xinuCMxgVCxFZvoHEDYp0tghRm3SaU=; b=ZostFo0YOyODAsd4WaodcFc2aB7L9N56Wa17YyaLvYomUXBrI5kWVqEZtfJRGeBGB5lsn0 R+OG+8uP4laxF/ttzWfkO2p77zL5nZQJnq6uFmUIhkQlKcUMRFcDg9r/3ggmGtAKHefsyG jlAXgRL7uxqUdNXd17CDLV9m1Ceo1MY= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-361-7PBixReXMZquahBhimCxWA-1; Wed, 20 Nov 2024 13:01:24 -0500 X-MC-Unique: 7PBixReXMZquahBhimCxWA-1 X-Mimecast-MFC-AGG-ID: 7PBixReXMZquahBhimCxWA Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 33F8319560AF for ; Wed, 20 Nov 2024 18:01:23 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.6]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id E86AC30000DF; Wed, 20 Nov 2024 18:01:21 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 6/9] libcamera: software_isp: Add CCM algorithm Date: Wed, 20 Nov 2024 19:01:01 +0100 Message-ID: <20241120180104.1221846-7-mzamazal@redhat.com> In-Reply-To: <20241120180104.1221846-1-mzamazal@redhat.com> References: <20241120180104.1221846-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 0jYo5ScXMFzldD6ordQDx-wbiQf7531RF_VtD7BF4iA_1732125683 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds color correction matrix (CCM) algorithm to software ISP. It is based on the corresponding algorithm in rkisp1. The primary difference against hardware pipelines is that applying the CCM is optional. Applying CCM causes a significant slowdown, time needed to process a frame raises by 40-90% on tested platforms. If CCM is really needed, it can be applied, if not, it's better to stick without it. This can be configured by presence or omission of Ccm algorithm in the tuning file. CCM is changed only if the determined temperature changes by at least 100 K (an arbitrarily selected value), to avoid recomputing the matrices and lookup tables all the time. Some ugly unsigned->signed int typecasting is needed to compare the temperatures. The outputs of the algorithm are not used yet, they will be enabled in followup patches. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/ccm.cpp | 88 +++++++++++++++++++++++++++ src/ipa/simple/algorithms/ccm.h | 44 ++++++++++++++ src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/ipa_context.h | 11 ++++ 4 files changed, 144 insertions(+) create mode 100644 src/ipa/simple/algorithms/ccm.cpp create mode 100644 src/ipa/simple/algorithms/ccm.h diff --git a/src/ipa/simple/algorithms/ccm.cpp b/src/ipa/simple/algorithms/ccm.cpp new file mode 100644 index 000000000..bd2c03fe4 --- /dev/null +++ b/src/ipa/simple/algorithms/ccm.cpp @@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Ideas On Board + * Copyright (C) 2024, Red Hat Inc. + * + * Color correction matrix + */ + +#include "ccm.h" + +#include + +#include + +#include + +namespace libcamera { + +namespace ipa::soft::algorithms { + +LOG_DEFINE_CATEGORY(IPASoftCcm) + +unsigned int Ccm::kTemperatureThreshold = 100; + +int Ccm::init([[maybe_unused]] IPAContext &context, const YamlObject &tuningData) +{ + int ret = ccm_.readYaml(tuningData["ccms"], "ct", "ccm"); + if (ret < 0) { + LOG(IPASoftCcm, Warning) + << "Failed to parse 'ccm' " + << "parameter from tuning file; falling back to unit matrix"; + ccmEnabled_ = false; + } else { + ccmEnabled_ = true; + } + + return 0; +} + +void Ccm::prepare(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, [[maybe_unused]] DebayerParams *params) +{ + context.activeState.ccm.enabled = ccmEnabled_; + + if (!ccmEnabled_) + return; + + unsigned int ct = context.activeState.awb.temperatureK; + + /* Change CCM only on bigger temperature changes. */ + if (frame > 0 && + abs(static_cast(ct) - static_cast(ct_)) < + static_cast(kTemperatureThreshold)) { + frameContext.ccm.ccm = context.activeState.ccm.ccm; + context.activeState.ccm.changed = false; + return; + } + + ct_ = ct; + Matrix ccm = ccm_.getInterpolated(ct); + + context.activeState.ccm.ccm = ccm; + frameContext.ccm.ccm = ccm; + context.activeState.ccm.changed = true; +} + +void Ccm::process([[maybe_unused]] IPAContext &context, + [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, + [[maybe_unused]] const SwIspStats *stats, + ControlList &metadata) +{ + if (!ccmEnabled_) + return; + + float m[9]; + for (unsigned int i = 0; i < 3; i++) { + for (unsigned int j = 0; j < 3; j++) + m[i * 3 + j] = frameContext.ccm.ccm[i][j]; + } + metadata.set(controls::ColourCorrectionMatrix, m); +} + +REGISTER_IPA_ALGORITHM(Ccm, "Ccm") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/ccm.h b/src/ipa/simple/algorithms/ccm.h new file mode 100644 index 000000000..6b7efbe21 --- /dev/null +++ b/src/ipa/simple/algorithms/ccm.h @@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Color correction matrix + */ + +#pragma once + +#include "libipa/interpolator.h" +#include "libipa/matrix.h" + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class Ccm : public Algorithm +{ +public: + Ccm() = default; + ~Ccm() = default; + + int init(IPAContext &context, const YamlObject &tuningData) override; + void prepare(IPAContext &context, + const uint32_t frame, + IPAFrameContext &frameContext, + DebayerParams *params) override; + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const SwIspStats *stats, + ControlList &metadata) override; + +private: + static unsigned int kTemperatureThreshold; + unsigned int ct_; + bool ccmEnabled_; + Interpolator> ccm_; +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index 37a2eb534..2d0adb059 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -4,5 +4,6 @@ soft_simple_ipa_algorithms = files([ 'awb.cpp', 'agc.cpp', 'blc.cpp', + 'ccm.cpp', 'lut.cpp', ]) diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index 268ba9cec..5f9ec0afc 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -12,6 +12,7 @@ #include #include +#include namespace libcamera { @@ -52,9 +53,19 @@ struct IPAActiveState { std::array gammaTable; uint8_t blackLevel; } gamma; + + struct { + Matrix ccm; + bool enabled; + bool changed; + } ccm; }; struct IPAFrameContext : public FrameContext { + struct { + Matrix ccm; + } ccm; + struct { uint32_t exposure; double gain;