{"id":23042,"url":"https://patchwork.libcamera.org/api/1.1/patches/23042/?format=json","web_url":"https://patchwork.libcamera.org/patch/23042/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20250326090849.15494-8-mzamazal@redhat.com>","date":"2025-03-26T09:08:44","name":"[v8,07/10] libcamera: software_isp: Add CCM algorithm","commit_ref":null,"pull_url":null,"state":"accepted","archived":false,"hash":"ffd6cb807b24225e1a776ba326d1d65109c9fbd8","submitter":{"id":177,"url":"https://patchwork.libcamera.org/api/1.1/people/177/?format=json","name":"Milan Zamazal","email":"mzamazal@redhat.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/23042/mbox/","series":[{"id":5083,"url":"https://patchwork.libcamera.org/api/1.1/series/5083/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=5083","date":"2025-03-26T09:08:37","name":"Software ISP support for CCM","version":8,"mbox":"https://patchwork.libcamera.org/series/5083/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/23042/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/23042/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id A42F1C3213\n\tfor ;\n\tWed, 26 Mar 2025 09:09:23 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 48EA06896D;\n\tWed, 26 Mar 2025 10:09:23 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.133.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 3A4FF6896C\n\tfor ;\n\tWed, 26 Mar 2025 10:09:20 +0100 (CET)","from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com\n\t(ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63])\n\tby relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3,\n\tcipher=TLS_AES_256_GCM_SHA384) id us-mta-392-Rw1Cl3egOSyQXNhmZcz5VQ-1;\n\tWed, 26 Mar 2025 05:09:18 -0400","from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com\n\t(mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com\n\t[10.30.177.111])\n\t(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n\tkey-exchange X25519 server-signature RSA-PSS (2048 bits)\n\tserver-digest SHA256) (No client certificate requested)\n\tby mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix)\n\twith ESMTPS id DA5C81945103; Wed, 26 Mar 2025 09:09:16 +0000 (UTC)","from mzamazal-thinkpadp1gen7.tpbc.com (unknown [10.44.34.15])\n\tby mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix)\n\twith ESMTP id A6FB2180094A; Wed, 26 Mar 2025 09:09:14 +0000 (UTC)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"WhowTPTp\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1742980159;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tcontent-transfer-encoding:content-transfer-encoding:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=UWFURJEAPEEWEp4W5DyBL159a3db24AjQmy88TenhXc=;\n\tb=WhowTPTpEEmpFTeRcGYOBdXz4bzQp2djMcLtZ8Oa+TvPBDQoobSyn1p+VIuMd+a5l0X7W3\n\txnd/H/wcknx1+gqKMsXv9w2xD8VYuTMhzX4KCgUBwrL1vEk/CNQ0/ZIpdJfxsqEk6C4LVs\n\tPTsAHoVO4phNOVnqJF4EX/nUWqmZHso=","X-MC-Unique":"Rw1Cl3egOSyQXNhmZcz5VQ-1","X-Mimecast-MFC-AGG-ID":"Rw1Cl3egOSyQXNhmZcz5VQ_1742980157","From":"Milan Zamazal ","To":"libcamera-devel@lists.libcamera.org","Cc":"Milan Zamazal ,\n\tRobert Mader ,\n\tHans de Goede ,\n\tLaurent Pinchart ,\n\tKieran Bingham ","Subject":"[PATCH v8 07/10] libcamera: software_isp: Add CCM algorithm","Date":"Wed, 26 Mar 2025 10:08:44 +0100","Message-ID":"<20250326090849.15494-8-mzamazal@redhat.com>","In-Reply-To":"<20250326090849.15494-1-mzamazal@redhat.com>","References":"<20250326090849.15494-1-mzamazal@redhat.com>","MIME-Version":"1.0","X-Scanned-By":"MIMEDefang 3.4.1 on 10.30.177.111","X-Mimecast-Spam-Score":"0","X-Mimecast-MFC-PROC-ID":"l5yMQ-cVb6nRnbdHK7MliuWXtSdjOU5SfUBi-rQboHQ_1742980157","X-Mimecast-Originator":"redhat.com","Content-Transfer-Encoding":"8bit","content-type":"text/plain; charset=\"US-ASCII\"; x-default=true","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"This patch adds color correction matrix (CCM) algorithm to software ISP.\nIt is based on the corresponding algorithm in rkisp1.\n\nThe primary difference against hardware pipelines is that applying the\nCCM is optional. Applying CCM causes a significant slowdown, time\nneeded to process a frame raises by 40-90% on tested platforms. If CCM\nis really needed, it can be applied, if not, it's better to stick\nwithout it. This can be configured by presence or omission of Ccm\nalgorithm in the tuning file.\n\nCCM is changed only if the determined temperature changes by at least\n100 K (an arbitrarily selected value), to avoid recomputing the matrices\nand lookup tables all the time.\n\nSince the CCM is float, rather than double, to use the same type as in\nthe rkisp1 pipeline, the type of color gains is changed from double to\nfloat.\n\nThe outputs of the algorithm are not used yet, they will be enabled in\nfollowup patches.\n\nSigned-off-by: Milan Zamazal \nReviewed-by: Kieran Bingham \nReviewed-by: Laurent Pinchart \nSigned-off-by: Milan Zamazal \n---\n src/ipa/simple/algorithms/awb.cpp | 4 +-\n src/ipa/simple/algorithms/ccm.cpp | 74 +++++++++++++++++++++++++++\n src/ipa/simple/algorithms/ccm.h | 43 ++++++++++++++++\n src/ipa/simple/algorithms/lut.cpp | 2 +-\n src/ipa/simple/algorithms/meson.build | 1 +\n src/ipa/simple/ipa_context.h | 13 ++++-\n 6 files changed, 133 insertions(+), 4 deletions(-)\n create mode 100644 src/ipa/simple/algorithms/ccm.cpp\n create mode 100644 src/ipa/simple/algorithms/ccm.h","diff":"diff --git a/src/ipa/simple/algorithms/awb.cpp b/src/ipa/simple/algorithms/awb.cpp\nindex 4e31e386..ec77c6e5 100644\n--- a/src/ipa/simple/algorithms/awb.cpp\n+++ b/src/ipa/simple/algorithms/awb.cpp\n@@ -59,9 +59,9 @@ void Awb::process(IPAContext &context,\n \t */\n \tauto &gains = context.activeState.awb.gains;\n \tgains = { {\n-\t\tsumR <= sumG / 4 ? 4.0 : static_cast(sumG) / sumR,\n+\t\tsumR <= sumG / 4 ? 4.0f : static_cast(sumG) / sumR,\n \t\t1.0,\n-\t\tsumB <= sumG / 4 ? 4.0 : static_cast(sumG) / sumB,\n+\t\tsumB <= sumG / 4 ? 4.0f : static_cast(sumG) / sumB,\n \t} };\n \n \tRGB rgbGains{ { 1 / gains.r(), 1 / gains.g(), 1 / gains.b() } };\ndiff --git a/src/ipa/simple/algorithms/ccm.cpp b/src/ipa/simple/algorithms/ccm.cpp\nnew file mode 100644\nindex 00000000..86e0395c\n--- /dev/null\n+++ b/src/ipa/simple/algorithms/ccm.cpp\n@@ -0,0 +1,74 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2024, Ideas On Board\n+ * Copyright (C) 2024-2025, Red Hat Inc.\n+ *\n+ * Color correction matrix\n+ */\n+\n+#include \"ccm.h\"\n+\n+#include \n+#include \n+\n+#include \n+\n+namespace {\n+\n+constexpr unsigned int kTemperatureThreshold = 100;\n+\n+}\n+\n+namespace libcamera {\n+\n+namespace ipa::soft::algorithms {\n+\n+LOG_DEFINE_CATEGORY(IPASoftCcm)\n+\n+int Ccm::init([[maybe_unused]] IPAContext &context, const YamlObject &tuningData)\n+{\n+\tint ret = ccm_.readYaml(tuningData[\"ccms\"], \"ct\", \"ccm\");\n+\tif (ret < 0) {\n+\t\tLOG(IPASoftCcm, Error)\n+\t\t\t<< \"Failed to parse 'ccm' parameter from tuning file.\";\n+\t\treturn ret;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+void Ccm::prepare(IPAContext &context, const uint32_t frame,\n+\t\t IPAFrameContext &frameContext, [[maybe_unused]] DebayerParams *params)\n+{\n+\tconst unsigned int ct = context.activeState.awb.temperatureK;\n+\n+\t/* Change CCM only on bigger temperature changes. */\n+\tif (frame > 0 &&\n+\t utils::abs_diff(ct, lastCt_) < kTemperatureThreshold) {\n+\t\tframeContext.ccm.ccm = context.activeState.ccm.ccm;\n+\t\tcontext.activeState.ccm.changed = false;\n+\t\treturn;\n+\t}\n+\n+\tlastCt_ = ct;\n+\tMatrix ccm = ccm_.getInterpolated(ct);\n+\n+\tcontext.activeState.ccm.ccm = ccm;\n+\tframeContext.ccm.ccm = ccm;\n+\tcontext.activeState.ccm.changed = true;\n+}\n+\n+void Ccm::process([[maybe_unused]] IPAContext &context,\n+\t\t [[maybe_unused]] const uint32_t frame,\n+\t\t IPAFrameContext &frameContext,\n+\t\t [[maybe_unused]] const SwIspStats *stats,\n+\t\t ControlList &metadata)\n+{\n+\tmetadata.set(controls::ColourCorrectionMatrix, frameContext.ccm.ccm.data());\n+}\n+\n+REGISTER_IPA_ALGORITHM(Ccm, \"Ccm\")\n+\n+} /* namespace ipa::soft::algorithms */\n+\n+} /* namespace libcamera */\ndiff --git a/src/ipa/simple/algorithms/ccm.h b/src/ipa/simple/algorithms/ccm.h\nnew file mode 100644\nindex 00000000..f4e2b85b\n--- /dev/null\n+++ b/src/ipa/simple/algorithms/ccm.h\n@@ -0,0 +1,43 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2024-2025, Red Hat Inc.\n+ *\n+ * Color correction matrix\n+ */\n+\n+#pragma once\n+\n+#include \"libcamera/internal/matrix.h\"\n+\n+#include \n+\n+#include \"algorithm.h\"\n+\n+namespace libcamera {\n+\n+namespace ipa::soft::algorithms {\n+\n+class Ccm : public Algorithm\n+{\n+public:\n+\tCcm() = default;\n+\t~Ccm() = default;\n+\n+\tint init(IPAContext &context, const YamlObject &tuningData) override;\n+\tvoid prepare(IPAContext &context,\n+\t\t const uint32_t frame,\n+\t\t IPAFrameContext &frameContext,\n+\t\t DebayerParams *params) override;\n+\tvoid process(IPAContext &context, const uint32_t frame,\n+\t\t IPAFrameContext &frameContext,\n+\t\t const SwIspStats *stats,\n+\t\t ControlList &metadata) override;\n+\n+private:\n+\tunsigned int lastCt_;\n+\tInterpolator> ccm_;\n+};\n+\n+} /* namespace ipa::soft::algorithms */\n+\n+} /* namespace libcamera */\ndiff --git a/src/ipa/simple/algorithms/lut.cpp b/src/ipa/simple/algorithms/lut.cpp\nindex 063bec73..3a3daed7 100644\n--- a/src/ipa/simple/algorithms/lut.cpp\n+++ b/src/ipa/simple/algorithms/lut.cpp\n@@ -103,7 +103,7 @@ void Lut::prepare(IPAContext &context,\n \t\tconst double div = static_cast(DebayerParams::kRGBLookupSize) /\n \t\t\t\t gammaTableSize;\n \t\t/* Apply gamma after gain! */\n-\t\tconst RGB lutGains = (gains * i / div).min(gammaTableSize - 1);\n+\t\tconst RGB lutGains = (gains * i / div).min(gammaTableSize - 1);\n \t\tparams->red[i] = gammaTable[static_cast(lutGains.r())];\n \t\tparams->green[i] = gammaTable[static_cast(lutGains.g())];\n \t\tparams->blue[i] = gammaTable[static_cast(lutGains.b())];\ndiff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build\nindex 37a2eb53..2d0adb05 100644\n--- a/src/ipa/simple/algorithms/meson.build\n+++ b/src/ipa/simple/algorithms/meson.build\n@@ -4,5 +4,6 @@ soft_simple_ipa_algorithms = files([\n 'awb.cpp',\n 'agc.cpp',\n 'blc.cpp',\n+ 'ccm.cpp',\n 'lut.cpp',\n ])\ndiff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h\nindex 6a285414..67183b3e 100644\n--- a/src/ipa/simple/ipa_context.h\n+++ b/src/ipa/simple/ipa_context.h\n@@ -13,6 +13,7 @@\n \n #include \n \n+#include \"libcamera/internal/matrix.h\"\n #include \"libcamera/internal/vector.h\"\n \n #include \n@@ -38,7 +39,7 @@ struct IPAActiveState {\n \t} blc;\n \n \tstruct {\n-\t\tRGB gains;\n+\t\tRGB gains;\n \t\tunsigned int temperatureK;\n \t} awb;\n \n@@ -48,6 +49,12 @@ struct IPAActiveState {\n \t\tuint8_t blackLevel;\n \t\tdouble contrast;\n \t} gamma;\n+\n+\tstruct {\n+\t\tMatrix ccm;\n+\t\tbool changed;\n+\t} ccm;\n+\n \tstruct {\n \t\t/* 0..2 range, 1.0 = normal */\n \t\tstd::optional contrast;\n@@ -55,6 +62,10 @@ struct IPAActiveState {\n };\n \n struct IPAFrameContext : public FrameContext {\n+\tstruct {\n+\t\tMatrix ccm;\n+\t} ccm;\n+\n \tstruct {\n \t\tint32_t exposure;\n \t\tdouble gain;\n","prefixes":["v8","07/10"]}