{"id":22707,"url":"https://patchwork.libcamera.org/api/1.1/patches/22707/?format=json","web_url":"https://patchwork.libcamera.org/patch/22707/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20250130181449.130492-11-mzamazal@redhat.com>","date":"2025-01-30T18:14:47","name":"[v5,10/10] libcamera: software_isp: Apply CCM in debayering","commit_ref":null,"pull_url":null,"state":"changes-requested","archived":false,"hash":"a328891b60229379c0c21746dac33f16bcd66ead","submitter":{"id":177,"url":"https://patchwork.libcamera.org/api/1.1/people/177/?format=json","name":"Milan Zamazal","email":"mzamazal@redhat.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/22707/mbox/","series":[{"id":4980,"url":"https://patchwork.libcamera.org/api/1.1/series/4980/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=4980","date":"2025-01-30T18:14:37","name":"Software ISP support for CCM","version":5,"mbox":"https://patchwork.libcamera.org/series/4980/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/22707/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/22707/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id F2BADBD808\n\tfor ;\n\tThu, 30 Jan 2025 18:15:38 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 99C916857D;\n\tThu, 30 Jan 2025 19:15:38 +0100 (CET)","from us-smtp-delivery-124.mimecast.com\n\t(us-smtp-delivery-124.mimecast.com [170.10.129.124])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id E73C368575\n\tfor ;\n\tThu, 30 Jan 2025 19:15:35 +0100 (CET)","from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com\n\t(ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63])\n\tby relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3,\n\tcipher=TLS_AES_256_GCM_SHA384) id us-mta-253-oml83FU4NxuCp6XRrGj5uA-1;\n\tThu, 30 Jan 2025 13:15:33 -0500","from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com\n\t(mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4])\n\t(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n\tkey-exchange X25519 server-signature RSA-PSS (2048 bits)\n\tserver-digest SHA256) (No client certificate requested)\n\tby mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix)\n\twith ESMTPS id 774BD19560AF; Thu, 30 Jan 2025 18:15:32 +0000 (UTC)","from mzamazal-thinkpadp1gen3.tpbc.com (unknown [10.45.224.122])\n\tby mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix)\n\twith ESMTP id 724EE30001BE; Thu, 30 Jan 2025 18:15:29 +0000 (UTC)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=redhat.com header.i=@redhat.com\n\theader.b=\"NwLMytV8\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n\ts=mimecast20190719; t=1738260935;\n\th=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n\tto:to:cc:cc:mime-version:mime-version:content-type:content-type:\n\tcontent-transfer-encoding:content-transfer-encoding:\n\tin-reply-to:in-reply-to:references:references;\n\tbh=ru5hpnF/sIFEjQqjEhzEwB1nzy8iIeBwdeGB5YPZpuE=;\n\tb=NwLMytV8V6/dH9yTmIHnQmn2T+Pz5jCQEV7Lqp7DYrphKPoQ8IaCejidrLuLzJgwEPgI2Y\n\tV1M8ubbzl8MFkuwtcq0tVtmGxrHemRIArchxZi9XkcO1AHkJrfGbvGCNnxfS1nWXM4PSi5\n\tgv8Go1/T2UotREuwrPcaqkcGnqv6Qrw=","X-MC-Unique":"oml83FU4NxuCp6XRrGj5uA-1","X-Mimecast-MFC-AGG-ID":"oml83FU4NxuCp6XRrGj5uA","From":"Milan Zamazal ","To":"libcamera-devel@lists.libcamera.org","Cc":"Milan Zamazal ,\n\tRobert Mader ,\n\tHans de Goede ,\n\tLaurent Pinchart ,\n\tKieran Bingham ","Subject":"[PATCH v5 10/10] libcamera: software_isp: Apply CCM in debayering","Date":"Thu, 30 Jan 2025 19:14:47 +0100","Message-ID":"<20250130181449.130492-11-mzamazal@redhat.com>","In-Reply-To":"<20250130181449.130492-1-mzamazal@redhat.com>","References":"<20250130181449.130492-1-mzamazal@redhat.com>","MIME-Version":"1.0","X-Scanned-By":"MIMEDefang 3.4.1 on 10.30.177.4","X-Mimecast-Spam-Score":"0","X-Mimecast-MFC-PROC-ID":"vwFNrjlQXwox_dCSYlrFxlVkXzej6Rs_w_xfzU02yUo_1738260932","X-Mimecast-Originator":"redhat.com","Content-Transfer-Encoding":"8bit","content-type":"text/plain; charset=\"US-ASCII\"; x-default=true","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"This patch applies color correction matrix (CCM) in debayering if the\nCCM is specified. Not using CCM must still be supported for performance\nreasons.\n\nThe CCM is applied as follows:\n\n [r1 r2 r3]\n [r g b] * [g1 g2 g3]\n [b1 b2 b3]\n\nThe CCM matrix (the right side of the multiplication) is constant during\nsingle frame processing, while the input pixel (the left side) changes.\nBecause each of the color channels is only 8-bit in software ISP, we can\nmake 9 lookup tables with 256 input values for multiplications of each\nof the r_i, g_i, b_i values. This way we don't have to multiply each\npixel, we can use table lookups and additions instead. Gamma (which is\nnon-linear and thus cannot be a part of the 9 lookup tables values) is\napplied on the final values rounded to integers using another lookup\ntable.\n\nWe use int16_t to store the precomputed multiplications. This seems to\nbe noticeably (>10%) faster than `float' for the price of slightly less\naccuracy and it covers the range of values that sane CCMs produce. The\nselection and structure of data is performance critical, for example\nusing bytes would add significant (>10%) speedup but would be too short\nto cover the value range.\n\nThe color lookup tables can be represented either as unions,\naccommodating tables for both the CCM and non-CCM cases, or as separate\ntables for each of the cases, leaving the tables for the other case\nunused. The latter is selected as a matter of preference.\n\nThe tables are copied (as before), which is not elegant but also not a\nbig problem. There are patches posted that use shared buffers for\nparameters passing in software ISP (see software ISP TODO #5) and they\ncan be adjusted for the new parameter format.\n\nColor gains from white balance are supposed not to be a part of the\nspecified CCM. They are applied on it using matrix multiplication,\nwhich is simple and in correspondence with future additions in the form\nof matrix multiplication, like saturation adjustment.\n\nWith this patch, the reported per-frame slowdown when applying CCM is\nabout 45% on Debix Model A and about 75% on TI AM69 SK.\n\nUsing std::clamp in debayering adds some performance penalty (a few\npercent). The clamping is necessary to eliminate out of range values\npossibly produced by the CCM. If it could be avoided by adjusting the\nprecomputed tables some way then performance could be improved a bit.\n\nSigned-off-by: Milan Zamazal \n---\n .../internal/software_isp/debayer_params.h | 15 ++++-\n src/ipa/simple/algorithms/lut.cpp | 65 ++++++++++++++-----\n src/ipa/simple/algorithms/lut.h | 1 +\n src/libcamera/software_isp/debayer.cpp | 52 ++++++++++++++-\n src/libcamera/software_isp/debayer_cpu.cpp | 35 ++++++++--\n src/libcamera/software_isp/debayer_cpu.h | 4 ++\n 6 files changed, 145 insertions(+), 27 deletions(-)","diff":"diff --git a/include/libcamera/internal/software_isp/debayer_params.h b/include/libcamera/internal/software_isp/debayer_params.h\nindex 7d8fdd48..9a7dbe93 100644\n--- a/include/libcamera/internal/software_isp/debayer_params.h\n+++ b/include/libcamera/internal/software_isp/debayer_params.h\n@@ -1,6 +1,6 @@\n /* SPDX-License-Identifier: LGPL-2.1-or-later */\n /*\n- * Copyright (C) 2023, 2024 Red Hat Inc.\n+ * Copyright (C) 2023-2025 Red Hat Inc.\n *\n * Authors:\n * Hans de Goede \n@@ -18,11 +18,24 @@ namespace libcamera {\n struct DebayerParams {\n \tstatic constexpr unsigned int kRGBLookupSize = 256;\n \n+\tstruct CcmColumn {\n+\t\tint16_t r;\n+\t\tint16_t g;\n+\t\tint16_t b;\n+\t};\n+\n \tusing ColorLookupTable = std::array;\n+\tusing CcmLookupTable = std::array;\n+\tusing GammaLookupTable = std::array;\n \n \tColorLookupTable red;\n \tColorLookupTable green;\n \tColorLookupTable blue;\n+\n+\tCcmLookupTable redCcm;\n+\tCcmLookupTable greenCcm;\n+\tCcmLookupTable blueCcm;\n+\tGammaLookupTable gammaLut;\n };\n \n } /* namespace libcamera */\ndiff --git a/src/ipa/simple/algorithms/lut.cpp b/src/ipa/simple/algorithms/lut.cpp\nindex e5e995c7..7719aecc 100644\n--- a/src/ipa/simple/algorithms/lut.cpp\n+++ b/src/ipa/simple/algorithms/lut.cpp\n@@ -1,6 +1,6 @@\n /* SPDX-License-Identifier: LGPL-2.1-or-later */\n /*\n- * Copyright (C) 2024, Red Hat Inc.\n+ * Copyright (C) 2024-2025, Red Hat Inc.\n *\n * Color lookup tables construction\n */\n@@ -80,6 +80,11 @@ void Lut::updateGammaTable(IPAContext &context)\n \tcontext.activeState.gamma.contrast = contrast;\n }\n \n+int16_t Lut::ccmValue(unsigned int i, float ccm) const\n+{\n+\treturn std::round(i * ccm);\n+}\n+\n void Lut::prepare(IPAContext &context,\n \t\t [[maybe_unused]] const uint32_t frame,\n \t\t [[maybe_unused]] IPAFrameContext &frameContext,\n@@ -91,28 +96,52 @@ void Lut::prepare(IPAContext &context,\n \t * observed, it's not permanently prone to minor fluctuations or\n \t * rounding errors.\n \t */\n-\tif (context.activeState.gamma.blackLevel != context.activeState.blc.level ||\n-\t context.activeState.gamma.contrast != context.activeState.knobs.contrast)\n+\tconst bool gammaUpdateNeeded =\n+\t\tcontext.activeState.gamma.blackLevel != context.activeState.blc.level ||\n+\t\tcontext.activeState.gamma.contrast != context.activeState.knobs.contrast;\n+\tif (gammaUpdateNeeded)\n \t\tupdateGammaTable(context);\n \n \tauto &gains = context.activeState.awb.gains;\n \tauto &gammaTable = context.activeState.gamma.gammaTable;\n \tconst unsigned int gammaTableSize = gammaTable.size();\n-\n-\tfor (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) {\n-\t\tconst double div = static_cast(DebayerParams::kRGBLookupSize) /\n-\t\t\t\t gammaTableSize;\n-\t\t/* Apply gamma after gain! */\n-\t\tunsigned int idx;\n-\t\tidx = std::min({ static_cast(i * gains.r() / div),\n-\t\t\t\t gammaTableSize - 1 });\n-\t\tparams->red[i] = gammaTable[idx];\n-\t\tidx = std::min({ static_cast(i * gains.g() / div),\n-\t\t\t\t gammaTableSize - 1 });\n-\t\tparams->green[i] = gammaTable[idx];\n-\t\tidx = std::min({ static_cast(i * gains.b() / div),\n-\t\t\t\t gammaTableSize - 1 });\n-\t\tparams->blue[i] = gammaTable[idx];\n+\tconst double div = static_cast(DebayerParams::kRGBLookupSize) /\n+\t\t\t gammaTableSize;\n+\n+\tif (!context.ccmEnabled) {\n+\t\tfor (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) {\n+\t\t\t/* Apply gamma after gain! */\n+\t\t\tunsigned int idx;\n+\t\t\tidx = std::min({ static_cast(i * gains.r() / div),\n+\t\t\t\t\t gammaTableSize - 1 });\n+\t\t\tparams->red[i] = gammaTable[idx];\n+\t\t\tidx = std::min({ static_cast(i * gains.g() / div),\n+\t\t\t\t\t gammaTableSize - 1 });\n+\t\t\tparams->green[i] = gammaTable[idx];\n+\t\t\tidx = std::min({ static_cast(i * gains.b() / div),\n+\t\t\t\t\t gammaTableSize - 1 });\n+\t\t\tparams->blue[i] = gammaTable[idx];\n+\t\t}\n+\t} else if (context.activeState.ccm.changed || gammaUpdateNeeded) {\n+\t\tMatrix gainCcm = { { gains.r(), 0, 0,\n+\t\t\t\t\t\t 0, gains.g(), 0,\n+\t\t\t\t\t\t 0, 0, gains.b() } };\n+\t\tauto ccm = gainCcm * context.activeState.ccm.ccm;\n+\t\tauto &red = params->redCcm;\n+\t\tauto &green = params->greenCcm;\n+\t\tauto &blue = params->blueCcm;\n+\t\tfor (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) {\n+\t\t\tred[i].r = ccmValue(i, ccm[0][0]);\n+\t\t\tred[i].g = ccmValue(i, ccm[1][0]);\n+\t\t\tred[i].b = ccmValue(i, ccm[2][0]);\n+\t\t\tgreen[i].r = ccmValue(i, ccm[0][1]);\n+\t\t\tgreen[i].g = ccmValue(i, ccm[1][1]);\n+\t\t\tgreen[i].b = ccmValue(i, ccm[2][1]);\n+\t\t\tblue[i].r = ccmValue(i, ccm[0][2]);\n+\t\t\tblue[i].g = ccmValue(i, ccm[1][2]);\n+\t\t\tblue[i].b = ccmValue(i, ccm[2][2]);\n+\t\t\tparams->gammaLut[i] = gammaTable[i / div];\n+\t\t}\n \t}\n }\n \ndiff --git a/src/ipa/simple/algorithms/lut.h b/src/ipa/simple/algorithms/lut.h\nindex 889f864b..77324800 100644\n--- a/src/ipa/simple/algorithms/lut.h\n+++ b/src/ipa/simple/algorithms/lut.h\n@@ -33,6 +33,7 @@ public:\n \n private:\n \tvoid updateGammaTable(IPAContext &context);\n+\tint16_t ccmValue(unsigned int i, float ccm) const;\n };\n \n } /* namespace ipa::soft::algorithms */\ndiff --git a/src/libcamera/software_isp/debayer.cpp b/src/libcamera/software_isp/debayer.cpp\nindex 45fe6960..4c84a556 100644\n--- a/src/libcamera/software_isp/debayer.cpp\n+++ b/src/libcamera/software_isp/debayer.cpp\n@@ -23,9 +23,39 @@ namespace libcamera {\n * \\brief Size of a color lookup table\n */\n \n+/**\n+ * \\struct DebayerParams::CcmColumn\n+ * \\brief Type of a single column of a color correction matrix (CCM)\n+ */\n+\n+/**\n+ * \\var DebayerParams::CcmColumn::r\n+ * \\brief Red (first) component of a CCM column\n+ */\n+\n+/**\n+ * \\var DebayerParams::CcmColumn::g\n+ * \\brief Green (second) component of a CCM column\n+ */\n+\n+/**\n+ * \\var DebayerParams::CcmColumn::b\n+ * \\brief Blue (third) component of a CCM column\n+ */\n+\n /**\n * \\typedef DebayerParams::ColorLookupTable\n- * \\brief Type of the lookup tables for red, green, blue values\n+ * \\brief Type of the simple lookup tables for red, green, blue values\n+ */\n+\n+/**\n+ * \\typedef DebayerParams::CcmLookupTable\n+ * \\brief Type of the CCM lookup tables for red, green, blue values\n+ */\n+\n+/**\n+ * \\typedef DebayerParams::GammaLookupTable\n+ * \\brief Type of the gamma lookup tables for CCM\n */\n \n /**\n@@ -43,6 +73,26 @@ namespace libcamera {\n * \\brief Lookup table for blue color, mapping input values to output values\n */\n \n+/**\n+ * \\var DebayerParams::redCcm\n+ * \\brief CCM lookup table for red color, mapping input values to output values\n+ */\n+\n+/**\n+ * \\var DebayerParams::greenCcm\n+ * \\brief CCM lookup table for green color, mapping input values to output values\n+ */\n+\n+/**\n+ * \\var DebayerParams::blueCcm\n+ * \\brief CCM lookup table for blue color, mapping input values to output values\n+ */\n+\n+/**\n+ * \\var DebayerParams::gammaLut\n+ * \\brief Gamma lookup table used with color correction matrix\n+ */\n+\n /**\n * \\class Debayer\n * \\brief Base debayering class\ndiff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp\nindex 0cd03a8f..5ddfdcf6 100644\n--- a/src/libcamera/software_isp/debayer_cpu.cpp\n+++ b/src/libcamera/software_isp/debayer_cpu.cpp\n@@ -11,6 +11,7 @@\n \n #include \"debayer_cpu.h\"\n \n+#include \n #include \n #include \n #include \n@@ -51,8 +52,12 @@ DebayerCpu::DebayerCpu(std::unique_ptr stats)\n \tenableInputMemcpy_ = true;\n \n \t/* Initialize color lookup tables */\n-\tfor (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++)\n+\tfor (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) {\n \t\tred_[i] = green_[i] = blue_[i] = i;\n+\t\tredCcm_[i] = { 1, 0, 0 };\n+\t\tgreenCcm_[i] = { 0, 1, 0 };\n+\t\tblueCcm_[i] = { 0, 0, 1 };\n+\t}\n }\n \n DebayerCpu::~DebayerCpu() = default;\n@@ -62,12 +67,24 @@ DebayerCpu::~DebayerCpu() = default;\n \tconst pixel_t *curr = (const pixel_t *)src[1] + xShift_; \\\n \tconst pixel_t *next = (const pixel_t *)src[2] + xShift_;\n \n-#define STORE_PIXEL(b, g, r) \\\n-\t*dst++ = blue_[b]; \\\n-\t*dst++ = green_[g]; \\\n-\t*dst++ = red_[r]; \\\n-\tif constexpr (addAlphaByte) \\\n-\t\t*dst++ = 255; \\\n+#define GAMMA(value) \\\n+\t*dst++ = gammaLut_[std::clamp(value, 0, static_cast(gammaLut_.size()) - 1)]\n+\n+#define STORE_PIXEL(b_, g_, r_) \\\n+\tif constexpr (ccmEnabled) { \\\n+\t\tconst DebayerParams::CcmColumn &blue = blueCcm_[b_]; \\\n+\t\tconst DebayerParams::CcmColumn &green = greenCcm_[g_]; \\\n+\t\tconst DebayerParams::CcmColumn &red = redCcm_[r_]; \\\n+\t\tGAMMA(blue.r + blue.g + blue.b); \\\n+\t\tGAMMA(green.r + green.g + green.b); \\\n+\t\tGAMMA(red.r + red.g + red.b); \\\n+\t} else { \\\n+\t\t*dst++ = blue_[b_]; \\\n+\t\t*dst++ = green_[g_]; \\\n+\t\t*dst++ = red_[r_]; \\\n+\t} \\\n+\tif constexpr (addAlphaByte) \\\n+\t\t*dst++ = 255; \\\n \tx++;\n \n /*\n@@ -757,6 +774,10 @@ void DebayerCpu::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output\n \tgreen_ = params.green;\n \tred_ = swapRedBlueGains_ ? params.blue : params.red;\n \tblue_ = swapRedBlueGains_ ? params.red : params.blue;\n+\tgreenCcm_ = params.greenCcm;\n+\tredCcm_ = swapRedBlueGains_ ? params.blueCcm : params.redCcm;\n+\tblueCcm_ = swapRedBlueGains_ ? params.redCcm : params.blueCcm;\n+\tgammaLut_ = params.gammaLut;\n \n \t/* Copy metadata from the input buffer */\n \tFrameMetadata &metadata = output->_d()->metadata();\ndiff --git a/src/libcamera/software_isp/debayer_cpu.h b/src/libcamera/software_isp/debayer_cpu.h\nindex 21c08a2d..9f73a2fd 100644\n--- a/src/libcamera/software_isp/debayer_cpu.h\n+++ b/src/libcamera/software_isp/debayer_cpu.h\n@@ -141,6 +141,10 @@ private:\n \tDebayerParams::ColorLookupTable red_;\n \tDebayerParams::ColorLookupTable green_;\n \tDebayerParams::ColorLookupTable blue_;\n+\tDebayerParams::CcmLookupTable redCcm_;\n+\tDebayerParams::CcmLookupTable greenCcm_;\n+\tDebayerParams::CcmLookupTable blueCcm_;\n+\tDebayerParams::GammaLookupTable gammaLut_;\n \tdebayerFn debayer0_;\n \tdebayerFn debayer1_;\n \tdebayerFn debayer2_;\n","prefixes":["v5","10/10"]}