From patchwork Wed Mar 26 09:08:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 23035 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2F2FBC3213 for ; Wed, 26 Mar 2025 09:09:08 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 88A3F68952; Wed, 26 Mar 2025 10:09:05 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="Q8hwptpR"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D7FBB6894B for ; Wed, 26 Mar 2025 10:09:01 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1742980140; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=iBLyT+6wu075uXDomZnGTYw0QarTmHzFS0y6QCaDGqw=; b=Q8hwptpR+FS6uPVMjhzVkYyCr3UX7DwrYMVTF3RZR5Tk18HvqZSRCNcJIjSNixM/XHJouD yl547tPYCjym+LpjyGiIC9RNwEbJtituFcNH6GNWMYOLZV4ZY97B48rp6olOwuUcN+gGaJ uzQz5GgztR6Y4VRp0G+rjup/SVuuwI0= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-345-zaBHDS7aNF21mj1fIHeIag-1; Wed, 26 Mar 2025 05:08:58 -0400 X-MC-Unique: zaBHDS7aNF21mj1fIHeIag-1 X-Mimecast-MFC-AGG-ID: zaBHDS7aNF21mj1fIHeIag_1742980137 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id BC566180AB16; Wed, 26 Mar 2025 09:08:57 +0000 (UTC) Received: from mzamazal-thinkpadp1gen7.tpbc.com (unknown [10.44.34.15]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 503DF1801747; Wed, 26 Mar 2025 09:08:55 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Robert Mader , Hans de Goede , Laurent Pinchart , Kieran Bingham Subject: [PATCH v8 00/10] Software ISP support for CCM Date: Wed, 26 Mar 2025 10:08:37 +0100 Message-ID: <20250326090849.15494-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 4Wa_xEWfGH7gcWzFljBx5BiZWzJbcEB09bEJDGbza64_1742980137 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This series implements application of color correction matrices in software ISP. Support for color temperature is added to auto white balance algorithm, this is needed to obtain the right CCM. A new Ccm algorithm is added to determine the matrix. Both roughly follow what the hardware pipelines do. Lut algorithm is modified to incorporate the CCM and perform some precomputations in the form of 4 lookup tables (one per each matrix column and one for gamma). A tricky part is to support both CCM and the original (no-CCM) transformations. CCM obviously adds a significant overhead; per-frame time increases by ~45% on Debix Model A and ~85% on TI AM69 SK. This means CCM must be optional, which is determined by presence of Ccm algorithm in the tuning file. The performance critical debayering code must not include extra conditionals and must work efficiently for both the cases. The macros in debayering code are refactored and restructured for this. An added bonus is that this also removes some annoying code duplication. And some more, partially already present, templating is used to select between CCM and non-CCM processing. Performance is much influenced by the data structures used in debayering. I tried to define something reasonable and performed only basic performance testing. Suggestions for possible improvements are welcome. There is already one from Hans about avoiding using std::clamp in debayering but I don’t have a working patch for this yet. When trying to use the following imx219 color correction matrix from rpi subtree, I get quite noticeable color casts: - Ccm: ccms: - ct: 2860 ccm: [ 2.12089, -0.52461, -0.59629, -0.85342, 2.80445, -0.95103, -0.26897, -1.14788, 2.41685 ] - ct: 2960 ccm: [ 2.26962, -0.54174, -0.72789, -0.77008, 2.60271, -0.83262, -0.26036, -1.51254, 2.77289 ] - ct: 3603 ccm: [ 2.18644, -0.66148, -0.52496, -0.77828, 2.69474, -0.91645, -0.25239, -0.83059, 2.08298 ] - ct: 4650 ccm: [ 2.18174, -0.70887, -0.47287, -0.70196, 2.76426, -1.06231, -0.25157, -0.71978, 1.97135 ] - ct: 5858 ccm: [ 2.32392, -0.88421, -0.43971, -0.63821, 2.58348, -0.94527, -0.28541, -0.54112, 1.82653 ] - ct: 7580 ccm: [ 2.21175, -0.53242, -0.67933, -0.57875, 3.07922, -1.50047, -0.27709, -0.73338, 2.01048 ] One of the possible explanations could be that the software ISP AWB (used to determine the color temperature) is no way accurate and produces some (different) color casts too. But it can also be caused by something else. Changes in v8: - vector.h include updated. - Values of the initial CCM lookup tables fixed. - Clarifications of docstrings as suggested by Laurent. Changes in v7: - The changed lookup table types updated in docstrings. Changes in v6: - A transposed CCM tables access was used in debayering, fixed. Unfortunately this slows down the processing by almost 10% in my environment, apparently due to worse locality. - When red-blue swapping is applied in debayering, `r' and `b' items in the CCM lookup tables must be swapped too, fixed. - The multiplication is now CCM * RGB rather than RGB * CCM; this matches the representation of CCMs in the tuning files. - CCM and gain value type changed from double to float to be consistent with the rkisp1 pipeline and controls::ColourCorrectionMatrix. - Some code simplifications suggested by Laurent. - Comments added to lookup tables declarations. Changes in v5: - Local variable to store temperature removed; the context variable used directly instead. - Patch to use RGB instead of a custom struct to store gains added. - Superfluous include of stdlib.h removed from ccm.cpp. - Error returned from Ccm::init when there is no CCM defined. - Ccm::kTemperatureThreshold moved to a constexpr global variable. - `const' added to const variables. - Separate color lookup tables for CCM / simple rather than unions. - ccmEnabled passed as a normal rather than template argument to setDebayerFunctions. - ccmEnabled is determined in init rather than in configure. Changes in v4: - utils::abs_diff used instead of the ugly typecasting. - A missing argument type added to Debayer::configure docstring. - CCM and related processing is disabled when tuning data is not available. - Unneeded initialization of context_.activeState.ccm.enabled to the default value removed. Changes in v3: - Rebased on master. - AWB gains applied. - ccmEnabled docstring moved to the right patch. - Small notes about std::variant and about std::clamp added to the commit message of the last patch. Changes in v2: - Fix of the gamma table index upper boundary. - Fix of red<->blue being temporarily swapped in an intermediate patch. - Fix of CCM transformation when swapRedBlueGains_ is true. - A related minor lookup data structures rearrangement, with an unintended but welcome effect of ~10% speedup with CCM in my environment. Milan Zamazal (10): libcamera: software_isp: Determine color temperature libcamera: software_isp: Use RGB type to represent gains libcamera: software_isp: Store color temperature to metadata libcamera: software_isp: lut: Remove maybe_unused on a used argument libcamera: software_isp: Use common code to store debayered pixels libcamera: software_isp: Use a macro to assign debayering methods libcamera: software_isp: Add CCM algorithm libcamera: software_isp: Add an example CCM to uncalibrated.yaml libcamera: software_isp: Track whether CCM is enabled libcamera: software_isp: Apply CCM in debayering .../internal/software_isp/debayer_params.h | 36 +++- .../internal/software_isp/software_isp.h | 1 + include/libcamera/ipa/soft.mojom | 2 +- src/ipa/simple/algorithms/awb.cpp | 29 +++- src/ipa/simple/algorithms/ccm.cpp | 76 +++++++++ src/ipa/simple/algorithms/ccm.h | 43 +++++ src/ipa/simple/algorithms/lut.cpp | 63 ++++--- src/ipa/simple/algorithms/lut.h | 1 + src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/data/uncalibrated.yaml | 9 + src/ipa/simple/ipa_context.h | 21 ++- src/ipa/simple/soft_simple.cpp | 8 +- src/libcamera/software_isp/debayer.cpp | 60 ++++++- src/libcamera/software_isp/debayer.h | 3 +- src/libcamera/software_isp/debayer_cpu.cpp | 158 +++++++++++------- src/libcamera/software_isp/debayer_cpu.h | 39 +++-- src/libcamera/software_isp/software_isp.cpp | 5 +- 17 files changed, 432 insertions(+), 123 deletions(-) create mode 100644 src/ipa/simple/algorithms/ccm.cpp create mode 100644 src/ipa/simple/algorithms/ccm.h