From patchwork Mon Jan 13 13:50:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 22536 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 94492C32F6 for ; Mon, 13 Jan 2025 13:51:28 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2A5AA68541; Mon, 13 Jan 2025 14:51:28 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="EUOK4EzI"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id ADEC36851F for ; Mon, 13 Jan 2025 14:51:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736776283; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tW5GKu8gi9RaBH6vdVUEjkLQv78IN0iNFmvRS0mhFrw=; b=EUOK4EzItQ4Oafu01Ea7MHemcS/LR6f4Pkzl4nIPudRdn9yOW3hx2Laiw9P1MuyI99HfQo D+0A7PEy8JPUAWF6PMXAEJfUTIBCoJiNuumyNOgfG2I5wwvrWeyqWvU+caQ3SHNu6yGSam nmzHhbVgLPMeHy5Gw8PPa4KjGzFd7ok= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-568-A50A95S6PDy2dpsvBq_GzQ-1; Mon, 13 Jan 2025 08:51:22 -0500 X-MC-Unique: A50A95S6PDy2dpsvBq_GzQ-1 X-Mimecast-MFC-AGG-ID: A50A95S6PDy2dpsvBq_GzQ Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 319FC195609E; Mon, 13 Jan 2025 13:51:21 +0000 (UTC) Received: from mzamazal-thinkpadp1gen3.tpbc.com (unknown [10.45.224.6]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 1A4B71956056; Mon, 13 Jan 2025 13:51:17 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Robert Mader , Hans de Goede , Laurent Pinchart , Kieran Bingham Subject: [PATCH v4 1/9] libcamera: software_isp: Determine color temperature Date: Mon, 13 Jan 2025 14:50:58 +0100 Message-ID: <20250113135108.13924-2-mzamazal@redhat.com> In-Reply-To: <20250113135108.13924-1-mzamazal@redhat.com> References: <20250113135108.13924-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: BPsJ6gWbU6q28GUWfSzXHHo0Fa2ewQsiNfIb9L36PKY_1736776281 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The AWB algorithm has data to determine color temperature of the image. Let's compute the temperature from it and store it into the context. This piece of information is currently unused but it will be needed in a followup patch introducing support for color correction matrix. Let's store the white balance related information under `awb' subsection of the active state, as the hardware pipelines do. Signed-off-by: Milan Zamazal Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/ipa/simple/algorithms/awb.cpp | 13 ++++++++++--- src/ipa/simple/algorithms/lut.cpp | 2 +- src/ipa/simple/ipa_context.h | 11 +++++++---- 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/src/ipa/simple/algorithms/awb.cpp b/src/ipa/simple/algorithms/awb.cpp index 195de41d..f4fe1af6 100644 --- a/src/ipa/simple/algorithms/awb.cpp +++ b/src/ipa/simple/algorithms/awb.cpp @@ -12,6 +12,7 @@ #include +#include "libipa/colours.h" #include "simple/ipa_context.h" namespace libcamera { @@ -23,7 +24,7 @@ namespace ipa::soft::algorithms { int Awb::configure(IPAContext &context, [[maybe_unused]] const IPAConfigInfo &configInfo) { - auto &gains = context.activeState.gains; + auto &gains = context.activeState.awb.gains; gains.red = gains.green = gains.blue = 1.0; return 0; @@ -54,12 +55,18 @@ void Awb::process(IPAContext &context, * Calculate red and blue gains for AWB. * Clamp max gain at 4.0, this also avoids 0 division. */ - auto &gains = context.activeState.gains; + auto &gains = context.activeState.awb.gains; gains.red = sumR <= sumG / 4 ? 4.0 : static_cast(sumG) / sumR; gains.blue = sumB <= sumG / 4 ? 4.0 : static_cast(sumG) / sumB; /* Green gain is fixed to 1.0 */ - LOG(IPASoftAwb, Debug) << "gain R/B " << gains.red << "/" << gains.blue; + RGB rgbGains{ { 1 / gains.red, 1 / gains.green, 1 / gains.blue } }; + uint32_t temperature = estimateCCT(rgbGains); + context.activeState.awb.temperatureK = temperature; + + LOG(IPASoftAwb, Debug) + << "gain R/B: " << gains.red << "/" << gains.blue + << "; temperature: " << context.activeState.awb.temperatureK; } REGISTER_IPA_ALGORITHM(Awb, "Awb") diff --git a/src/ipa/simple/algorithms/lut.cpp b/src/ipa/simple/algorithms/lut.cpp index 0ba2391f..d75ff710 100644 --- a/src/ipa/simple/algorithms/lut.cpp +++ b/src/ipa/simple/algorithms/lut.cpp @@ -95,7 +95,7 @@ void Lut::prepare(IPAContext &context, context.activeState.gamma.contrast != context.activeState.knobs.contrast) updateGammaTable(context); - auto &gains = context.activeState.gains; + auto &gains = context.activeState.awb.gains; auto &gammaTable = context.activeState.gamma.gammaTable; const unsigned int gammaTableSize = gammaTable.size(); diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index 4af51306..607af45a 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -36,10 +36,13 @@ struct IPAActiveState { } blc; struct { - double red; - double green; - double blue; - } gains; + struct { + double red; + double green; + double blue; + } gains; + unsigned int temperatureK; + } awb; static constexpr unsigned int kGammaLookupSize = 1024; struct { From patchwork Mon Jan 13 13:50:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 22537 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1A067C32F6 for ; Mon, 13 Jan 2025 13:51:32 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 965D868541; Mon, 13 Jan 2025 14:51:31 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="gduDW0Kd"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 973A86851C for ; Mon, 13 Jan 2025 14:51:27 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736776286; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RBG2NFjo1crXuGFY05OCFRYwkUiTlch/sgM30RDWovE=; b=gduDW0KdZdBaxMhvvLc2lgi3b/pT1MMiV8JdodsrtqPYcQOoPRsJ0zL72alVKskCrD82tC LUwrBF99rQOSLt5hgahp3BZ5J8f4FqExysQWp5Zb7CRHBPxflfcdpkFyGGQxpy04ORR4rC QezRMzhEky85TZZZiMH+ZW8qXwG3wYU= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-591-js_FSiuCOqeENXl--WGT0g-1; Mon, 13 Jan 2025 08:51:25 -0500 X-MC-Unique: js_FSiuCOqeENXl--WGT0g-1 X-Mimecast-MFC-AGG-ID: js_FSiuCOqeENXl--WGT0g Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 134921955D5E; Mon, 13 Jan 2025 13:51:24 +0000 (UTC) Received: from mzamazal-thinkpadp1gen3.tpbc.com (unknown [10.45.224.6]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id C1C0B1956056; Mon, 13 Jan 2025 13:51:21 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Robert Mader , Hans de Goede , Laurent Pinchart , Kieran Bingham Subject: [PATCH v4 2/9] libcamera: software_isp: Store color temperature to metadata Date: Mon, 13 Jan 2025 14:50:59 +0100 Message-ID: <20250113135108.13924-3-mzamazal@redhat.com> In-Reply-To: <20250113135108.13924-1-mzamazal@redhat.com> References: <20250113135108.13924-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: ZMqTj_VIwIy9OjCOuYi7DQspueaivokZSoyEz80TYsA_1736776284 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Image color temperature is a piece of information that should be reported in metadata, let's put it there. Metadata is currently not reported in simple pipeline but we should make at least newly added information ready to be reported. Signed-off-by: Milan Zamazal Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/ipa/simple/algorithms/awb.cpp | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/ipa/simple/algorithms/awb.cpp b/src/ipa/simple/algorithms/awb.cpp index f4fe1af6..30af3d31 100644 --- a/src/ipa/simple/algorithms/awb.cpp +++ b/src/ipa/simple/algorithms/awb.cpp @@ -12,6 +12,8 @@ #include +#include + #include "libipa/colours.h" #include "simple/ipa_context.h" @@ -34,7 +36,7 @@ void Awb::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, const SwIspStats *stats, - [[maybe_unused]] ControlList &metadata) + ControlList &metadata) { const SwIspStats::Histogram &histogram = stats->yHistogram; const uint8_t blackLevel = context.activeState.blc.level; @@ -63,6 +65,7 @@ void Awb::process(IPAContext &context, RGB rgbGains{ { 1 / gains.red, 1 / gains.green, 1 / gains.blue } }; uint32_t temperature = estimateCCT(rgbGains); context.activeState.awb.temperatureK = temperature; + metadata.set(controls::ColourTemperature, temperature); LOG(IPASoftAwb, Debug) << "gain R/B: " << gains.red << "/" << gains.blue From patchwork Mon Jan 13 13:51:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 22538 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 59F5FC3302 for ; Mon, 13 Jan 2025 13:51:34 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0941A6854A; Mon, 13 Jan 2025 14:51:34 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="Q3WB5q5d"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3853E68503 for ; Mon, 13 Jan 2025 14:51:32 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736776291; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h0hbzxDqPXAEUbeN6Thx3pZJw1gKBzwArsCFZJJljtU=; b=Q3WB5q5dU6+NSvcedh+YabkqlO0m645uF19a/u5hN8u79FVZBzvSSSwdpKJtQDgXFojrP+ WBenNM6G4LGw7ZoT9zgyp6O2gKaHSszJ0oVkjPnYAaJp+SnxVB3w/bjtzh9QwKpaDsO8JZ KwJo+DuADDnH8coQ/ojXxjkV5xP/myY= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-426-diVY0ABfMSWW_-wmLi2XsQ-1; Mon, 13 Jan 2025 08:51:28 -0500 X-MC-Unique: diVY0ABfMSWW_-wmLi2XsQ-1 X-Mimecast-MFC-AGG-ID: diVY0ABfMSWW_-wmLi2XsQ Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 22CF61955D82; Mon, 13 Jan 2025 13:51:27 +0000 (UTC) Received: from mzamazal-thinkpadp1gen3.tpbc.com (unknown [10.45.224.6]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 9273F1956056; Mon, 13 Jan 2025 13:51:24 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Robert Mader , Hans de Goede , Laurent Pinchart , Kieran Bingham Subject: [PATCH v4 3/9] libcamera: software_isp: lut: Remove maybe_unused on a used argument Date: Mon, 13 Jan 2025 14:51:00 +0100 Message-ID: <20250113135108.13924-4-mzamazal@redhat.com> In-Reply-To: <20250113135108.13924-1-mzamazal@redhat.com> References: <20250113135108.13924-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: T59nP6N3R1o07F3Sse3e47s-zoxBJu2u46oOVzZzkug_1736776287 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" `params' argument of Lut::prepare is actually used, let's remove maybe_unused from it. Signed-off-by: Milan Zamazal Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/ipa/simple/algorithms/lut.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ipa/simple/algorithms/lut.cpp b/src/ipa/simple/algorithms/lut.cpp index d75ff710..243f0818 100644 --- a/src/ipa/simple/algorithms/lut.cpp +++ b/src/ipa/simple/algorithms/lut.cpp @@ -83,7 +83,7 @@ void Lut::updateGammaTable(IPAContext &context) void Lut::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, - [[maybe_unused]] DebayerParams *params) + DebayerParams *params) { /* * Update the gamma table if needed. This means if black level changes From patchwork Mon Jan 13 13:51:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 22539 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E46A9C32F6 for ; Mon, 13 Jan 2025 13:51:37 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 99E4E6854B; Mon, 13 Jan 2025 14:51:37 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="Y0Qju6/1"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 15B6668541 for ; Mon, 13 Jan 2025 14:51:35 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736776294; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JRPXyvmu52/wDdirpWjkxW4wp9Well/O1B2jolE1zHA=; b=Y0Qju6/167NgTS69W8YAgPLucuGPVVTiJPfExh0Za0JCIZ3uJoWH6U3k5eWlmtgsMK4/Af ZTpxsjqs/cTEbv7y0jHgJP70DUtFpQvP2tE/11NbnacOHZZ4tP3WiU9ONHNFgDcBS7a8PM 7jXb/V07YBLsBxUjK8ye+2Vtw8Yg1rM= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-84-Jy_JgPvfPfWklz30NyUzNQ-1; Mon, 13 Jan 2025 08:51:31 -0500 X-MC-Unique: Jy_JgPvfPfWklz30NyUzNQ-1 X-Mimecast-MFC-AGG-ID: Jy_JgPvfPfWklz30NyUzNQ Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EF9B71956075; Mon, 13 Jan 2025 13:51:29 +0000 (UTC) Received: from mzamazal-thinkpadp1gen3.tpbc.com (unknown [10.45.224.6]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id A445B1955BE3; Mon, 13 Jan 2025 13:51:27 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Robert Mader , Hans de Goede , Laurent Pinchart , Kieran Bingham Subject: [PATCH v4 4/9] libcamera: software_isp: Use common code to store debayered pixels Date: Mon, 13 Jan 2025 14:51:01 +0100 Message-ID: <20250113135108.13924-5-mzamazal@redhat.com> In-Reply-To: <20250113135108.13924-1-mzamazal@redhat.com> References: <20250113135108.13924-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: sZOdgUsYMNzClETSn6WbkEckqJe6hmxNkIsLOaV_K-U_1736776290 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The debayering macros use the same pattern, let's extract it to a common macro. This reduces code duplication a bit now and it'll make changes of debayering easier when color correction matrix is introduced. Signed-off-by: Milan Zamazal Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/libcamera/software_isp/debayer_cpu.cpp | 56 +++++++++++----------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index 31ab96ab..0eabced2 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -62,57 +62,57 @@ DebayerCpu::~DebayerCpu() = default; const pixel_t *curr = (const pixel_t *)src[1] + xShift_; \ const pixel_t *next = (const pixel_t *)src[2] + xShift_; +#define STORE_PIXEL(b, g, r) \ + *dst++ = blue_[b]; \ + *dst++ = green_[g]; \ + *dst++ = red_[r]; \ + if constexpr (addAlphaByte) \ + *dst++ = 255; \ + x++; + /* * RGR * GBG * RGR */ -#define BGGR_BGR888(p, n, div) \ - *dst++ = blue_[curr[x] / (div)]; \ - *dst++ = green_[(prev[x] + curr[x - p] + curr[x + n] + next[x]) / (4 * (div))]; \ - *dst++ = red_[(prev[x - p] + prev[x + n] + next[x - p] + next[x + n]) / (4 * (div))]; \ - if constexpr (addAlphaByte) \ - *dst++ = 255; \ - x++; +#define BGGR_BGR888(p, n, div) \ + STORE_PIXEL( \ + curr[x] / (div), \ + (prev[x] + curr[x - p] + curr[x + n] + next[x]) / (4 * (div)), \ + (prev[x - p] + prev[x + n] + next[x - p] + next[x + n]) / (4 * (div))) /* * GBG * RGR * GBG */ -#define GRBG_BGR888(p, n, div) \ - *dst++ = blue_[(prev[x] + next[x]) / (2 * (div))]; \ - *dst++ = green_[curr[x] / (div)]; \ - *dst++ = red_[(curr[x - p] + curr[x + n]) / (2 * (div))]; \ - if constexpr (addAlphaByte) \ - *dst++ = 255; \ - x++; +#define GRBG_BGR888(p, n, div) \ + STORE_PIXEL( \ + (prev[x] + next[x]) / (2 * (div)), \ + curr[x] / (div), \ + (curr[x - p] + curr[x + n]) / (2 * (div))) /* * GRG * BGB * GRG */ -#define GBRG_BGR888(p, n, div) \ - *dst++ = blue_[(curr[x - p] + curr[x + n]) / (2 * (div))]; \ - *dst++ = green_[curr[x] / (div)]; \ - *dst++ = red_[(prev[x] + next[x]) / (2 * (div))]; \ - if constexpr (addAlphaByte) \ - *dst++ = 255; \ - x++; +#define GBRG_BGR888(p, n, div) \ + STORE_PIXEL( \ + (curr[x - p] + curr[x + n]) / (2 * (div)), \ + curr[x] / (div), \ + (prev[x] + next[x]) / (2 * (div))) /* * BGB * GRG * BGB */ -#define RGGB_BGR888(p, n, div) \ - *dst++ = blue_[(prev[x - p] + prev[x + n] + next[x - p] + next[x + n]) / (4 * (div))]; \ - *dst++ = green_[(prev[x] + curr[x - p] + curr[x + n] + next[x]) / (4 * (div))]; \ - *dst++ = red_[curr[x] / (div)]; \ - if constexpr (addAlphaByte) \ - *dst++ = 255; \ - x++; +#define RGGB_BGR888(p, n, div) \ + STORE_PIXEL( \ + (prev[x - p] + prev[x + n] + next[x - p] + next[x + n]) / (4 * (div)), \ + (prev[x] + curr[x - p] + curr[x + n] + next[x]) / (4 * (div)), \ + curr[x] / (div)) template void DebayerCpu::debayer8_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) From patchwork Mon Jan 13 13:51:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 22540 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8E1A9C3302 for ; Mon, 13 Jan 2025 13:51:39 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 397376854D; Mon, 13 Jan 2025 14:51:39 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="iBJt+540"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 965C06854A for ; Mon, 13 Jan 2025 14:51:37 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736776296; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lA/H+fbgE2qg/IlS0be+rqc4a6iPwCPmnwo82T/i2YA=; b=iBJt+540YbAJ5O/OuHVHF2JZc5cfStNOTF/2Ons1OdPKqWlmyHfP2hNPE7NOAdt8ag2z+b 68L4soIM8bihXOHCePgu6DZpMvj+IEqK07OYL/5C75u1ELkjDcAHVXpUjFRLV5khkAONiF oXOJicVHzABwehQWQ8Jz3Bel9MyP31M= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-684-36m8fym9Oa-CEi7-D6XRZw-1; Mon, 13 Jan 2025 08:51:33 -0500 X-MC-Unique: 36m8fym9Oa-CEi7-D6XRZw-1 X-Mimecast-MFC-AGG-ID: 36m8fym9Oa-CEi7-D6XRZw Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id AD925195605F; Mon, 13 Jan 2025 13:51:32 +0000 (UTC) Received: from mzamazal-thinkpadp1gen3.tpbc.com (unknown [10.45.224.6]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 4E7121956056; Mon, 13 Jan 2025 13:51:30 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Robert Mader , Hans de Goede , Laurent Pinchart , Kieran Bingham Subject: [PATCH v4 5/9] libcamera: software_isp: Use a macro to assign debayering methods Date: Mon, 13 Jan 2025 14:51:02 +0100 Message-ID: <20250113135108.13924-6-mzamazal@redhat.com> In-Reply-To: <20250113135108.13924-1-mzamazal@redhat.com> References: <20250113135108.13924-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: rLepVesLVoX2GH5M-8KS5Oe1VYhoEg-QRMkIlTddIY8_1736776292 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Assignments of the debayering methods to be used is a repetitive pattern that can be (arguably) better expressed by using a macro. This removes some duplication and also makes easier to introduce more complex assignment patterns. This will be useful once color correction matrix support is added. Signed-off-by: Milan Zamazal Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/libcamera/software_isp/debayer_cpu.cpp | 25 ++++++++++------------ 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index 0eabced2..01cfb36b 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -368,6 +368,10 @@ int DebayerCpu::setupStandardBayerOrder(BayerFormat::Order order) return 0; } +#define SET_DEBAYER_METHODS(method0, method1) \ + debayer0_ = addAlphaByte ? &DebayerCpu::method0 : &DebayerCpu::method0; \ + debayer1_ = addAlphaByte ? &DebayerCpu::method1 : &DebayerCpu::method1; + int DebayerCpu::setDebayerFunctions(PixelFormat inputFormat, PixelFormat outputFormat) { BayerFormat bayerFormat = @@ -423,16 +427,13 @@ int DebayerCpu::setDebayerFunctions(PixelFormat inputFormat, PixelFormat outputF isStandardBayerOrder(bayerFormat.order)) { switch (bayerFormat.bitDepth) { case 8: - debayer0_ = addAlphaByte ? &DebayerCpu::debayer8_BGBG_BGR888 : &DebayerCpu::debayer8_BGBG_BGR888; - debayer1_ = addAlphaByte ? &DebayerCpu::debayer8_GRGR_BGR888 : &DebayerCpu::debayer8_GRGR_BGR888; + SET_DEBAYER_METHODS(debayer8_BGBG_BGR888, debayer8_GRGR_BGR888) break; case 10: - debayer0_ = addAlphaByte ? &DebayerCpu::debayer10_BGBG_BGR888 : &DebayerCpu::debayer10_BGBG_BGR888; - debayer1_ = addAlphaByte ? &DebayerCpu::debayer10_GRGR_BGR888 : &DebayerCpu::debayer10_GRGR_BGR888; + SET_DEBAYER_METHODS(debayer10_BGBG_BGR888, debayer10_GRGR_BGR888) break; case 12: - debayer0_ = addAlphaByte ? &DebayerCpu::debayer12_BGBG_BGR888 : &DebayerCpu::debayer12_BGBG_BGR888; - debayer1_ = addAlphaByte ? &DebayerCpu::debayer12_GRGR_BGR888 : &DebayerCpu::debayer12_GRGR_BGR888; + SET_DEBAYER_METHODS(debayer12_BGBG_BGR888, debayer12_GRGR_BGR888) break; } setupStandardBayerOrder(bayerFormat.order); @@ -443,20 +444,16 @@ int DebayerCpu::setDebayerFunctions(PixelFormat inputFormat, PixelFormat outputF bayerFormat.packing == BayerFormat::Packing::CSI2) { switch (bayerFormat.order) { case BayerFormat::BGGR: - debayer0_ = addAlphaByte ? &DebayerCpu::debayer10P_BGBG_BGR888 : &DebayerCpu::debayer10P_BGBG_BGR888; - debayer1_ = addAlphaByte ? &DebayerCpu::debayer10P_GRGR_BGR888 : &DebayerCpu::debayer10P_GRGR_BGR888; + SET_DEBAYER_METHODS(debayer10P_BGBG_BGR888, debayer10P_GRGR_BGR888) return 0; case BayerFormat::GBRG: - debayer0_ = addAlphaByte ? &DebayerCpu::debayer10P_GBGB_BGR888 : &DebayerCpu::debayer10P_GBGB_BGR888; - debayer1_ = addAlphaByte ? &DebayerCpu::debayer10P_RGRG_BGR888 : &DebayerCpu::debayer10P_RGRG_BGR888; + SET_DEBAYER_METHODS(debayer10P_GBGB_BGR888, debayer10P_RGRG_BGR888) return 0; case BayerFormat::GRBG: - debayer0_ = addAlphaByte ? &DebayerCpu::debayer10P_GRGR_BGR888 : &DebayerCpu::debayer10P_GRGR_BGR888; - debayer1_ = addAlphaByte ? &DebayerCpu::debayer10P_BGBG_BGR888 : &DebayerCpu::debayer10P_BGBG_BGR888; + SET_DEBAYER_METHODS(debayer10P_GRGR_BGR888, debayer10P_BGBG_BGR888) return 0; case BayerFormat::RGGB: - debayer0_ = addAlphaByte ? &DebayerCpu::debayer10P_RGRG_BGR888 : &DebayerCpu::debayer10P_RGRG_BGR888; - debayer1_ = addAlphaByte ? &DebayerCpu::debayer10P_GBGB_BGR888 : &DebayerCpu::debayer10P_GBGB_BGR888; + SET_DEBAYER_METHODS(debayer10P_RGRG_BGR888, debayer10P_GBGB_BGR888) return 0; default: break; From patchwork Mon Jan 13 13:51:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 22542 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7EEB3C3302 for ; Mon, 13 Jan 2025 13:51:44 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 208F36854D; Mon, 13 Jan 2025 14:51:44 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="BhApVKs/"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B47776854A for ; Mon, 13 Jan 2025 14:51:41 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736776300; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pomRdzGX5QZl8kRHCDHGyODvuT2feY78MpIK+hPrMaU=; b=BhApVKs/zH0vXU/dpmwh8JVkaMJaBYyO8thM6/DT08HYCeeiR7ZptEdQ+cgQGzx/xPMUjY /Yj5w3fpifcDN5ZolV5UPVA6hjMgiRXEu8zuys6bbEK7kuA+2PfW7oWn8jjDGTNnikxkRr BcBMNZhfijdtwK0TW3vyneCfe2uYQqw= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-318-j6EDIHypOIWqzxI6bb_OBw-1; Mon, 13 Jan 2025 08:51:36 -0500 X-MC-Unique: j6EDIHypOIWqzxI6bb_OBw-1 X-Mimecast-MFC-AGG-ID: j6EDIHypOIWqzxI6bb_OBw Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 6AC411955DCC; Mon, 13 Jan 2025 13:51:35 +0000 (UTC) Received: from mzamazal-thinkpadp1gen3.tpbc.com (unknown [10.45.224.6]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 3173B1955BE3; Mon, 13 Jan 2025 13:51:32 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Robert Mader , Hans de Goede , Laurent Pinchart , Kieran Bingham Subject: [PATCH v4 6/9] libcamera: software_isp: Add CCM algorithm Date: Mon, 13 Jan 2025 14:51:03 +0100 Message-ID: <20250113135108.13924-7-mzamazal@redhat.com> In-Reply-To: <20250113135108.13924-1-mzamazal@redhat.com> References: <20250113135108.13924-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: SllGPZCoE-Y98biWOjfZrfcpV_y6l8GY3kcDqaDO058_1736776295 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds color correction matrix (CCM) algorithm to software ISP. It is based on the corresponding algorithm in rkisp1. The primary difference against hardware pipelines is that applying the CCM is optional. Applying CCM causes a significant slowdown, time needed to process a frame raises by 40-90% on tested platforms. If CCM is really needed, it can be applied, if not, it's better to stick without it. This can be configured by presence or omission of Ccm algorithm in the tuning file. CCM is changed only if the determined temperature changes by at least 100 K (an arbitrarily selected value), to avoid recomputing the matrices and lookup tables all the time. The outputs of the algorithm are not used yet, they will be enabled in followup patches. Signed-off-by: Milan Zamazal Reviewed-by: Kieran Bingham --- src/ipa/simple/algorithms/ccm.cpp | 88 +++++++++++++++++++++++++++ src/ipa/simple/algorithms/ccm.h | 45 ++++++++++++++ src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/ipa_context.h | 13 ++++ 4 files changed, 147 insertions(+) create mode 100644 src/ipa/simple/algorithms/ccm.cpp create mode 100644 src/ipa/simple/algorithms/ccm.h diff --git a/src/ipa/simple/algorithms/ccm.cpp b/src/ipa/simple/algorithms/ccm.cpp new file mode 100644 index 00000000..3c7fca2d --- /dev/null +++ b/src/ipa/simple/algorithms/ccm.cpp @@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Ideas On Board + * Copyright (C) 2024, Red Hat Inc. + * + * Color correction matrix + */ + +#include "ccm.h" + +#include + +#include +#include + +#include + +namespace libcamera { + +namespace ipa::soft::algorithms { + +LOG_DEFINE_CATEGORY(IPASoftCcm) + +unsigned int Ccm::kTemperatureThreshold = 100; + +int Ccm::init([[maybe_unused]] IPAContext &context, const YamlObject &tuningData) +{ + int ret = ccm_.readYaml(tuningData["ccms"], "ct", "ccm"); + if (ret < 0) { + LOG(IPASoftCcm, Warning) + << "Failed to parse 'ccm' " + << "parameter from tuning file; falling back to unit matrix"; + ccmEnabled_ = false; + } else { + ccmEnabled_ = true; + } + + return 0; +} + +void Ccm::prepare(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, [[maybe_unused]] DebayerParams *params) +{ + context.activeState.ccm.enabled = ccmEnabled_; + + if (!ccmEnabled_) + return; + + unsigned int ct = context.activeState.awb.temperatureK; + + /* Change CCM only on bigger temperature changes. */ + if (frame > 0 && + utils::abs_diff(ct, ct_) < kTemperatureThreshold) { + frameContext.ccm.ccm = context.activeState.ccm.ccm; + context.activeState.ccm.changed = false; + return; + } + + ct_ = ct; + Matrix ccm = ccm_.getInterpolated(ct); + + context.activeState.ccm.ccm = ccm; + frameContext.ccm.ccm = ccm; + context.activeState.ccm.changed = true; +} + +void Ccm::process([[maybe_unused]] IPAContext &context, + [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, + [[maybe_unused]] const SwIspStats *stats, + ControlList &metadata) +{ + if (!ccmEnabled_) + return; + + float m[9]; + for (unsigned int i = 0; i < 3; i++) { + for (unsigned int j = 0; j < 3; j++) + m[i * 3 + j] = frameContext.ccm.ccm[i][j]; + } + metadata.set(controls::ColourCorrectionMatrix, m); +} + +REGISTER_IPA_ALGORITHM(Ccm, "Ccm") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/ccm.h b/src/ipa/simple/algorithms/ccm.h new file mode 100644 index 00000000..23481a08 --- /dev/null +++ b/src/ipa/simple/algorithms/ccm.h @@ -0,0 +1,45 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Color correction matrix + */ + +#pragma once + +#include "libcamera/internal/matrix.h" + +#include + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class Ccm : public Algorithm +{ +public: + Ccm() = default; + ~Ccm() = default; + + int init(IPAContext &context, const YamlObject &tuningData) override; + void prepare(IPAContext &context, + const uint32_t frame, + IPAFrameContext &frameContext, + DebayerParams *params) override; + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const SwIspStats *stats, + ControlList &metadata) override; + +private: + static unsigned int kTemperatureThreshold; + unsigned int ct_; + bool ccmEnabled_; + Interpolator> ccm_; +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index 37a2eb53..2d0adb05 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -4,5 +4,6 @@ soft_simple_ipa_algorithms = files([ 'awb.cpp', 'agc.cpp', 'blc.cpp', + 'ccm.cpp', 'lut.cpp', ]) diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index 607af45a..0def3eef 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -13,6 +13,8 @@ #include +#include "libcamera/internal/matrix.h" + #include namespace libcamera { @@ -50,6 +52,13 @@ struct IPAActiveState { uint8_t blackLevel; double contrast; } gamma; + + struct { + Matrix ccm; + bool enabled; + bool changed; + } ccm; + struct { /* 0..2 range, 1.0 = normal */ std::optional contrast; @@ -57,6 +66,10 @@ struct IPAActiveState { }; struct IPAFrameContext : public FrameContext { + struct { + Matrix ccm; + } ccm; + struct { int32_t exposure; double gain; From patchwork Mon Jan 13 13:51:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 22541 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9D262C32F6 for ; Mon, 13 Jan 2025 13:51:43 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 513ED68551; Mon, 13 Jan 2025 14:51:43 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="FXwesRQM"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9FC8268541 for ; Mon, 13 Jan 2025 14:51:41 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736776300; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v61EWbTrDt9oFN82LsAZZhORc4hZjPD1tJyXe8P7+vQ=; b=FXwesRQM5pjG00mh0DZfzzQMLB0VE689hAL3SRXO8FTq0BnWuSWgKvh8XYMFHS1OyPHQ7u h6M+3LoLYNDWSDjXWstiY3czDdGtgFxvqUVKKhNe/2kKQ++T/Lv5iuC0HbD0eGLC7Wy5k+ E0DZ+Zsd0/9B4vChiHC6M4UsEIxS4Ks= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-538-8VAAJiuhNoCVVkdV1QljWQ-1; Mon, 13 Jan 2025 08:51:39 -0500 X-MC-Unique: 8VAAJiuhNoCVVkdV1QljWQ-1 X-Mimecast-MFC-AGG-ID: 8VAAJiuhNoCVVkdV1QljWQ Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 5D5D31955D66; Mon, 13 Jan 2025 13:51:38 +0000 (UTC) Received: from mzamazal-thinkpadp1gen3.tpbc.com (unknown [10.45.224.6]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 06B0E1956056; Mon, 13 Jan 2025 13:51:35 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Robert Mader , Hans de Goede , Laurent Pinchart , Kieran Bingham Subject: [PATCH v4 7/9] libcamera: software_isp: Add an example CCM to uncalibrated.yaml Date: Mon, 13 Jan 2025 14:51:04 +0100 Message-ID: <20250113135108.13924-8-mzamazal@redhat.com> In-Reply-To: <20250113135108.13924-1-mzamazal@redhat.com> References: <20250113135108.13924-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: XL9LlHjsIz59adef-pmcUapKX96dBKowJKaGQyN04ys_1736776298 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" For performance reasons, color correction matrix (CCM) is not applied by default in software ISP. But let's add a commented out example how to define it to the default tuning file. Signed-off-by: Milan Zamazal Acked-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/ipa/simple/data/uncalibrated.yaml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index 3f147112..8df3cd4e 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -5,6 +5,13 @@ version: 1 algorithms: - BlackLevel: - Awb: + # Color correction matrices can be defined here: + # - Ccm: + # ccms: + # - ct: 6500 + # ccm: [ 1, 0, 0, + # 0, 1, 0, + # 0, 0, 1] - Lut: - Agc: ... From patchwork Mon Jan 13 13:51:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 22543 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4CC19C32F6 for ; Mon, 13 Jan 2025 13:51:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DC3AA68557; Mon, 13 Jan 2025 14:51:46 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="Ex1DwNO8"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C7CCF68552 for ; Mon, 13 Jan 2025 14:51:44 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736776303; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wEV6O0yE5nijpliaEpOkziOSGfqqkgcYDqeBFZinTdA=; b=Ex1DwNO8Ny2ZdQhjKKN1P5mY0/R3TN+hln+afNjphourLtbeWLqXYlQz4xCHHLC9GBd0Tp ixLwa4x0zsGlZE9OiXFKBDBL4KaNzwWnJWwWrnb1mbOMwz2OC+T36ast99JQYrucTSXbUK bSFSLBaOVLI4EGIBFxqkmP8HF2knSsk= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-503-9OR0a3zlNlmn9zYGYZLbbA-1; Mon, 13 Jan 2025 08:51:42 -0500 X-MC-Unique: 9OR0a3zlNlmn9zYGYZLbbA-1 X-Mimecast-MFC-AGG-ID: 9OR0a3zlNlmn9zYGYZLbbA Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 362E01955DDE; Mon, 13 Jan 2025 13:51:41 +0000 (UTC) Received: from mzamazal-thinkpadp1gen3.tpbc.com (unknown [10.45.224.6]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id F37461956056; Mon, 13 Jan 2025 13:51:38 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Robert Mader , Hans de Goede , Laurent Pinchart , Kieran Bingham Subject: [PATCH v4 8/9] libcamera: software_isp: Track whether CCM is enabled Date: Mon, 13 Jan 2025 14:51:05 +0100 Message-ID: <20250113135108.13924-9-mzamazal@redhat.com> In-Reply-To: <20250113135108.13924-1-mzamazal@redhat.com> References: <20250113135108.13924-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: rVQw81vA-nJqIj4YKbdYMJARSbd62r7hA6Lvj5Wwd7k_1736776301 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Applying color correction matrix (CCM) in software ISP is optional due to performance reasons. CCM is applied if and only if `Ccm' algorithm is present in the tuning file and it defines some CCM. Software ISP debayering is a performance critical piece of code and we do not want to use dynamic conditionals there. Therefore we pass information about CCM application to debayering configuration and let it select the right versions of debayering functions using templates. This is a similar trick as the previously used one for adding or not adding an alpha channel to the output. Debayering gets this information but it ignores it in this patch. Actual processing with CCM is added in the followup patch. Signed-off-by: Milan Zamazal --- include/libcamera/ipa/soft.mojom | 2 +- src/ipa/simple/algorithms/ccm.cpp | 11 +++++-- src/ipa/simple/algorithms/ccm.h | 1 + src/ipa/simple/soft_simple.cpp | 6 ++-- src/libcamera/software_isp/debayer.cpp | 3 +- src/libcamera/software_isp/debayer.h | 3 +- src/libcamera/software_isp/debayer_cpu.cpp | 35 ++++++++++++--------- src/libcamera/software_isp/debayer_cpu.h | 24 +++++++------- src/libcamera/software_isp/software_isp.cpp | 5 +-- 9 files changed, 54 insertions(+), 36 deletions(-) diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index d52e6f1a..b5ed3905 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -21,7 +21,7 @@ interface IPASoftInterface { start() => (int32 ret); stop(); configure(IPAConfigInfo configInfo) - => (int32 ret); + => (int32 ret, bool ccmEnabled); [async] queueRequest(uint32 frame, libcamera.ControlList sensorControls); [async] computeParams(uint32 frame); diff --git a/src/ipa/simple/algorithms/ccm.cpp b/src/ipa/simple/algorithms/ccm.cpp index 3c7fca2d..96038966 100644 --- a/src/ipa/simple/algorithms/ccm.cpp +++ b/src/ipa/simple/algorithms/ccm.cpp @@ -38,12 +38,17 @@ int Ccm::init([[maybe_unused]] IPAContext &context, const YamlObject &tuningData return 0; } -void Ccm::prepare(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameContext, [[maybe_unused]] DebayerParams *params) +int Ccm::configure(IPAContext &context, + [[maybe_unused]] const IPAConfigInfo &configInfo) { context.activeState.ccm.enabled = ccmEnabled_; + return 0; +} - if (!ccmEnabled_) +void Ccm::prepare(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, [[maybe_unused]] DebayerParams *params) +{ + if (!context.activeState.ccm.enabled) return; unsigned int ct = context.activeState.awb.temperatureK; diff --git a/src/ipa/simple/algorithms/ccm.h b/src/ipa/simple/algorithms/ccm.h index 23481a08..fe81ebd5 100644 --- a/src/ipa/simple/algorithms/ccm.h +++ b/src/ipa/simple/algorithms/ccm.h @@ -24,6 +24,7 @@ public: ~Ccm() = default; int init(IPAContext &context, const YamlObject &tuningData) override; + int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index b26e4e15..9b32e02b 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -52,7 +52,7 @@ public: const SharedFD &fdParams, const ControlInfoMap &sensorInfoMap, ControlInfoMap *ipaControls) override; - int configure(const IPAConfigInfo &configInfo) override; + int configure(const IPAConfigInfo &configInfo, bool *ccmEnabled) override; int start() override; void stop() override; @@ -182,7 +182,7 @@ int IPASoftSimple::init(const IPASettings &settings, return 0; } -int IPASoftSimple::configure(const IPAConfigInfo &configInfo) +int IPASoftSimple::configure(const IPAConfigInfo &configInfo, bool *ccmEnabled) { sensorInfoMap_ = configInfo.sensorControls; @@ -248,6 +248,8 @@ int IPASoftSimple::configure(const IPAConfigInfo &configInfo) return ret; } + *ccmEnabled = context_.activeState.ccm.enabled; + LOG(IPASoft, Info) << "Exposure " << context_.configuration.agc.exposureMin << "-" << context_.configuration.agc.exposureMax diff --git a/src/libcamera/software_isp/debayer.cpp b/src/libcamera/software_isp/debayer.cpp index f0b83261..45fe6960 100644 --- a/src/libcamera/software_isp/debayer.cpp +++ b/src/libcamera/software_isp/debayer.cpp @@ -57,10 +57,11 @@ Debayer::~Debayer() } /** - * \fn int Debayer::configure(const StreamConfiguration &inputCfg, const std::vector> &outputCfgs) + * \fn int Debayer::configure(const StreamConfiguration &inputCfg, const std::vector> &outputCfgs, bool ccmEnabled) * \brief Configure the debayer object according to the passed in parameters * \param[in] inputCfg The input configuration * \param[in] outputCfgs The output configurations + * \param[in] ccmEnabled Whether a color correction matrix is applied * * \return 0 on success, a negative errno on failure */ diff --git a/src/libcamera/software_isp/debayer.h b/src/libcamera/software_isp/debayer.h index d7ca060d..ba033d44 100644 --- a/src/libcamera/software_isp/debayer.h +++ b/src/libcamera/software_isp/debayer.h @@ -33,7 +33,8 @@ public: virtual ~Debayer() = 0; virtual int configure(const StreamConfiguration &inputCfg, - const std::vector> &outputCfgs) = 0; + const std::vector> &outputCfgs, + bool ccmEnabled) = 0; virtual std::vector formats(PixelFormat inputFormat) = 0; diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index 01cfb36b..3c6597f9 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -114,7 +114,7 @@ DebayerCpu::~DebayerCpu() = default; (prev[x] + curr[x - p] + curr[x + n] + next[x]) / (4 * (div)), \ curr[x] / (div)) -template +template void DebayerCpu::debayer8_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) { DECLARE_SRC_POINTERS(uint8_t) @@ -125,7 +125,7 @@ void DebayerCpu::debayer8_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) } } -template +template void DebayerCpu::debayer8_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]) { DECLARE_SRC_POINTERS(uint8_t) @@ -136,7 +136,7 @@ void DebayerCpu::debayer8_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]) } } -template +template void DebayerCpu::debayer10_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) { DECLARE_SRC_POINTERS(uint16_t) @@ -148,7 +148,7 @@ void DebayerCpu::debayer10_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) } } -template +template void DebayerCpu::debayer10_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]) { DECLARE_SRC_POINTERS(uint16_t) @@ -160,7 +160,7 @@ void DebayerCpu::debayer10_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]) } } -template +template void DebayerCpu::debayer12_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) { DECLARE_SRC_POINTERS(uint16_t) @@ -172,7 +172,7 @@ void DebayerCpu::debayer12_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) } } -template +template void DebayerCpu::debayer12_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]) { DECLARE_SRC_POINTERS(uint16_t) @@ -184,7 +184,7 @@ void DebayerCpu::debayer12_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]) } } -template +template void DebayerCpu::debayer10P_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) { const int widthInBytes = window_.width * 5 / 4; @@ -210,7 +210,7 @@ void DebayerCpu::debayer10P_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) } } -template +template void DebayerCpu::debayer10P_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]) { const int widthInBytes = window_.width * 5 / 4; @@ -231,7 +231,7 @@ void DebayerCpu::debayer10P_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]) } } -template +template void DebayerCpu::debayer10P_GBGB_BGR888(uint8_t *dst, const uint8_t *src[]) { const int widthInBytes = window_.width * 5 / 4; @@ -252,7 +252,7 @@ void DebayerCpu::debayer10P_GBGB_BGR888(uint8_t *dst, const uint8_t *src[]) } } -template +template void DebayerCpu::debayer10P_RGRG_BGR888(uint8_t *dst, const uint8_t *src[]) { const int widthInBytes = window_.width * 5 / 4; @@ -368,10 +368,11 @@ int DebayerCpu::setupStandardBayerOrder(BayerFormat::Order order) return 0; } -#define SET_DEBAYER_METHODS(method0, method1) \ - debayer0_ = addAlphaByte ? &DebayerCpu::method0 : &DebayerCpu::method0; \ - debayer1_ = addAlphaByte ? &DebayerCpu::method1 : &DebayerCpu::method1; +#define SET_DEBAYER_METHODS(method0, method1) \ + debayer0_ = addAlphaByte ? &DebayerCpu::method0 : &DebayerCpu::method0; \ + debayer1_ = addAlphaByte ? &DebayerCpu::method1 : &DebayerCpu::method1; +template int DebayerCpu::setDebayerFunctions(PixelFormat inputFormat, PixelFormat outputFormat) { BayerFormat bayerFormat = @@ -464,7 +465,8 @@ int DebayerCpu::setDebayerFunctions(PixelFormat inputFormat, PixelFormat outputF } int DebayerCpu::configure(const StreamConfiguration &inputCfg, - const std::vector> &outputCfgs) + const std::vector> &outputCfgs, + bool ccmEnabled) { if (getInputConfig(inputCfg.pixelFormat, inputConfig_) != 0) return -EINVAL; @@ -503,7 +505,10 @@ int DebayerCpu::configure(const StreamConfiguration &inputCfg, return -EINVAL; } - if (setDebayerFunctions(inputCfg.pixelFormat, outputCfg.pixelFormat) != 0) + int ret = ccmEnabled + ? setDebayerFunctions(inputCfg.pixelFormat, outputCfg.pixelFormat) + : setDebayerFunctions(inputCfg.pixelFormat, outputCfg.pixelFormat); + if (ret != 0) return -EINVAL; window_.x = ((inputCfg.size.width - outputCfg.size.width) / 2) & diff --git a/src/libcamera/software_isp/debayer_cpu.h b/src/libcamera/software_isp/debayer_cpu.h index 2c47e7c6..b2ec8f1b 100644 --- a/src/libcamera/software_isp/debayer_cpu.h +++ b/src/libcamera/software_isp/debayer_cpu.h @@ -31,7 +31,8 @@ public: ~DebayerCpu(); int configure(const StreamConfiguration &inputCfg, - const std::vector> &outputCfgs); + const std::vector> &outputCfgs, + bool ccmEnabled); Size patternSize(PixelFormat inputFormat); std::vector formats(PixelFormat input); std::tuple @@ -85,28 +86,28 @@ private: using debayerFn = void (DebayerCpu::*)(uint8_t *dst, const uint8_t *src[]); /* 8-bit raw bayer format */ - template + template void debayer8_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]); - template + template void debayer8_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]); /* unpacked 10-bit raw bayer format */ - template + template void debayer10_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]); - template + template void debayer10_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]); /* unpacked 12-bit raw bayer format */ - template + template void debayer12_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]); - template + template void debayer12_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]); /* CSI-2 packed 10-bit raw bayer format (all the 4 orders) */ - template + template void debayer10P_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]); - template + template void debayer10P_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]); - template + template void debayer10P_GBGB_BGR888(uint8_t *dst, const uint8_t *src[]); - template + template void debayer10P_RGRG_BGR888(uint8_t *dst, const uint8_t *src[]); struct DebayerInputConfig { @@ -125,6 +126,7 @@ private: int getInputConfig(PixelFormat inputFormat, DebayerInputConfig &config); int getOutputConfig(PixelFormat outputFormat, DebayerOutputConfig &config); int setupStandardBayerOrder(BayerFormat::Order order); + template int setDebayerFunctions(PixelFormat inputFormat, PixelFormat outputFormat); void setupInputMemcpy(const uint8_t *linePointers[]); void shiftLinePointers(const uint8_t *linePointers[], const uint8_t *src); diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 2bea64d9..6a07de85 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -236,11 +236,12 @@ int SoftwareIsp::configure(const StreamConfiguration &inputCfg, { ASSERT(ipa_ && debayer_); - int ret = ipa_->configure(configInfo); + bool ccmEnabled; + int ret = ipa_->configure(configInfo, &ccmEnabled); if (ret < 0) return ret; - return debayer_->configure(inputCfg, outputCfgs); + return debayer_->configure(inputCfg, outputCfgs, ccmEnabled); } /** From patchwork Mon Jan 13 13:51:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 22544 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 92FB5C32F6 for ; Mon, 13 Jan 2025 13:51:51 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3C46C6854D; Mon, 13 Jan 2025 14:51:51 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="UeFlyWop"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3AB096854D for ; Mon, 13 Jan 2025 14:51:49 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736776308; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+80hUaXe28lFpYqnRQmjEjCtat61TO5MEGgCc+xrcJI=; b=UeFlyWopAryw4OiNo8+LZVGkKFoc382jo04tyZc9sLnOa77Cc/gGjYr8ipYzJ9tjZXKnI7 u/E2Bj0Qm6src3P/E2UGpEEiBfQ/fxJxxR3u0UX8Dx6u13nf5909O5Hga5DpM2pUvAHjbK fernLPUtMIVAs2plNaoR/BC51DRVdEQ= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-338-kBeBo49ROji8S0GkGW_CbQ-1; Mon, 13 Jan 2025 08:51:45 -0500 X-MC-Unique: kBeBo49ROji8S0GkGW_CbQ-1 X-Mimecast-MFC-AGG-ID: kBeBo49ROji8S0GkGW_CbQ Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 227001955F34; Mon, 13 Jan 2025 13:51:44 +0000 (UTC) Received: from mzamazal-thinkpadp1gen3.tpbc.com (unknown [10.45.224.6]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id A5A081956056; Mon, 13 Jan 2025 13:51:41 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Robert Mader , Hans de Goede , Laurent Pinchart , Kieran Bingham Subject: [PATCH v4 9/9] libcamera: software_isp: Apply CCM in debayering Date: Mon, 13 Jan 2025 14:51:06 +0100 Message-ID: <20250113135108.13924-10-mzamazal@redhat.com> In-Reply-To: <20250113135108.13924-1-mzamazal@redhat.com> References: <20250113135108.13924-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: EG_kMNQWHMs-ou4wgtEWOODpuJiSpeqBOWieuhaai88_1736776304 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch applies color correction matrix (CCM) in debayering if the CCM is specified. Not using CCM must still be supported for performance reasons. The CCM is applied as follows: [r1 r2 r3] [r g b] * [g1 g2 g3] [b1 b2 b3] The CCM matrix (the right side of the multiplication) is constant during single frame processing, while the input pixel (the left side) changes. Because each of the color channels is only 8-bit in software ISP, we can make 9 lookup tables with 256 input values for multiplications of each of the r_i, g_i, b_i values. This way we don't have to multiply each pixel, we can use table lookups and additions instead. Gamma (which is non-linear and thus cannot be a part of the 9 lookup tables values) is applied on the final values rounded to integers using another lookup table. We use int16_t to store the precomputed multiplications. This seems to be noticeably (>10%) faster than `float' for the price of slightly less accuracy and it covers the range of values that sane CCMs produce. The selection and structure of data is performance critical, for example using bytes would add significant (>10%) speedup but would be too short to cover the value range. The color lookup tables are changed to unions to be able to serve as both simple lookup tables or CCM lookup tables. This is arguable but it keeps the code easier to understand if nothing else. It is important to make the unions on the whole lookup tables rather than their values because the latter might cause a noticeable slowdown on simple lookup due to the sheer fact that the table is not compact. Using unions makes the initialization of the tables dubious because it is done before the decision about using CCM is made but the initial values are dubious anyway. Using std::variant instead of unions would be even more difficult, consider e.g. mmap allocation and the followup type-casting of the debayer params. The tables are copied (as before), which is not elegant but also not a big problem. There are patches posted that use shared buffers for parameters passing in software ISP (see software ISP TODO #5) and they can be adjusted for the new parameter format. Color gains from white balance are supposed not to be a part of the specified CCM. They are applied on it using matrix multiplication, which is simple and in correspondence with future additions in the form of matrix multiplication, like saturation adjustment. With this patch, the reported per-frame slowdown when applying CCM is about 45% on Debix Model A and about 75% on TI AM69 SK. Using std::clamp in debayering adds some performance penalty (a few percent). The clamping is necessary to eliminate out of range values possibly produced by the CCM. If it could be avoided by adjusting the precomputed tables some way then performance could be improved a bit. Signed-off-by: Milan Zamazal --- .../internal/software_isp/debayer_params.h | 20 +++++- src/ipa/simple/algorithms/lut.cpp | 63 ++++++++++++++----- src/ipa/simple/algorithms/lut.h | 1 + src/libcamera/software_isp/debayer.cpp | 50 +++++++++++++++ src/libcamera/software_isp/debayer_cpu.cpp | 34 +++++++--- src/libcamera/software_isp/debayer_cpu.h | 7 ++- src/libcamera/software_isp/software_isp.cpp | 6 +- 7 files changed, 147 insertions(+), 34 deletions(-) diff --git a/include/libcamera/internal/software_isp/debayer_params.h b/include/libcamera/internal/software_isp/debayer_params.h index 7d8fdd48..f8731eae 100644 --- a/include/libcamera/internal/software_isp/debayer_params.h +++ b/include/libcamera/internal/software_isp/debayer_params.h @@ -18,11 +18,25 @@ namespace libcamera { struct DebayerParams { static constexpr unsigned int kRGBLookupSize = 256; + struct CcmColumn { + int16_t r; + int16_t g; + int16_t b; + }; + using ColorLookupTable = std::array; + using CcmLookupTable = std::array; + using GammaLookupTable = std::array; + + union LookupTable { + ColorLookupTable simple; + CcmLookupTable ccm; + }; - ColorLookupTable red; - ColorLookupTable green; - ColorLookupTable blue; + LookupTable red; + LookupTable green; + LookupTable blue; + GammaLookupTable gammaLut; }; } /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/lut.cpp b/src/ipa/simple/algorithms/lut.cpp index 243f0818..ddb2adf1 100644 --- a/src/ipa/simple/algorithms/lut.cpp +++ b/src/ipa/simple/algorithms/lut.cpp @@ -80,6 +80,11 @@ void Lut::updateGammaTable(IPAContext &context) context.activeState.gamma.contrast = contrast; } +int16_t Lut::ccmValue(unsigned int i, float ccm) const +{ + return std::round(i * ccm); +} + void Lut::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, @@ -91,28 +96,52 @@ void Lut::prepare(IPAContext &context, * observed, it's not permanently prone to minor fluctuations or * rounding errors. */ - if (context.activeState.gamma.blackLevel != context.activeState.blc.level || - context.activeState.gamma.contrast != context.activeState.knobs.contrast) + const bool gammaUpdateNeeded = + context.activeState.gamma.blackLevel != context.activeState.blc.level || + context.activeState.gamma.contrast != context.activeState.knobs.contrast; + if (gammaUpdateNeeded) updateGammaTable(context); auto &gains = context.activeState.awb.gains; auto &gammaTable = context.activeState.gamma.gammaTable; const unsigned int gammaTableSize = gammaTable.size(); - - for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { - const double div = static_cast(DebayerParams::kRGBLookupSize) / - gammaTableSize; - /* Apply gamma after gain! */ - unsigned int idx; - idx = std::min({ static_cast(i * gains.red / div), - gammaTableSize - 1 }); - params->red[i] = gammaTable[idx]; - idx = std::min({ static_cast(i * gains.green / div), - gammaTableSize - 1 }); - params->green[i] = gammaTable[idx]; - idx = std::min({ static_cast(i * gains.blue / div), - gammaTableSize - 1 }); - params->blue[i] = gammaTable[idx]; + const double div = static_cast(DebayerParams::kRGBLookupSize) / + gammaTableSize; + + if (!context.activeState.ccm.enabled) { + for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { + /* Apply gamma after gain! */ + unsigned int idx; + idx = std::min({ static_cast(i * gains.red / div), + gammaTableSize - 1 }); + params->red.simple[i] = gammaTable[idx]; + idx = std::min({ static_cast(i * gains.green / div), + gammaTableSize - 1 }); + params->green.simple[i] = gammaTable[idx]; + idx = std::min({ static_cast(i * gains.blue / div), + gammaTableSize - 1 }); + params->blue.simple[i] = gammaTable[idx]; + } + } else if (context.activeState.ccm.changed || gammaUpdateNeeded) { + Matrix gainCcm = { { gains.red, 0, 0, + 0, gains.green, 0, + 0, 0, gains.blue } }; + auto ccm = gainCcm * context.activeState.ccm.ccm; + auto &red = params->red.ccm; + auto &green = params->green.ccm; + auto &blue = params->blue.ccm; + for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { + red[i].r = ccmValue(i, ccm[0][0]); + red[i].g = ccmValue(i, ccm[1][0]); + red[i].b = ccmValue(i, ccm[2][0]); + green[i].r = ccmValue(i, ccm[0][1]); + green[i].g = ccmValue(i, ccm[1][1]); + green[i].b = ccmValue(i, ccm[2][1]); + blue[i].r = ccmValue(i, ccm[0][2]); + blue[i].g = ccmValue(i, ccm[1][2]); + blue[i].b = ccmValue(i, ccm[2][2]); + params->gammaLut[i] = gammaTable[i / div]; + } } } diff --git a/src/ipa/simple/algorithms/lut.h b/src/ipa/simple/algorithms/lut.h index 889f864b..77324800 100644 --- a/src/ipa/simple/algorithms/lut.h +++ b/src/ipa/simple/algorithms/lut.h @@ -33,6 +33,7 @@ public: private: void updateGammaTable(IPAContext &context); + int16_t ccmValue(unsigned int i, float ccm) const; }; } /* namespace ipa::soft::algorithms */ diff --git a/src/libcamera/software_isp/debayer.cpp b/src/libcamera/software_isp/debayer.cpp index 45fe6960..559de3e6 100644 --- a/src/libcamera/software_isp/debayer.cpp +++ b/src/libcamera/software_isp/debayer.cpp @@ -23,11 +23,56 @@ namespace libcamera { * \brief Size of a color lookup table */ +/** + * \struct DebayerParams::CcmColumn + * \brief Type of a single column of a color correction matrix + */ + +/** + * \var DebayerParams::CcmColumn::r + * \brief Red (first) component of a CCM column + */ + +/** + * \var DebayerParams::CcmColumn::g + * \brief Green (second) component of a CCM column + */ + +/** + * \var DebayerParams::CcmColumn::b + * \brief Blue (third) component of a CCM column + */ + /** * \typedef DebayerParams::ColorLookupTable + * \brief Type of the simple lookup tables for red, green, blue values + */ + +/** + * \typedef DebayerParams::CcmLookupTable + * \brief Type of the CCM lookup tables for red, green, blue values + */ + +/** + * \typedef DebayerParams::GammaLookupTable + * \brief Type of the gamma lookup tables for CCM + */ + +/** + * \union DebayerParams::LookupTable * \brief Type of the lookup tables for red, green, blue values */ +/** + * \var DebayerParams::LookupTable::simple + * \brief Simple lookup table for red, green, blue values + */ + +/** + * \var DebayerParams::LookupTable::ccm + * \brief CCM lookup table for red, green, blue values + */ + /** * \var DebayerParams::red * \brief Lookup table for red color, mapping input values to output values @@ -43,6 +88,11 @@ namespace libcamera { * \brief Lookup table for blue color, mapping input values to output values */ +/** + * \var DebayerParams::gammaLut + * \brief Gamma lookup table used with color correction matrix + */ + /** * \class Debayer * \brief Base debayering class diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index 3c6597f9..38715b72 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -11,6 +11,7 @@ #include "debayer_cpu.h" +#include #include #include #include @@ -51,8 +52,12 @@ DebayerCpu::DebayerCpu(std::unique_ptr stats) enableInputMemcpy_ = true; /* Initialize color lookup tables */ - for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) - red_[i] = green_[i] = blue_[i] = i; + for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { + red_.simple[i] = green_.simple[i] = blue_.simple[i] = i; + red_.ccm[i].r = red_.ccm[i].g = red_.ccm[i].b = 0; + green_.ccm[i].r = green_.ccm[i].g = green_.ccm[i].b = 0; + blue_.ccm[i].r = blue_.ccm[i].g = blue_.ccm[i].b = 0; + } } DebayerCpu::~DebayerCpu() = default; @@ -62,12 +67,24 @@ DebayerCpu::~DebayerCpu() = default; const pixel_t *curr = (const pixel_t *)src[1] + xShift_; \ const pixel_t *next = (const pixel_t *)src[2] + xShift_; -#define STORE_PIXEL(b, g, r) \ - *dst++ = blue_[b]; \ - *dst++ = green_[g]; \ - *dst++ = red_[r]; \ - if constexpr (addAlphaByte) \ - *dst++ = 255; \ +#define GAMMA(value) \ + *dst++ = gammaLut_[std::clamp(value, 0, static_cast(gammaLut_.size()) - 1)] + +#define STORE_PIXEL(b_, g_, r_) \ + if constexpr (ccmEnabled) { \ + DebayerParams::CcmColumn &blue = blue_.ccm[b_]; \ + DebayerParams::CcmColumn &green = green_.ccm[g_]; \ + DebayerParams::CcmColumn &red = red_.ccm[r_]; \ + GAMMA(blue.r + blue.g + blue.b); \ + GAMMA(green.r + green.g + green.b); \ + GAMMA(red.r + red.g + red.b); \ + } else { \ + *dst++ = blue_.simple[b_]; \ + *dst++ = green_.simple[g_]; \ + *dst++ = red_.simple[r_]; \ + } \ + if constexpr (addAlphaByte) \ + *dst++ = 255; \ x++; /* @@ -752,6 +769,7 @@ void DebayerCpu::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output green_ = params.green; red_ = swapRedBlueGains_ ? params.blue : params.red; blue_ = swapRedBlueGains_ ? params.red : params.blue; + gammaLut_ = params.gammaLut; /* Copy metadata from the input buffer */ FrameMetadata &metadata = output->_d()->metadata(); diff --git a/src/libcamera/software_isp/debayer_cpu.h b/src/libcamera/software_isp/debayer_cpu.h index b2ec8f1b..35c51e4f 100644 --- a/src/libcamera/software_isp/debayer_cpu.h +++ b/src/libcamera/software_isp/debayer_cpu.h @@ -137,9 +137,10 @@ private: /* Max. supported Bayer pattern height is 4, debayering this requires 5 lines */ static constexpr unsigned int kMaxLineBuffers = 5; - DebayerParams::ColorLookupTable red_; - DebayerParams::ColorLookupTable green_; - DebayerParams::ColorLookupTable blue_; + DebayerParams::LookupTable red_; + DebayerParams::LookupTable green_; + DebayerParams::LookupTable blue_; + DebayerParams::GammaLookupTable gammaLut_; debayerFn debayer0_; debayerFn debayer1_; debayerFn debayer2_; diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 6a07de85..40e11b9e 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -82,9 +82,9 @@ SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor, for (unsigned int i = 0; i < 256; i++) gammaTable[i] = UINT8_MAX * std::pow(i / 256.0, 0.5); for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { - debayerParams_.red[i] = gammaTable[i]; - debayerParams_.green[i] = gammaTable[i]; - debayerParams_.blue[i] = gammaTable[i]; + debayerParams_.red.simple[i] = gammaTable[i]; + debayerParams_.green.simple[i] = gammaTable[i]; + debayerParams_.blue.simple[i] = gammaTable[i]; } if (!dmaHeap_.isValid()) {