From patchwork Wed Nov 20 18:00:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 22041 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 174E6C32F9 for ; Wed, 20 Nov 2024 18:01:17 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id CF4B165F7B; Wed, 20 Nov 2024 19:01:15 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="Vtwn3+Gl"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9073865F54 for ; Wed, 20 Nov 2024 19:01:13 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1732125672; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=PRY5C2S8m4Z8OGpmsYdsDHK7eOoWSHIGhkKgdIsjTUc=; b=Vtwn3+GlQalZrrQrIabZbXt+ANJkOUicDeD2qAQYuvu9wyMBNumlrbWdcn6xZ9Lmw3YeL1 FZDBVS468UzoEWFb+2VO0oD3cq2QLGC6eTmZ+ti/aDnc5MkXtQzS3OAe2x6/yy+WaC67qh J1K90f0cU3s9wAtEM9e0kngi2kIUu7E= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-178-hs-8Z6CwPAiyehPIy3c2Mw-1; Wed, 20 Nov 2024 13:01:10 -0500 X-MC-Unique: hs-8Z6CwPAiyehPIy3c2Mw-1 X-Mimecast-MFC-AGG-ID: hs-8Z6CwPAiyehPIy3c2Mw Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 00E5319560AD for ; Wed, 20 Nov 2024 18:01:10 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.6]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 9871030000DF; Wed, 20 Nov 2024 18:01:08 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 0/9] Software ISP support for CCM Date: Wed, 20 Nov 2024 19:00:55 +0100 Message-ID: <20241120180104.1221846-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: t0lsc8XCk2shE7uE-D1l5xs3aO5c5durwmD901cJ_Mc_1732125670 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This series implements application of color correction matrices in software ISP. Support for color temperature is added to auto white balance algorithm, this is needed to obtain the right CCM. A new Ccm algorithm is added to determine the matrix. Both roughly follow what the hardware pipelines do. Lut algorithm is modified to incorporate the CCM and perform some precomputations in the form of 10 lookup tables (one per each matrix element and one for gamma). A tricky part is to support both CCM and the original (no-CCM) transformations. CCM obviously adds a significant overhead; per-frame time increases by ~45% on Debix Model A and ~85% on TI AM69 SK. This means CCM must be optional, which is determined by presence of Ccm algorithm in the tuning file. The performance critical debayering code must not include extra conditionals and must work efficiently for both the cases. The macros in debayering code are refactored and restructured for this. An added bonus is that this also removes some annoying code duplication. And some more, partially already present, templating is used to select between CCM and non-CCM processing. Performance is much influenced by the data structures used in debayering. I tried to define something reasonable and performed only basic performance testing. Suggestions for possible improvements are welcome. See the commit messages for more details. Milan Zamazal (9): libcamera: software_isp: Determine color temperature libcamera: software_isp: Store color temperature to metadata libcamera: software_isp: lut: Remove maybe_unused on a used argument libcamera: software_isp: Use common code to store debayered pixels libcamera: software_isp: Use a macro to assign debayering methods libcamera: software_isp: Add CCM algorithm libcamera: software_isp: Add an example CCM to uncalibrated.yaml libcamera: software_isp: Track whether CCM is enabled libcamera: software_isp: Apply CCM in debayering .../internal/software_isp/debayer_params.h | 20 ++- include/libcamera/ipa/soft.mojom | 2 +- src/ipa/simple/algorithms/awb.cpp | 18 ++- src/ipa/simple/algorithms/ccm.cpp | 86 ++++++++++++ src/ipa/simple/algorithms/ccm.h | 45 ++++++ src/ipa/simple/algorithms/lut.cpp | 62 +++++--- src/ipa/simple/algorithms/lut.h | 1 + src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/data/uncalibrated.yaml | 7 + src/ipa/simple/ipa_context.h | 22 ++- src/ipa/simple/soft_simple.cpp | 8 +- src/libcamera/software_isp/debayer.cpp | 53 ++++++- src/libcamera/software_isp/debayer.h | 3 +- src/libcamera/software_isp/debayer_cpu.cpp | 132 ++++++++++-------- src/libcamera/software_isp/debayer_cpu.h | 31 ++-- src/libcamera/software_isp/software_isp.cpp | 11 +- 16 files changed, 393 insertions(+), 109 deletions(-) create mode 100644 src/ipa/simple/algorithms/ccm.cpp create mode 100644 src/ipa/simple/algorithms/ccm.h