From patchwork Fri Sep 6 12:09:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21173 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7CD76C324C for ; Fri, 6 Sep 2024 12:09:56 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 589D0634E5; Fri, 6 Sep 2024 14:09:55 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="Ws0VARC9"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 185DE633CC for ; Fri, 6 Sep 2024 14:09:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624591; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=QRsBcdjQdR/1LUjbsTAwtv2eKkaAU0qiD4C4IMYRqvQ=; b=Ws0VARC91kyVqEGK9sVKRkPaP/HmLnP9QPJ/m4+NalkdzmgHPd29/JPxImjVZ79ghYRW+L ju0CVHXzZhTJsWnEUhogxZ0mOiIWqR1jDBtXbCCIcgvtP4IN/uNmP+DVEFACz8oE6u382Y Cv5GKOQa2NKfbcCBWS+gM/3b3IPOOOc= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-190-oQDlvFNCMky4-jvd8l2Fhw-1; Fri, 06 Sep 2024 08:09:46 -0400 X-MC-Unique: oQDlvFNCMky4-jvd8l2Fhw-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8E71319560BD; Fri, 6 Sep 2024 12:09:45 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 1D4951956086; Fri, 6 Sep 2024 12:09:42 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 00/18] Software ISP refactoring Date: Fri, 6 Sep 2024 14:09:09 +0200 Message-ID: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The purpose of this patch series is to bring software ISP code structuring closer to the hardware pipelines. Most notably, the API around algorithm.h is used now. This should make software ISP easier to understand, extend, maintain, and code-share with the other pipelines. What is omitted in the patch series: - Any bigger or unrelated functional changes. The purpose of this series is code restructuring, which is enough already. - Stats and params buffers are still used in the original way. Making their handling closer to the hardware pipelines will be a subject of followup patches. This series is a preparation step for that. - Any IPA code sharing with hardware pipelines. If there is an opportunity for this, it can be addressed in followup patches. Available in git at https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/pdm-algorithm Pending, more input from reviewers needed (see v3 review discussions): - Should `frame' argument name be kept for consistency with other pipelines for now or already renamed to something like `sequenceNumber'? - Should black level be stored as a 0..1 number or a pixel value number? Changes in v6: - Use data->swIsp_ only if it is defined. - IPASoftSimple::prepare renamed to IPASoftSimple::fillParamsBuffer. - soft_ipa_interface.h include moved to the right patch. - Unneeded includes in algorithm *.h files dropped. - Duplicate kGammaLookupSize constant removed and gammaTable.size() used instead. - Gamma algorithm merged to Lut algorithm. - Duplicate logging of AWB gains removed. - Miscellaneous formatting and cosmetic changes suggested by Laurent. Changes in v5: - Construction of color lookup tables moved to a separate algorithm. Changes in v4: - Removed the mistakenly included patches by Umang. - IPASoftInterface::prepare() is async now. - IPAConfigInfo definition moved to an earlier patch to avoid unnecessary confusion. - colors.cpp/h split to gamma and awb. - Using the right black level variable in Agc::process. - Documentation of the newly introduced SwStatsCpu::finishFrame arguments added. - Formatting changes as requested. - A forgotten obsolete source code comment removed. - Improvements to some commit messages. Changes in v3: - SoftwareIsp::queueRequest changed to async. - IPAActiveState docstring reworded. - Minor formatting fixes. Changes in v2: - Several cosmetic changes and patch arrangement problems pointed out by Umang applied. - Added ipa_context.cpp, as a documentation file for ipa_context.h. - Added a clarification source comment why SoftwareIsp::queueBuffers needs to get the frame number as a separate argument. - core_ipa_interface.h no longer included in module.h. - The context used by "14/19 Move black level to an algorithm module" was changed and the black level changes tracking was put closer to the pre-refactoring version, which makes more sense. Milan Zamazal (18): libcamera: software_isp: Remove superfluous includes libcamera: software_isp: Move BlackLevel to libcamera::ipa::soft libcamera: software_isp: Define skeletons for IPA refactoring libcamera: software_isp: Let IPASoftSimple inherit Module libcamera: software_isp: Make stats frame and buffer aware libcamera: software_isp: Remove final dots in debayer.cpp docstrings libcamera: software_isp: Track and pass frame ids libcamera: software_isp: Create algorithms libcamera: software_isp: Call Algorithm::configure libcamera: software_isp: Call Algorithm::queueRequest libcamera: software_isp: Call Algorithm::prepare libcamera: software_isp: Call Algorithm::process libcamera: software_isp: Move black level to an algorithm module libcamera: software_isp: Move color handling to an algorithm module libcamera: software_isp: Use floating point for color parameters libcamera: software_isp: Use DelayedControls libcamera: software_isp: Move exposure+gain to an algorithm module libcamera: software_isp: Update black level only on exposure changes .../internal/software_isp/software_isp.h | 15 +- include/libcamera/ipa/soft.mojom | 12 +- src/ipa/simple/algorithms/agc.cpp | 139 +++++++++ src/ipa/simple/algorithms/agc.h | 33 ++ src/ipa/simple/algorithms/algorithm.h | 22 ++ src/ipa/simple/algorithms/awb.cpp | 69 +++++ src/ipa/simple/algorithms/awb.h | 32 ++ src/ipa/simple/algorithms/blc.cpp | 78 +++++ src/ipa/simple/algorithms/blc.h | 36 +++ src/ipa/simple/algorithms/lut.cpp | 81 +++++ src/ipa/simple/algorithms/lut.h | 35 +++ src/ipa/simple/algorithms/meson.build | 8 + src/ipa/simple/black_level.cpp | 88 ------ src/ipa/simple/black_level.h | 29 -- src/ipa/simple/data/uncalibrated.yaml | 5 + src/ipa/simple/ipa_context.cpp | 102 +++++++ src/ipa/simple/ipa_context.h | 69 +++++ src/ipa/simple/meson.build | 9 +- src/ipa/simple/module.h | 30 ++ src/ipa/simple/soft_simple.cpp | 283 ++++++------------ src/libcamera/pipeline/simple/simple.cpp | 48 ++- src/libcamera/software_isp/TODO | 39 --- src/libcamera/software_isp/debayer.cpp | 51 ++-- src/libcamera/software_isp/debayer.h | 2 +- src/libcamera/software_isp/debayer_cpu.cpp | 9 +- src/libcamera/software_isp/debayer_cpu.h | 2 +- src/libcamera/software_isp/software_isp.cpp | 43 ++- src/libcamera/software_isp/swstats_cpu.cpp | 6 +- src/libcamera/software_isp/swstats_cpu.h | 4 +- 29 files changed, 955 insertions(+), 424 deletions(-) create mode 100644 src/ipa/simple/algorithms/agc.cpp create mode 100644 src/ipa/simple/algorithms/agc.h create mode 100644 src/ipa/simple/algorithms/algorithm.h create mode 100644 src/ipa/simple/algorithms/awb.cpp create mode 100644 src/ipa/simple/algorithms/awb.h create mode 100644 src/ipa/simple/algorithms/blc.cpp create mode 100644 src/ipa/simple/algorithms/blc.h create mode 100644 src/ipa/simple/algorithms/lut.cpp create mode 100644 src/ipa/simple/algorithms/lut.h create mode 100644 src/ipa/simple/algorithms/meson.build delete mode 100644 src/ipa/simple/black_level.cpp delete mode 100644 src/ipa/simple/black_level.h create mode 100644 src/ipa/simple/ipa_context.cpp create mode 100644 src/ipa/simple/ipa_context.h create mode 100644 src/ipa/simple/module.h