From patchwork Thu Dec 14 12:13:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19315 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D9C46BD87C for ; Thu, 14 Dec 2023 12:13:57 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 22CBD62B2A; Thu, 14 Dec 2023 13:13:57 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1702556037; bh=jVKe7ndTsbo/tOm45KL9yYek36xMo4Uchd5JUSeokOw=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=vyBeVNeYQeJ8rBxpkP924eRhmp7wu5PmlSi62skZGggTgZyMtmK7nDng0uGbtkfgF 9IyAwX7sGlRy07ho/dZmXyQtK66+jXM4hSj2lqrSWJV3Ip3l++mwrsLA1O/gim8Jkk onb+0szE3OO1mnYBrMP95BsAyCJNcmrJoD03gRhvDvpV1ltxHaJ/Dzkvuyc4Kv3p5n E3nXOaxSZIs8KId4kw59XogaM+IqzR+QGYPnoFk+auAxWUZx8rvpnMejOaQbaAZIMu GdbiRv7eGmPkDeYyVvNuN3xVxMxz5NN2n0p3V8/nVm9xd1GiaJljJ3ouoDA5g0MBK+ 53b5MsU4oXYtg== Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0A81361D97 for ; Thu, 14 Dec 2023 13:13:54 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="b3liFGbk"; dkim-atps=neutral DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702556033; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=s1GvCQmRcvqRjWqOavfejeN2wqbdnwZkaDtFd1lAJe0=; b=b3liFGbkSotRIF+hbu0n6f4NVf7HT8Umw01frrcVGgGgh/Dm2ZbQShLWyfUGNZS/UBT+1o E5DQlhT4i8iuZ3msZqp0qo5byv2otS4sLQD7fL/35CXCdJIdwXx3Smh22goEW3r55FEc2H nMFjSAaaKvapFzth5d+ErR1TX4/LU5k= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-596-_MyYIk1yObanPj0HlgPUQQ-1; Thu, 14 Dec 2023 07:13:52 -0500 X-MC-Unique: _MyYIk1yObanPj0HlgPUQQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1DDF0101A555; Thu, 14 Dec 2023 12:13:52 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.194.72]) by smtp.corp.redhat.com (Postfix) with ESMTP id B0FEA1121306; Thu, 14 Dec 2023 12:13:50 +0000 (UTC) To: libcamera-devel@lists.libcamera.org, Andrey Konovalov Date: Thu, 14 Dec 2023 13:13:38 +0100 Message-ID: <20231214121350.206015-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [libcamera-devel] [PATCH 00/11] libcamera: introduce Software ISP and Software IPA X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Hans de Goede via libcamera-devel From: Hans de Goede Reply-To: Hans de Goede Cc: Maxime Ripard , g.martti@gmail.com, t.langendam@gmail.com, srinivas.kandagatla@linaro.org, Bryan O'Donoghue , admin@dennisbonke.com Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Hi All, Here is a new, now non RFC because believed to be mostly ready for merging, version of the Software ISP work. First of all apologies for sending this out so soon after RFC v2. While Andrey was working on the general SoftwareIsp framework I have been focussing on optimizing the code for gathering statistics and doing the debayering. I ended up splitting out the swstats and debayer code into separate classes and this first non RFC posting integrates the best of Andrey's and my work. And it seemed better to post a new version integrating our work soon, so that review efforts can be focussed on the new combined work. Changes in v1 vs RFC-v2: - Add and use SwStats[Cpu] and Debayer[Cpu] classes - Rename linaro soft-IPA and soft-ISP implementations to simple following the simple pipeline hander - Integrate Pavel's swstats and debayer improvements Known open items: - The AWB red/blue gain calculations needs to be moved to the IPA and IPA then needs to pass a DebayerParams struct back to the SwIsp - Properly document all methods / attributed for doxygen git branch for this v1 patch set: https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v04 RFC-v2 cover-letter with small updates: Here is an implementation of Software ISP and Software IPA which provide debayering and implementation of image processing algorithms for systems without a hardware ISP, or in cases when there are no public drivers for the hardware ISP present in the system. The implementation of the Software ISP is a reference one. A naive AWB alorithm is implemented as part of a function which does debayering and statistics calculations - the algorithm part is to be moved to the IPA in the next version of the patch set. And for debayering itself there is already a more efficient implementation by Hans de Goede. This patch set is currently using the earlier debayering implementation as it is less lines of code and is OK for the initial discussion. Only RAW10P format from the sensor is currently supported, but other debayering functions can be easily added (which of them to call is decided based on the input format). The Software IPA has only auto exposure and AGC. For the AGC the analogue gain control of the camera sensor is used (if available). The algorithm is very much simplified, and is mostly included as a reference code. The 6th patch renames some variables in the simple pipeline handler for the Software ISP to use the same buffer handling code as the Converter currently does. This lets one to avoid adding extra code to the pipeline handler, but also makes the Software ISP and the Converter mutually exclusive. The Software ISP / IPA are intended to be used with the simple pipeline handler. The pipeline handler doesn't interact with the Software IPA directly - the Software IPA is hidden behind the Software ISP interface. To use the Software ISP the build must be configured with -Dpipelines=simple/simple -Dipas=simple/simple and the Converter must not be used (the latter is hardcoded in the simple pipeline handler). If the build is configured with just -Dpipelines=simple the Software ISP / IPA are not used, and the simple pipeline handler works in the same way as without this patchset. "simple" in the -Dpipelines=simple/simple -Dipas=simple/simple is the name of the Software ISP / IPA implementation. -Dpipelines=simple/ -Dipas=simple/ should work too if another implementation were added. What implementation to use is the build time choice, only one implementation is included into the build. This patch set uses SharedMemObject class used by the RPi pipeline handler - the second patch in the series moves the header file to a common directory. This patch set has been tested on Qualcomm RB5 board with a mezzanine board equipped with RPi camera v2 (not the standard RB5 camera mezzanine). git branch for the RFC-v2 patch set: https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v03 RFC-v1 of the patch set: https://patchwork.libcamera.org/cover/19262/ Changes in v2 vs v1: - patches are restructured and reordered - the Software IPA is hidden behind the Software ISP interface. The pipeline handler doesn't interact with the Software IPA directly - instantiation of the Software ISP / IPA is configurable (at build time) - comment explaining the implementation limitations is added to SwIspLinaro::IspWorker::debayerRaw10P() Regards, Hans Andrey Konovalov (7): libcamera: introduce SoftwareIsp class libcamera: internal: Move SharedMemObject class to a common directory libcamera: ipa: add Soft IPA common files libcamera: ipa: Soft IPA: add a Simple Soft IPA implementation libcamera: software_isp: add Simple SoftwareIsp implementation libcamera: pipeline: simple: rename converterBuffers_ and related vars libcamera: pipeline: simple: enable use of Soft ISP and Soft IPA Hans de Goede (4): libcamera: software_isp: Add SwStats base class libcamera: software_isp: Add SwStatsCpu class libcamera: software_isp: Add Debayer base class libcamera: software_isp: Add DebayerCpu class include/libcamera/internal/meson.build | 3 + .../libcamera/internal}/shared_mem_object.h | 4 - include/libcamera/internal/software_isp.h | 109 +++++ .../libcamera/internal/software_isp/debayer.h | 90 ++++ .../internal/software_isp/debayer_cpu.h | 104 +++++ .../internal/software_isp/debayer_params.h | 24 + .../internal/software_isp/meson.build | 11 + .../internal/software_isp/swisp_simple.h | 67 +++ .../internal/software_isp/swisp_stats.h | 21 + .../libcamera/internal/software_isp/swstats.h | 150 ++++++ .../internal/software_isp/swstats_cpu.h | 46 ++ include/libcamera/ipa/meson.build | 1 + include/libcamera/ipa/soft.mojom | 27 ++ meson_options.txt | 3 +- src/ipa/simple/common/meson.build | 17 + src/ipa/simple/common/soft_base.cpp | 66 +++ src/ipa/simple/common/soft_base.h | 47 ++ src/ipa/simple/meson.build | 12 + src/ipa/simple/simple/data/meson.build | 9 + src/ipa/simple/simple/data/soft.conf | 3 + src/ipa/simple/simple/meson.build | 26 ++ src/ipa/simple/simple/soft_simple.cpp | 210 +++++++++ src/libcamera/meson.build | 2 + src/libcamera/pipeline/simple/simple.cpp | 164 +++++-- src/libcamera/software_isp.cpp | 62 +++ src/libcamera/software_isp/debayer.cpp | 22 + src/libcamera/software_isp/debayer_cpu.cpp | 440 ++++++++++++++++++ src/libcamera/software_isp/meson.build | 27 ++ src/libcamera/software_isp/swisp_simple.cpp | 219 +++++++++ src/libcamera/software_isp/swstats.cpp | 65 +++ src/libcamera/software_isp/swstats_cpu.cpp | 169 +++++++ 31 files changed, 2164 insertions(+), 56 deletions(-) rename {src/libcamera/pipeline/rpi/common => include/libcamera/internal}/shared_mem_object.h (98%) create mode 100644 include/libcamera/internal/software_isp.h create mode 100644 include/libcamera/internal/software_isp/debayer.h create mode 100644 include/libcamera/internal/software_isp/debayer_cpu.h create mode 100644 include/libcamera/internal/software_isp/debayer_params.h create mode 100644 include/libcamera/internal/software_isp/meson.build create mode 100644 include/libcamera/internal/software_isp/swisp_simple.h create mode 100644 include/libcamera/internal/software_isp/swisp_stats.h create mode 100644 include/libcamera/internal/software_isp/swstats.h create mode 100644 include/libcamera/internal/software_isp/swstats_cpu.h create mode 100644 include/libcamera/ipa/soft.mojom create mode 100644 src/ipa/simple/common/meson.build create mode 100644 src/ipa/simple/common/soft_base.cpp create mode 100644 src/ipa/simple/common/soft_base.h create mode 100644 src/ipa/simple/meson.build create mode 100644 src/ipa/simple/simple/data/meson.build create mode 100644 src/ipa/simple/simple/data/soft.conf create mode 100644 src/ipa/simple/simple/meson.build create mode 100644 src/ipa/simple/simple/soft_simple.cpp create mode 100644 src/libcamera/software_isp.cpp create mode 100644 src/libcamera/software_isp/debayer.cpp create mode 100644 src/libcamera/software_isp/debayer_cpu.cpp create mode 100644 src/libcamera/software_isp/meson.build create mode 100644 src/libcamera/software_isp/swisp_simple.cpp create mode 100644 src/libcamera/software_isp/swstats.cpp create mode 100644 src/libcamera/software_isp/swstats_cpu.cpp