From patchwork Sat Jan 13 14:22:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19390 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 39313BEFBE for ; Sat, 13 Jan 2024 14:22:33 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 50792628B7; Sat, 13 Jan 2024 15:22:32 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1705155752; bh=hnxx/ZHy5MCYYGp4CJ6lQ0YQA1GII3UbyO0p8dCLXvk=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=ZUSYoFCF0AG43PGRajSo4xeJXf6Wh2c4mDDe1OFtMw41UXLDqIBaN0iVn6IH+vItk gqHOdltWVEJ2pkBBZ8k0R9dBKP94XH2Gga/Mxr7+J/DoleS5of6eMLD9R6aJdStNcA VWSp0maPjpaotkVW8ifD+7fyqyV1JMbaF59sUimGliE9l6fSHPmeczF6BSic8lDeWy Bgd+zP0m9i800GtMP+8k5rTLaKlRLDUy90aBTJWzIz0kH2VAwAVkw0vt0ugrt0v8Ji XJ4AXu4wyU65d5nHkVj1qEw5JQx+W+nbUHpCmR/BsSWibSc/r4gB4b5xpKppBZqrc3 AA4F9J27iBZnA== Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 80FAF61D57 for ; Sat, 13 Jan 2024 15:22:30 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="aQjasETD"; dkim-atps=neutral DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1705155748; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=y40wOkSLf8IHLnTw3RThVaROajESMeA8cJBjuK6pUjg=; b=aQjasETDnPKVKh+QajikkCX9Yqd1utZmEXYSm9B+UIv/Xq6GM9TSiEV1ZohreZsU7VuipZ Bi4NDkXUsLzUEwgdtv8Xcn7kt0674tWLbFQdJQRStsJRGgKyGA5Kaa1zoCy4Idq5j6cPwD rh3ZXl5euK8Cf+L4E1N4P4Pg4kVzj4M= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-316-TLdWefYcPxK1_vLd-OhN8w-1; Sat, 13 Jan 2024 09:22:26 -0500 X-MC-Unique: TLdWefYcPxK1_vLd-OhN8w-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4728682DFE0; Sat, 13 Jan 2024 14:22:26 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.58]) by smtp.corp.redhat.com (Postfix) with ESMTP id EBA193C25; Sat, 13 Jan 2024 14:22:23 +0000 (UTC) To: libcamera-devel@lists.libcamera.org, Andrey Konovalov Date: Sat, 13 Jan 2024 15:22:00 +0100 Message-ID: <20240113142218.28063-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [libcamera-devel] [PATCH v2 00/18] X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Hans de Goede via libcamera-devel From: Hans de Goede Reply-To: Hans de Goede Cc: Maxime Ripard , g.martti@gmail.com, t.langendam@gmail.com, srinivas.kandagatla@linaro.org, Pavel Machek , Bryan O'Donoghue , admin@dennisbonke.com Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Hi All, Here is v2 of the patch-set to add Software ISP support to libcamera / to the simple pipeline-handler. Changes in v2 vs v1: - Integrated Dennis, Martti and Toon's auto-exposure algorithm based on the paper which they found which gives us a nice relatively simple AEC + AGC algorithm TODO: Add link to paper to source + commit-message - Integrated Dennis' doxygen comments for all new classes, no more warnings! - Move AWB gain calculations to the IPA (Andrey) - Added 8 bpp and 10 bpp unpacked raw bayer input support (Hans) - Use dma-buf for the output FrameBuffer-s (Andrey) - Memcpy data from input FrameBuffers to a heap buffer on a line by line basis to speedup debayering from uncached mem (Hans) - This addresses all "open issues" from the v1 posting Changes in v1 vs RFC-v2: - Add and use SwStats[Cpu] and Debayer[Cpu] classes - Rename linaro soft-IPA and soft-ISP implementations to simple following the simple pipeline hander - Integrate Pavel's swstats and debayer improvements This has been tested on: - Qualcomm RB5 board with a mezzanine board equipped with RPi camera v2 (Andrey) - Lenovo x13s, sc8280xp (Bryan) - Pinephone (Pavel) - Debix Model A (Milan) - Lenovo Thinkpad Yoga X1 yoga gen7/8, Alder Lake/Raptor Lake IPU6EP + ov2740 10bpp packed + unpacked (Dennis / Hans) - Dell Latitude 9420, Tiger Lake IPU6 + ov01a1s (RGBI) sensor (Martti, requires IGIG_GBGR_IGIG_GRGB patches on top) git branch for this v2 patch set: https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v05 git branch for v1 of the patch set: https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v04 Old RFC-v2 cover-letter with small updates: Here is an implementation of Software ISP and Software IPA which provide debayering and implementation of image processing algorithms for systems without a hardware ISP, or in cases when there are no public drivers for the hardware ISP present in the system. The implementation of the Software ISP is a reference one. A naive AWB alorithm is implemented as part of a function which does debayering and statistics calculations - the algorithm part is to be moved to the IPA in the next version of the patch set. And for debayering itself there is already a more efficient implementation by Hans de Goede. This patch set is currently using the earlier debayering implementation as it is less lines of code and is OK for the initial discussion. Only RAW10P format from the sensor is currently supported, but other debayering functions can be easily added (which of them to call is decided based on the input format). The Software IPA has only auto exposure and AGC. For the AGC the analogue gain control of the camera sensor is used (if available). The algorithm is very much simplified, and is mostly included as a reference code. The 6th patch renames some variables in the simple pipeline handler for the Software ISP to use the same buffer handling code as the Converter currently does. This lets one to avoid adding extra code to the pipeline handler, but also makes the Software ISP and the Converter mutually exclusive. The Software ISP / IPA are intended to be used with the simple pipeline handler. The pipeline handler doesn't interact with the Software IPA directly - the Software IPA is hidden behind the Software ISP interface. To use the Software ISP the build must be configured with -Dpipelines=simple/simple -Dipas=simple/simple and the Converter must not be used (the latter is hardcoded in the simple pipeline handler). If the build is configured with just -Dpipelines=simple the Software ISP / IPA are not used, and the simple pipeline handler works in the same way as without this patchset. "simple" in the -Dpipelines=simple/simple -Dipas=simple/simple is the name of the Software ISP / IPA implementation. -Dpipelines=simple/ -Dipas=simple/ should work too if another implementation were added. What implementation to use is the build time choice, only one implementation is included into the build. This patch set uses SharedMemObject class used by the RPi pipeline handler - the second patch in the series moves the header file to a common directory. This patch set has been tested on Qualcomm RB5 board with a mezzanine board equipped with RPi camera v2 (not the standard RB5 camera mezzanine). git branch for the RFC-v2 patch set: https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v03 RFC-v1 of the patch set: https://patchwork.libcamera.org/cover/19262/ Changes in RFC v2 vs RFC v1: - patches are restructured and reordered - the Software IPA is hidden behind the Software ISP interface. The pipeline handler doesn't interact with the Software IPA directly - instantiation of the Software ISP / IPA is configurable (at build time) - comment explaining the implementation limitations is added to SwIspLinaro::IspWorker::debayerRaw10P() Regards, Hans Andrey Konovalov (10): libcamera: pipeline: simple: fix size adjustment in validate() libcamera: internal: Move dma_heaps.[h,cpp] to common directories libcamera: dma_heaps: extend DmaHeap class to support system heap libcamera: internal: Move SharedMemObject class to a common directory libcamera: introduce SoftwareIsp class libcamera: ipa: add Soft IPA common files libcamera: ipa: Soft IPA: add a Simple Soft IPA implementation libcamera: software_isp: add Simple SoftwareIsp implementation libcamera: pipeline: simple: rename converterBuffers_ and related vars libcamera: pipeline: simple: enable use of Soft ISP and Soft IPA Dennis Bonke (1): libcamera: internal: Document the SharedMemObject class Hans de Goede (7): libcamera: software_isp: Add SwStats base class libcamera: software_isp: Add SwStatsCpu class libcamera: software_isp: Add Debayer base class libcamera: software_isp: Add DebayerCpu class libcamera: swstats_cpu: Add support for 8 and 10 bpp unpacked bayer input libcamera: debayer_cpu: Add support for 8 and 10 bpp unpacked bayer input libcamera: debayer_cpu: Add BGR888 output support Documentation/Doxyfile.in | 1 + .../libcamera/internal}/dma_heaps.h | 14 +- include/libcamera/internal/meson.build | 4 + .../libcamera/internal}/shared_mem_object.h | 57 +- include/libcamera/internal/software_isp.h | 231 ++++++ .../libcamera/internal/software_isp/debayer.h | 132 ++++ .../internal/software_isp/debayer_cpu.h | 141 ++++ .../internal/software_isp/debayer_params.h | 43 ++ .../internal/software_isp/meson.build | 11 + .../internal/software_isp/swisp_simple.h | 163 +++++ .../internal/software_isp/swisp_stats.h | 34 + .../libcamera/internal/software_isp/swstats.h | 215 ++++++ .../internal/software_isp/swstats_cpu.h | 51 ++ include/libcamera/ipa/meson.build | 1 + include/libcamera/ipa/soft.mojom | 29 + meson_options.txt | 3 +- src/ipa/simple/common/meson.build | 17 + src/ipa/simple/common/soft_base.cpp | 73 ++ src/ipa/simple/common/soft_base.h | 50 ++ src/ipa/simple/meson.build | 12 + src/ipa/simple/simple/data/meson.build | 9 + src/ipa/simple/simple/data/soft.conf | 3 + src/ipa/simple/simple/meson.build | 26 + src/ipa/simple/simple/soft_simple.cpp | 273 ++++++++ .../{pipeline/rpi/vc4 => }/dma_heaps.cpp | 55 +- src/libcamera/meson.build | 3 + src/libcamera/pipeline/rpi/vc4/meson.build | 1 - src/libcamera/pipeline/rpi/vc4/vc4.cpp | 5 +- src/libcamera/pipeline/simple/simple.cpp | 177 +++-- src/libcamera/software_isp.cpp | 62 ++ src/libcamera/software_isp/debayer.cpp | 22 + src/libcamera/software_isp/debayer_cpu.cpp | 661 ++++++++++++++++++ src/libcamera/software_isp/meson.build | 27 + src/libcamera/software_isp/swisp_simple.cpp | 238 +++++++ src/libcamera/software_isp/swstats.cpp | 22 + src/libcamera/software_isp/swstats_cpu.cpp | 261 +++++++ 36 files changed, 3035 insertions(+), 92 deletions(-) rename {src/libcamera/pipeline/rpi/vc4 => include/libcamera/internal}/dma_heaps.h (65%) rename {src/libcamera/pipeline/rpi/common => include/libcamera/internal}/shared_mem_object.h (62%) create mode 100644 include/libcamera/internal/software_isp.h create mode 100644 include/libcamera/internal/software_isp/debayer.h create mode 100644 include/libcamera/internal/software_isp/debayer_cpu.h create mode 100644 include/libcamera/internal/software_isp/debayer_params.h create mode 100644 include/libcamera/internal/software_isp/meson.build create mode 100644 include/libcamera/internal/software_isp/swisp_simple.h create mode 100644 include/libcamera/internal/software_isp/swisp_stats.h create mode 100644 include/libcamera/internal/software_isp/swstats.h create mode 100644 include/libcamera/internal/software_isp/swstats_cpu.h create mode 100644 include/libcamera/ipa/soft.mojom create mode 100644 src/ipa/simple/common/meson.build create mode 100644 src/ipa/simple/common/soft_base.cpp create mode 100644 src/ipa/simple/common/soft_base.h create mode 100644 src/ipa/simple/meson.build create mode 100644 src/ipa/simple/simple/data/meson.build create mode 100644 src/ipa/simple/simple/data/soft.conf create mode 100644 src/ipa/simple/simple/meson.build create mode 100644 src/ipa/simple/simple/soft_simple.cpp rename src/libcamera/{pipeline/rpi/vc4 => }/dma_heaps.cpp (54%) create mode 100644 src/libcamera/software_isp.cpp create mode 100644 src/libcamera/software_isp/debayer.cpp create mode 100644 src/libcamera/software_isp/debayer_cpu.cpp create mode 100644 src/libcamera/software_isp/meson.build create mode 100644 src/libcamera/software_isp/swisp_simple.cpp create mode 100644 src/libcamera/software_isp/swstats.cpp create mode 100644 src/libcamera/software_isp/swstats_cpu.cpp