From patchwork Thu Feb 29 18:36:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19571 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 6A177BD160 for ; Thu, 29 Feb 2024 18:37:02 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 34A3F6286C; Thu, 29 Feb 2024 19:37:01 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="YL0zXLtT"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7E02361C96 for ; Thu, 29 Feb 2024 19:36:59 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231818; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=qEBIt1cyF/bH8kf1s5RC8QgRBMnDX5MJauo9PnGcNdY=; b=YL0zXLtT4okA2TlQKs+FfvIfGg8AYEEWhmWZTII4nbEHJI2Kwob2OZHYQUeb7OLDQDfoLY l6Gv0N7ax32X6D8+AseYzfjlZC1oGVwVQ+hdixGLuMVpbsk+fsrRJp7hyniGLHmXVD2wGD s1PxaovWN0+CvvbhQs/LL60tx3pBR4w= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-113-RComt9vEPx-oroYdVHtKqw-1; Thu, 29 Feb 2024 13:36:56 -0500 X-MC-Unique: RComt9vEPx-oroYdVHtKqw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7A19F185A781; Thu, 29 Feb 2024 18:36:56 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 686A21C060AF; Thu, 29 Feb 2024 18:36:55 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 00/18] libcamera: introduce Software ISP and Software IPA Date: Thu, 29 Feb 2024 19:36:11 +0100 Message-ID: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Hi All, Here is v4 of the patch-set to add Software ISP support to libcamera / to the simple pipeline-handler. Changes in v4 vs v3: - Andrey: [05/18] "libcamera: shared_mem_object: reorganize the code and document the SharedMemObject class" - documentation fixes [09/18] "libcamera: ipa: add Soft IPA" - Use int32_t for again*_ and exposure*_ as this matches the value the corresponding ControlValue::get() returns - Check for mmap() returning MAP_FAILED on error - Drop std::move() called on const SharedFD & argument - Replace #defines (EXPOSURE_OPTIMAL_VALUE etc) with const expressions - Use std::clamp() to keep exposure_ and again_ between the *_min_ and the *_max_ - add comment on non-linear gain value vs gain code [10/18] "libcamera: introduce SoftwareIsp" - Fix the check of if sharedParams_ were created OK new [18/18] "libcamera: Soft IPA: use CameraSensorHelper for analogue gain" - One can check if his camera sensor has the CameraSensorHelper implemented with something like 'grep ^REGISTER_CAMERA_SENSOR_HELPER src/ipa/libipa/camera_sensor_helper.cpp'. If the camera sensor used has no CameraSensorHelper, the Soft IPA works as before this patch. This fallback option doesn't make the code nicer, and makes it harder to change the AE/AGC algorithm to work faster, but I decided to keep it for now. - Milan: [17/18] "libcamera: software_isp: Apply black level compensation" - New patch adding black-level compensation autmatically detecting the sensor blacklevel - Hans: - Small typo changes in various comments/docs - Replace typedef-s with using CI pipeline: at least with v3 there was 1 issue with this series vs the CI pipeline which is not resolved yet, unit test configuration failed to build: - https://gitlab.freedesktop.org/camera/libcamera-softisp/-/jobs/55028866#L357 The issue is that the simple pipeline handler will not build unless -Dipas=simple is passed when configuring. The missing header here is autogenerated. I see that the ipu3 pipeline handler also unconditionally includes the auto-generated: #include #include headers, so it will show similar erros when using -Dpipelines=ipu3 without -Dipas=ipu3 . So I wonder how to fix this. Is there some meson magic somewhere to automatically enable the ipu3 ipa when the ipu3 pipeline is enabled which we can copy for the simple pipeline ? Changes in v3 vs v2: - The Software ISP now is always build when building the simple pipeline handler, no more -Dpipelines=simple/simple. Instead whether the SoftISP is used or not is based on the media-controller driver. For now it is only enabled for the "qcom-camss" driver (Andrey) - SoftISP factory has been removed, there is a just a single SoftwareISP class now - Fix the multi-threading issues (Andrey) - Fully document the SharedMemObject and DmaHeap classes (Andrey) - Drop SwStats base (integrate into SwStatsCpu class) - Move headers for classes only used by the SoftISP into src/libcamera/software_isp - Move SwStats / SwDebayer docs into .cpp and extend it - Rename many foo_bar symbols to fooBar - Add constexpr kFooBar values for varies hardcoded sizes like the yHistoGram having 16 bins and the gammalookuptable having 1024 entries - Make startFrame() and finishFrame() normal methods instead of using function pointers for these - Document how/why an array of src pointers is passed to the debayer functions - 12bpp unpacked raw bayer input support (Kieran) - Add "Software ISP benchmarking" doc Changes in v2 vs v1: - Integrated Dennis, Martti and Toon's auto-exposure algorithm based on the paper which they found which gives us a nice relatively simple AEC + AGC algorithm TODO: Add link to paper to source + commit-message - Integrated Dennis' doxygen comments for all new classes, no more warnings! - Move AWB gain calculations to the IPA (Andrey) - Added 8 bpp and 10 bpp unpacked raw bayer input support (Hans) - Use dma-buf for the output FrameBuffer-s (Andrey) - Memcpy data from input FrameBuffers to a heap buffer on a line by line basis to speedup debayering from uncached mem (Hans) - This addresses all "open issues" from the v1 posting Changes in v1 vs RFC-v2: - Add and use SwStats[Cpu] and Debayer[Cpu] classes - Rename linaro soft-IPA and soft-ISP implementations to simple following the simple pipeline hander - Integrate Pavel's swstats and debayer improvements This has been tested on: - Qualcomm RB5 board with a mezzanine board equipped with RPi camera v2 (Andrey) - Lenovo x13s, sc8280xp (Bryan) - Pinephone (Pavel) - Debix Model A (Milan) - Lenovo Thinkpad Yoga X1 yoga gen7/8, Alder Lake/Raptor Lake IPU6EP + ov2740 10bpp packed + unpacked (Dennis / Hans) - Dell Latitude 9420, Tiger Lake IPU6 + ov01a1s (RGBI) sensor (Martti, requires IGIG_GBGR_IGIG_GRGB patches on top) git branch for this v4 patch set: https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v07 git branch for v3 of the patch set: https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v06 git branch for v2 of the patch set: https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v05 git branch for v1 of the patch set: https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v04 Old RFC-v2 cover-letter with small updates: Here is an implementation of Software ISP and Software IPA which provide debayering and implementation of image processing algorithms for systems without a hardware ISP, or in cases when there are no public drivers for the hardware ISP present in the system. The Software IPA has auto exposure/AGC and AWB. For the AGC the analogue gain control of the camera sensor is used (if available). The 6th patch renames some variables in the simple pipeline handler for the Software ISP to use the same buffer handling code as the Converter currently does. This lets one to avoid adding extra code to the pipeline handler, but also makes the Software ISP and the Converter mutually exclusive. The Software ISP / IPA are intended to be used with the simple pipeline handler. The pipeline handler doesn't interact with the Software IPA directly - the Software IPA is hidden behind the Software ISP interface. To use the Software ISP the build must be configured with -Dpipelines=simple -Dipas=simple . This patch set uses SharedMemObject class used by the RPi pipeline handler - the second patch in the series moves the header file to a common directory. This patch set has been tested on Qualcomm RB5 board with a mezzanine board equipped with RPi camera v2 (not the standard RB5 camera mezzanine). git branch for the RFC-v2 patch set: https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v03 RFC-v1 of the patch set: https://patchwork.libcamera.org/cover/19262/ Changes in RFC v2 vs RFC v1: - patches are restructured and reordered - the Software IPA is hidden behind the Software ISP interface. The pipeline handler doesn't interact with the Software IPA directly - instantiation of the Software ISP / IPA is configurable (at build time) - comment explaining the implementation limitations is added to SwIspLinaro::IspWorker::debayerRaw10P() Regards, Hans Andrei Konovalov (2): libcamera: shared_mem_object: reorganize the code and document the SharedMemObject class libcamera: Soft IPA: use CameraSensorHelper for analogue gain Andrey Konovalov (8): libcamera: pipeline: simple: fix size adjustment in validate() libcamera: internal: Move dma_heaps.[h,cpp] to common directories libcamera: dma_heaps: extend DmaHeap class to support system heap libcamera: internal: Move SharedMemObject class to a common directory libcamera: ipa: add Soft IPA libcamera: introduce SoftwareIsp libcamera: pipeline: simple: rename converterBuffers_ and related vars libcamera: pipeline: simple: enable use of Soft ISP and Soft IPA Hans de Goede (7): libcamera: software_isp: Add SwStatsCpu class libcamera: software_isp: Add Debayer base class libcamera: software_isp: Add DebayerCpu class libcamera: swstats_cpu: Add support for 8, 10 and 12 bpp unpacked bayer input libcamera: debayer_cpu: Add support for 8, 10 and 12 bpp unpacked bayer input libcamera: debayer_cpu: Add BGR888 output support libcamera: Add "Software ISP benchmarking" documentation Milan Zamazal (1): libcamera: software_isp: Apply black level compensation Documentation/Doxyfile.in | 1 + Documentation/index.rst | 1 + Documentation/meson.build | 1 + Documentation/software-isp-benchmarking.rst | 82 ++ .../libcamera/internal}/dma_heaps.h | 14 +- include/libcamera/internal/meson.build | 3 + .../libcamera/internal}/shared_mem_object.h | 103 ++- .../internal/software_isp/debayer_params.h | 52 ++ .../internal/software_isp/meson.build | 7 + .../internal/software_isp/software_isp.h | 99 +++ .../internal/software_isp/swisp_stats.h | 44 + include/libcamera/ipa/meson.build | 1 + include/libcamera/ipa/soft.mojom | 28 + meson_options.txt | 2 +- src/ipa/simple/black_level.cpp | 84 ++ src/ipa/simple/black_level.h | 28 + src/ipa/simple/data/meson.build | 9 + src/ipa/simple/data/soft.conf | 3 + src/ipa/simple/meson.build | 30 + src/ipa/simple/soft_simple.cpp | 359 ++++++++ src/libcamera/dma_heaps.cpp | 164 ++++ src/libcamera/meson.build | 3 + src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp | 90 -- src/libcamera/pipeline/rpi/vc4/meson.build | 1 - src/libcamera/pipeline/rpi/vc4/vc4.cpp | 5 +- src/libcamera/pipeline/simple/simple.cpp | 234 ++++-- src/libcamera/shared_mem_object.cpp | 190 +++++ src/libcamera/software_isp/debayer.cpp | 29 + src/libcamera/software_isp/debayer.h | 104 +++ src/libcamera/software_isp/debayer_cpu.cpp | 784 ++++++++++++++++++ src/libcamera/software_isp/debayer_cpu.h | 158 ++++ src/libcamera/software_isp/meson.build | 8 + src/libcamera/software_isp/software_isp.cpp | 349 ++++++++ src/libcamera/software_isp/swstats_cpu.cpp | 336 ++++++++ src/libcamera/software_isp/swstats_cpu.h | 168 ++++ 35 files changed, 3373 insertions(+), 201 deletions(-) create mode 100644 Documentation/software-isp-benchmarking.rst rename {src/libcamera/pipeline/rpi/vc4 => include/libcamera/internal}/dma_heaps.h (65%) rename {src/libcamera/pipeline/rpi/common => include/libcamera/internal}/shared_mem_object.h (63%) create mode 100644 include/libcamera/internal/software_isp/debayer_params.h create mode 100644 include/libcamera/internal/software_isp/meson.build create mode 100644 include/libcamera/internal/software_isp/software_isp.h create mode 100644 include/libcamera/internal/software_isp/swisp_stats.h create mode 100644 include/libcamera/ipa/soft.mojom create mode 100644 src/ipa/simple/black_level.cpp create mode 100644 src/ipa/simple/black_level.h create mode 100644 src/ipa/simple/data/meson.build create mode 100644 src/ipa/simple/data/soft.conf create mode 100644 src/ipa/simple/meson.build create mode 100644 src/ipa/simple/soft_simple.cpp create mode 100644 src/libcamera/dma_heaps.cpp delete mode 100644 src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp create mode 100644 src/libcamera/shared_mem_object.cpp create mode 100644 src/libcamera/software_isp/debayer.cpp create mode 100644 src/libcamera/software_isp/debayer.h create mode 100644 src/libcamera/software_isp/debayer_cpu.cpp create mode 100644 src/libcamera/software_isp/debayer_cpu.h create mode 100644 src/libcamera/software_isp/meson.build create mode 100644 src/libcamera/software_isp/software_isp.cpp create mode 100644 src/libcamera/software_isp/swstats_cpu.cpp create mode 100644 src/libcamera/software_isp/swstats_cpu.h