From patchwork Tue Mar 19 12:35:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19758 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C5331C3272 for ; Tue, 19 Mar 2024 12:37:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 87FD462CAD; Tue, 19 Mar 2024 13:37:09 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="MFSqkwmb"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 10D0162CA7 for ; Tue, 19 Mar 2024 13:37:07 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851827; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0Kj1eEWEWKcX/f7g8KK/cc+4a1BmEcUIJDl+g2GbuPo=; b=MFSqkwmbGnbjHZLlLZ+41DfpEnVyU2RlpUplbYj/4opvIQGLSIQ7r7me9/bWBx0bD+6ARY G3PNf0dunZwdN6HU4EojrSh/TbS0TKGu0tOPxJFiq/DAliayjjnwXSmmYQHTaXQ9SzO6/o E30/M+NbFmEyJpZoOakdUNYU1yWcFJk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-626-mK5HQCSsMyOfMN2yr_K8Cg-1; Tue, 19 Mar 2024 08:37:00 -0400 X-MC-Unique: mK5HQCSsMyOfMN2yr_K8Cg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6608A852005; Tue, 19 Mar 2024 12:37:00 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3D6621C060A4; Tue, 19 Mar 2024 12:36:58 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Andrey Konovalov , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Hans de Goede , Kieran Bingham Subject: [PATCH v6 01/18] libcamera: pipeline: simple: fix size adjustment in validate() Date: Tue, 19 Mar 2024 13:35:48 +0100 Message-ID: <20240319123622.675599-2-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov SimpleCameraConfiguration::validate() adjusts the configuration of its streams (if the size is not in the outputSizes) to the captureSize. But the captureSize itself can be not in the outputSizes, and then the adjusted configuration won't be valid resulting in camera configuration failure. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Milan Zamazal Reviewed-by: Pavel Machek Signed-off-by: Andrey Konovalov Signed-off-by: Hans de Goede Reviewed-by: Laurent Pinchart --- src/libcamera/pipeline/simple/simple.cpp | 37 ++++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 01f2a977..04e77f7e 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -882,6 +882,30 @@ SimpleCameraConfiguration::SimpleCameraConfiguration(Camera *camera, { } +namespace { + +static Size adjustSize(const Size &requestedSize, const SizeRange &supportedSizes) +{ + ASSERT(supportedSizes.min <= supportedSizes.max); + + if (supportedSizes.min == supportedSizes.max) + return supportedSizes.max; + + unsigned int hStep = supportedSizes.hStep; + unsigned int vStep = supportedSizes.vStep; + + if (hStep == 0) + hStep = supportedSizes.max.width - supportedSizes.min.width; + if (vStep == 0) + vStep = supportedSizes.max.height - supportedSizes.min.height; + + Size adjusted = requestedSize.boundedTo(supportedSizes.max).expandedTo(supportedSizes.min); + + return adjusted.shrunkBy(supportedSizes.min).alignedDownTo(hStep, vStep).grownBy(supportedSizes.min); +} + +} /* namespace */ + CameraConfiguration::Status SimpleCameraConfiguration::validate() { const CameraSensor *sensor = data_->sensor_.get(); @@ -998,10 +1022,19 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() } if (!pipeConfig_->outputSizes.contains(cfg.size)) { + Size adjustedSize = pipeConfig_->captureSize; + /* + * The converter (when present) may not be able to output + * a size identical to its input size. The capture size is thus + * not guaranteed to be a valid output size. In such cases, use + * the smaller valid output size closest to the requested. + */ + if (!pipeConfig_->outputSizes.contains(adjustedSize)) + adjustedSize = adjustSize(cfg.size, pipeConfig_->outputSizes); LOG(SimplePipeline, Debug) << "Adjusting size from " << cfg.size - << " to " << pipeConfig_->captureSize; - cfg.size = pipeConfig_->captureSize; + << " to " << adjustedSize; + cfg.size = adjustedSize; status = Adjusted; } From patchwork Tue Mar 19 12:35:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19759 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 15B7EC3272 for ; Tue, 19 Mar 2024 12:37:12 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AF58E62D27; Tue, 19 Mar 2024 13:37:11 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="VJLGpmgl"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6D8D662CA8 for ; Tue, 19 Mar 2024 13:37:09 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tkDNprHAzAHddTuspjiNlu8IWrLfm/Y53ZcpYrntM8E=; b=VJLGpmglGffCeGTKKlP/aTx2o8dbe2w5a0MBNapeUTg87A8YqGKNVybKsfFFVa3FlHaBYy 9rFxZoAQywRUBdmfbQIxq0DVhG0zT3eTelEHbNUxqjn+xd0B/Rc3ADjsQ7NS0r6ApA/FRD FGOX4OS2TK/DMBg3M61Or4R16Pqqs8E= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-137-cMMIxN91OaCB3M0nNTVzmg-1; Tue, 19 Mar 2024 08:37:03 -0400 X-MC-Unique: cMMIxN91OaCB3M0nNTVzmg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5EDD528C97CE; Tue, 19 Mar 2024 12:37:03 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id C5C451C060A4; Tue, 19 Mar 2024 12:37:00 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Andrey Konovalov , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Hans de Goede , Kieran Bingham , Naushir Patuck Subject: [PATCH v6 02/18] libcamera: internal: Move dma_heaps.[h, cpp] to common directories Date: Tue, 19 Mar 2024 13:35:49 +0100 Message-ID: <20240319123622.675599-3-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov DmaHeap class is useful outside the RPi pipeline handler too. Move dma_heaps.h and dma_heaps.cpp to common directories. Update the build files and RPi vc4 pipeline handler accordingly. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Kieran Bingham Reviewed-by: Naushir Patuck Reviewed-by: Pavel Machek Signed-off-by: Andrey Konovalov Signed-off-by: Hans de Goede Reviewed-by: Milan Zamazal --- .../libcamera/internal}/dma_heaps.h | 4 - include/libcamera/internal/meson.build | 1 + src/libcamera/dma_heaps.cpp | 127 ++++++++++++++++++ src/libcamera/meson.build | 1 + src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp | 90 ------------- src/libcamera/pipeline/rpi/vc4/meson.build | 1 - src/libcamera/pipeline/rpi/vc4/vc4.cpp | 5 +- 7 files changed, 131 insertions(+), 98 deletions(-) rename {src/libcamera/pipeline/rpi/vc4 => include/libcamera/internal}/dma_heaps.h (92%) create mode 100644 src/libcamera/dma_heaps.cpp delete mode 100644 src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp diff --git a/src/libcamera/pipeline/rpi/vc4/dma_heaps.h b/include/libcamera/internal/dma_heaps.h similarity index 92% rename from src/libcamera/pipeline/rpi/vc4/dma_heaps.h rename to include/libcamera/internal/dma_heaps.h index 0a4a8d86..cff8f140 100644 --- a/src/libcamera/pipeline/rpi/vc4/dma_heaps.h +++ b/include/libcamera/internal/dma_heaps.h @@ -13,8 +13,6 @@ namespace libcamera { -namespace RPi { - class DmaHeap { public: @@ -27,6 +25,4 @@ private: UniqueFD dmaHeapHandle_; }; -} /* namespace RPi */ - } /* namespace libcamera */ diff --git a/include/libcamera/internal/meson.build b/include/libcamera/internal/meson.build index 7f1f3440..33eb0fb3 100644 --- a/include/libcamera/internal/meson.build +++ b/include/libcamera/internal/meson.build @@ -25,6 +25,7 @@ libcamera_internal_headers = files([ 'device_enumerator.h', 'device_enumerator_sysfs.h', 'device_enumerator_udev.h', + 'dma_heaps.h', 'formats.h', 'framebuffer.h', 'ipa_manager.h', diff --git a/src/libcamera/dma_heaps.cpp b/src/libcamera/dma_heaps.cpp new file mode 100644 index 00000000..38ef175a --- /dev/null +++ b/src/libcamera/dma_heaps.cpp @@ -0,0 +1,127 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Raspberry Pi Ltd + * + * dma_heaps.h - Helper class for dma-heap allocations. + */ + +#include "libcamera/internal/dma_heaps.h" + +#include +#include +#include +#include + +#include +#include + +#include + +/** + * \file dma_heaps.cpp + * \brief CMA dma-heap allocator + */ + +/* + * /dev/dma_heap/linux,cma is the dma-heap allocator, which allows dmaheap-cma + * to only have to worry about importing. + * + * Annoyingly, should the cma heap size be specified on the kernel command line + * instead of DT, the heap gets named "reserved" instead. + */ +static constexpr std::array heapNames = { + "/dev/dma_heap/linux,cma", + "/dev/dma_heap/reserved" +}; + +namespace libcamera { + +LOG_DEFINE_CATEGORY(DmaHeap) + +/** + * \class DmaHeap + * \brief Helper class for CMA dma-heap allocations + */ + +/** + * \brief Construct a DmaHeap that owns a CMA dma-heap file descriptor + * + * Goes through the internal list of possible names of the CMA dma-heap devices + * until a CMA dma-heap device is successfully opened. If it fails to open any + * dma-heap device, an invalid DmaHeap object is constructed. A valid DmaHeap + * object owns a wrapped dma-heap file descriptor. + * + * Please check the new DmaHeap object with \ref DmaHeap::isValid before using it. + */ +DmaHeap::DmaHeap() +{ + for (const char *name : heapNames) { + int ret = ::open(name, O_RDWR | O_CLOEXEC, 0); + if (ret < 0) { + ret = errno; + LOG(DmaHeap, Debug) + << "Failed to open " << name << ": " + << strerror(ret); + continue; + } + + dmaHeapHandle_ = UniqueFD(ret); + break; + } + + if (!dmaHeapHandle_.isValid()) + LOG(DmaHeap, Error) << "Could not open any dmaHeap device"; +} + +/** + * \brief Destroy the DmaHeap instance + * + * Destroying a DmaHeap instance which owns a wrapped dma-heap file descriptor + * closes the descriptor automatically. + */ +DmaHeap::~DmaHeap() = default; + +/** + * \fn DmaHeap::isValid() + * \brief Check if the DmaHeap instance is valid + * \return True if the DmaHeap is valid, false otherwise + */ + +/** + * \brief Allocate a dma-buf from the DmaHeap + * \param [in] name The name to set for the allocated buffer + * \param [in] size The size of the buffer to allocate + * \return The \ref UniqueFD of the allocated buffer + * + * Allocates a dma-buf with read/write access. + * If the allocation fails returns invalid UniqueFD. + */ +UniqueFD DmaHeap::alloc(const char *name, std::size_t size) +{ + int ret; + + if (!name) + return {}; + + struct dma_heap_allocation_data alloc = {}; + + alloc.len = size; + alloc.fd_flags = O_CLOEXEC | O_RDWR; + + ret = ::ioctl(dmaHeapHandle_.get(), DMA_HEAP_IOCTL_ALLOC, &alloc); + if (ret < 0) { + LOG(DmaHeap, Error) << "dmaHeap allocation failure for " << name; + return {}; + } + + UniqueFD allocFd(alloc.fd); + ret = ::ioctl(allocFd.get(), DMA_BUF_SET_NAME, name); + if (ret < 0) { + LOG(DmaHeap, Error) << "dmaHeap naming failure for " << name; + return {}; + } + + return allocFd; +} + +} /* namespace libcamera */ diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build index 2e7b0c77..dd8107fa 100644 --- a/src/libcamera/meson.build +++ b/src/libcamera/meson.build @@ -15,6 +15,7 @@ libcamera_sources = files([ 'delayed_controls.cpp', 'device_enumerator.cpp', 'device_enumerator_sysfs.cpp', + 'dma_heaps.cpp', 'fence.cpp', 'formats.cpp', 'framebuffer.cpp', diff --git a/src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp b/src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp deleted file mode 100644 index 317b1fc1..00000000 --- a/src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp +++ /dev/null @@ -1,90 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1-or-later */ -/* - * Copyright (C) 2020, Raspberry Pi Ltd - * - * dma_heaps.h - Helper class for dma-heap allocations. - */ - -#include "dma_heaps.h" - -#include -#include -#include -#include -#include -#include - -#include - -/* - * /dev/dma-heap/linux,cma is the dma-heap allocator, which allows dmaheap-cma - * to only have to worry about importing. - * - * Annoyingly, should the cma heap size be specified on the kernel command line - * instead of DT, the heap gets named "reserved" instead. - */ -static constexpr std::array heapNames = { - "/dev/dma_heap/linux,cma", - "/dev/dma_heap/reserved" -}; - -namespace libcamera { - -LOG_DECLARE_CATEGORY(RPI) - -namespace RPi { - -DmaHeap::DmaHeap() -{ - for (const char *name : heapNames) { - int ret = ::open(name, O_RDWR | O_CLOEXEC, 0); - if (ret < 0) { - ret = errno; - LOG(RPI, Debug) << "Failed to open " << name << ": " - << strerror(ret); - continue; - } - - dmaHeapHandle_ = UniqueFD(ret); - break; - } - - if (!dmaHeapHandle_.isValid()) - LOG(RPI, Error) << "Could not open any dmaHeap device"; -} - -DmaHeap::~DmaHeap() = default; - -UniqueFD DmaHeap::alloc(const char *name, std::size_t size) -{ - int ret; - - if (!name) - return {}; - - struct dma_heap_allocation_data alloc = {}; - - alloc.len = size; - alloc.fd_flags = O_CLOEXEC | O_RDWR; - - ret = ::ioctl(dmaHeapHandle_.get(), DMA_HEAP_IOCTL_ALLOC, &alloc); - if (ret < 0) { - LOG(RPI, Error) << "dmaHeap allocation failure for " - << name; - return {}; - } - - UniqueFD allocFd(alloc.fd); - ret = ::ioctl(allocFd.get(), DMA_BUF_SET_NAME, name); - if (ret < 0) { - LOG(RPI, Error) << "dmaHeap naming failure for " - << name; - return {}; - } - - return allocFd; -} - -} /* namespace RPi */ - -} /* namespace libcamera */ diff --git a/src/libcamera/pipeline/rpi/vc4/meson.build b/src/libcamera/pipeline/rpi/vc4/meson.build index cdb049c5..386e2296 100644 --- a/src/libcamera/pipeline/rpi/vc4/meson.build +++ b/src/libcamera/pipeline/rpi/vc4/meson.build @@ -1,7 +1,6 @@ # SPDX-License-Identifier: CC0-1.0 libcamera_sources += files([ - 'dma_heaps.cpp', 'vc4.cpp', ]) diff --git a/src/libcamera/pipeline/rpi/vc4/vc4.cpp b/src/libcamera/pipeline/rpi/vc4/vc4.cpp index ad7dddde..947b1e73 100644 --- a/src/libcamera/pipeline/rpi/vc4/vc4.cpp +++ b/src/libcamera/pipeline/rpi/vc4/vc4.cpp @@ -12,12 +12,11 @@ #include #include "libcamera/internal/device_enumerator.h" +#include "libcamera/internal/dma_heaps.h" #include "../common/pipeline_base.h" #include "../common/rpi_stream.h" -#include "dma_heaps.h" - using namespace std::chrono_literals; namespace libcamera { @@ -87,7 +86,7 @@ public: RPi::Device isp_; /* DMAHEAP allocation helper. */ - RPi::DmaHeap dmaHeap_; + DmaHeap dmaHeap_; SharedFD lsTable_; struct Config { From patchwork Tue Mar 19 12:35:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19760 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 3CC75C3274 for ; Tue, 19 Mar 2024 12:37:13 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 52A5962CAF; Tue, 19 Mar 2024 13:37:12 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="XGm7GChI"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4AC3F62CAB for ; Tue, 19 Mar 2024 13:37:10 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851829; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gnMRqS0gfC2BwtQzyeaQSroHobedeLswmEMy8N4g8hI=; b=XGm7GChIOgFOl+ucAihzP8QJAL/RAhpw6H51zsgXB7LvtjMqpmlo1d5b9yt9FH90+GVWMl 0eAx2zfshn73x1uobU3cKdcGUNywz9NxMki7XqIZxch0O4Ddl3xo1UxHy0yuaNhEcdzPtF iBWzBuA42GcE5fymyoBs1GA3VKfxKmY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-118-eV9cmX5iN_i-z11wBHiGIQ-1; Tue, 19 Mar 2024 08:37:06 -0400 X-MC-Unique: eV9cmX5iN_i-z11wBHiGIQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B5204185A78E; Tue, 19 Mar 2024 12:37:05 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id BF66B1C060A4; Tue, 19 Mar 2024 12:37:03 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Andrey Konovalov , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Hans de Goede , Kieran Bingham Subject: [PATCH v6 03/18] libcamera: dma_heaps: extend DmaHeap class to support system heap Date: Tue, 19 Mar 2024 13:35:50 +0100 Message-ID: <20240319123622.675599-4-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov Add an argument to the constructor to specify dma heaps type(s) to use. Can be DmaHeapFlag::Cma and/or DmaHeapFlag::System. By default DmaHeapFlag::Cma is used. If both DmaHeapFlag::Cma and DmaHeapFlag::System are set, CMA heap is tried first. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Milan Zamazal Reviewed-by: Pavel Machek Signed-off-by: Andrey Konovalov Signed-off-by: Hans de Goede --- include/libcamera/internal/dma_heaps.h | 12 ++++- src/libcamera/dma_heaps.cpp | 67 ++++++++++++++++++++------ 2 files changed, 63 insertions(+), 16 deletions(-) diff --git a/include/libcamera/internal/dma_heaps.h b/include/libcamera/internal/dma_heaps.h index cff8f140..80bf29e7 100644 --- a/include/libcamera/internal/dma_heaps.h +++ b/include/libcamera/internal/dma_heaps.h @@ -9,6 +9,7 @@ #include +#include #include namespace libcamera { @@ -16,7 +17,14 @@ namespace libcamera { class DmaHeap { public: - DmaHeap(); + enum class DmaHeapFlag { + Cma = 1 << 0, + System = 1 << 1, + }; + + using DmaHeapFlags = Flags; + + DmaHeap(DmaHeapFlags flags = DmaHeapFlag::Cma); ~DmaHeap(); bool isValid() const { return dmaHeapHandle_.isValid(); } UniqueFD alloc(const char *name, std::size_t size); @@ -25,4 +33,6 @@ private: UniqueFD dmaHeapHandle_; }; +LIBCAMERA_FLAGS_ENABLE_OPERATORS(DmaHeap::DmaHeapFlag) + } /* namespace libcamera */ diff --git a/src/libcamera/dma_heaps.cpp b/src/libcamera/dma_heaps.cpp index 38ef175a..d0e33ce6 100644 --- a/src/libcamera/dma_heaps.cpp +++ b/src/libcamera/dma_heaps.cpp @@ -19,9 +19,11 @@ /** * \file dma_heaps.cpp - * \brief CMA dma-heap allocator + * \brief dma-heap allocator */ +namespace libcamera { + /* * /dev/dma_heap/linux,cma is the dma-heap allocator, which allows dmaheap-cma * to only have to worry about importing. @@ -29,42 +31,77 @@ * Annoyingly, should the cma heap size be specified on the kernel command line * instead of DT, the heap gets named "reserved" instead. */ -static constexpr std::array heapNames = { - "/dev/dma_heap/linux,cma", - "/dev/dma_heap/reserved" + +/** + * \struct DmaHeapInfo + * \brief Tells what type of dma-heap the dma-heap represented by the device node name is + * \var DmaHeapInfo::flag + * \brief The type of the dma-heap + * \var DmaHeapInfo::name + * \brief The dma-heap's device node name + */ +struct DmaHeapInfo { + DmaHeap::DmaHeapFlag flag; + const char *name; }; -namespace libcamera { +static constexpr std::array heapInfos = { + { /* CMA heap names first */ + { DmaHeap::DmaHeapFlag::Cma, "/dev/dma_heap/linux,cma" }, + { DmaHeap::DmaHeapFlag::Cma, "/dev/dma_heap/reserved" }, + { DmaHeap::DmaHeapFlag::System, "/dev/dma_heap/system" } } +}; LOG_DEFINE_CATEGORY(DmaHeap) /** * \class DmaHeap - * \brief Helper class for CMA dma-heap allocations + * \brief Helper class for dma-heap allocations */ /** - * \brief Construct a DmaHeap that owns a CMA dma-heap file descriptor + * \enum DmaHeap::DmaHeapFlag + * \brief Type of the dma-heap + * \var DmaHeap::Cma + * \brief Allocate from a CMA dma-heap + * \var DmaHeap::System + * \brief Allocate from the system dma-heap + */ + +/** + * \typedef DmaHeap::DmaHeapFlags + * \brief A bitwise combination of DmaHeap::DmaHeapFlag values + */ + +/** + * \brief Construct a DmaHeap that owns a CMA or system dma-heap file descriptor + * \param [in] flags The type(s) of the dma-heap(s) to allocate from * - * Goes through the internal list of possible names of the CMA dma-heap devices - * until a CMA dma-heap device is successfully opened. If it fails to open any - * dma-heap device, an invalid DmaHeap object is constructed. A valid DmaHeap - * object owns a wrapped dma-heap file descriptor. + * By default \a flags are set to DmaHeap::DmaHeapFlag::Cma. The constructor goes + * through the internal list of possible names of the CMA and system dma-heap devices + * until the dma-heap device of the requested type is successfully opened. If more + * than one dma-heap type is specified in flags the CMA heap is tried first. If it + * fails to open any dma-heap device an invalid DmaHeap object is constructed. + * A valid DmaHeap object owns a wrapped dma-heap file descriptor. * * Please check the new DmaHeap object with \ref DmaHeap::isValid before using it. */ -DmaHeap::DmaHeap() +DmaHeap::DmaHeap(DmaHeapFlags flags) { - for (const char *name : heapNames) { - int ret = ::open(name, O_RDWR | O_CLOEXEC, 0); + for (const auto &info : heapInfos) { + if (!(flags & info.flag)) + continue; + + int ret = ::open(info.name, O_RDWR | O_CLOEXEC, 0); if (ret < 0) { ret = errno; LOG(DmaHeap, Debug) - << "Failed to open " << name << ": " + << "Failed to open " << info.name << ": " << strerror(ret); continue; } + LOG(DmaHeap, Debug) << "Using " << info.name; dmaHeapHandle_ = UniqueFD(ret); break; } From patchwork Tue Mar 19 12:35:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19761 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 216D4C3272 for ; Tue, 19 Mar 2024 12:37:16 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C697562D33; Tue, 19 Mar 2024 13:37:15 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="NT5cipEq"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 20DE662CA8 for ; Tue, 19 Mar 2024 13:37:13 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851832; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YqAj9ao1FE0PF3fvfHq+qhW/iy4PpFlLz4pNBFAEl2g=; b=NT5cipEq8hyhOxdFM90AxkijUoAuThhsomgsCufRIe/p+hBnoLagj4P0V94jxEZCmDy5Tx scAZDkXu5zqm0zKmx6pYqwsvi5rxB/n1cjP1u41D9Ens+oZo5yVZUtvz1/w1QjJ5i+Y1CO zoqjv/VleG8ExFcaS/CvzWJhLCm9SEQ= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-523-1BR-7ll0PdC-_Xz4UGxMtg-1; Tue, 19 Mar 2024 08:37:08 -0400 X-MC-Unique: 1BR-7ll0PdC-_Xz4UGxMtg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1A7DB3C40B51; Tue, 19 Mar 2024 12:37:08 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1EAAF1C060A4; Tue, 19 Mar 2024 12:37:05 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Andrey Konovalov , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Hans de Goede , Kieran Bingham Subject: [PATCH v6 04/18] libcamera: internal: Move SharedMemObject class to a common directory Date: Tue, 19 Mar 2024 13:35:51 +0100 Message-ID: <20240319123622.675599-5-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov Move SharedMemObject class out of RPi namespace and put it into include/libcamera/internal so that everyone could use it. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Signed-off-by: Andrey Konovalov Signed-off-by: Hans de Goede Reviewed-by: Milan Zamazal Reviewed-by: Laurent Pinchart --- include/libcamera/internal/meson.build | 1 + .../libcamera/internal}/shared_mem_object.h | 6 +----- 2 files changed, 2 insertions(+), 5 deletions(-) rename {src/libcamera/pipeline/rpi/common => include/libcamera/internal}/shared_mem_object.h (97%) diff --git a/include/libcamera/internal/meson.build b/include/libcamera/internal/meson.build index 33eb0fb3..5807dfd9 100644 --- a/include/libcamera/internal/meson.build +++ b/include/libcamera/internal/meson.build @@ -39,6 +39,7 @@ libcamera_internal_headers = files([ 'process.h', 'pub_key.h', 'request.h', + 'shared_mem_object.h', 'source_paths.h', 'sysfs.h', 'v4l2_device.h', diff --git a/src/libcamera/pipeline/rpi/common/shared_mem_object.h b/include/libcamera/internal/shared_mem_object.h similarity index 97% rename from src/libcamera/pipeline/rpi/common/shared_mem_object.h rename to include/libcamera/internal/shared_mem_object.h index aa56c220..98636b44 100644 --- a/src/libcamera/pipeline/rpi/common/shared_mem_object.h +++ b/include/libcamera/internal/shared_mem_object.h @@ -6,8 +6,8 @@ */ #pragma once -#include #include +#include #include #include #include @@ -19,8 +19,6 @@ namespace libcamera { -namespace RPi { - template class SharedMemObject { @@ -123,6 +121,4 @@ private: T *obj_; }; -} /* namespace RPi */ - } /* namespace libcamera */ From patchwork Tue Mar 19 12:35:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19762 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7A441C3274 for ; Tue, 19 Mar 2024 12:37:17 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E734D62CB0; Tue, 19 Mar 2024 13:37:16 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="CeATCWUi"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7E00762CB0 for ; Tue, 19 Mar 2024 13:37:14 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851833; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+U3fZupmgEV+NXN5HgwWh32kdIy52Th1rYplyYtySdY=; b=CeATCWUidoYd7evuOEbW/v+JlvOhdytFIjyKHU/2lgxXgTo3/PasLnwS1YhdCtZJS8Tnap HZaWFleBac4NAkTLYzBi27a9mO5svStzWjXZHYiQYXkmh0LSDNTGe+a32KW71GPkdqwv/R dmjEidjpkM8ZJNA7oRrS4JrSaRzF6eM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-563-mrwWM3VaM1q2qOGvES67_g-1; Tue, 19 Mar 2024 08:37:10 -0400 X-MC-Unique: mrwWM3VaM1q2qOGvES67_g-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A2251186E123; Tue, 19 Mar 2024 12:37:09 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id 551D11C060A6; Tue, 19 Mar 2024 12:37:08 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Andrei Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Hans de Goede , Kieran Bingham , Dennis Bonke Subject: [PATCH v6 05/18] libcamera: shared_mem_object: reorganize the code and document the SharedMemObject class Date: Tue, 19 Mar 2024 13:35:52 +0100 Message-ID: <20240319123622.675599-6-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrei Konovalov Split the parts which doesn't otherwise depend on the type T or arguments Args out of the SharedMemObject class into a new SharedMem class. Doxygen documentation by Dennis Bonke and Andrei Konovalov. Reviewed-by: Pavel Machek Co-developed-by: Dennis Bonke Signed-off-by: Dennis Bonke Signed-off-by: Andrei Konovalov Signed-off-by: Hans de Goede Reviewed-by: Milan Zamazal --- .../libcamera/internal/shared_mem_object.h | 101 ++++++---- src/libcamera/meson.build | 1 + src/libcamera/shared_mem_object.cpp | 190 ++++++++++++++++++ 3 files changed, 253 insertions(+), 39 deletions(-) create mode 100644 src/libcamera/shared_mem_object.cpp diff --git a/include/libcamera/internal/shared_mem_object.h b/include/libcamera/internal/shared_mem_object.h index 98636b44..43b07c9d 100644 --- a/include/libcamera/internal/shared_mem_object.h +++ b/include/libcamera/internal/shared_mem_object.h @@ -6,12 +6,9 @@ */ #pragma once -#include #include #include #include -#include -#include #include #include @@ -19,58 +16,92 @@ namespace libcamera { +class SharedMem +{ +public: + SharedMem() + : mem_(nullptr) + { + } + + SharedMem(const std::string &name, std::size_t size); + + SharedMem(SharedMem &&rhs) + { + this->name_ = std::move(rhs.name_); + this->fd_ = std::move(rhs.fd_); + this->mem_ = rhs.mem_; + rhs.mem_ = nullptr; + } + + virtual ~SharedMem() + { + if (mem_) + munmap(mem_, size_); + } + + /* Make SharedMem non-copyable for now. */ + LIBCAMERA_DISABLE_COPY(SharedMem) + + SharedMem &operator=(SharedMem &&rhs) + { + this->name_ = std::move(rhs.name_); + this->fd_ = std::move(rhs.fd_); + this->mem_ = rhs.mem_; + rhs.mem_ = nullptr; + return *this; + } + + const SharedFD &fd() const + { + return fd_; + } + + void *mem() const + { + return mem_; + } + +private: + std::string name_; + SharedFD fd_; + size_t size_; +protected: + void *mem_; +}; + template -class SharedMemObject +class SharedMemObject : public SharedMem { public: static constexpr std::size_t SIZE = sizeof(T); SharedMemObject() - : obj_(nullptr) + : SharedMem(), obj_(nullptr) { } template SharedMemObject(const std::string &name, Args &&...args) - : name_(name), obj_(nullptr) + : SharedMem(name, SIZE), obj_(nullptr) { - void *mem; - int ret; - - ret = memfd_create(name_.c_str(), MFD_CLOEXEC); - if (ret < 0) - return; - - fd_ = SharedFD(std::move(ret)); - if (!fd_.isValid()) - return; - - ret = ftruncate(fd_.get(), SIZE); - if (ret < 0) + if (mem_ == nullptr) return; - mem = mmap(nullptr, SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, - fd_.get(), 0); - if (mem == MAP_FAILED) - return; - - obj_ = new (mem) T(std::forward(args)...); + obj_ = new (mem_) T(std::forward(args)...); } SharedMemObject(SharedMemObject &&rhs) + : SharedMem(std::move(rhs)) { - this->name_ = std::move(rhs.name_); - this->fd_ = std::move(rhs.fd_); this->obj_ = rhs.obj_; rhs.obj_ = nullptr; } ~SharedMemObject() { - if (obj_) { + if (obj_) obj_->~T(); - munmap(obj_, SIZE); - } } /* Make SharedMemObject non-copyable for now. */ @@ -78,8 +109,7 @@ public: SharedMemObject &operator=(SharedMemObject &&rhs) { - this->name_ = std::move(rhs.name_); - this->fd_ = std::move(rhs.fd_); + SharedMem::operator=(std::move(rhs)); this->obj_ = rhs.obj_; rhs.obj_ = nullptr; return *this; @@ -105,19 +135,12 @@ public: return *obj_; } - const SharedFD &fd() const - { - return fd_; - } - explicit operator bool() const { return !!obj_; } private: - std::string name_; - SharedFD fd_; T *obj_; }; diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build index dd8107fa..ce31180b 100644 --- a/src/libcamera/meson.build +++ b/src/libcamera/meson.build @@ -39,6 +39,7 @@ libcamera_sources = files([ 'process.cpp', 'pub_key.cpp', 'request.cpp', + 'shared_mem_object.cpp', 'source_paths.cpp', 'stream.cpp', 'sysfs.cpp', diff --git a/src/libcamera/shared_mem_object.cpp b/src/libcamera/shared_mem_object.cpp new file mode 100644 index 00000000..44fe74c2 --- /dev/null +++ b/src/libcamera/shared_mem_object.cpp @@ -0,0 +1,190 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Raspberry Pi Ltd + * + * shared_mem_object.cpp - Helper class for shared memory allocations + */ + +#include "libcamera/internal/shared_mem_object.h" + +#include +#include + +/** + * \file shared_mem_object.cpp + * \brief Helper class for shared memory allocations + */ + +namespace libcamera { + +/** + * \class SharedMem + * \brief Helper class for allocating shared memory + * + * Memory is allocated and exposed as a SharedFD for use across IPC boundaries. + * + * SharedMem allocates the shared memory of the given size and maps it. + * To check that the shared memory was allocated and mapped successfully, one + * needs to verify that the pointer to the shared memory returned by SharedMem::mem() + * is not nullptr. + * + * To access the shared memory from another process the SharedFD should be passed + * to that process, and then the shared memory should be mapped into that process + * address space by calling mmap(). + * + * A single memfd is created for every SharedMem. If there is a need to allocate + * a large number of objects in shared memory, these objects should be grouped + * together and use the shared memory allocated by a single SharedMem object if + * possible. This will help to minimize the number of created memfd's. + */ + +/** + * \fn SharedMem::SharedMem(const std::string &name, std::size_t size) + * \brief Constructor for the SharedMem + * \param[in] name Name of the SharedMem + * \param[in] size Size of the shared memory to allocate and map + */ + +/** + * \fn SharedMem::SharedMem(SharedMem &&rhs) + * \brief Move constructor for SharedMem + * \param[in] rhs The object to move + */ + +/** + * \fn SharedMem::~SharedMem() + * \brief SharedMem destructor + * + * Unmaps the allocated shared memory. Decrements the shared memory descriptor use + * count. + */ + +/** + * \fn SharedMem &SharedMem::operator=(SharedMem &&rhs) + * \brief Move constructor for SharedMem + * \param[in] rhs The object to move + */ + +/** + * \fn const SharedFD &SharedMem::fd() const + * \brief Gets the file descriptor for the underlying shared memory + * \return The file descriptor + */ + +/** + * \fn void *SharedMem::mem() const + * \brief Gets the pointer to the underlying shared memory + * \return The pointer to the shared memory + */ + +SharedMem::SharedMem(const std::string &name, std::size_t size) + : name_(name), size_(size), mem_(nullptr) +{ + int fd = memfd_create(name_.c_str(), MFD_CLOEXEC); + if (fd < 0) + return; + + fd_ = SharedFD(std::move(fd)); + if (!fd_.isValid()) + return; + + if (ftruncate(fd_.get(), size_) < 0) + return; + + mem_ = mmap(nullptr, size, PROT_READ | PROT_WRITE, MAP_SHARED, + fd_.get(), 0); + if (mem_ == MAP_FAILED) + mem_ = nullptr; +} + +/** + * \var SharedMem::mem_ + * \brief Pointer to the shared memory allocated + */ + +/** + * \class SharedMemObject + * \brief Helper class for allocating objects in shared memory + * + * Memory is allocated and exposed as a SharedFD for use across IPC boundaries. + * + * Given the type of the object to be created in shared memory and the arguments + * to pass to this object's constructor, SharedMemObject allocates the shared memory + * of the size of the object and constructs the object in this memory. To ensure + * that the SharedMemObject was created successfully, one needs to verify that the + * overloaded bool() operator returns true. The object created in the shared memory + * can be accessed using the SharedMemObject::operator*() indirection operator. Its + * members can be accessed with the SharedMemObject::operator->() member of pointer + * operator. + * + * To access the object from another process the SharedFD should be passed to that + * process, and the shared memory should be mapped by calling mmap(). + * + * A single memfd is created for every SharedMemObject. If there is a need to allocate + * a large number of objects in shared memory, these objects should be grouped into a + * single large object to keep the number of created memfd's reasonably small. + */ + +/** + * \var SharedMemObject::SIZE + * \brief The size of the object that is going to be stored here + */ + +/** + * \fn SharedMemObject< T >::SharedMemObject(const std::string &name, Args &&...args) + * \brief Constructor for the SharedMemObject + * \param[in] name Name of the SharedMemObject + * \param[in] args Args to pass to the constructor of the object in shared memory + */ + +/** + * \fn SharedMemObject::SharedMemObject(SharedMemObject &&rhs) + * \brief Move constructor for SharedMemObject + * \param[in] rhs The object to move + */ + +/** + * \fn SharedMemObject::~SharedMemObject() + * \brief SharedMemObject destructor + * + * Destroys the object created in the shared memory and then unmaps the shared memory. + * Decrements the shared memory descriptor use count. + */ + +/** + * \fn SharedMemObject::operator=(SharedMemObject &&rhs) + * \brief Operator= for SharedMemObject + * \param[in] rhs The SharedMemObject object to take the data from + */ + +/** + * \fn SharedMemObject::operator->() + * \brief Operator-> for SharedMemObject + * \return The pointer to the object + */ + +/** + * \fn const T *SharedMemObject::operator->() const + * \brief Operator-> for SharedMemObject + * \return The pointer to the const object + */ + +/** + * \fn SharedMemObject::operator*() + * \brief Operator* for SharedMemObject + * \return The reference to the object + */ + +/** + * \fn const T &SharedMemObject::operator*() const + * \brief Operator* for SharedMemObject + * \return Const reference to the object + */ + +/** + * \fn SharedMemObject::operator bool() + * \brief Operator bool() for SharedMemObject + * \return True if the object was created OK in the shared memory, false otherwise + */ + +} // namespace libcamera From patchwork Tue Mar 19 12:35:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19763 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 37061C3272 for ; Tue, 19 Mar 2024 12:37:19 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D918D62D38; Tue, 19 Mar 2024 13:37:18 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="EKjxMt88"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 118A562D38 for ; Tue, 19 Mar 2024 13:37:16 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851835; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bngHayw4i0MPDf0TWnF5vmL8whmJroy5K/tpAp46qIE=; b=EKjxMt88EdihWAf5Pe/ZyQBOr0FAdV+xUIx5etHfHvLNBuCbu02otRVfyo+VWTId0olGhM niAy90iiNiZVuYEcf0BK+Dum/Bg+MSJ1F54rVPKZnbT3v7JeWQmj8s4EPwxmo83nInsZ7s gjPKym2nuZEocnnkARiemYTWOtrb9p0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-388-ZEo5iZ_JNwC2uVU1dS-r0w-1; Tue, 19 Mar 2024 08:37:12 -0400 X-MC-Unique: ZEo5iZ_JNwC2uVU1dS-r0w-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9D5E58007AD; Tue, 19 Mar 2024 12:37:11 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id D9D101C060A6; Tue, 19 Mar 2024 12:37:09 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Hans de Goede , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Kieran Bingham , Andrey Konovalov , Dennis Bonke , Marttico , Toon Langendam Subject: [PATCH v6 06/18] libcamera: software_isp: Add SwStatsCpu class Date: Tue, 19 Mar 2024 13:35:53 +0100 Message-ID: <20240319123622.675599-7-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Hans de Goede Add a CPU based SwStats implementation for SoftwareISP / SoftIPA use. This implementation offers a configure function + functions to gather statistics on a line by line basis. This allows CPU based software debayering to call into interlace debayering and statistics gathering on a line by line bases while the input data is still hot in the cache. This implementation also allows specifying a window over which to gather statistics instead of processing the whole frame. Doxygen documentation by Dennis Bonke. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Reviewed-by: Milan Zamazal Co-developed-by: Andrey Konovalov Signed-off-by: Andrey Konovalov Co-developed-by: Pavel Machek Signed-off-by: Pavel Machek Co-developed-by: Dennis Bonke Signed-off-by: Dennis Bonke Co-developed-by: Marttico Signed-off-by: Marttico Co-developed-by: Toon Langendam Signed-off-by: Toon Langendam Signed-off-by: Hans de Goede --- include/libcamera/internal/meson.build | 1 + .../internal/software_isp/meson.build | 5 + .../internal/software_isp/swisp_stats.h | 38 ++++ src/libcamera/meson.build | 1 + src/libcamera/software_isp/meson.build | 12 + src/libcamera/software_isp/swstats_cpu.cpp | 208 ++++++++++++++++++ src/libcamera/software_isp/swstats_cpu.h | 159 +++++++++++++ 7 files changed, 424 insertions(+) create mode 100644 include/libcamera/internal/software_isp/meson.build create mode 100644 include/libcamera/internal/software_isp/swisp_stats.h create mode 100644 src/libcamera/software_isp/meson.build create mode 100644 src/libcamera/software_isp/swstats_cpu.cpp create mode 100644 src/libcamera/software_isp/swstats_cpu.h diff --git a/include/libcamera/internal/meson.build b/include/libcamera/internal/meson.build index 5807dfd9..160fdc37 100644 --- a/include/libcamera/internal/meson.build +++ b/include/libcamera/internal/meson.build @@ -50,3 +50,4 @@ libcamera_internal_headers = files([ ]) subdir('converter') +subdir('software_isp') diff --git a/include/libcamera/internal/software_isp/meson.build b/include/libcamera/internal/software_isp/meson.build new file mode 100644 index 00000000..66c9c3fb --- /dev/null +++ b/include/libcamera/internal/software_isp/meson.build @@ -0,0 +1,5 @@ +# SPDX-License-Identifier: CC0-1.0 + +libcamera_internal_headers += files([ + 'swisp_stats.h', +]) diff --git a/include/libcamera/internal/software_isp/swisp_stats.h b/include/libcamera/internal/software_isp/swisp_stats.h new file mode 100644 index 00000000..afe42c9a --- /dev/null +++ b/include/libcamera/internal/software_isp/swisp_stats.h @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * + * swisp_stats.h - Statistics data format used by the software ISP and software IPA + */ + +#pragma once + +namespace libcamera { + +/** + * \brief Struct that holds the statistics for the Software ISP. + */ +struct SwIspStats { + /** + * \brief Holds the sum of all sampled red pixels. + */ + unsigned long sumR_; + /** + * \brief Holds the sum of all sampled green pixels. + */ + unsigned long sumG_; + /** + * \brief Holds the sum of all sampled blue pixels. + */ + unsigned long sumB_; + /** + * \brief Number of bins in the yHistogram. + */ + static constexpr unsigned int kYHistogramSize = 16; + /** + * \brief A histogram of luminance values. + */ + std::array yHistogram; +}; + +} /* namespace libcamera */ diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build index ce31180b..a3b12bc1 100644 --- a/src/libcamera/meson.build +++ b/src/libcamera/meson.build @@ -70,6 +70,7 @@ subdir('ipa') subdir('pipeline') subdir('proxy') subdir('sensor') +subdir('software_isp') null_dep = dependency('', required : false) diff --git a/src/libcamera/software_isp/meson.build b/src/libcamera/software_isp/meson.build new file mode 100644 index 00000000..fcfff74a --- /dev/null +++ b/src/libcamera/software_isp/meson.build @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: CC0-1.0 + +softisp_enabled = pipelines.contains('simple') +summary({'SoftISP support' : softisp_enabled}, section : 'Configuration') + +if not (softisp_enabled) + subdir_done() +endif + +libcamera_sources += files([ + 'swstats_cpu.cpp', +]) diff --git a/src/libcamera/software_isp/swstats_cpu.cpp b/src/libcamera/software_isp/swstats_cpu.cpp new file mode 100644 index 00000000..448d0e4c --- /dev/null +++ b/src/libcamera/software_isp/swstats_cpu.cpp @@ -0,0 +1,208 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * Copyright (C) 2023, Red Hat Inc. + * + * Authors: + * Hans de Goede + * + * swstats_cpu.cpp - CPU based software statistics implementation + */ + +#include "swstats_cpu.h" + +#include + +#include + +#include "libcamera/internal/bayer_format.h" + +namespace libcamera { + +/** + * \class SwStatsCpu + * \brief Class for gathering statistics on the CPU + * + * CPU based software ISP statistics implementation. + * + * This class offers a configure function + functions to gather statistics + * on a line by line basis. This allows CPU based software debayering to + * interlace debayering and statistics gathering on a line by line basis + * while the input data is still hot in the cache. + * + * It is also possible to specify a window over which to gather + * statistics instead of processing the whole frame. + */ + +LOG_DEFINE_CATEGORY(SwStatsCpu) + +SwStatsCpu::SwStatsCpu() +{ + sharedStats_ = SharedMemObject("softIsp_stats"); + if (!sharedStats_.fd().isValid()) + LOG(SwStatsCpu, Error) + << "Failed to create shared memory for statistics"; +} + +static const unsigned int kRedYMul = 77; /* 0.299 * 256 */ +static const unsigned int kGreenYMul = 150; /* 0.587 * 256 */ +static const unsigned int kBlueYMul = 29; /* 0.114 * 256 */ + +#define SWSTATS_START_LINE_STATS(pixel_t) \ + pixel_t r, g, g2, b; \ + unsigned int yVal; \ + \ + unsigned int sumR = 0; \ + unsigned int sumG = 0; \ + unsigned int sumB = 0; + +#define SWSTATS_ACCUMULATE_LINE_STATS(div) \ + sumR += r; \ + sumG += g; \ + sumB += b; \ + \ + yVal = r * kRedYMul; \ + yVal += g * kGreenYMul; \ + yVal += b * kBlueYMul; \ + stats_.yHistogram[yVal * SwIspStats::kYHistogramSize / (256 * 256 * (div))]++; + +#define SWSTATS_FINISH_LINE_STATS() \ + stats_.sumR_ += sumR; \ + stats_.sumG_ += sumG; \ + stats_.sumB_ += sumB; + +void SwStatsCpu::statsBGGR10PLine0(const uint8_t *src[]) +{ + const uint8_t *src0 = src[1] + window_.x * 5 / 4; + const uint8_t *src1 = src[2] + window_.x * 5 / 4; + const int widthInBytes = window_.width * 5 / 4; + + if (swapLines_) + std::swap(src0, src1); + + SWSTATS_START_LINE_STATS(uint8_t) + + /* x += 5 sample every other 2x2 block */ + for (int x = 0; x < widthInBytes; x += 5) { + /* BGGR */ + b = src0[x]; + g = src0[x + 1]; + g2 = src1[x]; + r = src1[x + 1]; + g = (g + g2) / 2; + /* Data is already 8 bits, divide by 1 */ + SWSTATS_ACCUMULATE_LINE_STATS(1) + } + + SWSTATS_FINISH_LINE_STATS() +} + +void SwStatsCpu::statsGBRG10PLine0(const uint8_t *src[]) +{ + const uint8_t *src0 = src[1] + window_.x * 5 / 4; + const uint8_t *src1 = src[2] + window_.x * 5 / 4; + const int widthInBytes = window_.width * 5 / 4; + + if (swapLines_) + std::swap(src0, src1); + + SWSTATS_START_LINE_STATS(uint8_t) + + /* x += 5 sample every other 2x2 block */ + for (int x = 0; x < widthInBytes; x += 5) { + /* GBRG */ + g = src0[x]; + b = src0[x + 1]; + r = src1[x]; + g2 = src1[x + 1]; + g = (g + g2) / 2; + /* Data is already 8 bits, divide by 1 */ + SWSTATS_ACCUMULATE_LINE_STATS(1) + } + + SWSTATS_FINISH_LINE_STATS() +} + +/** + * \brief Reset state to start statistics gathering for a new frame. + * + * This may only be called after a successful setWindow() call. + */ +void SwStatsCpu::startFrame(void) +{ + stats_.sumR_ = 0; + stats_.sumB_ = 0; + stats_.sumG_ = 0; + stats_.yHistogram.fill(0); +} + +/** + * \brief Finish statistics calculation for the current frame. + * + * This may only be called after a successful setWindow() call. + */ +void SwStatsCpu::finishFrame(void) +{ + *sharedStats_ = stats_; + statsReady.emit(0); +} + +/** + * \brief Configure the statistics object for the passed in input format. + * \param[in] inputCfg The input format + * + * \return 0 on success, a negative errno value on failure + */ +int SwStatsCpu::configure(const StreamConfiguration &inputCfg) +{ + BayerFormat bayerFormat = + BayerFormat::fromPixelFormat(inputCfg.pixelFormat); + + if (bayerFormat.bitDepth == 10 && + bayerFormat.packing == BayerFormat::Packing::CSI2) { + patternSize_.height = 2; + patternSize_.width = 4; /* 5 bytes per *4* pixels */ + /* Skip every 3th and 4th line, sample every other 2x2 block */ + ySkipMask_ = 0x02; + xShift_ = 0; + + switch (bayerFormat.order) { + case BayerFormat::BGGR: + case BayerFormat::GRBG: + stats0_ = &SwStatsCpu::statsBGGR10PLine0; + swapLines_ = bayerFormat.order == BayerFormat::GRBG; + return 0; + case BayerFormat::GBRG: + case BayerFormat::RGGB: + stats0_ = &SwStatsCpu::statsGBRG10PLine0; + swapLines_ = bayerFormat.order == BayerFormat::RGGB; + return 0; + default: + break; + } + } + + LOG(SwStatsCpu, Info) + << "Unsupported input format " << inputCfg.pixelFormat.toString(); + return -EINVAL; +} + +/** + * \brief Specify window coordinates over which to gather statistics. + * \param[in] window The window object. + */ +void SwStatsCpu::setWindow(Rectangle window) +{ + window_ = window; + + window_.x &= ~(patternSize_.width - 1); + window_.x += xShift_; + window_.y &= ~(patternSize_.height - 1); + + /* width_ - xShift_ to make sure the window fits */ + window_.width -= xShift_; + window_.width &= ~(patternSize_.width - 1); + window_.height &= ~(patternSize_.height - 1); +} + +} /* namespace libcamera */ diff --git a/src/libcamera/software_isp/swstats_cpu.h b/src/libcamera/software_isp/swstats_cpu.h new file mode 100644 index 00000000..0ac9ae71 --- /dev/null +++ b/src/libcamera/software_isp/swstats_cpu.h @@ -0,0 +1,159 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * Copyright (C) 2023, Red Hat Inc. + * + * Authors: + * Hans de Goede + * + * swstats_cpu.h - CPU based software statistics implementation + */ + +#pragma once + +#include + +#include + +#include + +#include "libcamera/internal/shared_mem_object.h" +#include "libcamera/internal/software_isp/swisp_stats.h" + +namespace libcamera { + +class PixelFormat; +struct StreamConfiguration; + +class SwStatsCpu +{ +public: + SwStatsCpu(); + ~SwStatsCpu() = default; + + /** + * \brief Gets whether the statistics object is valid. + * + * \return true if it's valid, false otherwise + */ + bool isValid() const { return sharedStats_.fd().isValid(); } + + /** + * \brief Get the file descriptor for the statistics. + * + * \return the file descriptor + */ + const SharedFD &getStatsFD() { return sharedStats_.fd(); } + + /** + * \brief Get the pattern size. + * + * For some input-formats, e.g. Bayer data, processing is done multiple lines + * and/or columns at a time. Get width and height at which the (bayer) pattern + * repeats. Window values are rounded down to a multiple of this and the height + * also indicates if processLine2() should be called or not. + * This may only be called after a successful configure() call. + * + * \return the pattern size + */ + const Size &patternSize() { return patternSize_; } + + int configure(const StreamConfiguration &inputCfg); + void setWindow(Rectangle window); + void startFrame(); + void finishFrame(); + + /** + * \brief Process line 0. + * \param[in] y The y coordinate. + * \param[in] src The input data. + * + * This function processes line 0 for input formats with patternSize height == 1. + * It'll process line 0 and 1 for input formats with patternSize height >= 2. + * This function may only be called after a successful setWindow() call. + */ + void processLine0(unsigned int y, const uint8_t *src[]) + { + if ((y & ySkipMask_) || y < (unsigned int)window_.y || + y >= (window_.y + window_.height)) + return; + + (this->*stats0_)(src); + } + + /** + * \brief Process line 2 and 3. + * \param[in] y The y coordinate. + * \param[in] src The input data. + * + * This function processes line 2 and 3 for input formats with patternSize height == 4. + * This function may only be called after a successful setWindow() call. + */ + void processLine2(unsigned int y, const uint8_t *src[]) + { + if ((y & ySkipMask_) || y < (unsigned int)window_.y || + y >= (window_.y + window_.height)) + return; + + (this->*stats2_)(src); + } + + /** + * \brief Signals that the statistics are ready. + * + * The int parameter isn't actually used. + */ + Signal statsReady; + +private: + /** + * \brief Called when there is data to get statistics from. + * \param[in] src The input data + * + * These functions take an array of (patternSize_.height + 1) src + * pointers each pointing to a line in the source image. The middle + * element of the array will point to the actual line being processed. + * Earlier element(s) will point to the previous line(s) and later + * element(s) to the next line(s). + * + * See the documentation of DebayerCpu::debayerFn for more details. + */ + using statsProcessFn = void (SwStatsCpu::*)(const uint8_t *src[]); + + void statsBGGR10PLine0(const uint8_t *src[]); + void statsGBRG10PLine0(const uint8_t *src[]); + + /* Variables set by configure(), used every line */ + statsProcessFn stats0_; + statsProcessFn stats2_; + bool swapLines_; + + /** + * \brief Skip lines where this bitmask is set in y. + */ + unsigned int ySkipMask_; + + /** + * \brief Statistics window, set by setWindow(), used ever line. + */ + Rectangle window_; + + /** + * \brief The size of the bayer pattern. + * + * Valid sizes are: 2x2, 4x2 or 4x4. + */ + Size patternSize_; + + /** + * \brief The offset of x, applied to window_.x for bayer variants. + * + * This can either be 0 or 1. + */ + unsigned int xShift_; + + SharedMemObject sharedStats_; + SwIspStats stats_; +}; + +} /* namespace libcamera */ From patchwork Tue Mar 19 12:35:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19764 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E49F4C3274 for ; Tue, 19 Mar 2024 12:37:21 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8FCCD62D2E; Tue, 19 Mar 2024 13:37:21 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="EuNurSfj"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E7AD462D3A for ; Tue, 19 Mar 2024 13:37:18 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851837; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6ITbLpCOMCVA8pu8/eMYJ/9xyFnVpkXTY9d1A5OZORU=; b=EuNurSfjM/gOaq2Eixh2vAycUGsKRLOFKGPXx7wOpnJZM4m0V1tFymIvqNbz+S6WNiJZXk cI+7I4ZbLGeGLAWYqFrgvyrIQADYAmV+wPeIHYc/wcYbrWuM6OQ1bw3l+xO5hMCnxXErwU wKWPLsCCH2mxErJQXagD08UYIeD5PBE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-20-UL_lyKubOX-7Hk1H5L4_yw-1; Tue, 19 Mar 2024 08:37:13 -0400 X-MC-Unique: UL_lyKubOX-7Hk1H5L4_yw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 49C638007AD; Tue, 19 Mar 2024 12:37:13 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id D7E741C060A4; Tue, 19 Mar 2024 12:37:11 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Hans de Goede , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Kieran Bingham , Dennis Bonke , Andrey Konovalov Subject: [PATCH v6 07/18] libcamera: software_isp: Add Debayer base class Date: Tue, 19 Mar 2024 13:35:54 +0100 Message-ID: <20240319123622.675599-8-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Hans de Goede Add a base class for debayer implementations. This is intended to be suitable for both GPU (or otherwise) accelerated debayer implementations as well as CPU based debayering. Doxygen documentation by Dennis Bonke. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Reviewed-by: Milan Zamazal Co-developed-by: Dennis Bonke Signed-off-by: Dennis Bonke Co-developed-by: Andrey Konovalov Signed-off-by: Andrey Konovalov Signed-off-by: Hans de Goede --- .../internal/software_isp/debayer_params.h | 48 ++++++++ .../internal/software_isp/meson.build | 1 + src/libcamera/software_isp/debayer.cpp | 29 +++++ src/libcamera/software_isp/debayer.h | 104 ++++++++++++++++++ src/libcamera/software_isp/meson.build | 1 + 5 files changed, 183 insertions(+) create mode 100644 include/libcamera/internal/software_isp/debayer_params.h create mode 100644 src/libcamera/software_isp/debayer.cpp create mode 100644 src/libcamera/software_isp/debayer.h diff --git a/include/libcamera/internal/software_isp/debayer_params.h b/include/libcamera/internal/software_isp/debayer_params.h new file mode 100644 index 00000000..98965fa1 --- /dev/null +++ b/include/libcamera/internal/software_isp/debayer_params.h @@ -0,0 +1,48 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Red Hat Inc. + * + * Authors: + * Hans de Goede + * + * debayer_params.h - DebayerParams header + */ + +#pragma once + +namespace libcamera { + +/** + * \brief Struct to hold the debayer parameters. + */ +struct DebayerParams { + /** + * \brief const value for 1.0 gain + */ + static constexpr unsigned int kGain10 = 256; + + /** + * \brief Red Gain + * + * 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. + */ + unsigned int gainR; + /** + * \brief Green Gain + * + * 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. + */ + unsigned int gainG; + /** + * \brief Blue Gain + * + * 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. + */ + unsigned int gainB; + /** + * \brief Gamma correction, 1.0 is no correction + */ + float gamma; +}; + +} /* namespace libcamera */ diff --git a/include/libcamera/internal/software_isp/meson.build b/include/libcamera/internal/software_isp/meson.build index 66c9c3fb..a620e16d 100644 --- a/include/libcamera/internal/software_isp/meson.build +++ b/include/libcamera/internal/software_isp/meson.build @@ -1,5 +1,6 @@ # SPDX-License-Identifier: CC0-1.0 libcamera_internal_headers += files([ + 'debayer_params.h', 'swisp_stats.h', ]) diff --git a/src/libcamera/software_isp/debayer.cpp b/src/libcamera/software_isp/debayer.cpp new file mode 100644 index 00000000..64f0b5a0 --- /dev/null +++ b/src/libcamera/software_isp/debayer.cpp @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * Copyright (C) 2023, Red Hat Inc. + * + * Authors: + * Hans de Goede + * + * debayer.cpp - debayer base class + */ + +#include "debayer.h" + +namespace libcamera { + +/** + * \class Debayer + * \brief Base debayering class + * + * Base class that provides functions for setting up the debayering process. + */ + +LOG_DEFINE_CATEGORY(Debayer) + +Debayer::~Debayer() +{ +} + +} /* namespace libcamera */ diff --git a/src/libcamera/software_isp/debayer.h b/src/libcamera/software_isp/debayer.h new file mode 100644 index 00000000..8880ff99 --- /dev/null +++ b/src/libcamera/software_isp/debayer.h @@ -0,0 +1,104 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * Copyright (C) 2023, Red Hat Inc. + * + * Authors: + * Hans de Goede + * + * debayer.h - debayering base class + */ + +#pragma once + +#include + +#include +#include + +#include +#include + +#include "libcamera/internal/software_isp/debayer_params.h" + +namespace libcamera { + +class FrameBuffer; + +LOG_DECLARE_CATEGORY(Debayer) + +class Debayer +{ +public: + virtual ~Debayer() = 0; + + /** + * \brief Configure the debayer object according to the passed in parameters. + * \param[in] inputCfg The input configuration. + * \param[in] outputCfgs The output configurations. + * + * \return 0 on success, a negative errno on failure. + */ + virtual int configure(const StreamConfiguration &inputCfg, + const std::vector> &outputCfgs) = 0; + + /** + * \brief Get the width and height at which the bayer pattern repeats. + * \param[in] inputFormat The input format. + * + * Valid sizes are: 2x2, 4x2 or 4x4. + * + * \return pattern size or an empty size for unsupported inputFormats. + */ + virtual Size patternSize(PixelFormat inputFormat) = 0; + + /** + * \brief Get the supported output formats. + * \param[in] inputFormat The input format. + * + * \return all supported output formats or an empty vector if there are none. + */ + virtual std::vector formats(PixelFormat inputFormat) = 0; + + /** + * \brief Get the stride and the frame size. + * \param[in] outputFormat The output format. + * \param[in] size The output size. + * + * \return a tuple of the stride and the frame size, or a tuple with 0,0 if there is no valid output config. + */ + virtual std::tuple + strideAndFrameSize(const PixelFormat &outputFormat, const Size &size) = 0; + + /** + * \brief Process the bayer data into the requested format. + * \param[in] input The input buffer. + * \param[in] output The output buffer. + * \param[in] params The parameters to be used in debayering. + * + * \note DebayerParams is passed by value deliberately so that a copy is passed + * when this is run in another thread by invokeMethod(). + */ + virtual void process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) = 0; + + /** + * \brief Get the supported output sizes for the given input format and size. + * \param[in] inputFormat The input format. + * \param[in] inputSize The input size. + * + * \return The valid size ranges or an empty range if there are none. + */ + virtual SizeRange sizes(PixelFormat inputFormat, const Size &inputSize) = 0; + + /** + * \brief Signals when the input buffer is ready. + */ + Signal inputBufferReady; + + /** + * \brief Signals when the output buffer is ready. + */ + Signal outputBufferReady; +}; + +} /* namespace libcamera */ diff --git a/src/libcamera/software_isp/meson.build b/src/libcamera/software_isp/meson.build index fcfff74a..62095f61 100644 --- a/src/libcamera/software_isp/meson.build +++ b/src/libcamera/software_isp/meson.build @@ -8,5 +8,6 @@ if not (softisp_enabled) endif libcamera_sources += files([ + 'debayer.cpp', 'swstats_cpu.cpp', ]) From patchwork Tue Mar 19 12:35:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19765 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A2301C32C3 for ; Tue, 19 Mar 2024 12:37:22 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 24A0562D42; Tue, 19 Mar 2024 13:37:22 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="NEWXJ6MB"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8846A62CB0 for ; Tue, 19 Mar 2024 13:37:19 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851838; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gI+GBoxb+me9ediuP6HV22Cq8Og10QnScvd7wZYeqyk=; b=NEWXJ6MBNwLBwCkkULe6U3GFfnEyNp1gEgqzJ7b/Jbwmay5neeHAkAYoH98ochzYcvipvZ z9Vztw3oe62WAmNSr+zk1pDQvGcSRmjchG9w40lYoLQM5aCofdH7RlRP1jRAXMZ2eshJBY Atsmerq79bMqy9WAkuBgE8668qDKwbg= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-76-I0GOczNsMXOfEDzL7dk4vA-1; Tue, 19 Mar 2024 08:37:15 -0400 X-MC-Unique: I0GOczNsMXOfEDzL7dk4vA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E88023C40B51; Tue, 19 Mar 2024 12:37:14 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id 84D5F1C060A4; Tue, 19 Mar 2024 12:37:13 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Hans de Goede , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Kieran Bingham , Dennis Bonke , Andrey Konovalov Subject: [PATCH v6 08/18] libcamera: software_isp: Add DebayerCpu class Date: Tue, 19 Mar 2024 13:35:55 +0100 Message-ID: <20240319123622.675599-9-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Hans de Goede Add CPU based debayering implementation. This initial implementation only supports debayering packed 10 bits per pixel bayer data in the 4 standard bayer orders. Doxygen documentation by Dennis Bonke. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Co-developed-by: Dennis Bonke Signed-off-by: Dennis Bonke Co-developed-by: Andrey Konovalov Signed-off-by: Andrey Konovalov Co-developed-by: Pavel Machek Signed-off-by: Pavel Machek Signed-off-by: Hans de Goede Reviewed-by: Milan Zamazal --- src/libcamera/software_isp/debayer_cpu.cpp | 626 +++++++++++++++++++++ src/libcamera/software_isp/debayer_cpu.h | 143 +++++ src/libcamera/software_isp/meson.build | 1 + 3 files changed, 770 insertions(+) create mode 100644 src/libcamera/software_isp/debayer_cpu.cpp create mode 100644 src/libcamera/software_isp/debayer_cpu.h diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp new file mode 100644 index 00000000..f932362c --- /dev/null +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -0,0 +1,626 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * Copyright (C) 2023, Red Hat Inc. + * + * Authors: + * Hans de Goede + * + * debayer_cpu.cpp - CPU based debayering class + */ + +#include "debayer_cpu.h" + +#include +#include +#include + +#include + +#include "libcamera/internal/bayer_format.h" +#include "libcamera/internal/framebuffer.h" +#include "libcamera/internal/mapped_framebuffer.h" + +namespace libcamera { + +/** + * \class DebayerCpu + * \brief Class for debayering on the CPU + * + * Implementation for CPU based debayering + */ + +/** + * \brief Constructs a DebayerCpu object. + * \param[in] stats Pointer to the stats object to use. + */ +DebayerCpu::DebayerCpu(std::unique_ptr stats) + : stats_(std::move(stats)), gamma_correction_(1.0) +{ +#ifdef __x86_64__ + enableInputMemcpy_ = false; +#else + enableInputMemcpy_ = true; +#endif + /* Initialize gamma to 1.0 curve */ + for (unsigned int i = 0; i < kGammaLookupSize; i++) + gamma_[i] = i / (kGammaLookupSize / kRGBLookupSize); + + for (unsigned int i = 0; i < kMaxLineBuffers; i++) + lineBuffers_[i] = nullptr; +} + +DebayerCpu::~DebayerCpu() +{ + for (unsigned int i = 0; i < kMaxLineBuffers; i++) + free(lineBuffers_[i]); +} + +// RGR +// GBG +// RGR +#define BGGR_BGR888(p, n, div) \ + *dst++ = blue_[curr[x] / (div)]; \ + *dst++ = green_[(prev[x] + curr[x - p] + curr[x + n] + next[x]) / (4 * (div))]; \ + *dst++ = red_[(prev[x - p] + prev[x + n] + next[x - p] + next[x + n]) / (4 * (div))]; \ + x++; + +// GBG +// RGR +// GBG +#define GRBG_BGR888(p, n, div) \ + *dst++ = blue_[(prev[x] + next[x]) / (2 * (div))]; \ + *dst++ = green_[curr[x] / (div)]; \ + *dst++ = red_[(curr[x - p] + curr[x + n]) / (2 * (div))]; \ + x++; + +// GRG +// BGB +// GRG +#define GBRG_BGR888(p, n, div) \ + *dst++ = blue_[(curr[x - p] + curr[x + n]) / (2 * (div))]; \ + *dst++ = green_[curr[x] / (div)]; \ + *dst++ = red_[(prev[x] + next[x]) / (2 * (div))]; \ + x++; + +// BGB +// GRG +// BGB +#define RGGB_BGR888(p, n, div) \ + *dst++ = blue_[(prev[x - p] + prev[x + n] + next[x - p] + next[x + n]) / (4 * (div))]; \ + *dst++ = green_[(prev[x] + curr[x - p] + curr[x + n] + next[x]) / (4 * (div))]; \ + *dst++ = red_[curr[x] / (div)]; \ + x++; + +void DebayerCpu::debayer10P_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + const int width_in_bytes = window_.width * 5 / 4; + const uint8_t *prev = (const uint8_t *)src[0]; + const uint8_t *curr = (const uint8_t *)src[1]; + const uint8_t *next = (const uint8_t *)src[2]; + + /* + * For the first pixel getting a pixel from the previous column uses + * x - 2 to skip the 5th byte with least-significant bits for 4 pixels. + * Same for last pixel (uses x + 2) and looking at the next column. + */ + for (int x = 0; x < width_in_bytes;) { + /* First pixel */ + BGGR_BGR888(2, 1, 1) + /* Second pixel BGGR -> GBRG */ + GBRG_BGR888(1, 1, 1) + /* Same thing for third and fourth pixels */ + BGGR_BGR888(1, 1, 1) + GBRG_BGR888(1, 2, 1) + /* Skip 5th src byte with 4 x 2 least-significant-bits */ + x++; + } +} + +void DebayerCpu::debayer10P_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + const int width_in_bytes = window_.width * 5 / 4; + const uint8_t *prev = (const uint8_t *)src[0]; + const uint8_t *curr = (const uint8_t *)src[1]; + const uint8_t *next = (const uint8_t *)src[2]; + + for (int x = 0; x < width_in_bytes;) { + /* First pixel */ + GRBG_BGR888(2, 1, 1) + /* Second pixel GRBG -> RGGB */ + RGGB_BGR888(1, 1, 1) + /* Same thing for third and fourth pixels */ + GRBG_BGR888(1, 1, 1) + RGGB_BGR888(1, 2, 1) + /* Skip 5th src byte with 4 x 2 least-significant-bits */ + x++; + } +} + +void DebayerCpu::debayer10P_GBGB_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + const int width_in_bytes = window_.width * 5 / 4; + const uint8_t *prev = (const uint8_t *)src[0]; + const uint8_t *curr = (const uint8_t *)src[1]; + const uint8_t *next = (const uint8_t *)src[2]; + + for (int x = 0; x < width_in_bytes;) { + /* Even pixel */ + GBRG_BGR888(2, 1, 1) + /* Odd pixel GBGR -> BGGR */ + BGGR_BGR888(1, 1, 1) + /* Same thing for next 2 pixels */ + GBRG_BGR888(1, 1, 1) + BGGR_BGR888(1, 2, 1) + /* Skip 5th src byte with 4 x 2 least-significant-bits */ + x++; + } +} + +void DebayerCpu::debayer10P_RGRG_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + const int width_in_bytes = window_.width * 5 / 4; + const uint8_t *prev = (const uint8_t *)src[0]; + const uint8_t *curr = (const uint8_t *)src[1]; + const uint8_t *next = (const uint8_t *)src[2]; + + for (int x = 0; x < width_in_bytes;) { + /* Even pixel */ + RGGB_BGR888(2, 1, 1) + /* Odd pixel RGGB -> GRBG */ + GRBG_BGR888(1, 1, 1) + /* Same thing for next 2 pixels */ + RGGB_BGR888(1, 1, 1) + GRBG_BGR888(1, 2, 1) + /* Skip 5th src byte with 4 x 2 least-significant-bits */ + x++; + } +} + +static bool isStandardBayerOrder(BayerFormat::Order order) +{ + return order == BayerFormat::BGGR || order == BayerFormat::GBRG || + order == BayerFormat::GRBG || order == BayerFormat::RGGB; +} + +/* + * Setup the Debayer object according to the passed in parameters. + * Return 0 on success, a negative errno value on failure + * (unsupported parameters). + */ +int DebayerCpu::getInputConfig(PixelFormat inputFormat, DebayerInputConfig &config) +{ + BayerFormat bayerFormat = + BayerFormat::fromPixelFormat(inputFormat); + + if (bayerFormat.bitDepth == 10 && + bayerFormat.packing == BayerFormat::Packing::CSI2 && + isStandardBayerOrder(bayerFormat.order)) { + config.bpp = 10; + config.patternSize.width = 4; /* 5 bytes per *4* pixels */ + config.patternSize.height = 2; + config.outputFormats = std::vector({ formats::RGB888 }); + return 0; + } + + LOG(Debayer, Info) + << "Unsupported input format " << inputFormat.toString(); + return -EINVAL; +} + +int DebayerCpu::getOutputConfig(PixelFormat outputFormat, DebayerOutputConfig &config) +{ + if (outputFormat == formats::RGB888) { + config.bpp = 24; + return 0; + } + + LOG(Debayer, Info) + << "Unsupported output format " << outputFormat.toString(); + return -EINVAL; +} + +/* TODO: this ignores outputFormat since there is only 1 supported outputFormat for now */ +int DebayerCpu::setDebayerFunctions(PixelFormat inputFormat, [[maybe_unused]] PixelFormat outputFormat) +{ + BayerFormat bayerFormat = + BayerFormat::fromPixelFormat(inputFormat); + + if (bayerFormat.bitDepth == 10 && + bayerFormat.packing == BayerFormat::Packing::CSI2) { + switch (bayerFormat.order) { + case BayerFormat::BGGR: + debayer0_ = &DebayerCpu::debayer10P_BGBG_BGR888; + debayer1_ = &DebayerCpu::debayer10P_GRGR_BGR888; + return 0; + case BayerFormat::GBRG: + debayer0_ = &DebayerCpu::debayer10P_GBGB_BGR888; + debayer1_ = &DebayerCpu::debayer10P_RGRG_BGR888; + return 0; + case BayerFormat::GRBG: + debayer0_ = &DebayerCpu::debayer10P_GRGR_BGR888; + debayer1_ = &DebayerCpu::debayer10P_BGBG_BGR888; + return 0; + case BayerFormat::RGGB: + debayer0_ = &DebayerCpu::debayer10P_RGRG_BGR888; + debayer1_ = &DebayerCpu::debayer10P_GBGB_BGR888; + return 0; + default: + break; + } + } + + LOG(Debayer, Error) << "Unsupported input output format combination"; + return -EINVAL; +} + +int DebayerCpu::configure(const StreamConfiguration &inputCfg, + const std::vector> &outputCfgs) +{ + if (getInputConfig(inputCfg.pixelFormat, inputConfig_) != 0) + return -EINVAL; + + if (stats_->configure(inputCfg) != 0) + return -EINVAL; + + const Size &stats_pattern_size = stats_->patternSize(); + if (inputConfig_.patternSize.width != stats_pattern_size.width || + inputConfig_.patternSize.height != stats_pattern_size.height) { + LOG(Debayer, Error) + << "mismatching stats and debayer pattern sizes for " + << inputCfg.pixelFormat.toString(); + return -EINVAL; + } + + inputConfig_.stride = inputCfg.stride; + + if (outputCfgs.size() != 1) { + LOG(Debayer, Error) + << "Unsupported number of output streams: " + << outputCfgs.size(); + return -EINVAL; + } + + const StreamConfiguration &outputCfg = outputCfgs[0]; + SizeRange outSizeRange = sizes(inputCfg.pixelFormat, inputCfg.size); + std::tie(outputConfig_.stride, outputConfig_.frameSize) = + strideAndFrameSize(outputCfg.pixelFormat, outputCfg.size); + + if (!outSizeRange.contains(outputCfg.size) || outputConfig_.stride != outputCfg.stride) { + LOG(Debayer, Error) + << "Invalid output size/stride: " + << "\n " << outputCfg.size << " (" << outSizeRange << ")" + << "\n " << outputCfg.stride << " (" << outputConfig_.stride << ")"; + return -EINVAL; + } + + if (setDebayerFunctions(inputCfg.pixelFormat, outputCfg.pixelFormat) != 0) + return -EINVAL; + + window_.x = ((inputCfg.size.width - outputCfg.size.width) / 2) & + ~(inputConfig_.patternSize.width - 1); + window_.y = ((inputCfg.size.height - outputCfg.size.height) / 2) & + ~(inputConfig_.patternSize.height - 1); + window_.width = outputCfg.size.width; + window_.height = outputCfg.size.height; + + /* Don't pass x,y since process() already adjusts src before passing it */ + stats_->setWindow(Rectangle(window_.size())); + + /* pad with patternSize.Width on both left and right side */ + lineBufferPadding_ = inputConfig_.patternSize.width * inputConfig_.bpp / 8; + lineBufferLength_ = window_.width * inputConfig_.bpp / 8 + + 2 * lineBufferPadding_; + for (unsigned int i = 0; + i < (inputConfig_.patternSize.height + 1) && enableInputMemcpy_; + i++) { + free(lineBuffers_[i]); + lineBuffers_[i] = (uint8_t *)malloc(lineBufferLength_); + if (!lineBuffers_[i]) + return -ENOMEM; + } + + measuredFrames_ = 0; + frameProcessTime_ = 0; + + return 0; +} + +/* + * Get width and height at which the bayer-pattern repeats. + * Return pattern-size or an empty Size for an unsupported inputFormat. + */ +Size DebayerCpu::patternSize(PixelFormat inputFormat) +{ + DebayerCpu::DebayerInputConfig config; + + if (getInputConfig(inputFormat, config) != 0) + return {}; + + return config.patternSize; +} + +std::vector DebayerCpu::formats(PixelFormat inputFormat) +{ + DebayerCpu::DebayerInputConfig config; + + if (getInputConfig(inputFormat, config) != 0) + return std::vector(); + + return config.outputFormats; +} + +std::tuple +DebayerCpu::strideAndFrameSize(const PixelFormat &outputFormat, const Size &size) +{ + DebayerCpu::DebayerOutputConfig config; + + if (getOutputConfig(outputFormat, config) != 0) + return std::make_tuple(0, 0); + + /* round up to multiple of 8 for 64 bits alignment */ + unsigned int stride = (size.width * config.bpp / 8 + 7) & ~7; + + return std::make_tuple(stride, stride * size.height); +} + +void DebayerCpu::setupInputMemcpy(const uint8_t *linePointers[]) +{ + const unsigned int patternHeight = inputConfig_.patternSize.height; + + if (!enableInputMemcpy_) + return; + + for (unsigned int i = 0; i < patternHeight; i++) { + memcpy(lineBuffers_[i], linePointers[i + 1] - lineBufferPadding_, + lineBufferLength_); + linePointers[i + 1] = lineBuffers_[i] + lineBufferPadding_; + } + + /* Point lineBufferIndex_ to first unused lineBuffer */ + lineBufferIndex_ = patternHeight; +} + +void DebayerCpu::shiftLinePointers(const uint8_t *linePointers[], const uint8_t *src) +{ + const unsigned int patternHeight = inputConfig_.patternSize.height; + + for (unsigned int i = 0; i < patternHeight; i++) + linePointers[i] = linePointers[i + 1]; + + linePointers[patternHeight] = src + + (patternHeight / 2) * (int)inputConfig_.stride; +} + +void DebayerCpu::memcpyNextLine(const uint8_t *linePointers[]) +{ + const unsigned int patternHeight = inputConfig_.patternSize.height; + + if (!enableInputMemcpy_) + return; + + memcpy(lineBuffers_[lineBufferIndex_], linePointers[patternHeight] - lineBufferPadding_, + lineBufferLength_); + linePointers[patternHeight] = lineBuffers_[lineBufferIndex_] + lineBufferPadding_; + + lineBufferIndex_ = (lineBufferIndex_ + 1) % (patternHeight + 1); +} + +void DebayerCpu::process2(const uint8_t *src, uint8_t *dst) +{ + unsigned int y_end = window_.y + window_.height; + /* Holds [0] previous- [1] current- [2] next-line */ + const uint8_t *linePointers[3]; + + /* Adjust src to top left corner of the window */ + src += window_.y * inputConfig_.stride + window_.x * inputConfig_.bpp / 8; + + /* [x] becomes [x - 1] after initial shiftLinePointers() call */ + if (window_.y) { + linePointers[1] = src - inputConfig_.stride; /* previous-line */ + linePointers[2] = src; + } else { + /* window_.y == 0, use the next line as prev line */ + linePointers[1] = src + inputConfig_.stride; + linePointers[2] = src; + /* Last 2 lines also need special handling */ + y_end -= 2; + } + + setupInputMemcpy(linePointers); + + for (unsigned int y = window_.y; y < y_end; y += 2) { + shiftLinePointers(linePointers, src); + memcpyNextLine(linePointers); + stats_->processLine0(y, linePointers); + (this->*debayer0_)(dst, linePointers); + src += inputConfig_.stride; + dst += outputConfig_.stride; + + shiftLinePointers(linePointers, src); + memcpyNextLine(linePointers); + (this->*debayer1_)(dst, linePointers); + src += inputConfig_.stride; + dst += outputConfig_.stride; + } + + if (window_.y == 0) { + shiftLinePointers(linePointers, src); + memcpyNextLine(linePointers); + stats_->processLine0(y_end, linePointers); + (this->*debayer0_)(dst, linePointers); + src += inputConfig_.stride; + dst += outputConfig_.stride; + + shiftLinePointers(linePointers, src); + /* next line may point outside of src, use prev. */ + linePointers[2] = linePointers[0]; + (this->*debayer1_)(dst, linePointers); + src += inputConfig_.stride; + dst += outputConfig_.stride; + } +} + +void DebayerCpu::process4(const uint8_t *src, uint8_t *dst) +{ + const unsigned int y_end = window_.y + window_.height; + /* + * This holds pointers to [0] 2-lines-up [1] 1-line-up [2] current-line + * [3] 1-line-down [4] 2-lines-down. + */ + const uint8_t *linePointers[5]; + + /* Adjust src to top left corner of the window */ + src += window_.y * inputConfig_.stride + window_.x * inputConfig_.bpp / 8; + + /* [x] becomes [x - 1] after initial shiftLinePointers() call */ + linePointers[1] = src - 2 * inputConfig_.stride; + linePointers[2] = src - inputConfig_.stride; + linePointers[3] = src; + linePointers[4] = src + inputConfig_.stride; + + setupInputMemcpy(linePointers); + + for (unsigned int y = window_.y; y < y_end; y += 4) { + shiftLinePointers(linePointers, src); + memcpyNextLine(linePointers); + stats_->processLine0(y, linePointers); + (this->*debayer0_)(dst, linePointers); + src += inputConfig_.stride; + dst += outputConfig_.stride; + + shiftLinePointers(linePointers, src); + memcpyNextLine(linePointers); + (this->*debayer1_)(dst, linePointers); + src += inputConfig_.stride; + dst += outputConfig_.stride; + + shiftLinePointers(linePointers, src); + memcpyNextLine(linePointers); + stats_->processLine2(y, linePointers); + (this->*debayer2_)(dst, linePointers); + src += inputConfig_.stride; + dst += outputConfig_.stride; + + shiftLinePointers(linePointers, src); + memcpyNextLine(linePointers); + (this->*debayer3_)(dst, linePointers); + src += inputConfig_.stride; + dst += outputConfig_.stride; + } +} + +static inline int64_t timeDiff(timespec &after, timespec &before) +{ + return (after.tv_sec - before.tv_sec) * 1000000000LL + + (int64_t)after.tv_nsec - (int64_t)before.tv_nsec; +} + +void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) +{ + timespec frameStartTime; + + if (measuredFrames_ < DebayerCpu::kLastFrameToMeasure) { + frameStartTime = {}; + clock_gettime(CLOCK_MONOTONIC_RAW, &frameStartTime); + } + + /* Apply DebayerParams */ + if (params.gamma != gamma_correction_) { + for (unsigned int i = 0; i < kGammaLookupSize; i++) + gamma_[i] = UINT8_MAX * powf(i / (kGammaLookupSize - 1.0), params.gamma); + + gamma_correction_ = params.gamma; + } + + for (unsigned int i = 0; i < kRGBLookupSize; i++) { + constexpr unsigned int div = + kRGBLookupSize * DebayerParams::kGain10 / kGammaLookupSize; + unsigned int idx; + + /* Apply gamma after gain! */ + idx = std::min({ i * params.gainR / div, (kGammaLookupSize - 1) }); + red_[i] = gamma_[idx]; + + idx = std::min({ i * params.gainG / div, (kGammaLookupSize - 1) }); + green_[i] = gamma_[idx]; + + idx = std::min({ i * params.gainB / div, (kGammaLookupSize - 1) }); + blue_[i] = gamma_[idx]; + } + + /* Copy metadata from the input buffer */ + FrameMetadata &metadata = output->_d()->metadata(); + metadata.status = input->metadata().status; + metadata.sequence = input->metadata().sequence; + metadata.timestamp = input->metadata().timestamp; + + MappedFrameBuffer in(input, MappedFrameBuffer::MapFlag::Read); + MappedFrameBuffer out(output, MappedFrameBuffer::MapFlag::Write); + if (!in.isValid() || !out.isValid()) { + LOG(Debayer, Error) << "mmap-ing buffer(s) failed"; + metadata.status = FrameMetadata::FrameError; + return; + } + + stats_->startFrame(); + + if (inputConfig_.patternSize.height == 2) + process2(in.planes()[0].data(), out.planes()[0].data()); + else + process4(in.planes()[0].data(), out.planes()[0].data()); + + metadata.planes()[0].bytesused = out.planes()[0].size(); + + /* Measure before emitting signals */ + if (measuredFrames_ < DebayerCpu::kLastFrameToMeasure && + ++measuredFrames_ > DebayerCpu::kFramesToSkip) { + timespec frameEndTime = {}; + clock_gettime(CLOCK_MONOTONIC_RAW, &frameEndTime); + frameProcessTime_ += timeDiff(frameEndTime, frameStartTime); + if (measuredFrames_ == DebayerCpu::kLastFrameToMeasure) { + const unsigned int measuredFrames = DebayerCpu::kLastFrameToMeasure - + DebayerCpu::kFramesToSkip; + LOG(Debayer, Info) + << "Processed " << measuredFrames + << " frames in " << frameProcessTime_ / 1000 << "us, " + << frameProcessTime_ / (1000 * measuredFrames) + << " us/frame"; + } + } + + stats_->finishFrame(); + outputBufferReady.emit(output); + inputBufferReady.emit(input); +} + +SizeRange DebayerCpu::sizes(PixelFormat inputFormat, const Size &inputSize) +{ + Size pattern_size = patternSize(inputFormat); + unsigned int border_height = pattern_size.height; + + if (pattern_size.isNull()) + return {}; + + /* No need for top/bottom border with a pattern height of 2 */ + if (pattern_size.height == 2) + border_height = 0; + + /* + * For debayer interpolation a border is kept around the entire image + * and the minimum output size is pattern-height x pattern-width. + */ + if (inputSize.width < (3 * pattern_size.width) || + inputSize.height < (2 * border_height + pattern_size.height)) { + LOG(Debayer, Warning) + << "Input format size too small: " << inputSize.toString(); + return {}; + } + + return SizeRange(Size(pattern_size.width, pattern_size.height), + Size((inputSize.width - 2 * pattern_size.width) & ~(pattern_size.width - 1), + (inputSize.height - 2 * border_height) & ~(pattern_size.height - 1)), + pattern_size.width, pattern_size.height); +} + +} /* namespace libcamera */ diff --git a/src/libcamera/software_isp/debayer_cpu.h b/src/libcamera/software_isp/debayer_cpu.h new file mode 100644 index 00000000..8a51ed85 --- /dev/null +++ b/src/libcamera/software_isp/debayer_cpu.h @@ -0,0 +1,143 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * Copyright (C) 2023, Red Hat Inc. + * + * Authors: + * Hans de Goede + * + * debayer_cpu.h - CPU based debayering header + */ + +#pragma once + +#include +#include +#include + +#include + +#include "debayer.h" +#include "swstats_cpu.h" + +namespace libcamera { + +class DebayerCpu : public Debayer, public Object +{ +public: + DebayerCpu(std::unique_ptr stats); + ~DebayerCpu(); + + int configure(const StreamConfiguration &inputCfg, + const std::vector> &outputCfgs); + Size patternSize(PixelFormat inputFormat); + std::vector formats(PixelFormat input); + std::tuple + strideAndFrameSize(const PixelFormat &outputFormat, const Size &size); + void process(FrameBuffer *input, FrameBuffer *output, DebayerParams params); + SizeRange sizes(PixelFormat inputFormat, const Size &inputSize); + + /** + * \brief Get the file descriptor for the statistics. + * + * \return the file descriptor pointing to the statistics. + */ + const SharedFD &getStatsFD() { return stats_->getStatsFD(); } + + /** + * \brief Get the output frame size. + * + * \return The output frame size. + */ + unsigned int frameSize() { return outputConfig_.frameSize; } + +private: + /** + * \brief Called to debayer 1 line of Bayer input data to output format + * \param[out] dst Pointer to the start of the output line to write + * \param[in] src The input data + * + * Input data is an array of (patternSize_.height + 1) src + * pointers each pointing to a line in the Bayer source. The middle + * element of the array will point to the actual line being processed. + * Earlier element(s) will point to the previous line(s) and later + * element(s) to the next line(s). + * + * These functions take an array of src pointers, rather than + * a single src pointer + a stride for the source, so that when the src + * is slow uncached memory it can be copied to faster memory before + * debayering. Debayering a standard 2x2 Bayer pattern requires access + * to the previous and next src lines for interpolating the missing + * colors. To allow copying the src lines only once 3 temporary buffers + * each holding a single line are used, re-using the oldest buffer for + * the next line and the pointers are swizzled so that: + * src[0] = previous-line, src[1] = currrent-line, src[2] = next-line. + * This way the 3 pointers passed to the debayer functions form + * a sliding window over the src avoiding the need to copy each + * line more than once. + * + * Similarly for bayer patterns which repeat every 4 lines, 5 src + * pointers are passed holding: src[0] = 2-lines-up, src[1] = 1-line-up + * src[2] = current-line, src[3] = 1-line-down, src[4] = 2-lines-down. + */ + using debayerFn = void (DebayerCpu::*)(uint8_t *dst, const uint8_t *src[]); + + /* CSI-2 packed 10-bit raw bayer format (all the 4 orders) */ + void debayer10P_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]); + void debayer10P_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]); + void debayer10P_GBGB_BGR888(uint8_t *dst, const uint8_t *src[]); + void debayer10P_RGRG_BGR888(uint8_t *dst, const uint8_t *src[]); + + struct DebayerInputConfig { + Size patternSize; + unsigned int bpp; /* Memory used per pixel, not precision */ + unsigned int stride; + std::vector outputFormats; + }; + + struct DebayerOutputConfig { + unsigned int bpp; /* Memory used per pixel, not precision */ + unsigned int stride; + unsigned int frameSize; + }; + + int getInputConfig(PixelFormat inputFormat, DebayerInputConfig &config); + int getOutputConfig(PixelFormat outputFormat, DebayerOutputConfig &config); + int setDebayerFunctions(PixelFormat inputFormat, PixelFormat outputFormat); + void setupInputMemcpy(const uint8_t *linePointers[]); + void shiftLinePointers(const uint8_t *linePointers[], const uint8_t *src); + void memcpyNextLine(const uint8_t *linePointers[]); + void process2(const uint8_t *src, uint8_t *dst); + void process4(const uint8_t *src, uint8_t *dst); + + static constexpr unsigned int kGammaLookupSize = 1024; + static constexpr unsigned int kRGBLookupSize = 256; + /* Max. supported Bayer pattern height is 4, debayering this requires 5 lines */ + static constexpr unsigned int kMaxLineBuffers = 5; + + std::array gamma_; + std::array red_; + std::array green_; + std::array blue_; + debayerFn debayer0_; + debayerFn debayer1_; + debayerFn debayer2_; + debayerFn debayer3_; + Rectangle window_; + DebayerInputConfig inputConfig_; + DebayerOutputConfig outputConfig_; + std::unique_ptr stats_; + uint8_t *lineBuffers_[kMaxLineBuffers]; + unsigned int lineBufferLength_; + unsigned int lineBufferPadding_; + unsigned int lineBufferIndex_; + bool enableInputMemcpy_; + float gamma_correction_; + unsigned int measuredFrames_; + int64_t frameProcessTime_; + /* Skip 30 frames for things to stabilize then measure 30 frames */ + static constexpr unsigned int kFramesToSkip = 30; + static constexpr unsigned int kLastFrameToMeasure = 60; +}; + +} /* namespace libcamera */ diff --git a/src/libcamera/software_isp/meson.build b/src/libcamera/software_isp/meson.build index 62095f61..71b46539 100644 --- a/src/libcamera/software_isp/meson.build +++ b/src/libcamera/software_isp/meson.build @@ -9,5 +9,6 @@ endif libcamera_sources += files([ 'debayer.cpp', + 'debayer_cpu.cpp', 'swstats_cpu.cpp', ]) From patchwork Tue Mar 19 12:35:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19766 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C0993C3272 for ; Tue, 19 Mar 2024 12:37:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 58FBD62DD1; Tue, 19 Mar 2024 13:37:24 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="bUfl3D5z"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C61AC62D37 for ; Tue, 19 Mar 2024 13:37:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851841; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KDlz08+dcNRxNN1chjBqyd09aH8j/whCCEHQRUTGoKs=; b=bUfl3D5zyhM9sBzhWvPGRPrqdgmKfGElR93TYh/NDVI8FPcTbbv1mhsqUGkbfNgihE3Pb2 3HQSoekDnkfLdFzkEUydzgiL3xc1dxvBaNonGqAUalcPTjw7pSrkALx80a0BsRoYxyaMex OXisUQfey2CqWGBlR7DRRIUCSev1TyM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-590-nVsfXiVLOs6qbMvIj8vrZA-1; Tue, 19 Mar 2024 08:37:18 -0400 X-MC-Unique: nVsfXiVLOs6qbMvIj8vrZA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 84815101A588; Tue, 19 Mar 2024 12:37:17 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2C8F21C060A4; Tue, 19 Mar 2024 12:37:15 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Andrey Konovalov , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Hans de Goede , Kieran Bingham , Dennis Bonke , Marttico , Toon Langendam Subject: [PATCH v6 09/18] libcamera: ipa: add Soft IPA Date: Tue, 19 Mar 2024 13:35:56 +0100 Message-ID: <20240319123622.675599-10-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov Define the Soft IPA main and event interfaces, add the Soft IPA implementation. The current src/ipa/meson.build assumes the IPA name to match the pipeline name. For this reason "-Dipas=simple" is used for the Soft IPA module. Auto exposure/gain and AWB implementation by Dennis, Toon and Martti. Auto exposure/gain targets a Mean Sample Value of 2.5 following the MSV calculation algorithm from: https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Signed-off-by: Andrey Konovalov Co-developed-by: Dennis Bonke Signed-off-by: Dennis Bonke Co-developed-by: Marttico Signed-off-by: Marttico Co-developed-by: Toon Langendam Signed-off-by: Toon Langendam Signed-off-by: Hans de Goede --- Documentation/Doxyfile.in | 1 + include/libcamera/ipa/meson.build | 1 + include/libcamera/ipa/soft.mojom | 28 +++ meson_options.txt | 2 +- src/ipa/simple/data/meson.build | 9 + src/ipa/simple/data/soft.conf | 3 + src/ipa/simple/meson.build | 25 +++ src/ipa/simple/soft_simple.cpp | 326 ++++++++++++++++++++++++++++++ 8 files changed, 394 insertions(+), 1 deletion(-) create mode 100644 include/libcamera/ipa/soft.mojom create mode 100644 src/ipa/simple/data/meson.build create mode 100644 src/ipa/simple/data/soft.conf create mode 100644 src/ipa/simple/meson.build create mode 100644 src/ipa/simple/soft_simple.cpp diff --git a/Documentation/Doxyfile.in b/Documentation/Doxyfile.in index a86ea6c1..2be8d47b 100644 --- a/Documentation/Doxyfile.in +++ b/Documentation/Doxyfile.in @@ -44,6 +44,7 @@ EXCLUDE = @TOP_SRCDIR@/include/libcamera/base/span.h \ @TOP_SRCDIR@/src/libcamera/pipeline/ \ @TOP_SRCDIR@/src/libcamera/tracepoints.cpp \ @TOP_BUILDDIR@/include/libcamera/internal/tracepoints.h \ + @TOP_BUILDDIR@/include/libcamera/ipa/soft_ipa_interface.h \ @TOP_BUILDDIR@/src/libcamera/proxy/ EXCLUDE_PATTERNS = @TOP_BUILDDIR@/include/libcamera/ipa/*_serializer.h \ diff --git a/include/libcamera/ipa/meson.build b/include/libcamera/ipa/meson.build index f3b4881c..3352d08f 100644 --- a/include/libcamera/ipa/meson.build +++ b/include/libcamera/ipa/meson.build @@ -65,6 +65,7 @@ pipeline_ipa_mojom_mapping = { 'ipu3': 'ipu3.mojom', 'rkisp1': 'rkisp1.mojom', 'rpi/vc4': 'raspberrypi.mojom', + 'simple': 'soft.mojom', 'vimc': 'vimc.mojom', } diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom new file mode 100644 index 00000000..c249bd75 --- /dev/null +++ b/include/libcamera/ipa/soft.mojom @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ + +/* + * \todo Document the interface and remove the related EXCLUDE_PATTERNS entry. + */ + +module ipa.soft; + +import "include/libcamera/ipa/core.mojom"; + +interface IPASoftInterface { + init(libcamera.IPASettings settings, + libcamera.SharedFD fdStats, + libcamera.SharedFD fdParams, + libcamera.ControlInfoMap sensorCtrlInfoMap) + => (int32 ret); + start() => (int32 ret); + stop(); + configure(libcamera.ControlInfoMap sensorCtrlInfoMap) + => (int32 ret); + + [async] processStats(libcamera.ControlList sensorControls); +}; + +interface IPASoftEventInterface { + setSensorControls(libcamera.ControlList sensorControls); + setIspParams(int32 dummy); +}; diff --git a/meson_options.txt b/meson_options.txt index 99dab96d..2644bef0 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -27,7 +27,7 @@ option('gstreamer', option('ipas', type : 'array', - choices : ['ipu3', 'rkisp1', 'rpi/vc4', 'vimc'], + choices : ['ipu3', 'rkisp1', 'rpi/vc4', 'simple', 'vimc'], description : 'Select which IPA modules to build') option('lc-compliance', diff --git a/src/ipa/simple/data/meson.build b/src/ipa/simple/data/meson.build new file mode 100644 index 00000000..33548cc6 --- /dev/null +++ b/src/ipa/simple/data/meson.build @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: CC0-1.0 + +conf_files = files([ + 'soft.conf', +]) + +install_data(conf_files, + install_dir : ipa_data_dir / 'soft', + install_tag : 'runtime') diff --git a/src/ipa/simple/data/soft.conf b/src/ipa/simple/data/soft.conf new file mode 100644 index 00000000..0c70e7c0 --- /dev/null +++ b/src/ipa/simple/data/soft.conf @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: LGPL-2.1-or-later +# +# Dummy configuration file for the soft IPA. diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build new file mode 100644 index 00000000..3e863db7 --- /dev/null +++ b/src/ipa/simple/meson.build @@ -0,0 +1,25 @@ +# SPDX-License-Identifier: CC0-1.0 + +ipa_name = 'ipa_soft_simple' + +mod = shared_module(ipa_name, + ['soft_simple.cpp', libcamera_generated_ipa_headers], + name_prefix : '', + include_directories : [ipa_includes, libipa_includes], + dependencies : libcamera_private, + link_with : libipa, + install : true, + install_dir : ipa_install_dir) + +if ipa_sign_module + custom_target(ipa_name + '.so.sign', + input : mod, + output : ipa_name + '.so.sign', + command : [ipa_sign, ipa_priv_key, '@INPUT@', '@OUTPUT@'], + install : false, + build_by_default : true) +endif + +subdir('data') + +ipa_names += ipa_name diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp new file mode 100644 index 00000000..312df4ba --- /dev/null +++ b/src/ipa/simple/soft_simple.cpp @@ -0,0 +1,326 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * + * soft_simple.cpp - Simple Software Image Processing Algorithm module + */ + +#include + +#include +#include +#include + +#include +#include + +#include +#include +#include + +#include "libcamera/internal/camera_sensor.h" +#include "libcamera/internal/software_isp/debayer_params.h" +#include "libcamera/internal/software_isp/swisp_stats.h" + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoft) + +namespace ipa::soft { + +class IPASoftSimple : public ipa::soft::IPASoftInterface +{ +public: + IPASoftSimple() + : params_(static_cast(MAP_FAILED)), + stats_(static_cast(MAP_FAILED)), ignore_updates_(0) + { + } + + ~IPASoftSimple() + { + if (stats_ != MAP_FAILED) + munmap(stats_, sizeof(SwIspStats)); + if (params_ != MAP_FAILED) + munmap(params_, sizeof(DebayerParams)); + } + + int init(const IPASettings &settings, + const SharedFD &fdStats, + const SharedFD &fdParams, + const ControlInfoMap &sensorInfoMap) override; + int configure(const ControlInfoMap &sensorInfoMap) override; + + int start() override; + void stop() override; + + void processStats(const ControlList &sensorControls) override; + +private: + void updateExposure(double exposureMSV); + + SharedFD fdStats_; + SharedFD fdParams_; + DebayerParams *params_; + SwIspStats *stats_; + + int32_t exposure_min_, exposure_max_; + int32_t again_min_, again_max_; + int32_t again_, exposure_; + unsigned int ignore_updates_; +}; + +int IPASoftSimple::init([[maybe_unused]] const IPASettings &settings, + const SharedFD &fdStats, + const SharedFD &fdParams, + const ControlInfoMap &sensorInfoMap) +{ + fdStats_ = fdStats; + if (!fdStats_.isValid()) { + LOG(IPASoft, Error) << "Invalid Statistics handle"; + return -ENODEV; + } + + fdParams_ = fdParams; + if (!fdParams_.isValid()) { + LOG(IPASoft, Error) << "Invalid Parameters handle"; + return -ENODEV; + } + + params_ = static_cast(mmap(nullptr, sizeof(DebayerParams), + PROT_WRITE, MAP_SHARED, + fdParams_.get(), 0)); + if (params_ == MAP_FAILED) { + LOG(IPASoft, Error) << "Unable to map Parameters"; + return -errno; + } + + stats_ = static_cast(mmap(nullptr, sizeof(SwIspStats), + PROT_READ, MAP_SHARED, + fdStats_.get(), 0)); + if (stats_ == MAP_FAILED) { + LOG(IPASoft, Error) << "Unable to map Statistics"; + return -errno; + } + + if (sensorInfoMap.find(V4L2_CID_EXPOSURE) == sensorInfoMap.end()) { + LOG(IPASoft, Error) << "Don't have exposure control"; + return -EINVAL; + } + + if (sensorInfoMap.find(V4L2_CID_ANALOGUE_GAIN) == sensorInfoMap.end()) { + LOG(IPASoft, Error) << "Don't have gain control"; + return -EINVAL; + } + + return 0; +} + +int IPASoftSimple::configure(const ControlInfoMap &sensorInfoMap) +{ + const ControlInfo &exposure_info = sensorInfoMap.find(V4L2_CID_EXPOSURE)->second; + const ControlInfo &gain_info = sensorInfoMap.find(V4L2_CID_ANALOGUE_GAIN)->second; + + exposure_min_ = exposure_info.min().get(); + exposure_max_ = exposure_info.max().get(); + if (!exposure_min_) { + LOG(IPASoft, Warning) << "Minimum exposure is zero, that can't be linear"; + exposure_min_ = 1; + } + + again_min_ = gain_info.min().get(); + again_max_ = gain_info.max().get(); + /* + * The camera sensor gain (g) is usually not equal to the value written + * into the gain register (x). But the way how the AGC algorithm changes + * the gain value to make the total exposure closer to the optimum assumes + * that g(x) is not too far from linear function. If the minimal gain is 0, + * the g(x) is likely to be far from the linear, like g(x) = a / (b * x + c). + * To avoid unexpected changes to the gain by the AGC algorithm (abrupt near + * one edge, and very small near the other) we limit the range of the gain + * values used. + */ + if (!again_min_) { + LOG(IPASoft, Warning) << "Minimum gain is zero, that can't be linear"; + again_min_ = std::min(100, again_min_ / 2 + again_max_ / 2); + } + + LOG(IPASoft, Info) << "Exposure " << exposure_min_ << "-" << exposure_max_ + << ", gain " << again_min_ << "-" << again_max_; + + return 0; +} + +int IPASoftSimple::start() +{ + return 0; +} + +void IPASoftSimple::stop() +{ +} + +/* + * The number of bins to use for the optimal exposure calculations. + */ +static constexpr unsigned int kExposureBinsCount = 5; +/* + * The exposure is optimal when the mean sample value of the histogram is + * in the middle of the range. + */ +static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; +/* + * The below value implements the hysteresis for the exposure adjustment. + * It is small enough to have the exposure close to the optimal, and is big + * enough to prevent the exposure from wobbling around the optimal value. + */ +static constexpr float kExposureSatisfactory = 0.2; + +void IPASoftSimple::processStats(const ControlList &sensorControls) +{ + /* + * Calculate red and blue gains for AWB. + * Clamp max gain at 4.0, this also avoids 0 division. + */ + if (stats_->sumR_ <= stats_->sumG_ / 4) + params_->gainR = 1024; + else + params_->gainR = 256 * stats_->sumG_ / stats_->sumR_; + + if (stats_->sumB_ <= stats_->sumG_ / 4) + params_->gainB = 1024; + else + params_->gainB = 256 * stats_->sumG_ / stats_->sumB_; + + /* Green gain and gamma values are fixed */ + params_->gainG = 256; + params_->gamma = 0.5; + + setIspParams.emit(0); + + /* + * AE / AGC, use 2 frames delay to make sure that the exposure and + * the gain set have applied to the camera sensor. + */ + if (ignore_updates_ > 0) { + --ignore_updates_; + return; + } + + /* + * Calculate Mean Sample Value (MSV) according to formula from: + * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf + */ + constexpr unsigned int yHistValsPerBin = + SwIspStats::kYHistogramSize / kExposureBinsCount; + constexpr unsigned int yHistValsPerBinMod = + SwIspStats::kYHistogramSize / + (SwIspStats::kYHistogramSize % kExposureBinsCount + 1); + int ExposureBins[kExposureBinsCount] = {}; + unsigned int denom = 0; + unsigned int num = 0; + + for (unsigned int i = 0; i < SwIspStats::kYHistogramSize; i++) { + unsigned int idx = (i - (i / yHistValsPerBinMod)) / yHistValsPerBin; + ExposureBins[idx] += stats_->yHistogram[i]; + } + + for (unsigned int i = 0; i < kExposureBinsCount; i++) { + LOG(IPASoft, Debug) << i << ": " << ExposureBins[i]; + denom += ExposureBins[i]; + num += ExposureBins[i] * (i + 1); + } + + float exposureMSV = (float)num / denom; + + /* sanity check */ + if (!sensorControls.contains(V4L2_CID_EXPOSURE) || + !sensorControls.contains(V4L2_CID_ANALOGUE_GAIN)) { + LOG(IPASoft, Error) << "Control(s) missing"; + return; + } + + ControlList ctrls(sensorControls); + + exposure_ = ctrls.get(V4L2_CID_EXPOSURE).get(); + again_ = ctrls.get(V4L2_CID_ANALOGUE_GAIN).get(); + + updateExposure(exposureMSV); + + ctrls.set(V4L2_CID_EXPOSURE, exposure_); + ctrls.set(V4L2_CID_ANALOGUE_GAIN, again_); + + ignore_updates_ = 2; + + setSensorControls.emit(ctrls); + + LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV + << " exp " << exposure_ << " again " << again_ + << " gain R/B " << params_->gainR << "/" << params_->gainB; +} + +void IPASoftSimple::updateExposure(double exposureMSV) +{ + /* DENOMINATOR of 10 gives ~10% increment/decrement; DENOMINATOR of 5 - about ~20% */ + static constexpr uint8_t kExpDenominator = 10; + static constexpr uint8_t kExpNumeratorUp = kExpDenominator + 1; + static constexpr uint8_t kExpNumeratorDown = kExpDenominator - 1; + + int next; + + if (exposureMSV < kExposureOptimal - kExposureSatisfactory) { + next = exposure_ * kExpNumeratorUp / kExpDenominator; + if (next - exposure_ < 1) + exposure_ += 1; + else + exposure_ = next; + if (exposure_ >= exposure_max_) { + next = again_ * kExpNumeratorUp / kExpDenominator; + if (next - again_ < 1) + again_ += 1; + else + again_ = next; + } + } + + if (exposureMSV > kExposureOptimal + kExposureSatisfactory) { + if (exposure_ == exposure_max_ && again_ != again_min_) { + next = again_ * kExpNumeratorDown / kExpDenominator; + if (again_ - next < 1) + again_ -= 1; + else + again_ = next; + } else { + next = exposure_ * kExpNumeratorDown / kExpDenominator; + if (exposure_ - next < 1) + exposure_ -= 1; + else + exposure_ = next; + } + } + + exposure_ = std::clamp(exposure_, exposure_min_, exposure_max_); + again_ = std::clamp(again_, again_min_, again_max_); +} + +} /* namespace ipa::soft */ + +/* + * External IPA module interface + */ +extern "C" { +const struct IPAModuleInfo ipaModuleInfo = { + IPA_MODULE_API_VERSION, + 0, + "SimplePipelineHandler", + "simple", +}; + +IPAInterface *ipaCreate() +{ + return new ipa::soft::IPASoftSimple(); +} + +} /* extern "C" */ + +} /* namespace libcamera */ From patchwork Tue Mar 19 12:35:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19767 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 65B30C3274 for ; Tue, 19 Mar 2024 12:37:28 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2106862E75; Tue, 19 Mar 2024 13:37:28 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="ffxr4f9W"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 635B762D37 for ; Tue, 19 Mar 2024 13:37:26 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851845; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fN9PXOvA1XsUeqFt/5W1yLw5Whr0wX5jrpi5n4DaR5g=; b=ffxr4f9WNDxbN1XBvopx2X+vlvXilSIneffZ4C3GtVDyomOGDMH0M+e+3f5Fn9axXTDLQd IL9XenKZWfwgVG56CAg5MUIRyVPNm5MSOzBcc9UnWCiGsGP6r/lwFQAkq0IIBCOia54+u1 vz3MQBajhkKxWpeSDzVgLY/Cwv6EG78= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-453-PH7q5hTdPT2HdU8IGeOzBg-1; Tue, 19 Mar 2024 08:37:21 -0400 X-MC-Unique: PH7q5hTdPT2HdU8IGeOzBg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4593F3C40B50; Tue, 19 Mar 2024 12:37:20 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id DE6981C060A6; Tue, 19 Mar 2024 12:37:17 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Andrey Konovalov , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Hans de Goede , Kieran Bingham , Dennis Bonke Subject: [PATCH v6 10/18] libcamera: introduce SoftwareIsp Date: Tue, 19 Mar 2024 13:35:57 +0100 Message-ID: <20240319123622.675599-11-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov Doxygen documentation by Dennis Bonke. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Co-developed-by: Dennis Bonke Signed-off-by: Dennis Bonke Signed-off-by: Andrey Konovalov Signed-off-by: Hans de Goede --- .../internal/software_isp/meson.build | 1 + .../internal/software_isp/software_isp.h | 98 +++++ src/libcamera/software_isp/meson.build | 1 + src/libcamera/software_isp/software_isp.cpp | 349 ++++++++++++++++++ 4 files changed, 449 insertions(+) create mode 100644 include/libcamera/internal/software_isp/software_isp.h create mode 100644 src/libcamera/software_isp/software_isp.cpp diff --git a/include/libcamera/internal/software_isp/meson.build b/include/libcamera/internal/software_isp/meson.build index a620e16d..508ddddc 100644 --- a/include/libcamera/internal/software_isp/meson.build +++ b/include/libcamera/internal/software_isp/meson.build @@ -2,5 +2,6 @@ libcamera_internal_headers += files([ 'debayer_params.h', + 'software_isp.h', 'swisp_stats.h', ]) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h new file mode 100644 index 00000000..8d25e979 --- /dev/null +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -0,0 +1,98 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * + * software_isp.h - Simple software ISP implementation + */ + +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include +#include + +#include +#include + +#include "libcamera/internal/dma_heaps.h" +#include "libcamera/internal/pipeline_handler.h" +#include "libcamera/internal/shared_mem_object.h" +#include "libcamera/internal/software_isp/debayer_params.h" + +namespace libcamera { + +class DebayerCpu; +class FrameBuffer; +class PixelFormat; +struct StreamConfiguration; + +LOG_DECLARE_CATEGORY(SoftwareIsp) + +class SoftwareIsp +{ +public: + SoftwareIsp(PipelineHandler *pipe, const ControlInfoMap &sensorControls); + ~SoftwareIsp(); + + int loadConfiguration([[maybe_unused]] const std::string &filename) { return 0; } + + bool isValid() const; + + std::vector formats(PixelFormat input); + + SizeRange sizes(PixelFormat inputFormat, const Size &inputSize); + + std::tuple + strideAndFrameSize(const PixelFormat &outputFormat, const Size &size); + + int configure(const StreamConfiguration &inputCfg, + const std::vector> &outputCfgs, + const ControlInfoMap &sensorControls); + + int exportBuffers(unsigned int output, unsigned int count, + std::vector> *buffers); + + void processStats(const ControlList &sensorControls); + + int start(); + void stop(); + + int queueBuffers(FrameBuffer *input, + const std::map &outputs); + + void process(FrameBuffer *input, FrameBuffer *output); + + Signal inputBufferReady; + Signal outputBufferReady; + Signal ispStatsReady; + Signal setSensorControls; + +private: + void saveIspParams(int dummy); + void setSensorCtrls(const ControlList &sensorControls); + void statsReady(int dummy); + void inputReady(FrameBuffer *input); + void outputReady(FrameBuffer *output); + + std::unique_ptr debayer_; + Thread ispWorkerThread_; + SharedMemObject sharedParams_; + DebayerParams debayerParams_; + DmaHeap dmaHeap_; + + std::unique_ptr ipa_; +}; + +} /* namespace libcamera */ diff --git a/src/libcamera/software_isp/meson.build b/src/libcamera/software_isp/meson.build index 71b46539..e9266e54 100644 --- a/src/libcamera/software_isp/meson.build +++ b/src/libcamera/software_isp/meson.build @@ -10,5 +10,6 @@ endif libcamera_sources += files([ 'debayer.cpp', 'debayer_cpu.cpp', + 'software_isp.cpp', 'swstats_cpu.cpp', ]) diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp new file mode 100644 index 00000000..388b4496 --- /dev/null +++ b/src/libcamera/software_isp/software_isp.cpp @@ -0,0 +1,349 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * + * software_isp.cpp - Simple software ISP implementation + */ + +#include "libcamera/internal/software_isp/software_isp.h" + +#include +#include +#include + +#include +#include + +#include "libcamera/internal/bayer_format.h" +#include "libcamera/internal/framebuffer.h" +#include "libcamera/internal/ipa_manager.h" +#include "libcamera/internal/mapped_framebuffer.h" + +#include "debayer_cpu.h" + +/** + * \file software_isp.cpp + * \brief Simple software ISP implementation + */ + +namespace libcamera { + +LOG_DEFINE_CATEGORY(SoftwareIsp) + +/** + * \class SoftwareIsp + * \brief Class for the Software ISP + */ + +/** + * \var SoftwareIsp::inputBufferReady + * \brief A signal emitted when the input frame buffer completes + */ + +/** + * \var SoftwareIsp::outputBufferReady + * \brief A signal emitted when the output frame buffer completes + */ + +/** + * \var SoftwareIsp::ispStatsReady + * \brief A signal emitted when the statistics for IPA are ready + * + * The int parameter isn't actually used. + */ + +/** + * \var SoftwareIsp::setSensorControls + * \brief A signal emitted when the values to write to the sensor controls are ready + */ + +/** + * \brief Constructs SoftwareIsp object + * \param[in] pipe The pipeline handler in use + * \param[in] sensorControls ControlInfoMap describing the controls supported by the sensor + */ +SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const ControlInfoMap &sensorControls) + : debayer_(nullptr), + debayerParams_{ DebayerParams::kGain10, DebayerParams::kGain10, DebayerParams::kGain10, 0.5f }, + dmaHeap_(DmaHeap::DmaHeapFlag::Cma | DmaHeap::DmaHeapFlag::System) +{ + if (!dmaHeap_.isValid()) { + LOG(SoftwareIsp, Error) << "Failed to create DmaHeap object"; + return; + } + + sharedParams_ = SharedMemObject("softIsp_params"); + if (!sharedParams_) { + LOG(SoftwareIsp, Error) << "Failed to create shared memory for parameters"; + return; + } + + auto stats = std::make_unique(); + if (!stats->isValid()) { + LOG(SoftwareIsp, Error) << "Failed to create SwStatsCpu object"; + return; + } + stats->statsReady.connect(this, &SoftwareIsp::statsReady); + + debayer_ = std::make_unique(std::move(stats)); + debayer_->inputBufferReady.connect(this, &SoftwareIsp::inputReady); + debayer_->outputBufferReady.connect(this, &SoftwareIsp::outputReady); + + ipa_ = IPAManager::createIPA(pipe, 0, 0); + if (!ipa_) { + LOG(SoftwareIsp, Error) + << "Creating IPA for software ISP failed"; + debayer_.reset(); + return; + } + + int ret = ipa_->init(IPASettings{ "No cfg file", "No sensor model" }, + debayer_->getStatsFD(), + sharedParams_.fd(), + sensorControls); + if (ret) { + LOG(SoftwareIsp, Error) << "IPA init failed"; + debayer_.reset(); + return; + } + + ipa_->setIspParams.connect(this, &SoftwareIsp::saveIspParams); + ipa_->setSensorControls.connect(this, &SoftwareIsp::setSensorCtrls); + + debayer_->moveToThread(&ispWorkerThread_); +} + +SoftwareIsp::~SoftwareIsp() +{ + /* make sure to destroy the DebayerCpu before the ispWorkerThread_ is gone */ + debayer_.reset(); +} + +/** + * \fn int SoftwareIsp::loadConfiguration([[maybe_unused]] const std::string &filename) + * \brief Load a configuration from a file + * \param[in] filename The file to load the configuration data from + * + * Currently is a stub doing nothing and always returning "success". + * + * \return 0 on success + */ + +/** + * \brief Process the statistics gathered + * \param[in] sensorControls The sensor controls + * + * Requests the IPA to calculate new parameters for ISP and new control + * values for the sensor. + */ +void SoftwareIsp::processStats(const ControlList &sensorControls) +{ + ASSERT(ipa_); + ipa_->processStats(sensorControls); +} + +/** + * \brief Check the validity of Software Isp object + * \return True if Software Isp is valid, false otherwise + */ +bool SoftwareIsp::isValid() const +{ + return !!debayer_; +} + +/** + * \brief Get the output formats supported for the given input format + * \param[in] inputFormat The input format + * \return All the supported output formats or an empty vector if there are none + */ +std::vector SoftwareIsp::formats(PixelFormat inputFormat) +{ + ASSERT(debayer_ != nullptr); + + return debayer_->formats(inputFormat); +} + +/** + * \brief Get the supported output sizes for the given input format and size + * \param[in] inputFormat The input format + * \param[in] inputSize The input frame size + * \return The valid size range or an empty range if there are none + */ +SizeRange SoftwareIsp::sizes(PixelFormat inputFormat, const Size &inputSize) +{ + ASSERT(debayer_ != nullptr); + + return debayer_->sizes(inputFormat, inputSize); +} + +/** + * Get the output stride and the frame size in bytes for the given output format and size + * \param[in] outputFormat The output format + * \param[in] size The output size (width and height in pixels) + * \return A tuple of the stride and the frame size in bytes, or a tuple of 0,0 + * if there is no valid output config + */ +std::tuple +SoftwareIsp::strideAndFrameSize(const PixelFormat &outputFormat, const Size &size) +{ + ASSERT(debayer_ != nullptr); + + return debayer_->strideAndFrameSize(outputFormat, size); +} + +/** + * \brief Configure the SoftwareIsp object according to the passed in parameters + * \param[in] inputCfg The input configuration + * \param[in] outputCfgs The output configurations + * \param[in] sensorControls ControlInfoMap of the controls supported by the sensor + * \return 0 on success, a negative errno on failure + */ +int SoftwareIsp::configure(const StreamConfiguration &inputCfg, + const std::vector> &outputCfgs, + const ControlInfoMap &sensorControls) +{ + ASSERT(ipa_ != nullptr && debayer_ != nullptr); + + int ret = ipa_->configure(sensorControls); + if (ret < 0) + return ret; + + return debayer_->configure(inputCfg, outputCfgs); +} + +/** + * \brief Export the buffers from the Software ISP + * \param[in] output Output stream index exporting the buffers + * \param[in] count Number of buffers to allocate + * \param[out] buffers Vector to store the allocated buffers + * \return The number of allocated buffers on success or a negative error code + * otherwise + */ +int SoftwareIsp::exportBuffers(unsigned int output, unsigned int count, + std::vector> *buffers) +{ + ASSERT(debayer_ != nullptr); + + /* single output for now */ + if (output >= 1) + return -EINVAL; + + for (unsigned int i = 0; i < count; i++) { + const std::string name = "frame-" + std::to_string(i); + const size_t frameSize = debayer_->frameSize(); + + FrameBuffer::Plane outPlane; + outPlane.fd = SharedFD(dmaHeap_.alloc(name.c_str(), frameSize)); + if (!outPlane.fd.isValid()) { + LOG(SoftwareIsp, Error) + << "failed to allocate a dma_buf"; + return -ENOMEM; + } + outPlane.offset = 0; + outPlane.length = frameSize; + + std::vector planes{ outPlane }; + buffers->emplace_back(std::make_unique(std::move(planes))); + } + + return count; +} + +/** + * \brief Queue buffers to Software ISP + * \param[in] input The input framebuffer + * \param[in] outputs The container holding the output stream indexes and + * their respective frame buffer outputs + * \return 0 on success, a negative errno on failure + */ +int SoftwareIsp::queueBuffers(FrameBuffer *input, + const std::map &outputs) +{ + unsigned int mask = 0; + + /* + * Validate the outputs as a sanity check: at least one output is + * required, all outputs must reference a valid stream and no two + * outputs can reference the same stream. + */ + if (outputs.empty()) + return -EINVAL; + + for (auto [index, buffer] : outputs) { + if (!buffer) + return -EINVAL; + if (index >= 1) /* only single stream atm */ + return -EINVAL; + if (mask & (1 << index)) + return -EINVAL; + + mask |= 1 << index; + } + + process(input, outputs.at(0)); + + return 0; +} + +/** + * \brief Starts the Software ISP streaming operation + * \return 0 on success, any other value indicates an error + */ +int SoftwareIsp::start() +{ + int ret = ipa_->start(); + if (ret) + return ret; + + ispWorkerThread_.start(); + return 0; +} + +/** + * \brief Stops the Software ISP streaming operation + */ +void SoftwareIsp::stop() +{ + ispWorkerThread_.exit(); + ispWorkerThread_.wait(); + + ipa_->stop(); +} + +/** + * \brief Passes the input framebuffer to the ISP worker to process + * \param[in] input The input framebuffer + * \param[out] output The framebuffer to write the processed frame to + */ +void SoftwareIsp::process(FrameBuffer *input, FrameBuffer *output) +{ + debayer_->invokeMethod(&DebayerCpu::process, + ConnectionTypeQueued, input, output, debayerParams_); +} + +void SoftwareIsp::saveIspParams([[maybe_unused]] int dummy) +{ + debayerParams_ = *sharedParams_; +} + +void SoftwareIsp::setSensorCtrls(const ControlList &sensorControls) +{ + setSensorControls.emit(sensorControls); +} + +void SoftwareIsp::statsReady(int dummy) +{ + ispStatsReady.emit(dummy); +} + +void SoftwareIsp::inputReady(FrameBuffer *input) +{ + inputBufferReady.emit(input); +} + +void SoftwareIsp::outputReady(FrameBuffer *output) +{ + outputBufferReady.emit(output); +} + +} /* namespace libcamera */ From patchwork Tue Mar 19 12:35:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19769 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C2E7EC3274 for ; Tue, 19 Mar 2024 12:37:31 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 6C82E62D3A; Tue, 19 Mar 2024 13:37:31 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="ikHO5bqN"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 271D662D2E for ; Tue, 19 Mar 2024 13:37:29 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851848; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fph8l4Wu873rDNe4y0+qgCozyBIHvJYKKfjLo+wu2S4=; b=ikHO5bqNzvR4/xbSH+OSw4tv65Di+f8yHFng+AnEBKCkR9qHjFllw0+Qw9hBOiCadUviGH R1ny19Av9FNDa/way4ye65HllnXQZjasJ9Y7wkfeM/htGVxoEOautnDIjGwVYD27sx+y2y z1zGphVmpsIHF7G20GpNcLblVJKFhgU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-346-ecd5yOGmOf2sYYQUKFa6zg-1; Tue, 19 Mar 2024 08:37:23 -0400 X-MC-Unique: ecd5yOGmOf2sYYQUKFa6zg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9CCE6101A552; Tue, 19 Mar 2024 12:37:22 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id A4FA31C060A4; Tue, 19 Mar 2024 12:37:20 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Andrey Konovalov , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Hans de Goede , Kieran Bingham Subject: [PATCH v6 11/18] libcamera: pipeline: simple: rename converterBuffers_ and related vars Date: Tue, 19 Mar 2024 13:35:58 +0100 Message-ID: <20240319123622.675599-12-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov The converterBuffers_ and the converterQueue_ are not that specific to the Converter, and could be used by another entity doing the format conversion. Rename converterBuffers_, converterQueue_, and useConverter_ to conversionBuffers_, conversionQueue_ and useConversion_ to disassociate them from the Converter. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Kieran Bingham Reviewed-by: Pavel Machek Signed-off-by: Andrey Konovalov Signed-off-by: Hans de Goede Reviewed-by: Laurent Pinchart --- src/libcamera/pipeline/simple/simple.cpp | 63 ++++++++++++------------ 1 file changed, 32 insertions(+), 31 deletions(-) diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 04e77f7e..9c8f7ba3 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -270,17 +270,18 @@ public: std::vector configs_; std::map> formats_; + std::vector> conversionBuffers_; + std::queue> conversionQueue_; + bool useConversion_; + std::unique_ptr converter_; - std::vector> converterBuffers_; - bool useConverter_; - std::queue> converterQueue_; private: void tryPipeline(unsigned int code, const Size &size); static std::vector routedSourcePads(MediaPad *sink); - void converterInputDone(FrameBuffer *buffer); - void converterOutputDone(FrameBuffer *buffer); + void conversionInputDone(FrameBuffer *buffer); + void conversionOutputDone(FrameBuffer *buffer); }; class SimpleCameraConfiguration : public CameraConfiguration @@ -504,8 +505,8 @@ int SimpleCameraData::init() << "Failed to create converter, disabling format conversion"; converter_.reset(); } else { - converter_->inputBufferReady.connect(this, &SimpleCameraData::converterInputDone); - converter_->outputBufferReady.connect(this, &SimpleCameraData::converterOutputDone); + converter_->inputBufferReady.connect(this, &SimpleCameraData::conversionInputDone); + converter_->outputBufferReady.connect(this, &SimpleCameraData::conversionOutputDone); } } @@ -741,7 +742,7 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) * point converting an erroneous buffer. */ if (buffer->metadata().status != FrameMetadata::FrameSuccess) { - if (!useConverter_) { + if (!useConversion_) { /* No conversion, just complete the request. */ Request *request = buffer->request(); pipe->completeBuffer(request, buffer); @@ -757,16 +758,16 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) if (buffer->metadata().status != FrameMetadata::FrameCancelled) video_->queueBuffer(buffer); - if (converterQueue_.empty()) + if (conversionQueue_.empty()) return; Request *request = nullptr; - for (auto &item : converterQueue_.front()) { + for (auto &item : conversionQueue_.front()) { FrameBuffer *outputBuffer = item.second; request = outputBuffer->request(); pipe->completeBuffer(request, outputBuffer); } - converterQueue_.pop(); + conversionQueue_.pop(); if (request) pipe->completeRequest(request); @@ -783,9 +784,9 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) */ Request *request = buffer->request(); - if (useConverter_ && !converterQueue_.empty()) { + if (useConversion_ && !conversionQueue_.empty()) { const std::map &outputs = - converterQueue_.front(); + conversionQueue_.front(); if (!outputs.empty()) { FrameBuffer *outputBuffer = outputs.begin()->second; if (outputBuffer) @@ -802,14 +803,14 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) * conversion is needed. If there's no queued request, just requeue the * captured buffer for capture. */ - if (useConverter_) { - if (converterQueue_.empty()) { + if (useConversion_) { + if (conversionQueue_.empty()) { video_->queueBuffer(buffer); return; } - converter_->queueBuffers(buffer, converterQueue_.front()); - converterQueue_.pop(); + converter_->queueBuffers(buffer, conversionQueue_.front()); + conversionQueue_.pop(); return; } @@ -818,13 +819,13 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) pipe->completeRequest(request); } -void SimpleCameraData::converterInputDone(FrameBuffer *buffer) +void SimpleCameraData::conversionInputDone(FrameBuffer *buffer) { /* Queue the input buffer back for capture. */ video_->queueBuffer(buffer); } -void SimpleCameraData::converterOutputDone(FrameBuffer *buffer) +void SimpleCameraData::conversionOutputDone(FrameBuffer *buffer) { SimplePipelineHandler *pipe = SimpleCameraData::pipe(); @@ -1190,14 +1191,14 @@ int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c) /* Configure the converter if needed. */ std::vector> outputCfgs; - data->useConverter_ = config->needConversion(); + data->useConversion_ = config->needConversion(); for (unsigned int i = 0; i < config->size(); ++i) { StreamConfiguration &cfg = config->at(i); cfg.setStream(&data->streams_[i]); - if (data->useConverter_) + if (data->useConversion_) outputCfgs.push_back(cfg); } @@ -1223,7 +1224,7 @@ int SimplePipelineHandler::exportFrameBuffers(Camera *camera, Stream *stream, * Export buffers on the converter or capture video node, depending on * whether the converter is used or not. */ - if (data->useConverter_) + if (data->useConversion_) return data->converter_->exportBuffers(data->streamIndex(stream), count, buffers); else @@ -1244,13 +1245,13 @@ int SimplePipelineHandler::start(Camera *camera, [[maybe_unused]] const ControlL return -EBUSY; } - if (data->useConverter_) { + if (data->useConversion_) { /* * When using the converter allocate a fixed number of internal * buffers. */ ret = video->allocateBuffers(kNumInternalBuffers, - &data->converterBuffers_); + &data->conversionBuffers_); } else { /* Otherwise, prepare for using buffers from the only stream. */ Stream *stream = &data->streams_[0]; @@ -1269,7 +1270,7 @@ int SimplePipelineHandler::start(Camera *camera, [[maybe_unused]] const ControlL return ret; } - if (data->useConverter_) { + if (data->useConversion_) { ret = data->converter_->start(); if (ret < 0) { stop(camera); @@ -1277,7 +1278,7 @@ int SimplePipelineHandler::start(Camera *camera, [[maybe_unused]] const ControlL } /* Queue all internal buffers for capture. */ - for (std::unique_ptr &buffer : data->converterBuffers_) + for (std::unique_ptr &buffer : data->conversionBuffers_) video->queueBuffer(buffer.get()); } @@ -1289,7 +1290,7 @@ void SimplePipelineHandler::stopDevice(Camera *camera) SimpleCameraData *data = cameraData(camera); V4L2VideoDevice *video = data->video_; - if (data->useConverter_) + if (data->useConversion_) data->converter_->stop(); video->streamOff(); @@ -1297,7 +1298,7 @@ void SimplePipelineHandler::stopDevice(Camera *camera) video->bufferReady.disconnect(data, &SimpleCameraData::bufferReady); - data->converterBuffers_.clear(); + data->conversionBuffers_.clear(); releasePipeline(data); } @@ -1315,7 +1316,7 @@ int SimplePipelineHandler::queueRequestDevice(Camera *camera, Request *request) * queue, it will be handed to the converter in the capture * completion handler. */ - if (data->useConverter_) { + if (data->useConversion_) { buffers.emplace(data->streamIndex(stream), buffer); } else { ret = data->video_->queueBuffer(buffer); @@ -1324,8 +1325,8 @@ int SimplePipelineHandler::queueRequestDevice(Camera *camera, Request *request) } } - if (data->useConverter_) - data->converterQueue_.push(std::move(buffers)); + if (data->useConversion_) + data->conversionQueue_.push(std::move(buffers)); return 0; } From patchwork Tue Mar 19 12:35:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19768 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 119D3C3272 for ; Tue, 19 Mar 2024 12:37:31 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BA49362D37; Tue, 19 Mar 2024 13:37:30 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="fpYXzsIY"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2942562EAE for ; Tue, 19 Mar 2024 13:37:28 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851847; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pJpT1F4oQ7gc1ZpLO/VZUkXB70Pd8FArwesXC4kGy8s=; b=fpYXzsIYfFbbU5MBpVpkEHRwoiE1OcnnLe7PmxPAyA+5oMG4t/ryTkeDbTFC6WyzPY6w4+ 4zge8SVbrmQkkte2pLreVWut6PrIemfNPtCI8IhkayKSQHOJvREysH/Oo1/8W5obU6p3u7 A7a3old65jYYq7kyDOMvm3Ucwn69eWs= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-447-zE4OPp-NOUqDAO72AgLW3Q-1; Tue, 19 Mar 2024 08:37:25 -0400 X-MC-Unique: zE4OPp-NOUqDAO72AgLW3Q-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6549128C97C9; Tue, 19 Mar 2024 12:37:25 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id 07A981C060CE; Tue, 19 Mar 2024 12:37:22 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Andrey Konovalov , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Hans de Goede , Kieran Bingham Subject: [PATCH v6 12/18] libcamera: pipeline: simple: enable use of Soft ISP and Soft IPA Date: Tue, 19 Mar 2024 13:35:59 +0100 Message-ID: <20240319123622.675599-13-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov To enable the Simple Soft ISP and Soft IPA for simple pipeline handler configure the build with: -Dpipelines=simple -Dipas=simple Also using the Soft ISP for the particular hardware platform must be enabled in the supportedDevices[] table. If the pipeline uses Converter, Soft ISP and Soft IPA aren't available. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Signed-off-by: Andrey Konovalov Signed-off-by: Hans de Goede --- src/libcamera/pipeline/simple/simple.cpp | 138 ++++++++++++++++++----- 1 file changed, 109 insertions(+), 29 deletions(-) diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 9c8f7ba3..ac796b9b 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -34,6 +34,7 @@ #include "libcamera/internal/device_enumerator.h" #include "libcamera/internal/media_device.h" #include "libcamera/internal/pipeline_handler.h" +#include "libcamera/internal/software_isp/software_isp.h" #include "libcamera/internal/v4l2_subdevice.h" #include "libcamera/internal/v4l2_videodevice.h" @@ -185,18 +186,23 @@ struct SimplePipelineInfo { * and the number of streams it supports. */ std::vector> converters; + /* + * Using Software ISP is to be enabled per driver. + * The Software ISP can't be used together with the converters. + */ + bool swIspEnabled; }; namespace { static const SimplePipelineInfo supportedDevices[] = { - { "dcmipp", {} }, - { "imx7-csi", { { "pxp", 1 } } }, - { "j721e-csi2rx", {} }, - { "mtk-seninf", { { "mtk-mdp", 3 } } }, - { "mxc-isi", {} }, - { "qcom-camss", {} }, - { "sun6i-csi", {} }, + { "dcmipp", {}, false }, + { "imx7-csi", { { "pxp", 1 } }, false }, + { "j721e-csi2rx", {}, false }, + { "mtk-seninf", { { "mtk-mdp", 3 } }, false }, + { "mxc-isi", {}, false }, + { "qcom-camss", {}, true }, + { "sun6i-csi", {}, false }, }; } /* namespace */ @@ -275,6 +281,7 @@ public: bool useConversion_; std::unique_ptr converter_; + std::unique_ptr swIsp_; private: void tryPipeline(unsigned int code, const Size &size); @@ -282,6 +289,9 @@ private: void conversionInputDone(FrameBuffer *buffer); void conversionOutputDone(FrameBuffer *buffer); + + void ispStatsReady(int dummy); + void setSensorControls(const ControlList &sensorControls); }; class SimpleCameraConfiguration : public CameraConfiguration @@ -333,6 +343,7 @@ public: V4L2VideoDevice *video(const MediaEntity *entity); V4L2Subdevice *subdev(const MediaEntity *entity); MediaDevice *converter() { return converter_; } + bool swIspEnabled() { return swIspEnabled_; } protected: int queueRequestDevice(Camera *camera, Request *request) override; @@ -361,6 +372,7 @@ private: std::map entities_; MediaDevice *converter_; + bool swIspEnabled_; }; /* ----------------------------------------------------------------------------- @@ -510,6 +522,29 @@ int SimpleCameraData::init() } } + /* + * Instantiate Soft ISP if this is enabled for the given driver and no converter is used. + */ + if (!converter_ && pipe->swIspEnabled()) { + swIsp_ = std::make_unique(pipe, sensor_->controls()); + if (!swIsp_->isValid()) { + LOG(SimplePipeline, Warning) + << "Failed to create software ISP, disabling software debayering"; + swIsp_.reset(); + } else { + /* + * \todo explain why SimpleCameraData::conversionInputDone() can't be directly + * connected to inputBufferReady signal. + */ + swIsp_->inputBufferReady.connect(pipe, [this](FrameBuffer *buffer) { + this->conversionInputDone(buffer); + }); + swIsp_->outputBufferReady.connect(this, &SimpleCameraData::conversionOutputDone); + swIsp_->ispStatsReady.connect(this, &SimpleCameraData::ispStatsReady); + swIsp_->setSensorControls.connect(this, &SimpleCameraData::setSensorControls); + } + } + video_ = pipe->video(entities_.back().entity); ASSERT(video_); @@ -600,12 +635,20 @@ void SimpleCameraData::tryPipeline(unsigned int code, const Size &size) config.captureFormat = pixelFormat; config.captureSize = format.size; - if (!converter_) { - config.outputFormats = { pixelFormat }; - config.outputSizes = config.captureSize; - } else { + if (converter_) { config.outputFormats = converter_->formats(pixelFormat); config.outputSizes = converter_->sizes(format.size); + } else if (swIsp_) { + config.outputFormats = swIsp_->formats(pixelFormat); + config.outputSizes = swIsp_->sizes(pixelFormat, format.size); + if (config.outputFormats.empty()) { + /* Do not use swIsp for unsupported pixelFormat's: */ + config.outputFormats = { pixelFormat }; + config.outputSizes = config.captureSize; + } + } else { + config.outputFormats = { pixelFormat }; + config.outputSizes = config.captureSize; } configs_.push_back(config); @@ -751,9 +794,9 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) } /* - * The converter is in use. Requeue the internal buffer for - * capture (unless the stream is being stopped), and complete - * the request with all the user-facing buffers. + * The converter or Software ISP is in use. Requeue the internal + * buffer for capture (unless the stream is being stopped), and + * complete the request with all the user-facing buffers. */ if (buffer->metadata().status != FrameMetadata::FrameCancelled) video_->queueBuffer(buffer); @@ -799,9 +842,9 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) buffer->metadata().timestamp); /* - * Queue the captured and the request buffer to the converter if format - * conversion is needed. If there's no queued request, just requeue the - * captured buffer for capture. + * Queue the captured and the request buffer to the converter or Software + * ISP if format conversion is needed. If there's no queued request, just + * requeue the captured buffer for capture. */ if (useConversion_) { if (conversionQueue_.empty()) { @@ -809,7 +852,11 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) return; } - converter_->queueBuffers(buffer, conversionQueue_.front()); + if (converter_) + converter_->queueBuffers(buffer, conversionQueue_.front()); + else + swIsp_->queueBuffers(buffer, conversionQueue_.front()); + conversionQueue_.pop(); return; } @@ -835,6 +882,18 @@ void SimpleCameraData::conversionOutputDone(FrameBuffer *buffer) pipe->completeRequest(request); } +void SimpleCameraData::ispStatsReady([[maybe_unused]] int dummy) +{ + swIsp_->processStats(sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, + V4L2_CID_EXPOSURE })); +} + +void SimpleCameraData::setSensorControls(const ControlList &sensorControls) +{ + ControlList ctrls(sensorControls); + sensor_->setControls(&ctrls); +} + /* Retrieve all source pads connected to a sink pad through active routes. */ std::vector SimpleCameraData::routedSourcePads(MediaPad *sink) { @@ -1047,8 +1106,10 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() /* Set the stride, frameSize and bufferCount. */ if (needConversion_) { std::tie(cfg.stride, cfg.frameSize) = - data_->converter_->strideAndFrameSize(cfg.pixelFormat, - cfg.size); + (data_->converter_) ? data_->converter_->strideAndFrameSize(cfg.pixelFormat, + cfg.size) + : data_->swIsp_->strideAndFrameSize(cfg.pixelFormat, + cfg.size); if (cfg.stride == 0) return Invalid; } else { @@ -1211,7 +1272,9 @@ int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c) inputCfg.stride = captureFormat.planes[0].bpl; inputCfg.bufferCount = kNumInternalBuffers; - return data->converter_->configure(inputCfg, outputCfgs); + return (data->converter_) ? data->converter_->configure(inputCfg, outputCfgs) + : data->swIsp_->configure(inputCfg, outputCfgs, + data->sensor_->controls()); } int SimplePipelineHandler::exportFrameBuffers(Camera *camera, Stream *stream, @@ -1225,8 +1288,10 @@ int SimplePipelineHandler::exportFrameBuffers(Camera *camera, Stream *stream, * whether the converter is used or not. */ if (data->useConversion_) - return data->converter_->exportBuffers(data->streamIndex(stream), - count, buffers); + return (data->converter_) ? data->converter_->exportBuffers(data->streamIndex(stream), + count, buffers) + : data->swIsp_->exportBuffers(data->streamIndex(stream), + count, buffers); else return data->video_->exportBuffers(count, buffers); } @@ -1271,10 +1336,18 @@ int SimplePipelineHandler::start(Camera *camera, [[maybe_unused]] const ControlL } if (data->useConversion_) { - ret = data->converter_->start(); - if (ret < 0) { - stop(camera); - return ret; + if (data->converter_) { + ret = data->converter_->start(); + if (ret < 0) { + stop(camera); + return ret; + } + } else if (data->swIsp_) { + ret = data->swIsp_->start(); + if (ret < 0) { + stop(camera); + return ret; + } } /* Queue all internal buffers for capture. */ @@ -1290,8 +1363,13 @@ void SimplePipelineHandler::stopDevice(Camera *camera) SimpleCameraData *data = cameraData(camera); V4L2VideoDevice *video = data->video_; - if (data->useConversion_) - data->converter_->stop(); + if (data->useConversion_) { + if (data->converter_) + data->converter_->stop(); + else if (data->swIsp_) { + data->swIsp_->stop(); + } + } video->streamOff(); video->releaseBuffers(); @@ -1453,6 +1531,8 @@ bool SimplePipelineHandler::match(DeviceEnumerator *enumerator) } } + swIspEnabled_ = info->swIspEnabled; + /* Locate the sensors. */ std::vector sensors = locateSensors(); if (sensors.empty()) { From patchwork Tue Mar 19 12:36:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19770 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 75D36C3272 for ; Tue, 19 Mar 2024 12:37:35 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 188BA62D37; Tue, 19 Mar 2024 13:37:35 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="bagBq9Hl"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id EB6FB62D2E for ; Tue, 19 Mar 2024 13:37:32 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851851; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xpxRtnwE9qEZI2OeeWgRCKDmS6DLlgupCI1EjvDqThk=; b=bagBq9HlgfqrHnEQVKnhZywgGywsRKUJ8FzOE6O8JQgjw/5FXh84Ev+2kO0uaU5WLeGkhm msVfk30VD1GLGktGJfbTC/Z2oaFTfn4A0yKsygpj6srGAogXU+yOQzIx2X3M1fEk0jegC6 HB7e+OhuB2txX3n5MprOt4byPz8rqTc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-170-GVXZJ16cPDmcYtEr7igRnA-1; Tue, 19 Mar 2024 08:37:28 -0400 X-MC-Unique: GVXZJ16cPDmcYtEr7igRnA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 90E0480026B; Tue, 19 Mar 2024 12:37:27 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id DB4351C060A6; Tue, 19 Mar 2024 12:37:25 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Hans de Goede , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Kieran Bingham Subject: [PATCH v6 13/18] libcamera: swstats_cpu: Add support for 8, 10 and 12 bpp unpacked bayer input Date: Tue, 19 Mar 2024 13:36:00 +0100 Message-ID: <20240319123622.675599-14-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Hans de Goede Add support for 8, 10 and 12 bpp unpacked bayer input for all 4 standard bayer orders. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Reviewed-by: Milan Zamazal Signed-off-by: Kieran Bingham Signed-off-by: Hans de Goede --- src/libcamera/software_isp/swstats_cpu.cpp | 128 +++++++++++++++++++++ src/libcamera/software_isp/swstats_cpu.h | 9 ++ 2 files changed, 137 insertions(+) diff --git a/src/libcamera/software_isp/swstats_cpu.cpp b/src/libcamera/software_isp/swstats_cpu.cpp index 448d0e4c..be310f56 100644 --- a/src/libcamera/software_isp/swstats_cpu.cpp +++ b/src/libcamera/software_isp/swstats_cpu.cpp @@ -71,6 +71,83 @@ static const unsigned int kBlueYMul = 29; /* 0.114 * 256 */ stats_.sumG_ += sumG; \ stats_.sumB_ += sumB; +void SwStatsCpu::statsBGGR8Line0(const uint8_t *src[]) +{ + const uint8_t *src0 = src[1] + window_.x; + const uint8_t *src1 = src[2] + window_.x; + + SWSTATS_START_LINE_STATS(uint8_t) + + if (swapLines_) + std::swap(src0, src1); + + /* x += 4 sample every other 2x2 block */ + for (int x = 0; x < (int)window_.width; x += 4) { + b = src0[x]; + g = src0[x + 1]; + g2 = src1[x]; + r = src1[x + 1]; + + g = (g + g2) / 2; + + SWSTATS_ACCUMULATE_LINE_STATS(1) + } + + SWSTATS_FINISH_LINE_STATS() +} + +void SwStatsCpu::statsBGGR10Line0(const uint8_t *src[]) +{ + const uint16_t *src0 = (const uint16_t *)src[1] + window_.x; + const uint16_t *src1 = (const uint16_t *)src[2] + window_.x; + + SWSTATS_START_LINE_STATS(uint16_t) + + if (swapLines_) + std::swap(src0, src1); + + /* x += 4 sample every other 2x2 block */ + for (int x = 0; x < (int)window_.width; x += 4) { + b = src0[x]; + g = src0[x + 1]; + g2 = src1[x]; + r = src1[x + 1]; + + g = (g + g2) / 2; + + /* divide Y by 4 for 10 -> 8 bpp value */ + SWSTATS_ACCUMULATE_LINE_STATS(4) + } + + SWSTATS_FINISH_LINE_STATS() +} + +void SwStatsCpu::statsBGGR12Line0(const uint8_t *src[]) +{ + const uint16_t *src0 = (const uint16_t *)src[1] + window_.x; + const uint16_t *src1 = (const uint16_t *)src[2] + window_.x; + + SWSTATS_START_LINE_STATS(uint16_t) + + if (swapLines_) + std::swap(src0, src1); + + /* x += 4 sample every other 2x2 block */ + for (int x = 0; x < (int)window_.width; x += 4) { + b = src0[x]; + g = src0[x + 1]; + g2 = src1[x]; + r = src1[x + 1]; + + g = (g + g2) / 2; + + /* divide Y by 16 for 12 -> 8 bpp value */ + SWSTATS_ACCUMULATE_LINE_STATS(16) + } + + SWSTATS_FINISH_LINE_STATS() +} + void SwStatsCpu::statsBGGR10PLine0(const uint8_t *src[]) { const uint8_t *src0 = src[1] + window_.x * 5 / 4; @@ -147,6 +224,42 @@ void SwStatsCpu::finishFrame(void) statsReady.emit(0); } +/** + * \brief Setup SwStatsCpu object for standard Bayer orders + * \param[in] order The Bayer order + * + * Check if order is a standard Bayer order and setup xShift_ and swapLines_ + * so that a single BGGR stats function can be used for all 4 standard orders. + */ +int SwStatsCpu::setupStandardBayerOrder(BayerFormat::Order order) +{ + switch (order) { + case BayerFormat::BGGR: + xShift_ = 0; + swapLines_ = false; + break; + case BayerFormat::GBRG: + xShift_ = 1; /* BGGR -> GBRG */ + swapLines_ = false; + break; + case BayerFormat::GRBG: + xShift_ = 0; + swapLines_ = true; /* BGGR -> GRBG */ + break; + case BayerFormat::RGGB: + xShift_ = 1; /* BGGR -> GBRG */ + swapLines_ = true; /* GBRG -> RGGB */ + break; + default: + return -EINVAL; + } + + patternSize_.height = 2; + patternSize_.width = 2; + ySkipMask_ = 0x02; /* Skip every 3th and 4th line */ + return 0; +} + /** * \brief Configure the statistics object for the passed in input format. * \param[in] inputCfg The input format @@ -158,6 +271,21 @@ int SwStatsCpu::configure(const StreamConfiguration &inputCfg) BayerFormat bayerFormat = BayerFormat::fromPixelFormat(inputCfg.pixelFormat); + if (bayerFormat.packing == BayerFormat::Packing::None && + setupStandardBayerOrder(bayerFormat.order) == 0) { + switch (bayerFormat.bitDepth) { + case 8: + stats0_ = &SwStatsCpu::statsBGGR8Line0; + return 0; + case 10: + stats0_ = &SwStatsCpu::statsBGGR10Line0; + return 0; + case 12: + stats0_ = &SwStatsCpu::statsBGGR12Line0; + return 0; + } + } + if (bayerFormat.bitDepth == 10 && bayerFormat.packing == BayerFormat::Packing::CSI2) { patternSize_.height = 2; diff --git a/src/libcamera/software_isp/swstats_cpu.h b/src/libcamera/software_isp/swstats_cpu.h index 0ac9ae71..bbbcf69b 100644 --- a/src/libcamera/software_isp/swstats_cpu.h +++ b/src/libcamera/software_isp/swstats_cpu.h @@ -17,6 +17,7 @@ #include +#include "libcamera/internal/bayer_format.h" #include "libcamera/internal/shared_mem_object.h" #include "libcamera/internal/software_isp/swisp_stats.h" @@ -120,6 +121,14 @@ private: */ using statsProcessFn = void (SwStatsCpu::*)(const uint8_t *src[]); + int setupStandardBayerOrder(BayerFormat::Order order); + /* Bayer 8 bpp unpacked */ + void statsBGGR8Line0(const uint8_t *src[]); + /* Bayer 10 bpp unpacked */ + void statsBGGR10Line0(const uint8_t *src[]); + /* Bayer 12 bpp unpacked */ + void statsBGGR12Line0(const uint8_t *src[]); + /* Bayer 10 bpp packed */ void statsBGGR10PLine0(const uint8_t *src[]); void statsGBRG10PLine0(const uint8_t *src[]); From patchwork Tue Mar 19 12:36:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19771 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0DC79C3274 for ; Tue, 19 Mar 2024 12:37:38 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9335963037; Tue, 19 Mar 2024 13:37:37 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="VdQhsw1P"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id AFF0962D2E for ; Tue, 19 Mar 2024 13:37:35 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851854; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=R1lWVqASZaCycrrvpZCwKj5wP7FsiCNd/QQRdS6uIKU=; b=VdQhsw1PigebkmQjjRfKvj0jUgGMqc0CW/n+r2u9WS+QVC9OTYhNcMLdFRtoN7eZcDynk8 +RpDC/xfIV/P36TvvFUdv7W+fE++kvVx8V6prxv+LtBNKIQp6FiZfruvbb1HEzyLmgmSVT KvQcxOJZF6SKIo4fl1V8QDhP8qZ6C6g= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-203-Ywkq7XnJOtK3xb6vYOFIeA-1; Tue, 19 Mar 2024 08:37:30 -0400 X-MC-Unique: Ywkq7XnJOtK3xb6vYOFIeA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DD7ED1C2CED3; Tue, 19 Mar 2024 12:37:29 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id E6FE31C060A6; Tue, 19 Mar 2024 12:37:27 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Hans de Goede , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Kieran Bingham Subject: [PATCH v6 14/18] libcamera: debayer_cpu: Add support for 8, 10 and 12 bpp unpacked bayer input Date: Tue, 19 Mar 2024 13:36:01 +0100 Message-ID: <20240319123622.675599-15-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Hans de Goede Add support for 8, 10 and 12 bpp unpacked bayer input for all 4 standard bayer orders. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Reviewed-by: Milan Zamazal Signed-off-by: Kieran Bingham Signed-off-by: Hans de Goede --- src/libcamera/software_isp/debayer_cpu.cpp | 128 +++++++++++++++++++++ src/libcamera/software_isp/debayer_cpu.h | 13 +++ 2 files changed, 141 insertions(+) diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index f932362c..eb1c2718 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -56,6 +56,11 @@ DebayerCpu::~DebayerCpu() free(lineBuffers_[i]); } +#define DECLARE_SRC_POINTERS(pixel_t) \ + const pixel_t *prev = (const pixel_t *)src[0] + xShift_; \ + const pixel_t *curr = (const pixel_t *)src[1] + xShift_; \ + const pixel_t *next = (const pixel_t *)src[2] + xShift_; + // RGR // GBG // RGR @@ -92,6 +97,70 @@ DebayerCpu::~DebayerCpu() *dst++ = red_[curr[x] / (div)]; \ x++; +void DebayerCpu::debayer8_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + DECLARE_SRC_POINTERS(uint8_t) + + for (int x = 0; x < (int)window_.width;) { + BGGR_BGR888(1, 1, 1) + GBRG_BGR888(1, 1, 1) + } +} + +void DebayerCpu::debayer8_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + DECLARE_SRC_POINTERS(uint8_t) + + for (int x = 0; x < (int)window_.width;) { + GRBG_BGR888(1, 1, 1) + RGGB_BGR888(1, 1, 1) + } +} + +void DebayerCpu::debayer10_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + DECLARE_SRC_POINTERS(uint16_t) + + for (int x = 0; x < (int)window_.width;) { + /* divide values by 4 for 10 -> 8 bpp value */ + BGGR_BGR888(1, 1, 4) + GBRG_BGR888(1, 1, 4) + } +} + +void DebayerCpu::debayer10_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + DECLARE_SRC_POINTERS(uint16_t) + + for (int x = 0; x < (int)window_.width;) { + /* divide values by 4 for 10 -> 8 bpp value */ + GRBG_BGR888(1, 1, 4) + RGGB_BGR888(1, 1, 4) + } +} + +void DebayerCpu::debayer12_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + DECLARE_SRC_POINTERS(uint16_t) + + for (int x = 0; x < (int)window_.width;) { + /* divide values by 16 for 12 -> 8 bpp value */ + BGGR_BGR888(1, 1, 16) + GBRG_BGR888(1, 1, 16) + } +} + +void DebayerCpu::debayer12_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + DECLARE_SRC_POINTERS(uint16_t) + + for (int x = 0; x < (int)window_.width;) { + /* divide values by 16 for 12 -> 8 bpp value */ + GRBG_BGR888(1, 1, 16) + RGGB_BGR888(1, 1, 16) + } +} + void DebayerCpu::debayer10P_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) { const int width_in_bytes = window_.width * 5 / 4; @@ -193,6 +262,16 @@ int DebayerCpu::getInputConfig(PixelFormat inputFormat, DebayerInputConfig &conf BayerFormat bayerFormat = BayerFormat::fromPixelFormat(inputFormat); + if ((bayerFormat.bitDepth == 8 || bayerFormat.bitDepth == 10 || bayerFormat.bitDepth == 12) && + bayerFormat.packing == BayerFormat::Packing::None && + isStandardBayerOrder(bayerFormat.order)) { + config.bpp = (bayerFormat.bitDepth + 7) & ~7; + config.patternSize.width = 2; + config.patternSize.height = 2; + config.outputFormats = std::vector({ formats::RGB888 }); + return 0; + } + if (bayerFormat.bitDepth == 10 && bayerFormat.packing == BayerFormat::Packing::CSI2 && isStandardBayerOrder(bayerFormat.order)) { @@ -220,12 +299,61 @@ int DebayerCpu::getOutputConfig(PixelFormat outputFormat, DebayerOutputConfig &c return -EINVAL; } +/* + * Check for standard Bayer orders and set xShift_ and swap debayer0/1, so that + * a single pair of BGGR debayer functions can be used for all 4 standard orders. + */ +int DebayerCpu::setupStandardBayerOrder(BayerFormat::Order order) +{ + switch (order) { + case BayerFormat::BGGR: + break; + case BayerFormat::GBRG: + xShift_ = 1; /* BGGR -> GBRG */ + break; + case BayerFormat::GRBG: + std::swap(debayer0_, debayer1_); /* BGGR -> GRBG */ + break; + case BayerFormat::RGGB: + xShift_ = 1; /* BGGR -> GBRG */ + std::swap(debayer0_, debayer1_); /* GBRG -> RGGB */ + break; + default: + return -EINVAL; + } + + return 0; +} + /* TODO: this ignores outputFormat since there is only 1 supported outputFormat for now */ int DebayerCpu::setDebayerFunctions(PixelFormat inputFormat, [[maybe_unused]] PixelFormat outputFormat) { BayerFormat bayerFormat = BayerFormat::fromPixelFormat(inputFormat); + xShift_ = 0; + + if ((bayerFormat.bitDepth == 8 || bayerFormat.bitDepth == 10 || bayerFormat.bitDepth == 12) && + bayerFormat.packing == BayerFormat::Packing::None && + isStandardBayerOrder(bayerFormat.order)) { + switch (bayerFormat.bitDepth) { + case 8: + debayer0_ = &DebayerCpu::debayer8_BGBG_BGR888; + debayer1_ = &DebayerCpu::debayer8_GRGR_BGR888; + break; + case 10: + debayer0_ = &DebayerCpu::debayer10_BGBG_BGR888; + debayer1_ = &DebayerCpu::debayer10_GRGR_BGR888; + break; + case 12: + debayer0_ = &DebayerCpu::debayer12_BGBG_BGR888; + debayer1_ = &DebayerCpu::debayer12_GRGR_BGR888; + break; + } + setupStandardBayerOrder(bayerFormat.order); + return 0; + } + if (bayerFormat.bitDepth == 10 && bayerFormat.packing == BayerFormat::Packing::CSI2) { switch (bayerFormat.order) { diff --git a/src/libcamera/software_isp/debayer_cpu.h b/src/libcamera/software_isp/debayer_cpu.h index 8a51ed85..fd1fa180 100644 --- a/src/libcamera/software_isp/debayer_cpu.h +++ b/src/libcamera/software_isp/debayer_cpu.h @@ -17,6 +17,8 @@ #include +#include "libcamera/internal/bayer_format.h" + #include "debayer.h" #include "swstats_cpu.h" @@ -82,6 +84,15 @@ private: */ using debayerFn = void (DebayerCpu::*)(uint8_t *dst, const uint8_t *src[]); + /* 8-bit raw bayer format */ + void debayer8_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]); + void debayer8_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]); + /* unpacked 10-bit raw bayer format */ + void debayer10_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]); + void debayer10_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]); + /* unpacked 12-bit raw bayer format */ + void debayer12_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]); + void debayer12_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]); /* CSI-2 packed 10-bit raw bayer format (all the 4 orders) */ void debayer10P_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]); void debayer10P_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]); @@ -103,6 +114,7 @@ private: int getInputConfig(PixelFormat inputFormat, DebayerInputConfig &config); int getOutputConfig(PixelFormat outputFormat, DebayerOutputConfig &config); + int setupStandardBayerOrder(BayerFormat::Order order); int setDebayerFunctions(PixelFormat inputFormat, PixelFormat outputFormat); void setupInputMemcpy(const uint8_t *linePointers[]); void shiftLinePointers(const uint8_t *linePointers[], const uint8_t *src); @@ -131,6 +143,7 @@ private: unsigned int lineBufferLength_; unsigned int lineBufferPadding_; unsigned int lineBufferIndex_; + unsigned int xShift_; /* Offset of 0/1 applied to window_.x */ bool enableInputMemcpy_; float gamma_correction_; unsigned int measuredFrames_; From patchwork Tue Mar 19 12:36:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19772 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2CAE1C3272 for ; Tue, 19 Mar 2024 12:37:40 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B16CB62D2E; Tue, 19 Mar 2024 13:37:39 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="SnkDvzpr"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3827B62FFC for ; Tue, 19 Mar 2024 13:37:37 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851856; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1SNWhZi4uEprycadqZza9PiEjPAcyWfysP5QdmO0OZA=; b=SnkDvzprJC79yYH7oU+ZRillTDH1G5lFs8TOIK9hPnhCR+gipcyafxPOHg0YFEeOZUROee enqmLdQE322kY4Dua+4yLMR4HGWOhCyufrNxybC235mGGaEn6pSJhL5WkrPIgta815FrmJ Mhhyn3ObmY2JjYk18T7XXfh/+NZuMFw= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-101-6COubDljOl6v3oMT07M2bQ-1; Tue, 19 Mar 2024 08:37:33 -0400 X-MC-Unique: 6COubDljOl6v3oMT07M2bQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8ADA53C40B55; Tue, 19 Mar 2024 12:37:32 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id 498B61C060A6; Tue, 19 Mar 2024 12:37:30 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Hans de Goede , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Kieran Bingham , Stefan Klug Subject: [PATCH v6 15/18] libcamera: debayer_cpu: Add BGR888 output support Date: Tue, 19 Mar 2024 13:36:02 +0100 Message-ID: <20240319123622.675599-16-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Hans de Goede BGR888 is RGB888 with the red and blue pixels swapped, adjust the debayering to swap the red and blue pixels in the bayer pattern to add support for writing formats::BGR888. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Milan Zamazal Signed-off-by: Hans de Goede Reviewed-by: Stefan Klug --- src/libcamera/software_isp/debayer_cpu.cpp | 42 +++++++++++++++++++--- src/libcamera/software_isp/debayer_cpu.h | 1 + 2 files changed, 38 insertions(+), 5 deletions(-) diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index eb1c2718..a1692693 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -268,7 +268,7 @@ int DebayerCpu::getInputConfig(PixelFormat inputFormat, DebayerInputConfig &conf config.bpp = (bayerFormat.bitDepth + 7) & ~7; config.patternSize.width = 2; config.patternSize.height = 2; - config.outputFormats = std::vector({ formats::RGB888 }); + config.outputFormats = std::vector({ formats::RGB888, formats::BGR888 }); return 0; } @@ -278,7 +278,7 @@ int DebayerCpu::getInputConfig(PixelFormat inputFormat, DebayerInputConfig &conf config.bpp = 10; config.patternSize.width = 4; /* 5 bytes per *4* pixels */ config.patternSize.height = 2; - config.outputFormats = std::vector({ formats::RGB888 }); + config.outputFormats = std::vector({ formats::RGB888, formats::BGR888 }); return 0; } @@ -289,7 +289,7 @@ int DebayerCpu::getInputConfig(PixelFormat inputFormat, DebayerInputConfig &conf int DebayerCpu::getOutputConfig(PixelFormat outputFormat, DebayerOutputConfig &config) { - if (outputFormat == formats::RGB888) { + if (outputFormat == formats::RGB888 || outputFormat == formats::BGR888) { config.bpp = 24; return 0; } @@ -325,13 +325,41 @@ int DebayerCpu::setupStandardBayerOrder(BayerFormat::Order order) return 0; } -/* TODO: this ignores outputFormat since there is only 1 supported outputFormat for now */ -int DebayerCpu::setDebayerFunctions(PixelFormat inputFormat, [[maybe_unused]] PixelFormat outputFormat) +int DebayerCpu::setDebayerFunctions(PixelFormat inputFormat, PixelFormat outputFormat) { BayerFormat bayerFormat = BayerFormat::fromPixelFormat(inputFormat); xShift_ = 0; + swapRedBlueGains_ = false; + + switch (outputFormat) { + case formats::RGB888: + break; + case formats::BGR888: + /* Swap R and B in bayer order to generate BGR888 instead of RGB888 */ + swapRedBlueGains_ = true; + + switch (bayerFormat.order) { + case BayerFormat::BGGR: + bayerFormat.order = BayerFormat::RGGB; + break; + case BayerFormat::GBRG: + bayerFormat.order = BayerFormat::GRBG; + break; + case BayerFormat::GRBG: + bayerFormat.order = BayerFormat::GBRG; + break; + case BayerFormat::RGGB: + bayerFormat.order = BayerFormat::BGGR; + break; + default: + goto invalid_fmt; + } + break; + default: + goto invalid_fmt; + } if ((bayerFormat.bitDepth == 8 || bayerFormat.bitDepth == 10 || bayerFormat.bitDepth == 12) && bayerFormat.packing == BayerFormat::Packing::None && @@ -378,6 +406,7 @@ int DebayerCpu::setDebayerFunctions(PixelFormat inputFormat, [[maybe_unused]] Pi } } +invalid_fmt: LOG(Debayer, Error) << "Unsupported input output format combination"; return -EINVAL; } @@ -661,6 +690,9 @@ void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams gamma_correction_ = params.gamma; } + if (swapRedBlueGains_) + std::swap(params.gainR, params.gainB); + for (unsigned int i = 0; i < kRGBLookupSize; i++) { constexpr unsigned int div = kRGBLookupSize * DebayerParams::kGain10 / kGammaLookupSize; diff --git a/src/libcamera/software_isp/debayer_cpu.h b/src/libcamera/software_isp/debayer_cpu.h index fd1fa180..5f44fc65 100644 --- a/src/libcamera/software_isp/debayer_cpu.h +++ b/src/libcamera/software_isp/debayer_cpu.h @@ -145,6 +145,7 @@ private: unsigned int lineBufferIndex_; unsigned int xShift_; /* Offset of 0/1 applied to window_.x */ bool enableInputMemcpy_; + bool swapRedBlueGains_; float gamma_correction_; unsigned int measuredFrames_; int64_t frameProcessTime_; From patchwork Tue Mar 19 12:36:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19773 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9A6DFC3272 for ; Tue, 19 Mar 2024 12:37:43 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5307B6305E; Tue, 19 Mar 2024 13:37:43 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="D1zSebEG"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id CD1FA62CA7 for ; Tue, 19 Mar 2024 13:37:41 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851860; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=G0yImgZkvIz9scyIzliYmfyOITw5DAJIqXQDUmWHqfM=; b=D1zSebEG4hB6RBuLTp4OSwqZ3eNgI5Ik3QX8n0PC+tWrSJyGQQKoImR5QLpFvsfAcKBCIh YzgMGbKG0zTPD+vd+4NDtYXyP57YLEEi5eblciFBGGDvO10pWt3OgHsRYvXWsCsPmjXe/G r9BlA+JW0o0jihHlfZnq3Q64MkPg+yI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-259-OPaaI1JnMY62OLo6WZCclQ-1; Tue, 19 Mar 2024 08:37:35 -0400 X-MC-Unique: OPaaI1JnMY62OLo6WZCclQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 065858007B8; Tue, 19 Mar 2024 12:37:35 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id EB49A1C060A6; Tue, 19 Mar 2024 12:37:32 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Hans de Goede , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Kieran Bingham , Stefan Klug Subject: [PATCH v6 16/18] libcamera: Add "Software ISP benchmarking" documentation Date: Tue, 19 Mar 2024 13:36:03 +0100 Message-ID: <20240319123622.675599-17-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Hans de Goede Add a "Software ISP benchmarking" documentation section which describes the performance/power consumption measurements used during the Software ISP's development. Reviewed-by: Milan Zamazal Signed-off-by: Hans de Goede Reviewed-by: Stefan Klug Reviewed-by: Laurent Pinchart --- Documentation/index.rst | 1 + Documentation/meson.build | 1 + Documentation/software-isp-benchmarking.rst | 82 +++++++++++++++++++++ 3 files changed, 84 insertions(+) create mode 100644 Documentation/software-isp-benchmarking.rst diff --git a/Documentation/index.rst b/Documentation/index.rst index 63fac72d..5442ae75 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -24,3 +24,4 @@ Lens driver requirements Python Bindings Camera Sensor Model + SoftwareISP Benchmarking diff --git a/Documentation/meson.build b/Documentation/meson.build index 7a58fec8..3872e0a8 100644 --- a/Documentation/meson.build +++ b/Documentation/meson.build @@ -80,6 +80,7 @@ if sphinx.found() 'lens_driver_requirements.rst', 'python-bindings.rst', 'sensor_driver_requirements.rst', + 'software-isp-benchmarking.rst', '../README.rst', ] diff --git a/Documentation/software-isp-benchmarking.rst b/Documentation/software-isp-benchmarking.rst new file mode 100644 index 00000000..a9a5edbe --- /dev/null +++ b/Documentation/software-isp-benchmarking.rst @@ -0,0 +1,82 @@ +.. SPDX-License-Identifier: CC-BY-SA-4.0 + +.. _software-isp-benchmarking: + +Software ISP benchmarking +========================= + +The Software ISP is particularly sensitive to performance regressions +therefore it is a good idea to always benchmark the Software ISP +before and after making changes to it and ensure that there are +no performance regressions. + +DebayerCpu class builtin benchmark +---------------------------------- + +The DebayerCpu class has a builtin benchmark. This benchmark +measures the time spent on processing (collecting statistics +and debayering) only, it does not measure the time spent on +capturing or outputting the frames. + +The builtin benchmark always runs. So this can be used by simply +running "cam" or "qcam" with a pipeline using the Software ISP. + +When it runs it will skip measuring the first 30 frames to +allow the caches and the CPU temperature (turbo-ing) to warm-up +and then it measures 30 fps and shows the total and per frame +processing time using an info level log message: + +.. code-block:: text + + INFO Debayer debayer_cpu.cpp:907 Processed 30 frames in 244317us, 8143 us/frame + +To get stable measurements it is advised to disable any other processes which +may cause significant CPU usage (e.g. disable wifi, bluetooth and browsers). +When possible it is also advisable to disable CPU turbo-ing and +frequency-scaling. + +For example when benchmarking on a Lenovo ThinkPad X1 Yoga Gen 8, with +the charger plugged in, the CPU can be fixed to run at 2 GHz using: + +.. code-block:: shell + + sudo x86_energy_perf_policy --turbo-enable 0 + sudo cpupower frequency-set -d 2GHz -u 2GHz + +with these settings the builtin bench reports a processing time of ~7.8ms/frame +on this laptop for FHD SGRBG10 (unpacked) bayer data. + +Measuring power consumption +--------------------------- + +Since the Software ISP is often used on mobile devices it is also +important to measure power consumption and ensure that that does +not regress. + +For example to measure power consumption on a Lenovo ThinkPad X1 Yoga Gen 8 +it needs to be running on battery and it should be configured with its +platform-profile (/sys/firmware/acpi/platform_profile) set to balanced and +with its default turbo and frequency-scaling behavior to match real world usage. + +Then start qcam to capture a FHD picture at 30 fps and position the qcam window +so that it is fully visible. After this run the following command to monitor +the power consumption: + +.. code-block:: shell + + watch -n 10 cat /sys/class/power_supply/BAT0/power_now /sys/class/hwmon/hwmon6/fan?_input + +Note this not only measures the power consumption in µW it also monitors +the speed of this laptop's 2 fans. This is important because depending on +the ambient temperature the 2 fans may spin up while testing and this +will cause an additional power consumption of approx. 0.5 W messing up +the measurement. + +After starting qcam + the watch command let the laptop sit without using +it for 2 minutes for the readings to stabilize. Then check that the fans +have not turned on and manually take a couple of consecutive power readings +and average these. + +On the example Lenovo ThinkPad X1 Yoga Gen 8 laptop this results in +a measured power consumption of approx. 13 W while running qcam versus +approx. 4-5 W while setting idle with its OLED panel on. From patchwork Tue Mar 19 12:36:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19774 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 186F9C3272 for ; Tue, 19 Mar 2024 12:37:46 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id CEA4D62D2E; Tue, 19 Mar 2024 13:37:45 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="QU+w6Xu/"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 71F8D63079 for ; Tue, 19 Mar 2024 13:37:43 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851862; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pzkfNgbaoBQCgkqvscoy+omXdExcya0NexVn9GEpZ3o=; b=QU+w6Xu/P6yYuyk8CuW8mPC6aUIBcslSjUw1hQwxMcAZ6N5YOLP1FnutYuzEplWfuXBJA+ mG50WUT9ssbmmfkhAc4CpcKD0FVpcVm/oYQAEp8/bFZp2GPpLBEXkcTx5FRPShAxki2NoS aWZkvKCToe6cB4mtq+Ze3TbxG+ob2/M= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-636-mOhSJ7nUNv6vsC7JPxNrZw-1; Tue, 19 Mar 2024 08:37:37 -0400 X-MC-Unique: mOhSJ7nUNv6vsC7JPxNrZw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6246885200B; Tue, 19 Mar 2024 12:37:37 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id 66F021C060A4; Tue, 19 Mar 2024 12:37:35 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Andrey Konovalov , Bryan O'Donoghue , Maxime Ripard , Pavel Machek , Hans de Goede , Kieran Bingham Subject: [PATCH v6 17/18] libcamera: software_isp: Apply black level compensation Date: Tue, 19 Mar 2024 13:36:04 +0100 Message-ID: <20240319123622.675599-18-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Black may not be represented as 0 pixel value for given hardware, it may be higher. If this is not compensated then various problems may occur such as low contrast or suboptimal exposure. The black pixel value can be either retrieved from a tuning file for the given hardware, or automatically on fly. The former is the right and correct method, while the latter can be used when a tuning file is not available for the given hardware. Since there is currently no support for tuning files in software ISP, the automatic, hardware independent way, is always used. Support for tuning files should be added in future but it will require more work than this patch. The patch looks at the image histogram and assumes that black starts when pixel values start occurring on the left. A certain amount of the darkest pixels is ignored; it doesn't matter whether they represent various kinds of noise or are real, they are better to omit in any case to make the image looking better. It also doesn't matter whether the darkest pixels occur around the supposed black level or are spread between 0 and the black level, the difference is not important. An arbitrary threshold of 2% darkest pixels is applied; there is no magic about that value. The patch assumes that the black values for different colors are the same and doesn't attempt any other non-primitive enhancements. It cannot completely replace tuning files and simplicity, while providing visible benefit, is its goal. Anything more sophisticated is left for future patches. A possible cheap enhancement, if needed, could be setting exposure + gain to minimum values temporarily, before setting the black level. In theory, the black level should be fixed but it may not be reached in all images. For this reason, the patch updates black level only if the observed value is lower than the current one; it should be never increased. The purpose of the patch is to compensate for hardware properties. General image contrast enhancements are out of scope of this patch. Stats are still gathered as an uncorrected histogram, to avoid any confusion and to represent the raw image data. Exposure must be determined after the black level correction -- it has no influence on the sub-black area and must be correct after applying the black level correction. The granularity of the histogram is increased from 16 to 64 to provide a better precision (there is no theory behind either of those numbers). Reviewed-by: Hans de Goede Signed-off-by: Milan Zamazal Signed-off-by: Hans de Goede --- .../internal/software_isp/debayer_params.h | 4 + .../internal/software_isp/swisp_stats.h | 10 ++- src/ipa/simple/black_level.cpp | 88 +++++++++++++++++++ src/ipa/simple/black_level.h | 28 ++++++ src/ipa/simple/meson.build | 7 +- src/ipa/simple/soft_simple.cpp | 28 ++++-- src/libcamera/software_isp/debayer_cpu.cpp | 13 ++- src/libcamera/software_isp/debayer_cpu.h | 1 + src/libcamera/software_isp/software_isp.cpp | 2 +- 9 files changed, 164 insertions(+), 17 deletions(-) create mode 100644 src/ipa/simple/black_level.cpp create mode 100644 src/ipa/simple/black_level.h diff --git a/include/libcamera/internal/software_isp/debayer_params.h b/include/libcamera/internal/software_isp/debayer_params.h index 98965fa1..5e38e08b 100644 --- a/include/libcamera/internal/software_isp/debayer_params.h +++ b/include/libcamera/internal/software_isp/debayer_params.h @@ -43,6 +43,10 @@ struct DebayerParams { * \brief Gamma correction, 1.0 is no correction */ float gamma; + /** + * \brief Level of the black point, 0..255, 0 is no correction. + */ + unsigned int blackLevel; }; } /* namespace libcamera */ diff --git a/include/libcamera/internal/software_isp/swisp_stats.h b/include/libcamera/internal/software_isp/swisp_stats.h index afe42c9a..25cd5abd 100644 --- a/include/libcamera/internal/software_isp/swisp_stats.h +++ b/include/libcamera/internal/software_isp/swisp_stats.h @@ -7,6 +7,8 @@ #pragma once +#include + namespace libcamera { /** @@ -28,11 +30,15 @@ struct SwIspStats { /** * \brief Number of bins in the yHistogram. */ - static constexpr unsigned int kYHistogramSize = 16; + static constexpr unsigned int kYHistogramSize = 64; + /** + * \brief Type of the histogram. + */ + using histogram = std::array; /** * \brief A histogram of luminance values. */ - std::array yHistogram; + histogram yHistogram; }; } /* namespace libcamera */ diff --git a/src/ipa/simple/black_level.cpp b/src/ipa/simple/black_level.cpp new file mode 100644 index 00000000..9d4aa800 --- /dev/null +++ b/src/ipa/simple/black_level.cpp @@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * black_level.cpp - black level handling + */ + +#include "black_level.h" + +#include + +#include + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoftBL) + +/** + * \class BlackLevel + * \brief Object providing black point level for software ISP + * + * Black level can be provided in hardware tuning files or, if no tuning file is + * available for the given hardware, guessed automatically, with less accuracy. + * As tuning files are not yet implemented for software ISP, BlackLevel + * currently provides only guessed black levels. + * + * This class serves for tracking black level as a property of the underlying + * hardware, not as means of enhancing a particular scene or image. + * + * The class is supposed to be instantiated for the given camera stream. + * The black level can be retrieved using BlackLevel::get() method. It is + * initially 0 and may change when updated using BlackLevel::update() method. + */ + +BlackLevel::BlackLevel() + : blackLevel_(255), blackLevelSet_(false) +{ +} + +/** + * \brief Return the current black level + * + * \return The black level, in the range from 0 (minimum) to 255 (maximum). + * If the black level couldn't be determined yet, return 0. + */ +unsigned int BlackLevel::get() const +{ + return blackLevelSet_ ? blackLevel_ : 0; +} + +/** + * \brief Update black level from the provided histogram + * \param[in] yHistogram The histogram to be used for updating black level + * + * The black level is property of the given hardware, not image. It is updated + * only if it has not been yet set or if it is lower than the lowest value seen + * so far. + */ +void BlackLevel::update(SwIspStats::histogram &yHistogram) +{ + /* + * The constant is selected to be "good enough", not overly conservative or + * aggressive. There is no magic about the given value. + */ + constexpr float ignoredPercentage_ = 0.02; + const unsigned int total = + std::accumulate(begin(yHistogram), end(yHistogram), 0); + const unsigned int pixelThreshold = ignoredPercentage_ * total; + const unsigned int histogramRatio = 256 / SwIspStats::kYHistogramSize; + const unsigned int currentBlackIdx = blackLevel_ / histogramRatio; + + for (unsigned int i = 0, seen = 0; + i < currentBlackIdx && i < SwIspStats::kYHistogramSize; + i++) { + seen += yHistogram[i]; + if (seen >= pixelThreshold) { + blackLevel_ = i * histogramRatio; + blackLevelSet_ = true; + LOG(IPASoftBL, Debug) + << "Auto-set black level: " + << i << "/" << SwIspStats::kYHistogramSize + << " (" << 100 * (seen - yHistogram[i]) / total << "% below, " + << 100 * seen / total << "% at or below)"; + break; + } + }; +} +} // namespace libcamera diff --git a/src/ipa/simple/black_level.h b/src/ipa/simple/black_level.h new file mode 100644 index 00000000..b3785db0 --- /dev/null +++ b/src/ipa/simple/black_level.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * black_level.h - black level handling + */ + +#pragma once + +#include + +#include "libcamera/internal/software_isp/swisp_stats.h" + +namespace libcamera { + +class BlackLevel +{ +public: + BlackLevel(); + unsigned int get() const; + void update(std::array &yHistogram); + +private: + unsigned int blackLevel_; + bool blackLevelSet_; +}; + +} // namespace libcamera diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build index 3e863db7..44b5f1d7 100644 --- a/src/ipa/simple/meson.build +++ b/src/ipa/simple/meson.build @@ -2,8 +2,13 @@ ipa_name = 'ipa_soft_simple' +soft_simple_sources = files([ + 'soft_simple.cpp', + 'black_level.cpp', +]) + mod = shared_module(ipa_name, - ['soft_simple.cpp', libcamera_generated_ipa_headers], + [soft_simple_sources, libcamera_generated_ipa_headers], name_prefix : '', include_directories : [ipa_includes, libipa_includes], dependencies : libcamera_private, diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 312df4ba..ac027568 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -22,6 +22,8 @@ #include "libcamera/internal/software_isp/debayer_params.h" #include "libcamera/internal/software_isp/swisp_stats.h" +#include "black_level.h" + namespace libcamera { LOG_DEFINE_CATEGORY(IPASoft) @@ -33,7 +35,8 @@ class IPASoftSimple : public ipa::soft::IPASoftInterface public: IPASoftSimple() : params_(static_cast(MAP_FAILED)), - stats_(static_cast(MAP_FAILED)), ignore_updates_(0) + stats_(static_cast(MAP_FAILED)), + blackLevel_(BlackLevel()), ignore_updates_(0) { } @@ -63,6 +66,7 @@ private: SharedFD fdParams_; DebayerParams *params_; SwIspStats *stats_; + BlackLevel blackLevel_; int32_t exposure_min_, exposure_max_; int32_t again_min_, again_max_; @@ -196,6 +200,10 @@ void IPASoftSimple::processStats(const ControlList &sensorControls) params_->gainG = 256; params_->gamma = 0.5; + if (ignore_updates_ > 0) + blackLevel_.update(stats_->yHistogram); + params_->blackLevel = blackLevel_.get(); + setIspParams.emit(0); /* @@ -211,18 +219,19 @@ void IPASoftSimple::processStats(const ControlList &sensorControls) * Calculate Mean Sample Value (MSV) according to formula from: * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf */ - constexpr unsigned int yHistValsPerBin = - SwIspStats::kYHistogramSize / kExposureBinsCount; - constexpr unsigned int yHistValsPerBinMod = - SwIspStats::kYHistogramSize / - (SwIspStats::kYHistogramSize % kExposureBinsCount + 1); + const unsigned int blackLevelHistIdx = + params_->blackLevel / (256 / SwIspStats::kYHistogramSize); + const unsigned int histogramSize = SwIspStats::kYHistogramSize - blackLevelHistIdx; + const unsigned int yHistValsPerBin = histogramSize / kExposureBinsCount; + const unsigned int yHistValsPerBinMod = + histogramSize / (histogramSize % kExposureBinsCount + 1); int ExposureBins[kExposureBinsCount] = {}; unsigned int denom = 0; unsigned int num = 0; - for (unsigned int i = 0; i < SwIspStats::kYHistogramSize; i++) { + for (unsigned int i = 0; i < histogramSize; i++) { unsigned int idx = (i - (i / yHistValsPerBinMod)) / yHistValsPerBin; - ExposureBins[idx] += stats_->yHistogram[i]; + ExposureBins[idx] += stats_->yHistogram[blackLevelHistIdx + i]; } for (unsigned int i = 0; i < kExposureBinsCount; i++) { @@ -256,7 +265,8 @@ void IPASoftSimple::processStats(const ControlList &sensorControls) LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV << " exp " << exposure_ << " again " << again_ - << " gain R/B " << params_->gainR << "/" << params_->gainB; + << " gain R/B " << params_->gainR << "/" << params_->gainB + << " black level " << params_->blackLevel; } void IPASoftSimple::updateExposure(double exposureMSV) diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index a1692693..3be3cdfe 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -35,7 +35,7 @@ namespace libcamera { * \param[in] stats Pointer to the stats object to use. */ DebayerCpu::DebayerCpu(std::unique_ptr stats) - : stats_(std::move(stats)), gamma_correction_(1.0) + : stats_(std::move(stats)), gamma_correction_(1.0), blackLevel_(0) { #ifdef __x86_64__ enableInputMemcpy_ = false; @@ -683,11 +683,16 @@ void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams } /* Apply DebayerParams */ - if (params.gamma != gamma_correction_) { - for (unsigned int i = 0; i < kGammaLookupSize; i++) - gamma_[i] = UINT8_MAX * powf(i / (kGammaLookupSize - 1.0), params.gamma); + if (params.gamma != gamma_correction_ || params.blackLevel != blackLevel_) { + const unsigned int blackIndex = + params.blackLevel * kGammaLookupSize / 256; + std::fill(gamma_.begin(), gamma_.begin() + blackIndex, 0); + const float divisor = kGammaLookupSize - blackIndex - 1.0; + for (unsigned int i = blackIndex; i < kGammaLookupSize; i++) + gamma_[i] = UINT8_MAX * powf((i - blackIndex) / divisor, params.gamma); gamma_correction_ = params.gamma; + blackLevel_ = params.blackLevel; } if (swapRedBlueGains_) diff --git a/src/libcamera/software_isp/debayer_cpu.h b/src/libcamera/software_isp/debayer_cpu.h index 5f44fc65..ea02f909 100644 --- a/src/libcamera/software_isp/debayer_cpu.h +++ b/src/libcamera/software_isp/debayer_cpu.h @@ -147,6 +147,7 @@ private: bool enableInputMemcpy_; bool swapRedBlueGains_; float gamma_correction_; + unsigned int blackLevel_; unsigned int measuredFrames_; int64_t frameProcessTime_; /* Skip 30 frames for things to stabilize then measure 30 frames */ diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 388b4496..9b49be41 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -64,7 +64,7 @@ LOG_DEFINE_CATEGORY(SoftwareIsp) */ SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const ControlInfoMap &sensorControls) : debayer_(nullptr), - debayerParams_{ DebayerParams::kGain10, DebayerParams::kGain10, DebayerParams::kGain10, 0.5f }, + debayerParams_{ DebayerParams::kGain10, DebayerParams::kGain10, DebayerParams::kGain10, 0.5f, 0 }, dmaHeap_(DmaHeap::DmaHeapFlag::Cma | DmaHeap::DmaHeapFlag::System) { if (!dmaHeap_.isValid()) { From patchwork Tue Mar 19 12:36:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 19775 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id EA651C3274 for ; Tue, 19 Mar 2024 12:37:46 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 930FD63055; Tue, 19 Mar 2024 13:37:46 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="VtBOWqGU"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 33D1562D2E for ; Tue, 19 Mar 2024 13:37:45 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1710851864; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L6QSVmfOdEG+8GxEYT4oUFQEHFx1W2n55DRmuzC8+Cs=; b=VtBOWqGU3PyqViarLvsMB8mgr5mYlPuiNF1Y02Ft4KT+NPHCd9aCs6o2grn73fa/g5yELC 58abGoUhLlRpzpPfPeL7Il7xY+oRRrIr5GMQwltyx/p4sc2nBR0QzwZ7usDVeQkjlhdxo8 MRo+sSTJxdijjOrjXoTMNd59rR43Gu8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-550--NdhT1vhOJy1MmnHHFKE3w-1; Tue, 19 Mar 2024 08:37:40 -0400 X-MC-Unique: -NdhT1vhOJy1MmnHHFKE3w-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E69F4852000; Tue, 19 Mar 2024 12:37:39 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.226.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id BD9F31C060CE; Tue, 19 Mar 2024 12:37:37 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Andrei Konovalov , Bryan O'Donoghue , Maxime Ripard , Milan Zamazal , Pavel Machek , Hans de Goede , Kieran Bingham Subject: [PATCH v6 18/18] libcamera: Soft IPA: use CameraSensorHelper for analogue gain Date: Tue, 19 Mar 2024 13:36:05 +0100 Message-ID: <20240319123622.675599-19-mzamazal@redhat.com> In-Reply-To: <20240319123622.675599-1-mzamazal@redhat.com> References: <20240319123622.675599-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrei Konovalov Use CameraSensorHelper to convert the analogue gain code read from the camera sensor into real analogue gain value. In the future this makes it possible to use faster AE/AGC algorithm. For now the same AE/AGC algorithm is used, but even then the CameraSensorHelper lets us use the full range of analogue gain values. If there is no CameraSensorHelper for the camera sensor in use, a warning log message is printed, and the AE/AGC works exactly as before this change. Signed-off-by: Andrei Konovalov Reviewed-by: Hans de Goede Signed-off-by: Hans de Goede Reviewed-by: Milan Zamazal --- .../internal/software_isp/software_isp.h | 3 +- src/ipa/simple/soft_simple.cpp | 77 ++++++++++++------- src/libcamera/pipeline/simple/simple.cpp | 2 +- src/libcamera/software_isp/software_isp.cpp | 8 +- 4 files changed, 57 insertions(+), 33 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index 8d25e979..2a6db7ba 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -26,6 +26,7 @@ #include #include +#include "libcamera/internal/camera_sensor.h" #include "libcamera/internal/dma_heaps.h" #include "libcamera/internal/pipeline_handler.h" #include "libcamera/internal/shared_mem_object.h" @@ -43,7 +44,7 @@ LOG_DECLARE_CATEGORY(SoftwareIsp) class SoftwareIsp { public: - SoftwareIsp(PipelineHandler *pipe, const ControlInfoMap &sensorControls); + SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor); ~SoftwareIsp(); int loadConfiguration([[maybe_unused]] const std::string &filename) { return 0; } diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index ac027568..e4d64762 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -22,6 +22,8 @@ #include "libcamera/internal/software_isp/debayer_params.h" #include "libcamera/internal/software_isp/swisp_stats.h" +#include "libipa/camera_sensor_helper.h" + #include "black_level.h" namespace libcamera { @@ -67,18 +69,27 @@ private: DebayerParams *params_; SwIspStats *stats_; BlackLevel blackLevel_; + std::unique_ptr camHelper_; int32_t exposure_min_, exposure_max_; - int32_t again_min_, again_max_; - int32_t again_, exposure_; + int32_t exposure_; + double again_min_, again_max_, againMinStep_; + double again_; unsigned int ignore_updates_; }; -int IPASoftSimple::init([[maybe_unused]] const IPASettings &settings, +int IPASoftSimple::init(const IPASettings &settings, const SharedFD &fdStats, const SharedFD &fdParams, const ControlInfoMap &sensorInfoMap) { + camHelper_ = CameraSensorHelperFactoryBase::create(settings.sensorModel); + if (camHelper_ == nullptr) { + LOG(IPASoft, Warning) + << "Failed to create camera sensor helper for " + << settings.sensorModel; + } + fdStats_ = fdStats; if (!fdStats_.isValid()) { LOG(IPASoft, Error) << "Invalid Statistics handle"; @@ -132,25 +143,35 @@ int IPASoftSimple::configure(const ControlInfoMap &sensorInfoMap) exposure_min_ = 1; } - again_min_ = gain_info.min().get(); - again_max_ = gain_info.max().get(); - /* - * The camera sensor gain (g) is usually not equal to the value written - * into the gain register (x). But the way how the AGC algorithm changes - * the gain value to make the total exposure closer to the optimum assumes - * that g(x) is not too far from linear function. If the minimal gain is 0, - * the g(x) is likely to be far from the linear, like g(x) = a / (b * x + c). - * To avoid unexpected changes to the gain by the AGC algorithm (abrupt near - * one edge, and very small near the other) we limit the range of the gain - * values used. - */ - if (!again_min_) { - LOG(IPASoft, Warning) << "Minimum gain is zero, that can't be linear"; - again_min_ = std::min(100, again_min_ / 2 + again_max_ / 2); + int32_t again_min = gain_info.min().get(); + int32_t again_max = gain_info.max().get(); + + if (camHelper_) { + again_min_ = camHelper_->gain(again_min); + again_max_ = camHelper_->gain(again_max); + againMinStep_ = (again_max_ - again_min_) / 100.0; + } else { + /* + * The camera sensor gain (g) is usually not equal to the value written + * into the gain register (x). But the way how the AGC algorithm changes + * the gain value to make the total exposure closer to the optimum assumes + * that g(x) is not too far from linear function. If the minimal gain is 0, + * the g(x) is likely to be far from the linear, like g(x) = a / (b * x + c). + * To avoid unexpected changes to the gain by the AGC algorithm (abrupt near + * one edge, and very small near the other) we limit the range of the gain + * values used. + */ + again_max_ = again_max; + if (!again_min) { + LOG(IPASoft, Warning) << "Minimum gain is zero, that can't be linear"; + again_min_ = std::min(100, again_min / 2 + again_max / 2); + } + againMinStep_ = 1.0; } LOG(IPASoft, Info) << "Exposure " << exposure_min_ << "-" << exposure_max_ - << ", gain " << again_min_ << "-" << again_max_; + << ", gain " << again_min_ << "-" << again_max_ + << " (" << againMinStep_ << ")"; return 0; } @@ -252,12 +273,14 @@ void IPASoftSimple::processStats(const ControlList &sensorControls) ControlList ctrls(sensorControls); exposure_ = ctrls.get(V4L2_CID_EXPOSURE).get(); - again_ = ctrls.get(V4L2_CID_ANALOGUE_GAIN).get(); + int32_t again = ctrls.get(V4L2_CID_ANALOGUE_GAIN).get(); + again_ = camHelper_ ? camHelper_->gain(again) : again; updateExposure(exposureMSV); ctrls.set(V4L2_CID_EXPOSURE, exposure_); - ctrls.set(V4L2_CID_ANALOGUE_GAIN, again_); + ctrls.set(V4L2_CID_ANALOGUE_GAIN, + static_cast(camHelper_ ? camHelper_->gainCode(again_) : again_)); ignore_updates_ = 2; @@ -276,7 +299,7 @@ void IPASoftSimple::updateExposure(double exposureMSV) static constexpr uint8_t kExpNumeratorUp = kExpDenominator + 1; static constexpr uint8_t kExpNumeratorDown = kExpDenominator - 1; - int next; + double next; if (exposureMSV < kExposureOptimal - kExposureSatisfactory) { next = exposure_ * kExpNumeratorUp / kExpDenominator; @@ -286,18 +309,18 @@ void IPASoftSimple::updateExposure(double exposureMSV) exposure_ = next; if (exposure_ >= exposure_max_) { next = again_ * kExpNumeratorUp / kExpDenominator; - if (next - again_ < 1) - again_ += 1; + if (next - again_ < againMinStep_) + again_ += againMinStep_; else again_ = next; } } if (exposureMSV > kExposureOptimal + kExposureSatisfactory) { - if (exposure_ == exposure_max_ && again_ != again_min_) { + if (exposure_ == exposure_max_ && again_ > again_min_) { next = again_ * kExpNumeratorDown / kExpDenominator; - if (again_ - next < 1) - again_ -= 1; + if (again_ - next < againMinStep_) + again_ -= againMinStep_; else again_ = next; } else { diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index ac796b9b..22639407 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -526,7 +526,7 @@ int SimpleCameraData::init() * Instantiate Soft ISP if this is enabled for the given driver and no converter is used. */ if (!converter_ && pipe->swIspEnabled()) { - swIsp_ = std::make_unique(pipe, sensor_->controls()); + swIsp_ = std::make_unique(pipe, sensor_.get()); if (!swIsp_->isValid()) { LOG(SimplePipeline, Warning) << "Failed to create software ISP, disabling software debayering"; diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 9b49be41..ea4d96e4 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -60,9 +60,9 @@ LOG_DEFINE_CATEGORY(SoftwareIsp) /** * \brief Constructs SoftwareIsp object * \param[in] pipe The pipeline handler in use - * \param[in] sensorControls ControlInfoMap describing the controls supported by the sensor + * \param[in] sensor Pointer to the CameraSensor instance owned by the pipeline handler */ -SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const ControlInfoMap &sensorControls) +SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor) : debayer_(nullptr), debayerParams_{ DebayerParams::kGain10, DebayerParams::kGain10, DebayerParams::kGain10, 0.5f, 0 }, dmaHeap_(DmaHeap::DmaHeapFlag::Cma | DmaHeap::DmaHeapFlag::System) @@ -97,10 +97,10 @@ SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const ControlInfoMap &sensorCont return; } - int ret = ipa_->init(IPASettings{ "No cfg file", "No sensor model" }, + int ret = ipa_->init(IPASettings{ "No cfg file", sensor->model() }, debayer_->getStatsFD(), sharedParams_.fd(), - sensorControls); + sensor->controls()); if (ret) { LOG(SoftwareIsp, Error) << "IPA init failed"; debayer_.reset();