From patchwork Thu Feb 29 18:36:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19572 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1A051BD160 for ; Thu, 29 Feb 2024 18:37:05 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DA8C262868; Thu, 29 Feb 2024 19:37:04 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="FHjfFGAK"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id AFD0D62867 for ; Thu, 29 Feb 2024 19:37:03 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231822; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rXFxaDFzAvk1gQ0trIvCVHe0LKfhxrZKoTdAlTgIiSc=; b=FHjfFGAKB5C3Jr8qtmn2l5rXYZkaQ2Dq3ih8UK/rSSbf8pMCdHN0kcu866yzV1SeR07Zuj /Pzlm+w0GDSBSlX2cZ4+XCUnWxmBhxn9knh0vZEeNi4Az/lSfMGwIqpFnuKYY6rDn5RPcx xMkoLDsbn2em3lEYNmekJKjjhWI6tM0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-215-6XqhDAykOo-xQkHcLR3H3Q-1; Thu, 29 Feb 2024 13:36:58 -0500 X-MC-Unique: 6XqhDAykOo-xQkHcLR3H3Q-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C063929AA3B4; Thu, 29 Feb 2024 18:36:57 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id ABA801C060B1; Thu, 29 Feb 2024 18:36:56 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 01/18] libcamera: pipeline: simple: fix size adjustment in validate() Date: Thu, 29 Feb 2024 19:36:12 +0100 Message-ID: <20240229183654.7206-2-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov SimpleCameraConfiguration::validate() adjusts the configuration of its streams (if the size is not in the outputSizes) to the captureSize. But the captureSize itself can be not in the outputSizes, and then the adjusted configuration won't be valid resulting in camera configuration failure. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Milan Zamazal Reviewed-by: Pavel Machek Signed-off-by: Andrey Konovalov Signed-off-by: Hans de Goede --- src/libcamera/pipeline/simple/simple.cpp | 37 ++++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index e37b0d1c..aa69bcc4 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -882,6 +882,30 @@ SimpleCameraConfiguration::SimpleCameraConfiguration(Camera *camera, { } +namespace { + +static Size adjustSize(const Size &requestedSize, const SizeRange &supportedSizes) +{ + ASSERT(supportedSizes.min <= supportedSizes.max); + + if (supportedSizes.min == supportedSizes.max) + return supportedSizes.max; + + unsigned int hStep = supportedSizes.hStep; + unsigned int vStep = supportedSizes.vStep; + + if (hStep == 0) + hStep = supportedSizes.max.width - supportedSizes.min.width; + if (vStep == 0) + vStep = supportedSizes.max.height - supportedSizes.min.height; + + Size adjusted = requestedSize.boundedTo(supportedSizes.max).expandedTo(supportedSizes.min); + + return adjusted.shrunkBy(supportedSizes.min).alignedDownTo(hStep, vStep).grownBy(supportedSizes.min); +} + +} /* namespace */ + CameraConfiguration::Status SimpleCameraConfiguration::validate() { const CameraSensor *sensor = data_->sensor_.get(); @@ -998,10 +1022,19 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() } if (!pipeConfig_->outputSizes.contains(cfg.size)) { + Size adjustedSize = pipeConfig_->captureSize; + /* + * The converter (when present) may not be able to output + * a size identical to its input size. The capture size is thus + * not guaranteed to be a valid output size. In such cases, use + * the smaller valid output size closest to the requested. + */ + if (!pipeConfig_->outputSizes.contains(adjustedSize)) + adjustedSize = adjustSize(cfg.size, pipeConfig_->outputSizes); LOG(SimplePipeline, Debug) << "Adjusting size from " << cfg.size - << " to " << pipeConfig_->captureSize; - cfg.size = pipeConfig_->captureSize; + << " to " << adjustedSize; + cfg.size = adjustedSize; status = Adjusted; } From patchwork Thu Feb 29 18:36:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19574 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id AE8EFBD160 for ; Thu, 29 Feb 2024 18:37:08 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 6715062878; Thu, 29 Feb 2024 19:37:08 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="hf9EGwFw"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id BE8C462872 for ; Thu, 29 Feb 2024 19:37:06 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231825; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xMifitZ0zWR9d0xdSpclF5nXoTvfh1W4gf3CcsRMFrs=; b=hf9EGwFwq1mM0RjoBBHKik0M0gqACVX5SnXbODHnXh5y9Dedh/lF9x2wWvlCwNI7vdjrng sna6yGSxRcUeKCed+AgQnnSSYY89b4HUElNEeY8Y0hVo0POLbUMiCw57wl6TLlNyZX6d+K 5UqAH1nrUV0Kj7ZUCF3b4MRHkFoU4fY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-77-24P3RzDFO_mxSjhEWrsDxg-1; Thu, 29 Feb 2024 13:37:00 -0500 X-MC-Unique: 24P3RzDFO_mxSjhEWrsDxg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6787082DFE1; Thu, 29 Feb 2024 18:36:59 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id F21831C060AF; Thu, 29 Feb 2024 18:36:57 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 02/18] libcamera: internal: Move dma_heaps.[h, cpp] to common directories Date: Thu, 29 Feb 2024 19:36:13 +0100 Message-ID: <20240229183654.7206-3-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov DmaHeap class is useful outside the RPi pipeline handler too. Move dma_heaps.h and dma_heaps.cpp to common directories. Update the build files and RPi vc4 pipeline handler accordingly. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Kieran Bingham Reviewed-by: Naushir Patuck Reviewed-by: Pavel Machek Signed-off-by: Andrey Konovalov Signed-off-by: Hans de Goede --- .../libcamera/internal}/dma_heaps.h | 4 - include/libcamera/internal/meson.build | 1 + src/libcamera/dma_heaps.cpp | 127 ++++++++++++++++++ src/libcamera/meson.build | 1 + src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp | 90 ------------- src/libcamera/pipeline/rpi/vc4/meson.build | 1 - src/libcamera/pipeline/rpi/vc4/vc4.cpp | 5 +- 7 files changed, 131 insertions(+), 98 deletions(-) rename {src/libcamera/pipeline/rpi/vc4 => include/libcamera/internal}/dma_heaps.h (92%) create mode 100644 src/libcamera/dma_heaps.cpp delete mode 100644 src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp diff --git a/src/libcamera/pipeline/rpi/vc4/dma_heaps.h b/include/libcamera/internal/dma_heaps.h similarity index 92% rename from src/libcamera/pipeline/rpi/vc4/dma_heaps.h rename to include/libcamera/internal/dma_heaps.h index 0a4a8d86..cff8f140 100644 --- a/src/libcamera/pipeline/rpi/vc4/dma_heaps.h +++ b/include/libcamera/internal/dma_heaps.h @@ -13,8 +13,6 @@ namespace libcamera { -namespace RPi { - class DmaHeap { public: @@ -27,6 +25,4 @@ private: UniqueFD dmaHeapHandle_; }; -} /* namespace RPi */ - } /* namespace libcamera */ diff --git a/include/libcamera/internal/meson.build b/include/libcamera/internal/meson.build index 7f1f3440..33eb0fb3 100644 --- a/include/libcamera/internal/meson.build +++ b/include/libcamera/internal/meson.build @@ -25,6 +25,7 @@ libcamera_internal_headers = files([ 'device_enumerator.h', 'device_enumerator_sysfs.h', 'device_enumerator_udev.h', + 'dma_heaps.h', 'formats.h', 'framebuffer.h', 'ipa_manager.h', diff --git a/src/libcamera/dma_heaps.cpp b/src/libcamera/dma_heaps.cpp new file mode 100644 index 00000000..38ef175a --- /dev/null +++ b/src/libcamera/dma_heaps.cpp @@ -0,0 +1,127 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Raspberry Pi Ltd + * + * dma_heaps.h - Helper class for dma-heap allocations. + */ + +#include "libcamera/internal/dma_heaps.h" + +#include +#include +#include +#include + +#include +#include + +#include + +/** + * \file dma_heaps.cpp + * \brief CMA dma-heap allocator + */ + +/* + * /dev/dma_heap/linux,cma is the dma-heap allocator, which allows dmaheap-cma + * to only have to worry about importing. + * + * Annoyingly, should the cma heap size be specified on the kernel command line + * instead of DT, the heap gets named "reserved" instead. + */ +static constexpr std::array heapNames = { + "/dev/dma_heap/linux,cma", + "/dev/dma_heap/reserved" +}; + +namespace libcamera { + +LOG_DEFINE_CATEGORY(DmaHeap) + +/** + * \class DmaHeap + * \brief Helper class for CMA dma-heap allocations + */ + +/** + * \brief Construct a DmaHeap that owns a CMA dma-heap file descriptor + * + * Goes through the internal list of possible names of the CMA dma-heap devices + * until a CMA dma-heap device is successfully opened. If it fails to open any + * dma-heap device, an invalid DmaHeap object is constructed. A valid DmaHeap + * object owns a wrapped dma-heap file descriptor. + * + * Please check the new DmaHeap object with \ref DmaHeap::isValid before using it. + */ +DmaHeap::DmaHeap() +{ + for (const char *name : heapNames) { + int ret = ::open(name, O_RDWR | O_CLOEXEC, 0); + if (ret < 0) { + ret = errno; + LOG(DmaHeap, Debug) + << "Failed to open " << name << ": " + << strerror(ret); + continue; + } + + dmaHeapHandle_ = UniqueFD(ret); + break; + } + + if (!dmaHeapHandle_.isValid()) + LOG(DmaHeap, Error) << "Could not open any dmaHeap device"; +} + +/** + * \brief Destroy the DmaHeap instance + * + * Destroying a DmaHeap instance which owns a wrapped dma-heap file descriptor + * closes the descriptor automatically. + */ +DmaHeap::~DmaHeap() = default; + +/** + * \fn DmaHeap::isValid() + * \brief Check if the DmaHeap instance is valid + * \return True if the DmaHeap is valid, false otherwise + */ + +/** + * \brief Allocate a dma-buf from the DmaHeap + * \param [in] name The name to set for the allocated buffer + * \param [in] size The size of the buffer to allocate + * \return The \ref UniqueFD of the allocated buffer + * + * Allocates a dma-buf with read/write access. + * If the allocation fails returns invalid UniqueFD. + */ +UniqueFD DmaHeap::alloc(const char *name, std::size_t size) +{ + int ret; + + if (!name) + return {}; + + struct dma_heap_allocation_data alloc = {}; + + alloc.len = size; + alloc.fd_flags = O_CLOEXEC | O_RDWR; + + ret = ::ioctl(dmaHeapHandle_.get(), DMA_HEAP_IOCTL_ALLOC, &alloc); + if (ret < 0) { + LOG(DmaHeap, Error) << "dmaHeap allocation failure for " << name; + return {}; + } + + UniqueFD allocFd(alloc.fd); + ret = ::ioctl(allocFd.get(), DMA_BUF_SET_NAME, name); + if (ret < 0) { + LOG(DmaHeap, Error) << "dmaHeap naming failure for " << name; + return {}; + } + + return allocFd; +} + +} /* namespace libcamera */ diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build index 45f63e93..3c5e43df 100644 --- a/src/libcamera/meson.build +++ b/src/libcamera/meson.build @@ -17,6 +17,7 @@ libcamera_sources = files([ 'delayed_controls.cpp', 'device_enumerator.cpp', 'device_enumerator_sysfs.cpp', + 'dma_heaps.cpp', 'fence.cpp', 'formats.cpp', 'framebuffer.cpp', diff --git a/src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp b/src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp deleted file mode 100644 index 317b1fc1..00000000 --- a/src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp +++ /dev/null @@ -1,90 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1-or-later */ -/* - * Copyright (C) 2020, Raspberry Pi Ltd - * - * dma_heaps.h - Helper class for dma-heap allocations. - */ - -#include "dma_heaps.h" - -#include -#include -#include -#include -#include -#include - -#include - -/* - * /dev/dma-heap/linux,cma is the dma-heap allocator, which allows dmaheap-cma - * to only have to worry about importing. - * - * Annoyingly, should the cma heap size be specified on the kernel command line - * instead of DT, the heap gets named "reserved" instead. - */ -static constexpr std::array heapNames = { - "/dev/dma_heap/linux,cma", - "/dev/dma_heap/reserved" -}; - -namespace libcamera { - -LOG_DECLARE_CATEGORY(RPI) - -namespace RPi { - -DmaHeap::DmaHeap() -{ - for (const char *name : heapNames) { - int ret = ::open(name, O_RDWR | O_CLOEXEC, 0); - if (ret < 0) { - ret = errno; - LOG(RPI, Debug) << "Failed to open " << name << ": " - << strerror(ret); - continue; - } - - dmaHeapHandle_ = UniqueFD(ret); - break; - } - - if (!dmaHeapHandle_.isValid()) - LOG(RPI, Error) << "Could not open any dmaHeap device"; -} - -DmaHeap::~DmaHeap() = default; - -UniqueFD DmaHeap::alloc(const char *name, std::size_t size) -{ - int ret; - - if (!name) - return {}; - - struct dma_heap_allocation_data alloc = {}; - - alloc.len = size; - alloc.fd_flags = O_CLOEXEC | O_RDWR; - - ret = ::ioctl(dmaHeapHandle_.get(), DMA_HEAP_IOCTL_ALLOC, &alloc); - if (ret < 0) { - LOG(RPI, Error) << "dmaHeap allocation failure for " - << name; - return {}; - } - - UniqueFD allocFd(alloc.fd); - ret = ::ioctl(allocFd.get(), DMA_BUF_SET_NAME, name); - if (ret < 0) { - LOG(RPI, Error) << "dmaHeap naming failure for " - << name; - return {}; - } - - return allocFd; -} - -} /* namespace RPi */ - -} /* namespace libcamera */ diff --git a/src/libcamera/pipeline/rpi/vc4/meson.build b/src/libcamera/pipeline/rpi/vc4/meson.build index cdb049c5..386e2296 100644 --- a/src/libcamera/pipeline/rpi/vc4/meson.build +++ b/src/libcamera/pipeline/rpi/vc4/meson.build @@ -1,7 +1,6 @@ # SPDX-License-Identifier: CC0-1.0 libcamera_sources += files([ - 'dma_heaps.cpp', 'vc4.cpp', ]) diff --git a/src/libcamera/pipeline/rpi/vc4/vc4.cpp b/src/libcamera/pipeline/rpi/vc4/vc4.cpp index 6b6ecd3d..e51d54c0 100644 --- a/src/libcamera/pipeline/rpi/vc4/vc4.cpp +++ b/src/libcamera/pipeline/rpi/vc4/vc4.cpp @@ -12,12 +12,11 @@ #include #include "libcamera/internal/device_enumerator.h" +#include "libcamera/internal/dma_heaps.h" #include "../common/pipeline_base.h" #include "../common/rpi_stream.h" -#include "dma_heaps.h" - using namespace std::chrono_literals; namespace libcamera { @@ -87,7 +86,7 @@ public: RPi::Device isp_; /* DMAHEAP allocation helper. */ - RPi::DmaHeap dmaHeap_; + DmaHeap dmaHeap_; SharedFD lsTable_; struct Config { From patchwork Thu Feb 29 18:36:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19573 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 21D75C3260 for ; Thu, 29 Feb 2024 18:37:06 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 68D3862867; Thu, 29 Feb 2024 19:37:05 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="c6J8V2zh"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D44D462868 for ; Thu, 29 Feb 2024 19:37:03 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231822; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i8a6rvDnjyUzJvzAnQCPnxvyftgBl6a4uc/OiLbTtz0=; b=c6J8V2zh8zm6EcGYorwI1ABL8HXvZ3QReldQa5/pPfutq/VoQh2uVUHei/TdwALVnhDm+R rwPgaCtFhCwEG6fuLSttEKec4uxTKMYTMjqHszCeS4sEPGc9rW77yMem9tcr6HDqJuGrUp dsITEXQ54EbRO51UQMpfmeG6vsG8qUY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-474-SjZ0M7FKO3mPx2dVUeaomQ-1; Thu, 29 Feb 2024 13:37:01 -0500 X-MC-Unique: SjZ0M7FKO3mPx2dVUeaomQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AE53682DFE0; Thu, 29 Feb 2024 18:37:00 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 981B01C060B1; Thu, 29 Feb 2024 18:36:59 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 03/18] libcamera: dma_heaps: extend DmaHeap class to support system heap Date: Thu, 29 Feb 2024 19:36:14 +0100 Message-ID: <20240229183654.7206-4-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov Add an argument to the constructor to specify dma heaps type(s) to use. Can be DmaHeapFlag::Cma and/or DmaHeapFlag::System. By default DmaHeapFlag::Cma is used. If both DmaHeapFlag::Cma and DmaHeapFlag::System are set, CMA heap is tried first. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Milan Zamazal Reviewed-by: Pavel Machek Signed-off-by: Andrey Konovalov Signed-off-by: Hans de Goede --- include/libcamera/internal/dma_heaps.h | 12 ++++- src/libcamera/dma_heaps.cpp | 67 ++++++++++++++++++++------ 2 files changed, 63 insertions(+), 16 deletions(-) diff --git a/include/libcamera/internal/dma_heaps.h b/include/libcamera/internal/dma_heaps.h index cff8f140..80bf29e7 100644 --- a/include/libcamera/internal/dma_heaps.h +++ b/include/libcamera/internal/dma_heaps.h @@ -9,6 +9,7 @@ #include +#include #include namespace libcamera { @@ -16,7 +17,14 @@ namespace libcamera { class DmaHeap { public: - DmaHeap(); + enum class DmaHeapFlag { + Cma = 1 << 0, + System = 1 << 1, + }; + + using DmaHeapFlags = Flags; + + DmaHeap(DmaHeapFlags flags = DmaHeapFlag::Cma); ~DmaHeap(); bool isValid() const { return dmaHeapHandle_.isValid(); } UniqueFD alloc(const char *name, std::size_t size); @@ -25,4 +33,6 @@ private: UniqueFD dmaHeapHandle_; }; +LIBCAMERA_FLAGS_ENABLE_OPERATORS(DmaHeap::DmaHeapFlag) + } /* namespace libcamera */ diff --git a/src/libcamera/dma_heaps.cpp b/src/libcamera/dma_heaps.cpp index 38ef175a..d0e33ce6 100644 --- a/src/libcamera/dma_heaps.cpp +++ b/src/libcamera/dma_heaps.cpp @@ -19,9 +19,11 @@ /** * \file dma_heaps.cpp - * \brief CMA dma-heap allocator + * \brief dma-heap allocator */ +namespace libcamera { + /* * /dev/dma_heap/linux,cma is the dma-heap allocator, which allows dmaheap-cma * to only have to worry about importing. @@ -29,42 +31,77 @@ * Annoyingly, should the cma heap size be specified on the kernel command line * instead of DT, the heap gets named "reserved" instead. */ -static constexpr std::array heapNames = { - "/dev/dma_heap/linux,cma", - "/dev/dma_heap/reserved" + +/** + * \struct DmaHeapInfo + * \brief Tells what type of dma-heap the dma-heap represented by the device node name is + * \var DmaHeapInfo::flag + * \brief The type of the dma-heap + * \var DmaHeapInfo::name + * \brief The dma-heap's device node name + */ +struct DmaHeapInfo { + DmaHeap::DmaHeapFlag flag; + const char *name; }; -namespace libcamera { +static constexpr std::array heapInfos = { + { /* CMA heap names first */ + { DmaHeap::DmaHeapFlag::Cma, "/dev/dma_heap/linux,cma" }, + { DmaHeap::DmaHeapFlag::Cma, "/dev/dma_heap/reserved" }, + { DmaHeap::DmaHeapFlag::System, "/dev/dma_heap/system" } } +}; LOG_DEFINE_CATEGORY(DmaHeap) /** * \class DmaHeap - * \brief Helper class for CMA dma-heap allocations + * \brief Helper class for dma-heap allocations */ /** - * \brief Construct a DmaHeap that owns a CMA dma-heap file descriptor + * \enum DmaHeap::DmaHeapFlag + * \brief Type of the dma-heap + * \var DmaHeap::Cma + * \brief Allocate from a CMA dma-heap + * \var DmaHeap::System + * \brief Allocate from the system dma-heap + */ + +/** + * \typedef DmaHeap::DmaHeapFlags + * \brief A bitwise combination of DmaHeap::DmaHeapFlag values + */ + +/** + * \brief Construct a DmaHeap that owns a CMA or system dma-heap file descriptor + * \param [in] flags The type(s) of the dma-heap(s) to allocate from * - * Goes through the internal list of possible names of the CMA dma-heap devices - * until a CMA dma-heap device is successfully opened. If it fails to open any - * dma-heap device, an invalid DmaHeap object is constructed. A valid DmaHeap - * object owns a wrapped dma-heap file descriptor. + * By default \a flags are set to DmaHeap::DmaHeapFlag::Cma. The constructor goes + * through the internal list of possible names of the CMA and system dma-heap devices + * until the dma-heap device of the requested type is successfully opened. If more + * than one dma-heap type is specified in flags the CMA heap is tried first. If it + * fails to open any dma-heap device an invalid DmaHeap object is constructed. + * A valid DmaHeap object owns a wrapped dma-heap file descriptor. * * Please check the new DmaHeap object with \ref DmaHeap::isValid before using it. */ -DmaHeap::DmaHeap() +DmaHeap::DmaHeap(DmaHeapFlags flags) { - for (const char *name : heapNames) { - int ret = ::open(name, O_RDWR | O_CLOEXEC, 0); + for (const auto &info : heapInfos) { + if (!(flags & info.flag)) + continue; + + int ret = ::open(info.name, O_RDWR | O_CLOEXEC, 0); if (ret < 0) { ret = errno; LOG(DmaHeap, Debug) - << "Failed to open " << name << ": " + << "Failed to open " << info.name << ": " << strerror(ret); continue; } + LOG(DmaHeap, Debug) << "Using " << info.name; dmaHeapHandle_ = UniqueFD(ret); break; } From patchwork Thu Feb 29 18:36:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19575 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 472F2BD160 for ; Thu, 29 Feb 2024 18:37:11 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0150B628AC; Thu, 29 Feb 2024 19:37:11 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="SvpLsj3v"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 559EB62872 for ; Thu, 29 Feb 2024 19:37:08 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231827; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nvmetijkEVtRUnqZqiD6jk/RwERJw1bedWgNe+9Yybs=; b=SvpLsj3v+S9zUOyWIMm6cpE5J0Y1mcwZ8xWzlAbHWLfcVQRJFc+RISrUYUvAJRk+nozdPv 6fBiUpvPD2Pnld/Nz4LSR0QWmSJuEMeiwVpGNF7VVyVFJZQhGWTvEgfFXQqFeDrQzXlMcK XmVIf9hu2XlvVhrtI6KeTKpaMgtiZEo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-453-QhsPJqF0Pgqr7iJ8btEvVQ-1; Thu, 29 Feb 2024 13:37:02 -0500 X-MC-Unique: QhsPJqF0Pgqr7iJ8btEvVQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 00D65811E81; Thu, 29 Feb 2024 18:37:02 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id E14401C06710; Thu, 29 Feb 2024 18:37:00 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 04/18] libcamera: internal: Move SharedMemObject class to a common directory Date: Thu, 29 Feb 2024 19:36:15 +0100 Message-ID: <20240229183654.7206-5-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov Move SharedMemObject class out of RPi namespace and put it into include/libcamera/internal so that everyone could use it. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Signed-off-by: Andrey Konovalov Signed-off-by: Hans de Goede --- include/libcamera/internal/meson.build | 1 + .../common => include/libcamera/internal}/shared_mem_object.h | 4 ---- 2 files changed, 1 insertion(+), 4 deletions(-) rename {src/libcamera/pipeline/rpi/common => include/libcamera/internal}/shared_mem_object.h (98%) diff --git a/include/libcamera/internal/meson.build b/include/libcamera/internal/meson.build index 33eb0fb3..5807dfd9 100644 --- a/include/libcamera/internal/meson.build +++ b/include/libcamera/internal/meson.build @@ -39,6 +39,7 @@ libcamera_internal_headers = files([ 'process.h', 'pub_key.h', 'request.h', + 'shared_mem_object.h', 'source_paths.h', 'sysfs.h', 'v4l2_device.h', diff --git a/src/libcamera/pipeline/rpi/common/shared_mem_object.h b/include/libcamera/internal/shared_mem_object.h similarity index 98% rename from src/libcamera/pipeline/rpi/common/shared_mem_object.h rename to include/libcamera/internal/shared_mem_object.h index aa56c220..bfb639ee 100644 --- a/src/libcamera/pipeline/rpi/common/shared_mem_object.h +++ b/include/libcamera/internal/shared_mem_object.h @@ -19,8 +19,6 @@ namespace libcamera { -namespace RPi { - template class SharedMemObject { @@ -123,6 +121,4 @@ private: T *obj_; }; -} /* namespace RPi */ - } /* namespace libcamera */ From patchwork Thu Feb 29 18:36:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19576 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C72EAC3260 for ; Thu, 29 Feb 2024 18:37:11 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8394C62874; Thu, 29 Feb 2024 19:37:11 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="i21jqm9U"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E889462868 for ; Thu, 29 Feb 2024 19:37:08 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231827; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bsBYaIS6H267us+9PC6tsECbjiethd+JbCXpmA+c3cE=; b=i21jqm9U0aiTaQUm7tdQgsAxBZIXjE7/tyWWS2yijwApzXSAU8Vbp+7GgfKTs5IvThTNEU 4PXimtFKR1IRT4q0dyHPDCUpQg40M8G3szfikQzXqEhskMhxBzgZdQ3ZXwhbvRDqaimy/S bsKeJD1A+HHGb/QQtbhvik85vya2rEM= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-543-QS1iu_eKNmWrgMZULDheBA-1; Thu, 29 Feb 2024 13:37:05 -0500 X-MC-Unique: QS1iu_eKNmWrgMZULDheBA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 691EA3C025BE; Thu, 29 Feb 2024 18:37:04 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3360F1C060B1; Thu, 29 Feb 2024 18:37:02 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 05/18] libcamera: shared_mem_object: reorganize the code and document the SharedMemObject class Date: Thu, 29 Feb 2024 19:36:16 +0100 Message-ID: <20240229183654.7206-6-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue , Dennis Bonke Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrei Konovalov Split the parts which doesn't otherwise depend on the type T or arguments Args out of the SharedMemObject class into a new SharedMem class. Doxygen documentation by Dennis Bonke and Andrei Konovalov. Reviewed-by: Pavel Machek Co-developed-by: Dennis Bonke Signed-off-by: Dennis Bonke Signed-off-by: Andrei Konovalov --- .../libcamera/internal/shared_mem_object.h | 101 ++++++---- src/libcamera/meson.build | 1 + src/libcamera/shared_mem_object.cpp | 190 ++++++++++++++++++ 3 files changed, 253 insertions(+), 39 deletions(-) create mode 100644 src/libcamera/shared_mem_object.cpp diff --git a/include/libcamera/internal/shared_mem_object.h b/include/libcamera/internal/shared_mem_object.h index bfb639ee..bb9cddcd 100644 --- a/include/libcamera/internal/shared_mem_object.h +++ b/include/libcamera/internal/shared_mem_object.h @@ -7,11 +7,8 @@ #pragma once #include -#include #include #include -#include -#include #include #include @@ -19,58 +16,92 @@ namespace libcamera { +class SharedMem +{ +public: + SharedMem() + : mem_(nullptr) + { + } + + SharedMem(const std::string &name, std::size_t size); + + SharedMem(SharedMem &&rhs) + { + this->name_ = std::move(rhs.name_); + this->fd_ = std::move(rhs.fd_); + this->mem_ = rhs.mem_; + rhs.mem_ = nullptr; + } + + virtual ~SharedMem() + { + if (mem_) + munmap(mem_, size_); + } + + /* Make SharedMem non-copyable for now. */ + LIBCAMERA_DISABLE_COPY(SharedMem) + + SharedMem &operator=(SharedMem &&rhs) + { + this->name_ = std::move(rhs.name_); + this->fd_ = std::move(rhs.fd_); + this->mem_ = rhs.mem_; + rhs.mem_ = nullptr; + return *this; + } + + const SharedFD &fd() const + { + return fd_; + } + + void *mem() const + { + return mem_; + } + +private: + std::string name_; + SharedFD fd_; + size_t size_; +protected: + void *mem_; +}; + template -class SharedMemObject +class SharedMemObject : public SharedMem { public: static constexpr std::size_t SIZE = sizeof(T); SharedMemObject() - : obj_(nullptr) + : SharedMem(), obj_(nullptr) { } template SharedMemObject(const std::string &name, Args &&...args) - : name_(name), obj_(nullptr) + : SharedMem(name, SIZE), obj_(nullptr) { - void *mem; - int ret; - - ret = memfd_create(name_.c_str(), MFD_CLOEXEC); - if (ret < 0) + if (mem_ == nullptr) return; - fd_ = SharedFD(std::move(ret)); - if (!fd_.isValid()) - return; - - ret = ftruncate(fd_.get(), SIZE); - if (ret < 0) - return; - - mem = mmap(nullptr, SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, - fd_.get(), 0); - if (mem == MAP_FAILED) - return; - - obj_ = new (mem) T(std::forward(args)...); + obj_ = new (mem_) T(std::forward(args)...); } SharedMemObject(SharedMemObject &&rhs) + : SharedMem(std::move(rhs)) { - this->name_ = std::move(rhs.name_); - this->fd_ = std::move(rhs.fd_); this->obj_ = rhs.obj_; rhs.obj_ = nullptr; } ~SharedMemObject() { - if (obj_) { + if (obj_) obj_->~T(); - munmap(obj_, SIZE); - } } /* Make SharedMemObject non-copyable for now. */ @@ -78,8 +109,7 @@ public: SharedMemObject &operator=(SharedMemObject &&rhs) { - this->name_ = std::move(rhs.name_); - this->fd_ = std::move(rhs.fd_); + SharedMem::operator=(std::move(rhs)); this->obj_ = rhs.obj_; rhs.obj_ = nullptr; return *this; @@ -105,19 +135,12 @@ public: return *obj_; } - const SharedFD &fd() const - { - return fd_; - } - explicit operator bool() const { return !!obj_; } private: - std::string name_; - SharedFD fd_; T *obj_; }; diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build index 3c5e43df..94a95ae3 100644 --- a/src/libcamera/meson.build +++ b/src/libcamera/meson.build @@ -41,6 +41,7 @@ libcamera_sources = files([ 'process.cpp', 'pub_key.cpp', 'request.cpp', + 'shared_mem_object.cpp', 'source_paths.cpp', 'stream.cpp', 'sysfs.cpp', diff --git a/src/libcamera/shared_mem_object.cpp b/src/libcamera/shared_mem_object.cpp new file mode 100644 index 00000000..44fe74c2 --- /dev/null +++ b/src/libcamera/shared_mem_object.cpp @@ -0,0 +1,190 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Raspberry Pi Ltd + * + * shared_mem_object.cpp - Helper class for shared memory allocations + */ + +#include "libcamera/internal/shared_mem_object.h" + +#include +#include + +/** + * \file shared_mem_object.cpp + * \brief Helper class for shared memory allocations + */ + +namespace libcamera { + +/** + * \class SharedMem + * \brief Helper class for allocating shared memory + * + * Memory is allocated and exposed as a SharedFD for use across IPC boundaries. + * + * SharedMem allocates the shared memory of the given size and maps it. + * To check that the shared memory was allocated and mapped successfully, one + * needs to verify that the pointer to the shared memory returned by SharedMem::mem() + * is not nullptr. + * + * To access the shared memory from another process the SharedFD should be passed + * to that process, and then the shared memory should be mapped into that process + * address space by calling mmap(). + * + * A single memfd is created for every SharedMem. If there is a need to allocate + * a large number of objects in shared memory, these objects should be grouped + * together and use the shared memory allocated by a single SharedMem object if + * possible. This will help to minimize the number of created memfd's. + */ + +/** + * \fn SharedMem::SharedMem(const std::string &name, std::size_t size) + * \brief Constructor for the SharedMem + * \param[in] name Name of the SharedMem + * \param[in] size Size of the shared memory to allocate and map + */ + +/** + * \fn SharedMem::SharedMem(SharedMem &&rhs) + * \brief Move constructor for SharedMem + * \param[in] rhs The object to move + */ + +/** + * \fn SharedMem::~SharedMem() + * \brief SharedMem destructor + * + * Unmaps the allocated shared memory. Decrements the shared memory descriptor use + * count. + */ + +/** + * \fn SharedMem &SharedMem::operator=(SharedMem &&rhs) + * \brief Move constructor for SharedMem + * \param[in] rhs The object to move + */ + +/** + * \fn const SharedFD &SharedMem::fd() const + * \brief Gets the file descriptor for the underlying shared memory + * \return The file descriptor + */ + +/** + * \fn void *SharedMem::mem() const + * \brief Gets the pointer to the underlying shared memory + * \return The pointer to the shared memory + */ + +SharedMem::SharedMem(const std::string &name, std::size_t size) + : name_(name), size_(size), mem_(nullptr) +{ + int fd = memfd_create(name_.c_str(), MFD_CLOEXEC); + if (fd < 0) + return; + + fd_ = SharedFD(std::move(fd)); + if (!fd_.isValid()) + return; + + if (ftruncate(fd_.get(), size_) < 0) + return; + + mem_ = mmap(nullptr, size, PROT_READ | PROT_WRITE, MAP_SHARED, + fd_.get(), 0); + if (mem_ == MAP_FAILED) + mem_ = nullptr; +} + +/** + * \var SharedMem::mem_ + * \brief Pointer to the shared memory allocated + */ + +/** + * \class SharedMemObject + * \brief Helper class for allocating objects in shared memory + * + * Memory is allocated and exposed as a SharedFD for use across IPC boundaries. + * + * Given the type of the object to be created in shared memory and the arguments + * to pass to this object's constructor, SharedMemObject allocates the shared memory + * of the size of the object and constructs the object in this memory. To ensure + * that the SharedMemObject was created successfully, one needs to verify that the + * overloaded bool() operator returns true. The object created in the shared memory + * can be accessed using the SharedMemObject::operator*() indirection operator. Its + * members can be accessed with the SharedMemObject::operator->() member of pointer + * operator. + * + * To access the object from another process the SharedFD should be passed to that + * process, and the shared memory should be mapped by calling mmap(). + * + * A single memfd is created for every SharedMemObject. If there is a need to allocate + * a large number of objects in shared memory, these objects should be grouped into a + * single large object to keep the number of created memfd's reasonably small. + */ + +/** + * \var SharedMemObject::SIZE + * \brief The size of the object that is going to be stored here + */ + +/** + * \fn SharedMemObject< T >::SharedMemObject(const std::string &name, Args &&...args) + * \brief Constructor for the SharedMemObject + * \param[in] name Name of the SharedMemObject + * \param[in] args Args to pass to the constructor of the object in shared memory + */ + +/** + * \fn SharedMemObject::SharedMemObject(SharedMemObject &&rhs) + * \brief Move constructor for SharedMemObject + * \param[in] rhs The object to move + */ + +/** + * \fn SharedMemObject::~SharedMemObject() + * \brief SharedMemObject destructor + * + * Destroys the object created in the shared memory and then unmaps the shared memory. + * Decrements the shared memory descriptor use count. + */ + +/** + * \fn SharedMemObject::operator=(SharedMemObject &&rhs) + * \brief Operator= for SharedMemObject + * \param[in] rhs The SharedMemObject object to take the data from + */ + +/** + * \fn SharedMemObject::operator->() + * \brief Operator-> for SharedMemObject + * \return The pointer to the object + */ + +/** + * \fn const T *SharedMemObject::operator->() const + * \brief Operator-> for SharedMemObject + * \return The pointer to the const object + */ + +/** + * \fn SharedMemObject::operator*() + * \brief Operator* for SharedMemObject + * \return The reference to the object + */ + +/** + * \fn const T &SharedMemObject::operator*() const + * \brief Operator* for SharedMemObject + * \return Const reference to the object + */ + +/** + * \fn SharedMemObject::operator bool() + * \brief Operator bool() for SharedMemObject + * \return True if the object was created OK in the shared memory, false otherwise + */ + +} // namespace libcamera From patchwork Thu Feb 29 18:36:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19577 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D168BBD160 for ; Thu, 29 Feb 2024 18:37:15 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 7697D62868; Thu, 29 Feb 2024 19:37:15 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="HTolGNmb"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 506E962868 for ; Thu, 29 Feb 2024 19:37:13 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231832; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=G6UPJKoTHTivwEjPrQmzRvX/gXaEW6vG+cGpIhqyd8M=; b=HTolGNmbexAeo+LSgKi4ZTDlXcXf/pNcrY2QMgFVK0WG95aFGOi8+FO7GCfZoaSVKl3df1 EDX5dHXvLXvMi9WMlzASLrDosuUrlbd6hqRyQZxS4rrQNFZQU+fOGsqKCLB2kCtLNZ6BUG OPSus5PJ0fHGLz2Jxe3T8Ehvi0T59FI= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-408-fQHAZfKoOMSrZpbUrmO1Uw-1; Thu, 29 Feb 2024 13:37:08 -0500 X-MC-Unique: fQHAZfKoOMSrZpbUrmO1Uw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5D5B93C5CF23; Thu, 29 Feb 2024 18:37:06 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id EF6491C060B1; Thu, 29 Feb 2024 18:37:04 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 06/18] libcamera: software_isp: Add SwStatsCpu class Date: Thu, 29 Feb 2024 19:36:17 +0100 Message-ID: <20240229183654.7206-7-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Marttico , Toon Langendam , Pavel Machek , Bryan O'Donoghue , Dennis Bonke Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a CPU based SwStats implementation for SoftwareISP / SoftIPA use. This implementation offers a configure function + functions to gather statistics on a line by line basis. This allows CPU based software debayering to call into interlace debayering and statistics gathering on a line by line bases while the input data is still hot in the cache. This implementation also allows specifying a window over which to gather statistics instead of processing the whole frame. Doxygen documentation by Dennis Bonke. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Co-developed-by: Andrey Konovalov Signed-off-by: Andrey Konovalov Co-developed-by: Pavel Machek Signed-off-by: Pavel Machek Co-developed-by: Dennis Bonke Signed-off-by: Dennis Bonke Co-developed-by: Marttico Signed-off-by: Marttico Co-developed-by: Toon Langendam Signed-off-by: Toon Langendam Signed-off-by: Hans de Goede Reviewed-by: Milan Zamazal --- Changes in v3: - Merge SwStats base-class into SwStatsCpu - Move swstats_cpu.h to src/libcamera/software_isp/ - Move documentation to .cpp file and extend it - Rename a bunch of foo_bar symbols to fooBar - Change a couple of defines and hardcoded values to constexpr - Remove statsBayer10P in swstats_cpu.cpp, instead move the loop to statsBGGR10PLine0 / statsGBRG10PLine0 - Make startFrame() and finishFrame() normal methods instead of using function pointers for these --- include/libcamera/internal/meson.build | 1 + .../internal/software_isp/meson.build | 5 + .../internal/software_isp/swisp_stats.h | 38 ++++ src/libcamera/meson.build | 1 + src/libcamera/software_isp/meson.build | 5 + src/libcamera/software_isp/swstats_cpu.cpp | 208 ++++++++++++++++++ src/libcamera/software_isp/swstats_cpu.h | 159 +++++++++++++ 7 files changed, 417 insertions(+) create mode 100644 include/libcamera/internal/software_isp/meson.build create mode 100644 include/libcamera/internal/software_isp/swisp_stats.h create mode 100644 src/libcamera/software_isp/meson.build create mode 100644 src/libcamera/software_isp/swstats_cpu.cpp create mode 100644 src/libcamera/software_isp/swstats_cpu.h diff --git a/include/libcamera/internal/meson.build b/include/libcamera/internal/meson.build index 5807dfd9..160fdc37 100644 --- a/include/libcamera/internal/meson.build +++ b/include/libcamera/internal/meson.build @@ -50,3 +50,4 @@ libcamera_internal_headers = files([ ]) subdir('converter') +subdir('software_isp') diff --git a/include/libcamera/internal/software_isp/meson.build b/include/libcamera/internal/software_isp/meson.build new file mode 100644 index 00000000..66c9c3fb --- /dev/null +++ b/include/libcamera/internal/software_isp/meson.build @@ -0,0 +1,5 @@ +# SPDX-License-Identifier: CC0-1.0 + +libcamera_internal_headers += files([ + 'swisp_stats.h', +]) diff --git a/include/libcamera/internal/software_isp/swisp_stats.h b/include/libcamera/internal/software_isp/swisp_stats.h new file mode 100644 index 00000000..afe42c9a --- /dev/null +++ b/include/libcamera/internal/software_isp/swisp_stats.h @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * + * swisp_stats.h - Statistics data format used by the software ISP and software IPA + */ + +#pragma once + +namespace libcamera { + +/** + * \brief Struct that holds the statistics for the Software ISP. + */ +struct SwIspStats { + /** + * \brief Holds the sum of all sampled red pixels. + */ + unsigned long sumR_; + /** + * \brief Holds the sum of all sampled green pixels. + */ + unsigned long sumG_; + /** + * \brief Holds the sum of all sampled blue pixels. + */ + unsigned long sumB_; + /** + * \brief Number of bins in the yHistogram. + */ + static constexpr unsigned int kYHistogramSize = 16; + /** + * \brief A histogram of luminance values. + */ + std::array yHistogram; +}; + +} /* namespace libcamera */ diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build index 94a95ae3..91e4cc60 100644 --- a/src/libcamera/meson.build +++ b/src/libcamera/meson.build @@ -71,6 +71,7 @@ subdir('converter') subdir('ipa') subdir('pipeline') subdir('proxy') +subdir('software_isp') null_dep = dependency('', required : false) diff --git a/src/libcamera/software_isp/meson.build b/src/libcamera/software_isp/meson.build new file mode 100644 index 00000000..e1fb8ccc --- /dev/null +++ b/src/libcamera/software_isp/meson.build @@ -0,0 +1,5 @@ +# SPDX-License-Identifier: CC0-1.0 + +libcamera_sources += files([ + 'swstats_cpu.cpp', +]) diff --git a/src/libcamera/software_isp/swstats_cpu.cpp b/src/libcamera/software_isp/swstats_cpu.cpp new file mode 100644 index 00000000..448d0e4c --- /dev/null +++ b/src/libcamera/software_isp/swstats_cpu.cpp @@ -0,0 +1,208 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * Copyright (C) 2023, Red Hat Inc. + * + * Authors: + * Hans de Goede + * + * swstats_cpu.cpp - CPU based software statistics implementation + */ + +#include "swstats_cpu.h" + +#include + +#include + +#include "libcamera/internal/bayer_format.h" + +namespace libcamera { + +/** + * \class SwStatsCpu + * \brief Class for gathering statistics on the CPU + * + * CPU based software ISP statistics implementation. + * + * This class offers a configure function + functions to gather statistics + * on a line by line basis. This allows CPU based software debayering to + * interlace debayering and statistics gathering on a line by line basis + * while the input data is still hot in the cache. + * + * It is also possible to specify a window over which to gather + * statistics instead of processing the whole frame. + */ + +LOG_DEFINE_CATEGORY(SwStatsCpu) + +SwStatsCpu::SwStatsCpu() +{ + sharedStats_ = SharedMemObject("softIsp_stats"); + if (!sharedStats_.fd().isValid()) + LOG(SwStatsCpu, Error) + << "Failed to create shared memory for statistics"; +} + +static const unsigned int kRedYMul = 77; /* 0.299 * 256 */ +static const unsigned int kGreenYMul = 150; /* 0.587 * 256 */ +static const unsigned int kBlueYMul = 29; /* 0.114 * 256 */ + +#define SWSTATS_START_LINE_STATS(pixel_t) \ + pixel_t r, g, g2, b; \ + unsigned int yVal; \ + \ + unsigned int sumR = 0; \ + unsigned int sumG = 0; \ + unsigned int sumB = 0; + +#define SWSTATS_ACCUMULATE_LINE_STATS(div) \ + sumR += r; \ + sumG += g; \ + sumB += b; \ + \ + yVal = r * kRedYMul; \ + yVal += g * kGreenYMul; \ + yVal += b * kBlueYMul; \ + stats_.yHistogram[yVal * SwIspStats::kYHistogramSize / (256 * 256 * (div))]++; + +#define SWSTATS_FINISH_LINE_STATS() \ + stats_.sumR_ += sumR; \ + stats_.sumG_ += sumG; \ + stats_.sumB_ += sumB; + +void SwStatsCpu::statsBGGR10PLine0(const uint8_t *src[]) +{ + const uint8_t *src0 = src[1] + window_.x * 5 / 4; + const uint8_t *src1 = src[2] + window_.x * 5 / 4; + const int widthInBytes = window_.width * 5 / 4; + + if (swapLines_) + std::swap(src0, src1); + + SWSTATS_START_LINE_STATS(uint8_t) + + /* x += 5 sample every other 2x2 block */ + for (int x = 0; x < widthInBytes; x += 5) { + /* BGGR */ + b = src0[x]; + g = src0[x + 1]; + g2 = src1[x]; + r = src1[x + 1]; + g = (g + g2) / 2; + /* Data is already 8 bits, divide by 1 */ + SWSTATS_ACCUMULATE_LINE_STATS(1) + } + + SWSTATS_FINISH_LINE_STATS() +} + +void SwStatsCpu::statsGBRG10PLine0(const uint8_t *src[]) +{ + const uint8_t *src0 = src[1] + window_.x * 5 / 4; + const uint8_t *src1 = src[2] + window_.x * 5 / 4; + const int widthInBytes = window_.width * 5 / 4; + + if (swapLines_) + std::swap(src0, src1); + + SWSTATS_START_LINE_STATS(uint8_t) + + /* x += 5 sample every other 2x2 block */ + for (int x = 0; x < widthInBytes; x += 5) { + /* GBRG */ + g = src0[x]; + b = src0[x + 1]; + r = src1[x]; + g2 = src1[x + 1]; + g = (g + g2) / 2; + /* Data is already 8 bits, divide by 1 */ + SWSTATS_ACCUMULATE_LINE_STATS(1) + } + + SWSTATS_FINISH_LINE_STATS() +} + +/** + * \brief Reset state to start statistics gathering for a new frame. + * + * This may only be called after a successful setWindow() call. + */ +void SwStatsCpu::startFrame(void) +{ + stats_.sumR_ = 0; + stats_.sumB_ = 0; + stats_.sumG_ = 0; + stats_.yHistogram.fill(0); +} + +/** + * \brief Finish statistics calculation for the current frame. + * + * This may only be called after a successful setWindow() call. + */ +void SwStatsCpu::finishFrame(void) +{ + *sharedStats_ = stats_; + statsReady.emit(0); +} + +/** + * \brief Configure the statistics object for the passed in input format. + * \param[in] inputCfg The input format + * + * \return 0 on success, a negative errno value on failure + */ +int SwStatsCpu::configure(const StreamConfiguration &inputCfg) +{ + BayerFormat bayerFormat = + BayerFormat::fromPixelFormat(inputCfg.pixelFormat); + + if (bayerFormat.bitDepth == 10 && + bayerFormat.packing == BayerFormat::Packing::CSI2) { + patternSize_.height = 2; + patternSize_.width = 4; /* 5 bytes per *4* pixels */ + /* Skip every 3th and 4th line, sample every other 2x2 block */ + ySkipMask_ = 0x02; + xShift_ = 0; + + switch (bayerFormat.order) { + case BayerFormat::BGGR: + case BayerFormat::GRBG: + stats0_ = &SwStatsCpu::statsBGGR10PLine0; + swapLines_ = bayerFormat.order == BayerFormat::GRBG; + return 0; + case BayerFormat::GBRG: + case BayerFormat::RGGB: + stats0_ = &SwStatsCpu::statsGBRG10PLine0; + swapLines_ = bayerFormat.order == BayerFormat::RGGB; + return 0; + default: + break; + } + } + + LOG(SwStatsCpu, Info) + << "Unsupported input format " << inputCfg.pixelFormat.toString(); + return -EINVAL; +} + +/** + * \brief Specify window coordinates over which to gather statistics. + * \param[in] window The window object. + */ +void SwStatsCpu::setWindow(Rectangle window) +{ + window_ = window; + + window_.x &= ~(patternSize_.width - 1); + window_.x += xShift_; + window_.y &= ~(patternSize_.height - 1); + + /* width_ - xShift_ to make sure the window fits */ + window_.width -= xShift_; + window_.width &= ~(patternSize_.width - 1); + window_.height &= ~(patternSize_.height - 1); +} + +} /* namespace libcamera */ diff --git a/src/libcamera/software_isp/swstats_cpu.h b/src/libcamera/software_isp/swstats_cpu.h new file mode 100644 index 00000000..df13263b --- /dev/null +++ b/src/libcamera/software_isp/swstats_cpu.h @@ -0,0 +1,159 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * Copyright (C) 2023, Red Hat Inc. + * + * Authors: + * Hans de Goede + * + * swstats_cpu.h - CPU based software statistics implementation + */ + +#pragma once + +#include + +#include + +#include + +#include "libcamera/internal/shared_mem_object.h" +#include "libcamera/internal/software_isp/swisp_stats.h" + +namespace libcamera { + +class PixelFormat; +struct StreamConfiguration; + +class SwStatsCpu +{ +public: + SwStatsCpu(); + ~SwStatsCpu() = default; + + /** + * \brief Gets wether the statistics object is valid. + * + * \return true if it's valid, false otherwise + */ + bool isValid() const { return sharedStats_.fd().isValid(); } + + /** + * \brief Get the file descriptor for the statistics. + * + * \return the file descriptor + */ + const SharedFD &getStatsFD() { return sharedStats_.fd(); } + + /** + * \brief Get the pattern size. + * + * For some input-formats, e.g. Bayer data, processing is done multiple lines + * and/or columns at a time. Get width and height at which the (bayer) pattern + * repeats. Window values are rounded down to a multiple of this and the height + * also indicates if processLine2() should be called or not. + * This may only be called after a successful configure() call. + * + * \return the pattern size + */ + const Size &patternSize() { return patternSize_; } + + int configure(const StreamConfiguration &inputCfg); + void setWindow(Rectangle window); + void startFrame(); + void finishFrame(); + + /** + * \brief Process line 0. + * \param[in] y The y coordinate. + * \param[in] src The input data. + * + * This function processes line 0 for input formats with patternSize height == 1. + * It'll process line 0 and 1 for input formats with patternSize height >= 2. + * This function may only be called after a successful setWindow() call. + */ + void processLine0(unsigned int y, const uint8_t *src[]) + { + if ((y & ySkipMask_) || y < (unsigned int)window_.y || + y >= (window_.y + window_.height)) + return; + + (this->*stats0_)(src); + } + + /** + * \brief Process line 2 and 3. + * \param[in] y The y coordinate. + * \param[in] src The input data. + * + * This function processes line 2 and 3 for input formats with patternSize height == 4. + * This function may only be called after a successful setWindow() call. + */ + void processLine2(unsigned int y, const uint8_t *src[]) + { + if ((y & ySkipMask_) || y < (unsigned int)window_.y || + y >= (window_.y + window_.height)) + return; + + (this->*stats2_)(src); + } + + /** + * \brief Signals that the statistics are ready. + * + * The int parameter isn't actually used. + */ + Signal statsReady; + +private: + /** + * \brief Called when there is data to get statistics from. + * \param[in] src The input data + * + * These functions take an array of (patternSize_.height + 1) src + * pointers each pointing to a line in the source image. The middle + * element of the array will point to the actual line being processed. + * Earlier element(s) will point to the previous line(s) and later + * element(s) to the next line(s). + * + * See the documentation of DebayerCpu::debayerFn for more details. + */ + using statsProcessFn = void (SwStatsCpu::*)(const uint8_t *src[]); + + void statsBGGR10PLine0(const uint8_t *src[]); + void statsGBRG10PLine0(const uint8_t *src[]); + + /* Variables set by configure(), used every line */ + statsProcessFn stats0_; + statsProcessFn stats2_; + bool swapLines_; + + /** + * \brief Skip lines where this bitmask is set in y. + */ + unsigned int ySkipMask_; + + /** + * \brief Statistics window, set by setWindow(), used ever line. + */ + Rectangle window_; + + /** + * \brief The size of the bayer pattern. + * + * Valid sizes are: 2x2, 4x2 or 4x4. + */ + Size patternSize_; + + /** + * \brief The offset of x, applied to window_.x for bayer variants. + * + * This can either be 0 or 1. + */ + unsigned int xShift_; + + SharedMemObject sharedStats_; + SwIspStats stats_; +}; + +} /* namespace libcamera */ From patchwork Thu Feb 29 18:36:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19578 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B18B1C3260 for ; Thu, 29 Feb 2024 18:37:16 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 6804D62876; Thu, 29 Feb 2024 19:37:16 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="dmEAFV2j"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id F1B4062868 for ; Thu, 29 Feb 2024 19:37:13 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231833; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=M5wKa4MbKUa4G6tPgYUTOSoeeGaWk+A4NCSElR4JASo=; b=dmEAFV2jXAeYjJQWa+iGG9B9RpZ49PXBOppU6dbVjPRnTt6d8sWCaEHjgVGgqUGsbEW7D4 qT8JXnO4p68WXm6fVsTB0L/OvFeMY+r4R/fRqcUlxiUt3I79hb9Ed02TyuPnXgiXn2SU1r cYsFsM1Tvxqpdn8QYue/Jv2LdnjWa/w= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-458-keRlyK3KML6FRBnIGUFnpQ-1; Thu, 29 Feb 2024 13:37:08 -0500 X-MC-Unique: keRlyK3KML6FRBnIGUFnpQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BA18D1C05EA6; Thu, 29 Feb 2024 18:37:07 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8F3571C060AF; Thu, 29 Feb 2024 18:37:06 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 07/18] libcamera: software_isp: Add Debayer base class Date: Thu, 29 Feb 2024 19:36:18 +0100 Message-ID: <20240229183654.7206-8-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue , Dennis Bonke Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a base class for debayer implementations. This is intended to be suitable for both GPU (or otherwise) accelerated debayer implementations as well as CPU based debayering. Doxygen documentation by Dennis Bonke. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Co-developed-by: Dennis Bonke Signed-off-by: Dennis Bonke Co-developed-by: Andrey Konovalov Signed-off-by: Andrey Konovalov Signed-off-by: Hans de Goede Reviewed-by: Milan Zamazal --- Changes in v3: - Move debayer.h to src/libcamera/software_isp/ - Move class documentation (only a brief for now) to .cpp file --- .../internal/software_isp/debayer_params.h | 48 ++++++++ .../internal/software_isp/meson.build | 1 + src/libcamera/software_isp/debayer.cpp | 29 +++++ src/libcamera/software_isp/debayer.h | 104 ++++++++++++++++++ src/libcamera/software_isp/meson.build | 1 + 5 files changed, 183 insertions(+) create mode 100644 include/libcamera/internal/software_isp/debayer_params.h create mode 100644 src/libcamera/software_isp/debayer.cpp create mode 100644 src/libcamera/software_isp/debayer.h diff --git a/include/libcamera/internal/software_isp/debayer_params.h b/include/libcamera/internal/software_isp/debayer_params.h new file mode 100644 index 00000000..98e7fc92 --- /dev/null +++ b/include/libcamera/internal/software_isp/debayer_params.h @@ -0,0 +1,48 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Red Hat Inc. + * + * Authors: + * Hans de Goede + * + * swstats.h - software statistics base class + */ + +#pragma once + +namespace libcamera { + +/** + * \brief Struct to hold the debayer parameters. + */ +struct DebayerParams { + /** + * \brief const value for 1.0 gain + */ + static constexpr unsigned int kGain10 = 256; + + /** + * \brief Red Gain + * + * 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. + */ + unsigned int gainR; + /** + * \brief Green Gain + * + * 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. + */ + unsigned int gainG; + /** + * \brief Blue Gain + * + * 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. + */ + unsigned int gainB; + /** + * \brief Gamma correction, 1.0 is no correction + */ + float gamma; +}; + +} /* namespace libcamera */ diff --git a/include/libcamera/internal/software_isp/meson.build b/include/libcamera/internal/software_isp/meson.build index 66c9c3fb..a620e16d 100644 --- a/include/libcamera/internal/software_isp/meson.build +++ b/include/libcamera/internal/software_isp/meson.build @@ -1,5 +1,6 @@ # SPDX-License-Identifier: CC0-1.0 libcamera_internal_headers += files([ + 'debayer_params.h', 'swisp_stats.h', ]) diff --git a/src/libcamera/software_isp/debayer.cpp b/src/libcamera/software_isp/debayer.cpp new file mode 100644 index 00000000..64f0b5a0 --- /dev/null +++ b/src/libcamera/software_isp/debayer.cpp @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * Copyright (C) 2023, Red Hat Inc. + * + * Authors: + * Hans de Goede + * + * debayer.cpp - debayer base class + */ + +#include "debayer.h" + +namespace libcamera { + +/** + * \class Debayer + * \brief Base debayering class + * + * Base class that provides functions for setting up the debayering process. + */ + +LOG_DEFINE_CATEGORY(Debayer) + +Debayer::~Debayer() +{ +} + +} /* namespace libcamera */ diff --git a/src/libcamera/software_isp/debayer.h b/src/libcamera/software_isp/debayer.h new file mode 100644 index 00000000..8880ff99 --- /dev/null +++ b/src/libcamera/software_isp/debayer.h @@ -0,0 +1,104 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * Copyright (C) 2023, Red Hat Inc. + * + * Authors: + * Hans de Goede + * + * debayer.h - debayering base class + */ + +#pragma once + +#include + +#include +#include + +#include +#include + +#include "libcamera/internal/software_isp/debayer_params.h" + +namespace libcamera { + +class FrameBuffer; + +LOG_DECLARE_CATEGORY(Debayer) + +class Debayer +{ +public: + virtual ~Debayer() = 0; + + /** + * \brief Configure the debayer object according to the passed in parameters. + * \param[in] inputCfg The input configuration. + * \param[in] outputCfgs The output configurations. + * + * \return 0 on success, a negative errno on failure. + */ + virtual int configure(const StreamConfiguration &inputCfg, + const std::vector> &outputCfgs) = 0; + + /** + * \brief Get the width and height at which the bayer pattern repeats. + * \param[in] inputFormat The input format. + * + * Valid sizes are: 2x2, 4x2 or 4x4. + * + * \return pattern size or an empty size for unsupported inputFormats. + */ + virtual Size patternSize(PixelFormat inputFormat) = 0; + + /** + * \brief Get the supported output formats. + * \param[in] inputFormat The input format. + * + * \return all supported output formats or an empty vector if there are none. + */ + virtual std::vector formats(PixelFormat inputFormat) = 0; + + /** + * \brief Get the stride and the frame size. + * \param[in] outputFormat The output format. + * \param[in] size The output size. + * + * \return a tuple of the stride and the frame size, or a tuple with 0,0 if there is no valid output config. + */ + virtual std::tuple + strideAndFrameSize(const PixelFormat &outputFormat, const Size &size) = 0; + + /** + * \brief Process the bayer data into the requested format. + * \param[in] input The input buffer. + * \param[in] output The output buffer. + * \param[in] params The parameters to be used in debayering. + * + * \note DebayerParams is passed by value deliberately so that a copy is passed + * when this is run in another thread by invokeMethod(). + */ + virtual void process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) = 0; + + /** + * \brief Get the supported output sizes for the given input format and size. + * \param[in] inputFormat The input format. + * \param[in] inputSize The input size. + * + * \return The valid size ranges or an empty range if there are none. + */ + virtual SizeRange sizes(PixelFormat inputFormat, const Size &inputSize) = 0; + + /** + * \brief Signals when the input buffer is ready. + */ + Signal inputBufferReady; + + /** + * \brief Signals when the output buffer is ready. + */ + Signal outputBufferReady; +}; + +} /* namespace libcamera */ diff --git a/src/libcamera/software_isp/meson.build b/src/libcamera/software_isp/meson.build index e1fb8ccc..92fc90f3 100644 --- a/src/libcamera/software_isp/meson.build +++ b/src/libcamera/software_isp/meson.build @@ -1,5 +1,6 @@ # SPDX-License-Identifier: CC0-1.0 libcamera_sources += files([ + 'debayer.cpp', 'swstats_cpu.cpp', ]) From patchwork Thu Feb 29 18:36:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19579 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4CF83BD160 for ; Thu, 29 Feb 2024 18:37:18 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0B90762878; Thu, 29 Feb 2024 19:37:18 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="R/GvNHB0"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 952AA62876 for ; Thu, 29 Feb 2024 19:37:14 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231833; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q2NxR4jSh0dXG0JtPdTGBxCb5PUz0H9e3VLY7BIFcrI=; b=R/GvNHB0O7/EWSu8//Cic/TYpYN7QeM2kBuT0uXu+4/QjpKkZw8WGd+s1HDOOlOpM0QFFO N1iOHOJ/+qihxkTO7qqWxb0xYSEjJc+QEhQ0musWGBTltnbeHXtuLlDCwodkUrPvUl67F3 QUPr+VEWFMe0deen2QPkYXgSk8XkGWM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-511-x5eVJzdaPguxrolWSjp5Sw-1; Thu, 29 Feb 2024 13:37:09 -0500 X-MC-Unique: x5eVJzdaPguxrolWSjp5Sw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 48C1A82DFEE; Thu, 29 Feb 2024 18:37:09 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id EC4241C060B1; Thu, 29 Feb 2024 18:37:07 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 08/18] libcamera: software_isp: Add DebayerCpu class Date: Thu, 29 Feb 2024 19:36:19 +0100 Message-ID: <20240229183654.7206-9-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue , Dennis Bonke Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add CPU based debayering implementation. This initial implementation only supports debayering packed 10 bits per pixel bayer data in the 4 standard bayer orders. Doxygen documentation by Dennis Bonke. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Co-developed-by: Dennis Bonke Signed-off-by: Dennis Bonke Co-developed-by: Andrey Konovalov Signed-off-by: Andrey Konovalov Co-developed-by: Pavel Machek Signed-off-by: Pavel Machek Signed-off-by: Hans de Goede --- Changes in v3: - Move debayer_cpu.h to src/libcamera/software_isp/ - Move documentation to .cpp file - Document how/why an array of src pointers is passed to the debayer functions --- src/libcamera/software_isp/debayer_cpu.cpp | 619 +++++++++++++++++++++ src/libcamera/software_isp/debayer_cpu.h | 143 +++++ src/libcamera/software_isp/meson.build | 1 + 3 files changed, 763 insertions(+) create mode 100644 src/libcamera/software_isp/debayer_cpu.cpp create mode 100644 src/libcamera/software_isp/debayer_cpu.h diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp new file mode 100644 index 00000000..53e90776 --- /dev/null +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -0,0 +1,619 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * Copyright (C) 2023, Red Hat Inc. + * + * Authors: + * Hans de Goede + * + * debayer_cpu.cpp - CPU based debayering class + */ + +#include "debayer_cpu.h" + +#include +#include +#include + +#include + +#include "libcamera/internal/bayer_format.h" +#include "libcamera/internal/framebuffer.h" +#include "libcamera/internal/mapped_framebuffer.h" + +namespace libcamera { + +/** + * \class DebayerCpu + * \brief Class for debayering on the CPU + * + * Implementation for CPU based debayering + */ + +/** + * \brief Constructs a DebayerCpu object. + * \param[in] stats Pointer to the stats object to use. + */ +DebayerCpu::DebayerCpu(std::unique_ptr stats) + : stats_(std::move(stats)), gamma_correction_(1.0) +{ +#ifdef __x86_64__ + enableInputMemcpy_ = false; +#else + enableInputMemcpy_ = true; +#endif + /* Initialize gamma to 1.0 curve */ + for (unsigned int i = 0; i < kGammaLookupSize; i++) + gamma_[i] = i / 4; + + for (unsigned int i = 0; i < kMaxLineBuffers; i++) + lineBuffers_[i] = nullptr; +} + +DebayerCpu::~DebayerCpu() +{ + for (unsigned int i = 0; i < kMaxLineBuffers; i++) + free(lineBuffers_[i]); +} + +// RGR +// GBG +// RGR +#define BGGR_BGR888(p, n, div) \ + *dst++ = blue_[curr[x] / (div)]; \ + *dst++ = green_[(prev[x] + curr[x - p] + curr[x + n] + next[x]) / (4 * (div))]; \ + *dst++ = red_[(prev[x - p] + prev[x + n] + next[x - p] + next[x + n]) / (4 * (div))]; \ + x++; + +// GBG +// RGR +// GBG +#define GRBG_BGR888(p, n, div) \ + *dst++ = blue_[(prev[x] + next[x]) / (2 * (div))]; \ + *dst++ = green_[curr[x] / (div)]; \ + *dst++ = red_[(curr[x - p] + curr[x + n]) / (2 * (div))]; \ + x++; + +// GRG +// BGB +// GRG +#define GBRG_BGR888(p, n, div) \ + *dst++ = blue_[(curr[x - p] + curr[x + n]) / (2 * (div))]; \ + *dst++ = green_[curr[x] / (div)]; \ + *dst++ = red_[(prev[x] + next[x]) / (2 * (div))]; \ + x++; + +// BGB +// GRG +// BGB +#define RGGB_BGR888(p, n, div) \ + *dst++ = blue_[(prev[x - p] + prev[x + n] + next[x - p] + next[x + n]) / (4 * (div))]; \ + *dst++ = green_[(prev[x] + curr[x - p] + curr[x + n] + next[x]) / (4 * (div))]; \ + *dst++ = red_[curr[x] / (div)]; \ + x++; + +void DebayerCpu::debayer10P_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + const int width_in_bytes = window_.width * 5 / 4; + const uint8_t *prev = (const uint8_t *)src[0]; + const uint8_t *curr = (const uint8_t *)src[1]; + const uint8_t *next = (const uint8_t *)src[2]; + + /* + * For the first pixel getting a pixel from the previous column uses + * x - 2 to skip the 5th byte with least-significant bits for 4 pixels. + * Same for last pixel (uses x + 2) and looking at the next column. + * x++ in the for-loop skips the 5th byte with 4 x 2 lsb-s for 10bit packed. + */ + for (int x = 0; x < width_in_bytes; x++) { + /* First pixel */ + BGGR_BGR888(2, 1, 1) + /* Second pixel BGGR -> GBRG */ + GBRG_BGR888(1, 1, 1) + /* Same thing for third and fourth pixels */ + BGGR_BGR888(1, 1, 1) + GBRG_BGR888(1, 2, 1) + } +} + +void DebayerCpu::debayer10P_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + const int width_in_bytes = window_.width * 5 / 4; + const uint8_t *prev = (const uint8_t *)src[0]; + const uint8_t *curr = (const uint8_t *)src[1]; + const uint8_t *next = (const uint8_t *)src[2]; + + for (int x = 0; x < width_in_bytes; x++) { + /* First pixel */ + GRBG_BGR888(2, 1, 1) + /* Second pixel GRBG -> RGGB */ + RGGB_BGR888(1, 1, 1) + /* Same thing for third and fourth pixels */ + GRBG_BGR888(1, 1, 1) + RGGB_BGR888(1, 2, 1) + } +} + +void DebayerCpu::debayer10P_GBGB_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + const int width_in_bytes = window_.width * 5 / 4; + const uint8_t *prev = (const uint8_t *)src[0]; + const uint8_t *curr = (const uint8_t *)src[1]; + const uint8_t *next = (const uint8_t *)src[2]; + + for (int x = 0; x < width_in_bytes; x++) { + /* Even pixel */ + GBRG_BGR888(2, 1, 1) + /* Odd pixel GBGR -> BGGR */ + BGGR_BGR888(1, 1, 1) + /* Same thing for next 2 pixels */ + GBRG_BGR888(1, 1, 1) + BGGR_BGR888(1, 2, 1) + } +} + +void DebayerCpu::debayer10P_RGRG_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + const int width_in_bytes = window_.width * 5 / 4; + const uint8_t *prev = (const uint8_t *)src[0]; + const uint8_t *curr = (const uint8_t *)src[1]; + const uint8_t *next = (const uint8_t *)src[2]; + + for (int x = 0; x < width_in_bytes; x++) { + /* Even pixel */ + RGGB_BGR888(2, 1, 1) + /* Odd pixel RGGB -> GRBG*/ + GRBG_BGR888(1, 1, 1) + /* Same thing for next 2 pixels */ + RGGB_BGR888(1, 1, 1) + GRBG_BGR888(1, 2, 1) + } +} + +static bool isStandardBayerOrder(BayerFormat::Order order) +{ + return order == BayerFormat::BGGR || order == BayerFormat::GBRG || + order == BayerFormat::GRBG || order == BayerFormat::RGGB; +} + +/* + * Setup the Debayer object according to the passed in parameters. + * Return 0 on success, a negative errno value on failure + * (unsupported parameters). + */ +int DebayerCpu::getInputConfig(PixelFormat inputFormat, DebayerInputConfig &config) +{ + BayerFormat bayerFormat = + BayerFormat::fromPixelFormat(inputFormat); + + if (bayerFormat.bitDepth == 10 && + bayerFormat.packing == BayerFormat::Packing::CSI2 && + isStandardBayerOrder(bayerFormat.order)) { + config.bpp = 10; + config.patternSize.width = 4; /* 5 bytes per *4* pixels */ + config.patternSize.height = 2; + config.outputFormats = std::vector({ formats::RGB888 }); + return 0; + } + + LOG(Debayer, Info) + << "Unsupported input format " << inputFormat.toString(); + return -EINVAL; +} + +int DebayerCpu::getOutputConfig(PixelFormat outputFormat, DebayerOutputConfig &config) +{ + if (outputFormat == formats::RGB888) { + config.bpp = 24; + return 0; + } + + LOG(Debayer, Info) + << "Unsupported output format " << outputFormat.toString(); + return -EINVAL; +} + +/* TODO: this ignores outputFormat since there is only 1 supported outputFormat for now */ +int DebayerCpu::setDebayerFunctions(PixelFormat inputFormat, [[maybe_unused]] PixelFormat outputFormat) +{ + BayerFormat bayerFormat = + BayerFormat::fromPixelFormat(inputFormat); + + if (bayerFormat.bitDepth == 10 && + bayerFormat.packing == BayerFormat::Packing::CSI2) { + switch (bayerFormat.order) { + case BayerFormat::BGGR: + debayer0_ = &DebayerCpu::debayer10P_BGBG_BGR888; + debayer1_ = &DebayerCpu::debayer10P_GRGR_BGR888; + return 0; + case BayerFormat::GBRG: + debayer0_ = &DebayerCpu::debayer10P_GBGB_BGR888; + debayer1_ = &DebayerCpu::debayer10P_RGRG_BGR888; + return 0; + case BayerFormat::GRBG: + debayer0_ = &DebayerCpu::debayer10P_GRGR_BGR888; + debayer1_ = &DebayerCpu::debayer10P_BGBG_BGR888; + return 0; + case BayerFormat::RGGB: + debayer0_ = &DebayerCpu::debayer10P_RGRG_BGR888; + debayer1_ = &DebayerCpu::debayer10P_GBGB_BGR888; + return 0; + default: + break; + } + } + + LOG(Debayer, Error) << "Unsupported input output format combination"; + return -EINVAL; +} + +int DebayerCpu::configure(const StreamConfiguration &inputCfg, + const std::vector> &outputCfgs) +{ + if (getInputConfig(inputCfg.pixelFormat, inputConfig_) != 0) + return -EINVAL; + + if (stats_->configure(inputCfg) != 0) + return -EINVAL; + + const Size &stats_pattern_size = stats_->patternSize(); + if (inputConfig_.patternSize.width != stats_pattern_size.width || + inputConfig_.patternSize.height != stats_pattern_size.height) { + LOG(Debayer, Error) + << "mismatching stats and debayer pattern sizes for " + << inputCfg.pixelFormat.toString(); + return -EINVAL; + } + + inputConfig_.stride = inputCfg.stride; + + if (outputCfgs.size() != 1) { + LOG(Debayer, Error) + << "Unsupported number of output streams: " + << outputCfgs.size(); + return -EINVAL; + } + + const StreamConfiguration &outputCfg = outputCfgs[0]; + SizeRange outSizeRange = sizes(inputCfg.pixelFormat, inputCfg.size); + std::tie(outputConfig_.stride, outputConfig_.frameSize) = + strideAndFrameSize(outputCfg.pixelFormat, outputCfg.size); + + if (!outSizeRange.contains(outputCfg.size) || outputConfig_.stride != outputCfg.stride) { + LOG(Debayer, Error) + << "Invalid output size/stride: " + << "\n " << outputCfg.size << " (" << outSizeRange << ")" + << "\n " << outputCfg.stride << " (" << outputConfig_.stride << ")"; + return -EINVAL; + } + + if (setDebayerFunctions(inputCfg.pixelFormat, outputCfg.pixelFormat) != 0) + return -EINVAL; + + window_.x = ((inputCfg.size.width - outputCfg.size.width) / 2) & + ~(inputConfig_.patternSize.width - 1); + window_.y = ((inputCfg.size.height - outputCfg.size.height) / 2) & + ~(inputConfig_.patternSize.height - 1); + window_.width = outputCfg.size.width; + window_.height = outputCfg.size.height; + + /* Don't pass x,y since process() already adjusts src before passing it */ + stats_->setWindow(Rectangle(window_.size())); + + /* pad with patternSize.Width on both left and right side */ + lineBufferPadding_ = inputConfig_.patternSize.width * inputConfig_.bpp / 8; + lineBufferLength_ = window_.width * inputConfig_.bpp / 8 + + 2 * lineBufferPadding_; + for (unsigned int i = 0; + i < (inputConfig_.patternSize.height + 1) && enableInputMemcpy_; + i++) { + free(lineBuffers_[i]); + lineBuffers_[i] = (uint8_t *)malloc(lineBufferLength_); + if (!lineBuffers_[i]) + return -ENOMEM; + } + + measuredFrames_ = 0; + frameProcessTime_ = 0; + + return 0; +} + +/* + * Get width and height at which the bayer-pattern repeats. + * Return pattern-size or an empty Size for an unsupported inputFormat. + */ +Size DebayerCpu::patternSize(PixelFormat inputFormat) +{ + DebayerCpu::DebayerInputConfig config; + + if (getInputConfig(inputFormat, config) != 0) + return {}; + + return config.patternSize; +} + +std::vector DebayerCpu::formats(PixelFormat inputFormat) +{ + DebayerCpu::DebayerInputConfig config; + + if (getInputConfig(inputFormat, config) != 0) + return std::vector(); + + return config.outputFormats; +} + +std::tuple +DebayerCpu::strideAndFrameSize(const PixelFormat &outputFormat, const Size &size) +{ + DebayerCpu::DebayerOutputConfig config; + + if (getOutputConfig(outputFormat, config) != 0) + return std::make_tuple(0, 0); + + /* round up to multiple of 8 for 64 bits alignment */ + unsigned int stride = (size.width * config.bpp / 8 + 7) & ~7; + + return std::make_tuple(stride, stride * size.height); +} + +void DebayerCpu::setupInputMemcpy(const uint8_t *linePointers[]) +{ + const unsigned int patternHeight = inputConfig_.patternSize.height; + + if (!enableInputMemcpy_) + return; + + for (unsigned int i = 0; i < patternHeight; i++) { + memcpy(lineBuffers_[i], linePointers[i + 1] - lineBufferPadding_, + lineBufferLength_); + linePointers[i + 1] = lineBuffers_[i] + lineBufferPadding_; + } + + /* Point lineBufferIndex_ to first unused lineBuffer */ + lineBufferIndex_ = patternHeight; +} + +void DebayerCpu::shiftLinePointers(const uint8_t *linePointers[], const uint8_t *src) +{ + const unsigned int patternHeight = inputConfig_.patternSize.height; + + for (unsigned int i = 0; i < patternHeight; i++) + linePointers[i] = linePointers[i + 1]; + + linePointers[patternHeight] = src + + (patternHeight / 2) * (int)inputConfig_.stride; +} + +void DebayerCpu::memcpyNextLine(const uint8_t *linePointers[]) +{ + const unsigned int patternHeight = inputConfig_.patternSize.height; + + if (!enableInputMemcpy_) + return; + + memcpy(lineBuffers_[lineBufferIndex_], linePointers[patternHeight] - lineBufferPadding_, + lineBufferLength_); + linePointers[patternHeight] = lineBuffers_[lineBufferIndex_] + lineBufferPadding_; + + lineBufferIndex_ = (lineBufferIndex_ + 1) % (patternHeight + 1); +} + +void DebayerCpu::process2(const uint8_t *src, uint8_t *dst) +{ + unsigned int y_end = window_.y + window_.height; + /* Holds [0] previous- [1] current- [2] next-line */ + const uint8_t *linePointers[3]; + + /* Adjust src to top left corner of the window */ + src += window_.y * inputConfig_.stride + window_.x * inputConfig_.bpp / 8; + + /* [x] becomes [x - 1] after initial shiftLinePointers() call */ + if (window_.y) { + linePointers[1] = src - inputConfig_.stride; /* previous-line */ + linePointers[2] = src; + } else { + /* window_.y == 0, use the next line as prev line */ + linePointers[1] = src + inputConfig_.stride; + linePointers[2] = src; + /* Last 2 lines also need special handling */ + y_end -= 2; + } + + setupInputMemcpy(linePointers); + + for (unsigned int y = window_.y; y < y_end; y += 2) { + shiftLinePointers(linePointers, src); + memcpyNextLine(linePointers); + stats_->processLine0(y, linePointers); + (this->*debayer0_)(dst, linePointers); + src += inputConfig_.stride; + dst += outputConfig_.stride; + + shiftLinePointers(linePointers, src); + memcpyNextLine(linePointers); + (this->*debayer1_)(dst, linePointers); + src += inputConfig_.stride; + dst += outputConfig_.stride; + } + + if (window_.y == 0) { + shiftLinePointers(linePointers, src); + memcpyNextLine(linePointers); + stats_->processLine0(y_end, linePointers); + (this->*debayer0_)(dst, linePointers); + src += inputConfig_.stride; + dst += outputConfig_.stride; + + shiftLinePointers(linePointers, src); + /* next line may point outside of src, use prev. */ + linePointers[2] = linePointers[0]; + (this->*debayer1_)(dst, linePointers); + src += inputConfig_.stride; + dst += outputConfig_.stride; + } +} + +void DebayerCpu::process4(const uint8_t *src, uint8_t *dst) +{ + const unsigned int y_end = window_.y + window_.height; + /* + * This holds pointers to [0] 2-lines-up [1] 1-line-up [2] current-line + * [3] 1-line-down [4] 2-lines-down. + */ + const uint8_t *linePointers[5]; + + /* Adjust src to top left corner of the window */ + src += window_.y * inputConfig_.stride + window_.x * inputConfig_.bpp / 8; + + /* [x] becomes [x - 1] after initial shiftLinePointers() call */ + linePointers[1] = src - 2 * inputConfig_.stride; + linePointers[2] = src - inputConfig_.stride; + linePointers[3] = src; + linePointers[4] = src + inputConfig_.stride; + + setupInputMemcpy(linePointers); + + for (unsigned int y = window_.y; y < y_end; y += 4) { + shiftLinePointers(linePointers, src); + memcpyNextLine(linePointers); + stats_->processLine0(y, linePointers); + (this->*debayer0_)(dst, linePointers); + src += inputConfig_.stride; + dst += outputConfig_.stride; + + shiftLinePointers(linePointers, src); + memcpyNextLine(linePointers); + (this->*debayer1_)(dst, linePointers); + src += inputConfig_.stride; + dst += outputConfig_.stride; + + shiftLinePointers(linePointers, src); + memcpyNextLine(linePointers); + stats_->processLine2(y, linePointers); + (this->*debayer2_)(dst, linePointers); + src += inputConfig_.stride; + dst += outputConfig_.stride; + + shiftLinePointers(linePointers, src); + memcpyNextLine(linePointers); + (this->*debayer3_)(dst, linePointers); + src += inputConfig_.stride; + dst += outputConfig_.stride; + } +} + +static inline int64_t timeDiff(timespec &after, timespec &before) +{ + return (after.tv_sec - before.tv_sec) * 1000000000LL + + (int64_t)after.tv_nsec - (int64_t)before.tv_nsec; +} + +void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) +{ + timespec frameStartTime; + + if (measuredFrames_ < DebayerCpu::kLastFrameToMeasure) { + frameStartTime = {}; + clock_gettime(CLOCK_MONOTONIC_RAW, &frameStartTime); + } + + /* Apply DebayerParams */ + if (params.gamma != gamma_correction_) { + for (unsigned int i = 0; i < kGammaLookupSize; i++) + gamma_[i] = UINT8_MAX * powf(i / (kGammaLookupSize - 1.0), params.gamma); + + gamma_correction_ = params.gamma; + } + + for (unsigned int i = 0; i < kRGBLookupSize; i++) { + constexpr unsigned int div = + kRGBLookupSize * DebayerParams::kGain10 / kGammaLookupSize; + unsigned int idx; + + /* Apply gamma after gain! */ + idx = std::min({ i * params.gainR / div, (kGammaLookupSize - 1) }); + red_[i] = gamma_[idx]; + + idx = std::min({ i * params.gainG / div, (kGammaLookupSize - 1) }); + green_[i] = gamma_[idx]; + + idx = std::min({ i * params.gainB / div, (kGammaLookupSize - 1) }); + blue_[i] = gamma_[idx]; + } + + /* Copy metadata from the input buffer */ + FrameMetadata &metadata = output->_d()->metadata(); + metadata.status = input->metadata().status; + metadata.sequence = input->metadata().sequence; + metadata.timestamp = input->metadata().timestamp; + + MappedFrameBuffer in(input, MappedFrameBuffer::MapFlag::Read); + MappedFrameBuffer out(output, MappedFrameBuffer::MapFlag::Write); + if (!in.isValid() || !out.isValid()) { + LOG(Debayer, Error) << "mmap-ing buffer(s) failed"; + metadata.status = FrameMetadata::FrameError; + return; + } + + stats_->startFrame(); + + if (inputConfig_.patternSize.height == 2) + process2(in.planes()[0].data(), out.planes()[0].data()); + else + process4(in.planes()[0].data(), out.planes()[0].data()); + + metadata.planes()[0].bytesused = out.planes()[0].size(); + + /* Measure before emitting signals */ + if (measuredFrames_ < DebayerCpu::kLastFrameToMeasure && + ++measuredFrames_ > DebayerCpu::kFramesToSkip) { + timespec frameEndTime = {}; + clock_gettime(CLOCK_MONOTONIC_RAW, &frameEndTime); + frameProcessTime_ += timeDiff(frameEndTime, frameStartTime); + if (measuredFrames_ == DebayerCpu::kLastFrameToMeasure) { + const unsigned int measuredFrames = DebayerCpu::kLastFrameToMeasure - + DebayerCpu::kFramesToSkip; + LOG(Debayer, Info) + << "Processed " << measuredFrames + << " frames in " << frameProcessTime_ / 1000 << "us, " + << frameProcessTime_ / (1000 * measuredFrames) + << " us/frame"; + } + } + + stats_->finishFrame(); + outputBufferReady.emit(output); + inputBufferReady.emit(input); +} + +SizeRange DebayerCpu::sizes(PixelFormat inputFormat, const Size &inputSize) +{ + Size pattern_size = patternSize(inputFormat); + unsigned int border_height = pattern_size.height; + + if (pattern_size.isNull()) + return {}; + + /* No need for top/bottom border with a pattern height of 2 */ + if (pattern_size.height == 2) + border_height = 0; + + /* + * For debayer interpolation a border is kept around the entire image + * and the minimum output size is pattern-height x pattern-width. + */ + if (inputSize.width < (3 * pattern_size.width) || + inputSize.height < (2 * border_height + pattern_size.height)) { + LOG(Debayer, Warning) + << "Input format size too small: " << inputSize.toString(); + return {}; + } + + return SizeRange(Size(pattern_size.width, pattern_size.height), + Size((inputSize.width - 2 * pattern_size.width) & ~(pattern_size.width - 1), + (inputSize.height - 2 * border_height) & ~(pattern_size.height - 1)), + pattern_size.width, pattern_size.height); +} + +} /* namespace libcamera */ diff --git a/src/libcamera/software_isp/debayer_cpu.h b/src/libcamera/software_isp/debayer_cpu.h new file mode 100644 index 00000000..98782838 --- /dev/null +++ b/src/libcamera/software_isp/debayer_cpu.h @@ -0,0 +1,143 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * Copyright (C) 2023, Red Hat Inc. + * + * Authors: + * Hans de Goede + * + * debayer_cpu.h - CPU based debayering header + */ + +#pragma once + +#include +#include +#include + +#include + +#include "debayer.h" +#include "swstats_cpu.h" + +namespace libcamera { + +class DebayerCpu : public Debayer, public Object +{ +public: + DebayerCpu(std::unique_ptr stats); + ~DebayerCpu(); + + int configure(const StreamConfiguration &inputCfg, + const std::vector> &outputCfgs); + Size patternSize(PixelFormat inputFormat); + std::vector formats(PixelFormat input); + std::tuple + strideAndFrameSize(const PixelFormat &outputFormat, const Size &size); + void process(FrameBuffer *input, FrameBuffer *output, DebayerParams params); + SizeRange sizes(PixelFormat inputFormat, const Size &inputSize); + + /** + * \brief Get the file descriptor for the statistics. + * + * \return the file descriptor pointing to the statistics. + */ + const SharedFD &getStatsFD() { return stats_->getStatsFD(); } + + /** + * \brief Get the output frame size. + * + * \return The output frame size. + */ + unsigned int frameSize() { return outputConfig_.frameSize; } + +private: + /** + * \brief Called to debayer 1 line of Bayer input data to output format + * \param[out] dst Pointer to the start of the output line to write + * \param[in] src The input data + * + * Input data is an array of (patternSize_.height + 1) src + * pointers each pointing to a line in the Bayer source. The middle + * element of the array will point to the actual line being processed. + * Earlier element(s) will point to the previous line(s) and later + * element(s) to the next line(s). + * + * These functions take an array of src pointers, rather then + * a single src pointer + a stride for the source, so that when the src + * is slow uncached memory it can be copied to faster memory before + * debayering. Debayering a standard 2x2 Bayer pattern requires access + * to the previous and next src lines for interpolating the missing + * colors. To allow copying the src lines only once 3 temporary buffers + * each holding a single line are used, re-using the oldest buffer for + * the next line and the pointers are swizzled so that: + * src[0] = previous-line, src[1] = currrent-line, src[2] = next-line. + * This way the 3 pointers passed to the debayer functions form + * a sliding window over the src avoiding the need to copy each + * line more than once. + * + * Similarly for bayer patterns which repeat every 4 lines, 5 src + * pointers are passed holding: src[0] = 2-lines-up, src[1] = 1-line-up + * src[2] = current-line, src[3] = 1-line-down, src[4] = 2-lines-down. + */ + using debayerFn = void (DebayerCpu::*)(uint8_t *dst, const uint8_t *src[]); + + /* CSI-2 packed 10-bit raw bayer format (all the 4 orders) */ + void debayer10P_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]); + void debayer10P_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]); + void debayer10P_GBGB_BGR888(uint8_t *dst, const uint8_t *src[]); + void debayer10P_RGRG_BGR888(uint8_t *dst, const uint8_t *src[]); + + struct DebayerInputConfig { + Size patternSize; + unsigned int bpp; /* Memory used per pixel, not precision */ + unsigned int stride; + std::vector outputFormats; + }; + + struct DebayerOutputConfig { + unsigned int bpp; /* Memory used per pixel, not precision */ + unsigned int stride; + unsigned int frameSize; + }; + + int getInputConfig(PixelFormat inputFormat, DebayerInputConfig &config); + int getOutputConfig(PixelFormat outputFormat, DebayerOutputConfig &config); + int setDebayerFunctions(PixelFormat inputFormat, PixelFormat outputFormat); + void setupInputMemcpy(const uint8_t *linePointers[]); + void shiftLinePointers(const uint8_t *linePointers[], const uint8_t *src); + void memcpyNextLine(const uint8_t *linePointers[]); + void process2(const uint8_t *src, uint8_t *dst); + void process4(const uint8_t *src, uint8_t *dst); + + static constexpr unsigned int kGammaLookupSize = 1024; + static constexpr unsigned int kRGBLookupSize = 256; + /* Max. supported Bayer pattern height is 4, debayering this requires 5 lines */ + static constexpr unsigned int kMaxLineBuffers = 5; + + std::array gamma_; + std::array red_; + std::array green_; + std::array blue_; + debayerFn debayer0_; + debayerFn debayer1_; + debayerFn debayer2_; + debayerFn debayer3_; + Rectangle window_; + DebayerInputConfig inputConfig_; + DebayerOutputConfig outputConfig_; + std::unique_ptr stats_; + uint8_t *lineBuffers_[kMaxLineBuffers]; + unsigned int lineBufferLength_; + unsigned int lineBufferPadding_; + unsigned int lineBufferIndex_; + bool enableInputMemcpy_; + float gamma_correction_; + unsigned int measuredFrames_; + int64_t frameProcessTime_; + /* Skip 30 frames for things to stabilize then measure 30 frames */ + static constexpr unsigned int kFramesToSkip = 30; + static constexpr unsigned int kLastFrameToMeasure = 60; +}; + +} /* namespace libcamera */ diff --git a/src/libcamera/software_isp/meson.build b/src/libcamera/software_isp/meson.build index 92fc90f3..dd5e70c1 100644 --- a/src/libcamera/software_isp/meson.build +++ b/src/libcamera/software_isp/meson.build @@ -2,5 +2,6 @@ libcamera_sources += files([ 'debayer.cpp', + 'debayer_cpu.cpp', 'swstats_cpu.cpp', ]) From patchwork Thu Feb 29 18:36:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19580 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D382CC3260 for ; Thu, 29 Feb 2024 18:37:18 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8CCD262867; Thu, 29 Feb 2024 19:37:18 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="TbtNvAUh"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id CD3AB628AC for ; Thu, 29 Feb 2024 19:37:15 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231834; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JviXiHTX6Nsd0y9dDvx11znbf+iA+Q/laJ0u+6Dq5+s=; b=TbtNvAUh+4WOUaWMj23AYarsujPtQ+kckqDhcc8ts+Rejtl9CFpe6enhZutJuNWv8kr5OM bPiSnBCR54/zkRC9cLMqVvytgJfa1Zq076iZz1XJo421+ID/1QlI7A5XlBHjuXQxxSjSDQ x0xiSbErrIBoJFZOQdZl35/NC2U7XWo= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-216-unBdeZk6PNGg-UWX2EConA-1; Thu, 29 Feb 2024 13:37:11 -0500 X-MC-Unique: unBdeZk6PNGg-UWX2EConA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DD3171C05EAC; Thu, 29 Feb 2024 18:37:10 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7C3AF1C06714; Thu, 29 Feb 2024 18:37:09 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 09/18] libcamera: ipa: add Soft IPA Date: Thu, 29 Feb 2024 19:36:20 +0100 Message-ID: <20240229183654.7206-10-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Marttico , Toon Langendam , Pavel Machek , Bryan O'Donoghue , Dennis Bonke Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov Define the Soft IPA main and event interfaces, add the Soft IPA implementation. The current src/ipa/meson.build assumes the IPA name to match the pipeline name. For this reason "-Dipas=simple" is used for the Soft IPA module. Auto exposure/gain and AWB implementation by Dennis, Toon and Martti. Auto exposure/gain targets a Mean Sample Value of 2.5 following the MSV calculation algorithm from: https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Signed-off-by: Andrey Konovalov Co-developed-by: Dennis Bonke Signed-off-by: Dennis Bonke Co-developed-by: Marttico Signed-off-by: Marttico Co-developed-by: Toon Langendam Signed-off-by: Toon Langendam Signed-off-by: Hans de Goede --- Changes in v4: - Use int32_t for again*_ and exposure*_ as this matches the value the corresponding ControlValue::get() returns - Check for mmap() returning MAP_FAILED on error - Drop std::move() called on const SharedFD & argument - Replace #defines (EXPOSURE_OPTIMAL_VALUE etc) with const expressions - Use std::clamp() to keep exposure_ and again_ between the *_min_ and the *_max_ - add comment on non-linear gain value vs gain code Changes in v3: - Use new SwIspStats::kYHistogramSize constexpr and adjust the auto-exposure/-gain code so that it can deal with that having a different value then 16 (modify the loop to divide the histogram in 5 bins to not have hardcoded values) - Rename a few foo_bar symbols to fooBar --- Documentation/Doxyfile.in | 1 + include/libcamera/ipa/meson.build | 1 + include/libcamera/ipa/soft.mojom | 28 +++ meson_options.txt | 2 +- src/ipa/simple/data/meson.build | 9 + src/ipa/simple/data/soft.conf | 3 + src/ipa/simple/meson.build | 25 +++ src/ipa/simple/soft_simple.cpp | 326 ++++++++++++++++++++++++++++++ 8 files changed, 394 insertions(+), 1 deletion(-) create mode 100644 include/libcamera/ipa/soft.mojom create mode 100644 src/ipa/simple/data/meson.build create mode 100644 src/ipa/simple/data/soft.conf create mode 100644 src/ipa/simple/meson.build create mode 100644 src/ipa/simple/soft_simple.cpp diff --git a/Documentation/Doxyfile.in b/Documentation/Doxyfile.in index a86ea6c1..2be8d47b 100644 --- a/Documentation/Doxyfile.in +++ b/Documentation/Doxyfile.in @@ -44,6 +44,7 @@ EXCLUDE = @TOP_SRCDIR@/include/libcamera/base/span.h \ @TOP_SRCDIR@/src/libcamera/pipeline/ \ @TOP_SRCDIR@/src/libcamera/tracepoints.cpp \ @TOP_BUILDDIR@/include/libcamera/internal/tracepoints.h \ + @TOP_BUILDDIR@/include/libcamera/ipa/soft_ipa_interface.h \ @TOP_BUILDDIR@/src/libcamera/proxy/ EXCLUDE_PATTERNS = @TOP_BUILDDIR@/include/libcamera/ipa/*_serializer.h \ diff --git a/include/libcamera/ipa/meson.build b/include/libcamera/ipa/meson.build index f3b4881c..3352d08f 100644 --- a/include/libcamera/ipa/meson.build +++ b/include/libcamera/ipa/meson.build @@ -65,6 +65,7 @@ pipeline_ipa_mojom_mapping = { 'ipu3': 'ipu3.mojom', 'rkisp1': 'rkisp1.mojom', 'rpi/vc4': 'raspberrypi.mojom', + 'simple': 'soft.mojom', 'vimc': 'vimc.mojom', } diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom new file mode 100644 index 00000000..c249bd75 --- /dev/null +++ b/include/libcamera/ipa/soft.mojom @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ + +/* + * \todo Document the interface and remove the related EXCLUDE_PATTERNS entry. + */ + +module ipa.soft; + +import "include/libcamera/ipa/core.mojom"; + +interface IPASoftInterface { + init(libcamera.IPASettings settings, + libcamera.SharedFD fdStats, + libcamera.SharedFD fdParams, + libcamera.ControlInfoMap sensorCtrlInfoMap) + => (int32 ret); + start() => (int32 ret); + stop(); + configure(libcamera.ControlInfoMap sensorCtrlInfoMap) + => (int32 ret); + + [async] processStats(libcamera.ControlList sensorControls); +}; + +interface IPASoftEventInterface { + setSensorControls(libcamera.ControlList sensorControls); + setIspParams(int32 dummy); +}; diff --git a/meson_options.txt b/meson_options.txt index 99dab96d..2644bef0 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -27,7 +27,7 @@ option('gstreamer', option('ipas', type : 'array', - choices : ['ipu3', 'rkisp1', 'rpi/vc4', 'vimc'], + choices : ['ipu3', 'rkisp1', 'rpi/vc4', 'simple', 'vimc'], description : 'Select which IPA modules to build') option('lc-compliance', diff --git a/src/ipa/simple/data/meson.build b/src/ipa/simple/data/meson.build new file mode 100644 index 00000000..33548cc6 --- /dev/null +++ b/src/ipa/simple/data/meson.build @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: CC0-1.0 + +conf_files = files([ + 'soft.conf', +]) + +install_data(conf_files, + install_dir : ipa_data_dir / 'soft', + install_tag : 'runtime') diff --git a/src/ipa/simple/data/soft.conf b/src/ipa/simple/data/soft.conf new file mode 100644 index 00000000..0c70e7c0 --- /dev/null +++ b/src/ipa/simple/data/soft.conf @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: LGPL-2.1-or-later +# +# Dummy configuration file for the soft IPA. diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build new file mode 100644 index 00000000..3e863db7 --- /dev/null +++ b/src/ipa/simple/meson.build @@ -0,0 +1,25 @@ +# SPDX-License-Identifier: CC0-1.0 + +ipa_name = 'ipa_soft_simple' + +mod = shared_module(ipa_name, + ['soft_simple.cpp', libcamera_generated_ipa_headers], + name_prefix : '', + include_directories : [ipa_includes, libipa_includes], + dependencies : libcamera_private, + link_with : libipa, + install : true, + install_dir : ipa_install_dir) + +if ipa_sign_module + custom_target(ipa_name + '.so.sign', + input : mod, + output : ipa_name + '.so.sign', + command : [ipa_sign, ipa_priv_key, '@INPUT@', '@OUTPUT@'], + install : false, + build_by_default : true) +endif + +subdir('data') + +ipa_names += ipa_name diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp new file mode 100644 index 00000000..312df4ba --- /dev/null +++ b/src/ipa/simple/soft_simple.cpp @@ -0,0 +1,326 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * + * soft_simple.cpp - Simple Software Image Processing Algorithm module + */ + +#include + +#include +#include +#include + +#include +#include + +#include +#include +#include + +#include "libcamera/internal/camera_sensor.h" +#include "libcamera/internal/software_isp/debayer_params.h" +#include "libcamera/internal/software_isp/swisp_stats.h" + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoft) + +namespace ipa::soft { + +class IPASoftSimple : public ipa::soft::IPASoftInterface +{ +public: + IPASoftSimple() + : params_(static_cast(MAP_FAILED)), + stats_(static_cast(MAP_FAILED)), ignore_updates_(0) + { + } + + ~IPASoftSimple() + { + if (stats_ != MAP_FAILED) + munmap(stats_, sizeof(SwIspStats)); + if (params_ != MAP_FAILED) + munmap(params_, sizeof(DebayerParams)); + } + + int init(const IPASettings &settings, + const SharedFD &fdStats, + const SharedFD &fdParams, + const ControlInfoMap &sensorInfoMap) override; + int configure(const ControlInfoMap &sensorInfoMap) override; + + int start() override; + void stop() override; + + void processStats(const ControlList &sensorControls) override; + +private: + void updateExposure(double exposureMSV); + + SharedFD fdStats_; + SharedFD fdParams_; + DebayerParams *params_; + SwIspStats *stats_; + + int32_t exposure_min_, exposure_max_; + int32_t again_min_, again_max_; + int32_t again_, exposure_; + unsigned int ignore_updates_; +}; + +int IPASoftSimple::init([[maybe_unused]] const IPASettings &settings, + const SharedFD &fdStats, + const SharedFD &fdParams, + const ControlInfoMap &sensorInfoMap) +{ + fdStats_ = fdStats; + if (!fdStats_.isValid()) { + LOG(IPASoft, Error) << "Invalid Statistics handle"; + return -ENODEV; + } + + fdParams_ = fdParams; + if (!fdParams_.isValid()) { + LOG(IPASoft, Error) << "Invalid Parameters handle"; + return -ENODEV; + } + + params_ = static_cast(mmap(nullptr, sizeof(DebayerParams), + PROT_WRITE, MAP_SHARED, + fdParams_.get(), 0)); + if (params_ == MAP_FAILED) { + LOG(IPASoft, Error) << "Unable to map Parameters"; + return -errno; + } + + stats_ = static_cast(mmap(nullptr, sizeof(SwIspStats), + PROT_READ, MAP_SHARED, + fdStats_.get(), 0)); + if (stats_ == MAP_FAILED) { + LOG(IPASoft, Error) << "Unable to map Statistics"; + return -errno; + } + + if (sensorInfoMap.find(V4L2_CID_EXPOSURE) == sensorInfoMap.end()) { + LOG(IPASoft, Error) << "Don't have exposure control"; + return -EINVAL; + } + + if (sensorInfoMap.find(V4L2_CID_ANALOGUE_GAIN) == sensorInfoMap.end()) { + LOG(IPASoft, Error) << "Don't have gain control"; + return -EINVAL; + } + + return 0; +} + +int IPASoftSimple::configure(const ControlInfoMap &sensorInfoMap) +{ + const ControlInfo &exposure_info = sensorInfoMap.find(V4L2_CID_EXPOSURE)->second; + const ControlInfo &gain_info = sensorInfoMap.find(V4L2_CID_ANALOGUE_GAIN)->second; + + exposure_min_ = exposure_info.min().get(); + exposure_max_ = exposure_info.max().get(); + if (!exposure_min_) { + LOG(IPASoft, Warning) << "Minimum exposure is zero, that can't be linear"; + exposure_min_ = 1; + } + + again_min_ = gain_info.min().get(); + again_max_ = gain_info.max().get(); + /* + * The camera sensor gain (g) is usually not equal to the value written + * into the gain register (x). But the way how the AGC algorithm changes + * the gain value to make the total exposure closer to the optimum assumes + * that g(x) is not too far from linear function. If the minimal gain is 0, + * the g(x) is likely to be far from the linear, like g(x) = a / (b * x + c). + * To avoid unexpected changes to the gain by the AGC algorithm (abrupt near + * one edge, and very small near the other) we limit the range of the gain + * values used. + */ + if (!again_min_) { + LOG(IPASoft, Warning) << "Minimum gain is zero, that can't be linear"; + again_min_ = std::min(100, again_min_ / 2 + again_max_ / 2); + } + + LOG(IPASoft, Info) << "Exposure " << exposure_min_ << "-" << exposure_max_ + << ", gain " << again_min_ << "-" << again_max_; + + return 0; +} + +int IPASoftSimple::start() +{ + return 0; +} + +void IPASoftSimple::stop() +{ +} + +/* + * The number of bins to use for the optimal exposure calculations. + */ +static constexpr unsigned int kExposureBinsCount = 5; +/* + * The exposure is optimal when the mean sample value of the histogram is + * in the middle of the range. + */ +static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; +/* + * The below value implements the hysteresis for the exposure adjustment. + * It is small enough to have the exposure close to the optimal, and is big + * enough to prevent the exposure from wobbling around the optimal value. + */ +static constexpr float kExposureSatisfactory = 0.2; + +void IPASoftSimple::processStats(const ControlList &sensorControls) +{ + /* + * Calculate red and blue gains for AWB. + * Clamp max gain at 4.0, this also avoids 0 division. + */ + if (stats_->sumR_ <= stats_->sumG_ / 4) + params_->gainR = 1024; + else + params_->gainR = 256 * stats_->sumG_ / stats_->sumR_; + + if (stats_->sumB_ <= stats_->sumG_ / 4) + params_->gainB = 1024; + else + params_->gainB = 256 * stats_->sumG_ / stats_->sumB_; + + /* Green gain and gamma values are fixed */ + params_->gainG = 256; + params_->gamma = 0.5; + + setIspParams.emit(0); + + /* + * AE / AGC, use 2 frames delay to make sure that the exposure and + * the gain set have applied to the camera sensor. + */ + if (ignore_updates_ > 0) { + --ignore_updates_; + return; + } + + /* + * Calculate Mean Sample Value (MSV) according to formula from: + * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf + */ + constexpr unsigned int yHistValsPerBin = + SwIspStats::kYHistogramSize / kExposureBinsCount; + constexpr unsigned int yHistValsPerBinMod = + SwIspStats::kYHistogramSize / + (SwIspStats::kYHistogramSize % kExposureBinsCount + 1); + int ExposureBins[kExposureBinsCount] = {}; + unsigned int denom = 0; + unsigned int num = 0; + + for (unsigned int i = 0; i < SwIspStats::kYHistogramSize; i++) { + unsigned int idx = (i - (i / yHistValsPerBinMod)) / yHistValsPerBin; + ExposureBins[idx] += stats_->yHistogram[i]; + } + + for (unsigned int i = 0; i < kExposureBinsCount; i++) { + LOG(IPASoft, Debug) << i << ": " << ExposureBins[i]; + denom += ExposureBins[i]; + num += ExposureBins[i] * (i + 1); + } + + float exposureMSV = (float)num / denom; + + /* sanity check */ + if (!sensorControls.contains(V4L2_CID_EXPOSURE) || + !sensorControls.contains(V4L2_CID_ANALOGUE_GAIN)) { + LOG(IPASoft, Error) << "Control(s) missing"; + return; + } + + ControlList ctrls(sensorControls); + + exposure_ = ctrls.get(V4L2_CID_EXPOSURE).get(); + again_ = ctrls.get(V4L2_CID_ANALOGUE_GAIN).get(); + + updateExposure(exposureMSV); + + ctrls.set(V4L2_CID_EXPOSURE, exposure_); + ctrls.set(V4L2_CID_ANALOGUE_GAIN, again_); + + ignore_updates_ = 2; + + setSensorControls.emit(ctrls); + + LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV + << " exp " << exposure_ << " again " << again_ + << " gain R/B " << params_->gainR << "/" << params_->gainB; +} + +void IPASoftSimple::updateExposure(double exposureMSV) +{ + /* DENOMINATOR of 10 gives ~10% increment/decrement; DENOMINATOR of 5 - about ~20% */ + static constexpr uint8_t kExpDenominator = 10; + static constexpr uint8_t kExpNumeratorUp = kExpDenominator + 1; + static constexpr uint8_t kExpNumeratorDown = kExpDenominator - 1; + + int next; + + if (exposureMSV < kExposureOptimal - kExposureSatisfactory) { + next = exposure_ * kExpNumeratorUp / kExpDenominator; + if (next - exposure_ < 1) + exposure_ += 1; + else + exposure_ = next; + if (exposure_ >= exposure_max_) { + next = again_ * kExpNumeratorUp / kExpDenominator; + if (next - again_ < 1) + again_ += 1; + else + again_ = next; + } + } + + if (exposureMSV > kExposureOptimal + kExposureSatisfactory) { + if (exposure_ == exposure_max_ && again_ != again_min_) { + next = again_ * kExpNumeratorDown / kExpDenominator; + if (again_ - next < 1) + again_ -= 1; + else + again_ = next; + } else { + next = exposure_ * kExpNumeratorDown / kExpDenominator; + if (exposure_ - next < 1) + exposure_ -= 1; + else + exposure_ = next; + } + } + + exposure_ = std::clamp(exposure_, exposure_min_, exposure_max_); + again_ = std::clamp(again_, again_min_, again_max_); +} + +} /* namespace ipa::soft */ + +/* + * External IPA module interface + */ +extern "C" { +const struct IPAModuleInfo ipaModuleInfo = { + IPA_MODULE_API_VERSION, + 0, + "SimplePipelineHandler", + "simple", +}; + +IPAInterface *ipaCreate() +{ + return new ipa::soft::IPASoftSimple(); +} + +} /* extern "C" */ + +} /* namespace libcamera */ From patchwork Thu Feb 29 18:36:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19581 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5F887BD160 for ; Thu, 29 Feb 2024 18:37:20 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2B56562B3D; Thu, 29 Feb 2024 19:37:20 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="EjPE9aBH"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id EE53C62872 for ; Thu, 29 Feb 2024 19:37:17 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231837; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V69pnA03S6Y8j6r82EHmdspL5CRhzayZCFIK7ufDdwM=; b=EjPE9aBH6MV0fZr6Djxfn4uqGEqRShimRLY7gtYNu/yP81cS5siHLrg2ZDQOhIYw/xGO5P Mlo94x0htbq18+0FmZusR01ZvfN+A5AqclkZQnbe6zRD15tMMQGpa0VQ/fMwKNyQ8D2IzM Fer+Ov7lw+7k+aAsYefmOyuk+Xi1phI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-639-6--vggSING2tUV-ZTUE2-Q-1; Thu, 29 Feb 2024 13:37:12 -0500 X-MC-Unique: 6--vggSING2tUV-ZTUE2-Q-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 47600879849; Thu, 29 Feb 2024 18:37:12 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1AB671C060AF; Thu, 29 Feb 2024 18:37:11 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 10/18] libcamera: introduce SoftwareIsp Date: Thu, 29 Feb 2024 19:36:21 +0100 Message-ID: <20240229183654.7206-11-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue , Dennis Bonke Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov Doxygen documentation by Dennis Bonke. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Co-developed-by: Dennis Bonke Signed-off-by: Dennis Bonke Signed-off-by: Andrey Konovalov Signed-off-by: Hans de Goede --- Changes in v4: - Fix the check of if sharedParams_ were created OK Changes in v3: - SoftISP factory has been removed, there is a just a single SoftwareISP class now - The build must be configured with "-Dpipelines=simple -Dipas=simple", and the Soft ISP is always compiled. Using the Soft ISP must be enabled per platform through the supportedDevices[] table in the simple pipeline handler. - Move software_isp.h/.cpp into the software_isp include/src subdirs --- .../internal/software_isp/meson.build | 1 + .../internal/software_isp/software_isp.h | 98 +++++ src/libcamera/software_isp/meson.build | 1 + src/libcamera/software_isp/software_isp.cpp | 349 ++++++++++++++++++ 4 files changed, 449 insertions(+) create mode 100644 include/libcamera/internal/software_isp/software_isp.h create mode 100644 src/libcamera/software_isp/software_isp.cpp diff --git a/include/libcamera/internal/software_isp/meson.build b/include/libcamera/internal/software_isp/meson.build index a620e16d..508ddddc 100644 --- a/include/libcamera/internal/software_isp/meson.build +++ b/include/libcamera/internal/software_isp/meson.build @@ -2,5 +2,6 @@ libcamera_internal_headers += files([ 'debayer_params.h', + 'software_isp.h', 'swisp_stats.h', ]) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h new file mode 100644 index 00000000..8d25e979 --- /dev/null +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -0,0 +1,98 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * + * software_isp.h - Simple software ISP implementation + */ + +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include +#include + +#include +#include + +#include "libcamera/internal/dma_heaps.h" +#include "libcamera/internal/pipeline_handler.h" +#include "libcamera/internal/shared_mem_object.h" +#include "libcamera/internal/software_isp/debayer_params.h" + +namespace libcamera { + +class DebayerCpu; +class FrameBuffer; +class PixelFormat; +struct StreamConfiguration; + +LOG_DECLARE_CATEGORY(SoftwareIsp) + +class SoftwareIsp +{ +public: + SoftwareIsp(PipelineHandler *pipe, const ControlInfoMap &sensorControls); + ~SoftwareIsp(); + + int loadConfiguration([[maybe_unused]] const std::string &filename) { return 0; } + + bool isValid() const; + + std::vector formats(PixelFormat input); + + SizeRange sizes(PixelFormat inputFormat, const Size &inputSize); + + std::tuple + strideAndFrameSize(const PixelFormat &outputFormat, const Size &size); + + int configure(const StreamConfiguration &inputCfg, + const std::vector> &outputCfgs, + const ControlInfoMap &sensorControls); + + int exportBuffers(unsigned int output, unsigned int count, + std::vector> *buffers); + + void processStats(const ControlList &sensorControls); + + int start(); + void stop(); + + int queueBuffers(FrameBuffer *input, + const std::map &outputs); + + void process(FrameBuffer *input, FrameBuffer *output); + + Signal inputBufferReady; + Signal outputBufferReady; + Signal ispStatsReady; + Signal setSensorControls; + +private: + void saveIspParams(int dummy); + void setSensorCtrls(const ControlList &sensorControls); + void statsReady(int dummy); + void inputReady(FrameBuffer *input); + void outputReady(FrameBuffer *output); + + std::unique_ptr debayer_; + Thread ispWorkerThread_; + SharedMemObject sharedParams_; + DebayerParams debayerParams_; + DmaHeap dmaHeap_; + + std::unique_ptr ipa_; +}; + +} /* namespace libcamera */ diff --git a/src/libcamera/software_isp/meson.build b/src/libcamera/software_isp/meson.build index dd5e70c1..cca2c304 100644 --- a/src/libcamera/software_isp/meson.build +++ b/src/libcamera/software_isp/meson.build @@ -3,5 +3,6 @@ libcamera_sources += files([ 'debayer.cpp', 'debayer_cpu.cpp', + 'software_isp.cpp', 'swstats_cpu.cpp', ]) diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp new file mode 100644 index 00000000..388b4496 --- /dev/null +++ b/src/libcamera/software_isp/software_isp.cpp @@ -0,0 +1,349 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2023, Linaro Ltd + * + * software_isp.cpp - Simple software ISP implementation + */ + +#include "libcamera/internal/software_isp/software_isp.h" + +#include +#include +#include + +#include +#include + +#include "libcamera/internal/bayer_format.h" +#include "libcamera/internal/framebuffer.h" +#include "libcamera/internal/ipa_manager.h" +#include "libcamera/internal/mapped_framebuffer.h" + +#include "debayer_cpu.h" + +/** + * \file software_isp.cpp + * \brief Simple software ISP implementation + */ + +namespace libcamera { + +LOG_DEFINE_CATEGORY(SoftwareIsp) + +/** + * \class SoftwareIsp + * \brief Class for the Software ISP + */ + +/** + * \var SoftwareIsp::inputBufferReady + * \brief A signal emitted when the input frame buffer completes + */ + +/** + * \var SoftwareIsp::outputBufferReady + * \brief A signal emitted when the output frame buffer completes + */ + +/** + * \var SoftwareIsp::ispStatsReady + * \brief A signal emitted when the statistics for IPA are ready + * + * The int parameter isn't actually used. + */ + +/** + * \var SoftwareIsp::setSensorControls + * \brief A signal emitted when the values to write to the sensor controls are ready + */ + +/** + * \brief Constructs SoftwareIsp object + * \param[in] pipe The pipeline handler in use + * \param[in] sensorControls ControlInfoMap describing the controls supported by the sensor + */ +SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const ControlInfoMap &sensorControls) + : debayer_(nullptr), + debayerParams_{ DebayerParams::kGain10, DebayerParams::kGain10, DebayerParams::kGain10, 0.5f }, + dmaHeap_(DmaHeap::DmaHeapFlag::Cma | DmaHeap::DmaHeapFlag::System) +{ + if (!dmaHeap_.isValid()) { + LOG(SoftwareIsp, Error) << "Failed to create DmaHeap object"; + return; + } + + sharedParams_ = SharedMemObject("softIsp_params"); + if (!sharedParams_) { + LOG(SoftwareIsp, Error) << "Failed to create shared memory for parameters"; + return; + } + + auto stats = std::make_unique(); + if (!stats->isValid()) { + LOG(SoftwareIsp, Error) << "Failed to create SwStatsCpu object"; + return; + } + stats->statsReady.connect(this, &SoftwareIsp::statsReady); + + debayer_ = std::make_unique(std::move(stats)); + debayer_->inputBufferReady.connect(this, &SoftwareIsp::inputReady); + debayer_->outputBufferReady.connect(this, &SoftwareIsp::outputReady); + + ipa_ = IPAManager::createIPA(pipe, 0, 0); + if (!ipa_) { + LOG(SoftwareIsp, Error) + << "Creating IPA for software ISP failed"; + debayer_.reset(); + return; + } + + int ret = ipa_->init(IPASettings{ "No cfg file", "No sensor model" }, + debayer_->getStatsFD(), + sharedParams_.fd(), + sensorControls); + if (ret) { + LOG(SoftwareIsp, Error) << "IPA init failed"; + debayer_.reset(); + return; + } + + ipa_->setIspParams.connect(this, &SoftwareIsp::saveIspParams); + ipa_->setSensorControls.connect(this, &SoftwareIsp::setSensorCtrls); + + debayer_->moveToThread(&ispWorkerThread_); +} + +SoftwareIsp::~SoftwareIsp() +{ + /* make sure to destroy the DebayerCpu before the ispWorkerThread_ is gone */ + debayer_.reset(); +} + +/** + * \fn int SoftwareIsp::loadConfiguration([[maybe_unused]] const std::string &filename) + * \brief Load a configuration from a file + * \param[in] filename The file to load the configuration data from + * + * Currently is a stub doing nothing and always returning "success". + * + * \return 0 on success + */ + +/** + * \brief Process the statistics gathered + * \param[in] sensorControls The sensor controls + * + * Requests the IPA to calculate new parameters for ISP and new control + * values for the sensor. + */ +void SoftwareIsp::processStats(const ControlList &sensorControls) +{ + ASSERT(ipa_); + ipa_->processStats(sensorControls); +} + +/** + * \brief Check the validity of Software Isp object + * \return True if Software Isp is valid, false otherwise + */ +bool SoftwareIsp::isValid() const +{ + return !!debayer_; +} + +/** + * \brief Get the output formats supported for the given input format + * \param[in] inputFormat The input format + * \return All the supported output formats or an empty vector if there are none + */ +std::vector SoftwareIsp::formats(PixelFormat inputFormat) +{ + ASSERT(debayer_ != nullptr); + + return debayer_->formats(inputFormat); +} + +/** + * \brief Get the supported output sizes for the given input format and size + * \param[in] inputFormat The input format + * \param[in] inputSize The input frame size + * \return The valid size range or an empty range if there are none + */ +SizeRange SoftwareIsp::sizes(PixelFormat inputFormat, const Size &inputSize) +{ + ASSERT(debayer_ != nullptr); + + return debayer_->sizes(inputFormat, inputSize); +} + +/** + * Get the output stride and the frame size in bytes for the given output format and size + * \param[in] outputFormat The output format + * \param[in] size The output size (width and height in pixels) + * \return A tuple of the stride and the frame size in bytes, or a tuple of 0,0 + * if there is no valid output config + */ +std::tuple +SoftwareIsp::strideAndFrameSize(const PixelFormat &outputFormat, const Size &size) +{ + ASSERT(debayer_ != nullptr); + + return debayer_->strideAndFrameSize(outputFormat, size); +} + +/** + * \brief Configure the SoftwareIsp object according to the passed in parameters + * \param[in] inputCfg The input configuration + * \param[in] outputCfgs The output configurations + * \param[in] sensorControls ControlInfoMap of the controls supported by the sensor + * \return 0 on success, a negative errno on failure + */ +int SoftwareIsp::configure(const StreamConfiguration &inputCfg, + const std::vector> &outputCfgs, + const ControlInfoMap &sensorControls) +{ + ASSERT(ipa_ != nullptr && debayer_ != nullptr); + + int ret = ipa_->configure(sensorControls); + if (ret < 0) + return ret; + + return debayer_->configure(inputCfg, outputCfgs); +} + +/** + * \brief Export the buffers from the Software ISP + * \param[in] output Output stream index exporting the buffers + * \param[in] count Number of buffers to allocate + * \param[out] buffers Vector to store the allocated buffers + * \return The number of allocated buffers on success or a negative error code + * otherwise + */ +int SoftwareIsp::exportBuffers(unsigned int output, unsigned int count, + std::vector> *buffers) +{ + ASSERT(debayer_ != nullptr); + + /* single output for now */ + if (output >= 1) + return -EINVAL; + + for (unsigned int i = 0; i < count; i++) { + const std::string name = "frame-" + std::to_string(i); + const size_t frameSize = debayer_->frameSize(); + + FrameBuffer::Plane outPlane; + outPlane.fd = SharedFD(dmaHeap_.alloc(name.c_str(), frameSize)); + if (!outPlane.fd.isValid()) { + LOG(SoftwareIsp, Error) + << "failed to allocate a dma_buf"; + return -ENOMEM; + } + outPlane.offset = 0; + outPlane.length = frameSize; + + std::vector planes{ outPlane }; + buffers->emplace_back(std::make_unique(std::move(planes))); + } + + return count; +} + +/** + * \brief Queue buffers to Software ISP + * \param[in] input The input framebuffer + * \param[in] outputs The container holding the output stream indexes and + * their respective frame buffer outputs + * \return 0 on success, a negative errno on failure + */ +int SoftwareIsp::queueBuffers(FrameBuffer *input, + const std::map &outputs) +{ + unsigned int mask = 0; + + /* + * Validate the outputs as a sanity check: at least one output is + * required, all outputs must reference a valid stream and no two + * outputs can reference the same stream. + */ + if (outputs.empty()) + return -EINVAL; + + for (auto [index, buffer] : outputs) { + if (!buffer) + return -EINVAL; + if (index >= 1) /* only single stream atm */ + return -EINVAL; + if (mask & (1 << index)) + return -EINVAL; + + mask |= 1 << index; + } + + process(input, outputs.at(0)); + + return 0; +} + +/** + * \brief Starts the Software ISP streaming operation + * \return 0 on success, any other value indicates an error + */ +int SoftwareIsp::start() +{ + int ret = ipa_->start(); + if (ret) + return ret; + + ispWorkerThread_.start(); + return 0; +} + +/** + * \brief Stops the Software ISP streaming operation + */ +void SoftwareIsp::stop() +{ + ispWorkerThread_.exit(); + ispWorkerThread_.wait(); + + ipa_->stop(); +} + +/** + * \brief Passes the input framebuffer to the ISP worker to process + * \param[in] input The input framebuffer + * \param[out] output The framebuffer to write the processed frame to + */ +void SoftwareIsp::process(FrameBuffer *input, FrameBuffer *output) +{ + debayer_->invokeMethod(&DebayerCpu::process, + ConnectionTypeQueued, input, output, debayerParams_); +} + +void SoftwareIsp::saveIspParams([[maybe_unused]] int dummy) +{ + debayerParams_ = *sharedParams_; +} + +void SoftwareIsp::setSensorCtrls(const ControlList &sensorControls) +{ + setSensorControls.emit(sensorControls); +} + +void SoftwareIsp::statsReady(int dummy) +{ + ispStatsReady.emit(dummy); +} + +void SoftwareIsp::inputReady(FrameBuffer *input) +{ + inputBufferReady.emit(input); +} + +void SoftwareIsp::outputReady(FrameBuffer *output) +{ + outputBufferReady.emit(output); +} + +} /* namespace libcamera */ From patchwork Thu Feb 29 18:36:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19582 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 02A7DC3260 for ; Thu, 29 Feb 2024 18:37:22 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AF2A96287A; Thu, 29 Feb 2024 19:37:21 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="DfpYn7d2"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id AE5C262872 for ; Thu, 29 Feb 2024 19:37:19 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231838; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6QpjvS8IQhzK31wvEVJS2o+aB5hSdgVjyf0HOcI1aFQ=; b=DfpYn7d2vdSiK3zebY8Sut7+SAxHpuOkHt20XMrCW70ujKVwNMWup4dTtRw9znREfk4Ofh sFKUZy8XRHBqrcftZHVJqPjiS9p81uyczrRBidMTa+bfEmUKhJHX+eSltAI8/gw7cU0tc3 5dRGatLC/+xi7rIJ/KnSfpSG9+e6Mgo= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-295-nN5TOSM3PE2jRuFRDJG7zA-1; Thu, 29 Feb 2024 13:37:14 -0500 X-MC-Unique: nN5TOSM3PE2jRuFRDJG7zA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A8ADB3C025BE; Thu, 29 Feb 2024 18:37:13 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 792281C06532; Thu, 29 Feb 2024 18:37:12 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 11/18] libcamera: pipeline: simple: rename converterBuffers_ and related vars Date: Thu, 29 Feb 2024 19:36:22 +0100 Message-ID: <20240229183654.7206-12-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov The converterBuffers_ and the converterQueue_ are not that specific to the Converter, and could be used by another entity doing the format conversion. Rename converterBuffers_, converterQueue_, and useConverter_ to conversionBuffers_, conversionQueue_ and useConversion_ to disassociate them from the Converter. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Kieran Bingham Reviewed-by: Pavel Machek Signed-off-by: Andrey Konovalov Signed-off-by: Hans de Goede --- src/libcamera/pipeline/simple/simple.cpp | 63 ++++++++++++------------ 1 file changed, 32 insertions(+), 31 deletions(-) diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index aa69bcc4..f4e56e3c 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -270,17 +270,18 @@ public: std::vector configs_; std::map> formats_; + std::vector> conversionBuffers_; + std::queue> conversionQueue_; + bool useConversion_; + std::unique_ptr converter_; - std::vector> converterBuffers_; - bool useConverter_; - std::queue> converterQueue_; private: void tryPipeline(unsigned int code, const Size &size); static std::vector routedSourcePads(MediaPad *sink); - void converterInputDone(FrameBuffer *buffer); - void converterOutputDone(FrameBuffer *buffer); + void conversionInputDone(FrameBuffer *buffer); + void conversionOutputDone(FrameBuffer *buffer); }; class SimpleCameraConfiguration : public CameraConfiguration @@ -504,8 +505,8 @@ int SimpleCameraData::init() << "Failed to create converter, disabling format conversion"; converter_.reset(); } else { - converter_->inputBufferReady.connect(this, &SimpleCameraData::converterInputDone); - converter_->outputBufferReady.connect(this, &SimpleCameraData::converterOutputDone); + converter_->inputBufferReady.connect(this, &SimpleCameraData::conversionInputDone); + converter_->outputBufferReady.connect(this, &SimpleCameraData::conversionOutputDone); } } @@ -741,7 +742,7 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) * point converting an erroneous buffer. */ if (buffer->metadata().status != FrameMetadata::FrameSuccess) { - if (!useConverter_) { + if (!useConversion_) { /* No conversion, just complete the request. */ Request *request = buffer->request(); pipe->completeBuffer(request, buffer); @@ -757,16 +758,16 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) if (buffer->metadata().status != FrameMetadata::FrameCancelled) video_->queueBuffer(buffer); - if (converterQueue_.empty()) + if (conversionQueue_.empty()) return; Request *request = nullptr; - for (auto &item : converterQueue_.front()) { + for (auto &item : conversionQueue_.front()) { FrameBuffer *outputBuffer = item.second; request = outputBuffer->request(); pipe->completeBuffer(request, outputBuffer); } - converterQueue_.pop(); + conversionQueue_.pop(); if (request) pipe->completeRequest(request); @@ -783,9 +784,9 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) */ Request *request = buffer->request(); - if (useConverter_ && !converterQueue_.empty()) { + if (useConversion_ && !conversionQueue_.empty()) { const std::map &outputs = - converterQueue_.front(); + conversionQueue_.front(); if (!outputs.empty()) { FrameBuffer *outputBuffer = outputs.begin()->second; if (outputBuffer) @@ -802,14 +803,14 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) * conversion is needed. If there's no queued request, just requeue the * captured buffer for capture. */ - if (useConverter_) { - if (converterQueue_.empty()) { + if (useConversion_) { + if (conversionQueue_.empty()) { video_->queueBuffer(buffer); return; } - converter_->queueBuffers(buffer, converterQueue_.front()); - converterQueue_.pop(); + converter_->queueBuffers(buffer, conversionQueue_.front()); + conversionQueue_.pop(); return; } @@ -818,13 +819,13 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) pipe->completeRequest(request); } -void SimpleCameraData::converterInputDone(FrameBuffer *buffer) +void SimpleCameraData::conversionInputDone(FrameBuffer *buffer) { /* Queue the input buffer back for capture. */ video_->queueBuffer(buffer); } -void SimpleCameraData::converterOutputDone(FrameBuffer *buffer) +void SimpleCameraData::conversionOutputDone(FrameBuffer *buffer) { SimplePipelineHandler *pipe = SimpleCameraData::pipe(); @@ -1190,14 +1191,14 @@ int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c) /* Configure the converter if needed. */ std::vector> outputCfgs; - data->useConverter_ = config->needConversion(); + data->useConversion_ = config->needConversion(); for (unsigned int i = 0; i < config->size(); ++i) { StreamConfiguration &cfg = config->at(i); cfg.setStream(&data->streams_[i]); - if (data->useConverter_) + if (data->useConversion_) outputCfgs.push_back(cfg); } @@ -1223,7 +1224,7 @@ int SimplePipelineHandler::exportFrameBuffers(Camera *camera, Stream *stream, * Export buffers on the converter or capture video node, depending on * whether the converter is used or not. */ - if (data->useConverter_) + if (data->useConversion_) return data->converter_->exportBuffers(data->streamIndex(stream), count, buffers); else @@ -1244,13 +1245,13 @@ int SimplePipelineHandler::start(Camera *camera, [[maybe_unused]] const ControlL return -EBUSY; } - if (data->useConverter_) { + if (data->useConversion_) { /* * When using the converter allocate a fixed number of internal * buffers. */ ret = video->allocateBuffers(kNumInternalBuffers, - &data->converterBuffers_); + &data->conversionBuffers_); } else { /* Otherwise, prepare for using buffers from the only stream. */ Stream *stream = &data->streams_[0]; @@ -1269,7 +1270,7 @@ int SimplePipelineHandler::start(Camera *camera, [[maybe_unused]] const ControlL return ret; } - if (data->useConverter_) { + if (data->useConversion_) { ret = data->converter_->start(); if (ret < 0) { stop(camera); @@ -1277,7 +1278,7 @@ int SimplePipelineHandler::start(Camera *camera, [[maybe_unused]] const ControlL } /* Queue all internal buffers for capture. */ - for (std::unique_ptr &buffer : data->converterBuffers_) + for (std::unique_ptr &buffer : data->conversionBuffers_) video->queueBuffer(buffer.get()); } @@ -1289,7 +1290,7 @@ void SimplePipelineHandler::stopDevice(Camera *camera) SimpleCameraData *data = cameraData(camera); V4L2VideoDevice *video = data->video_; - if (data->useConverter_) + if (data->useConversion_) data->converter_->stop(); video->streamOff(); @@ -1297,7 +1298,7 @@ void SimplePipelineHandler::stopDevice(Camera *camera) video->bufferReady.disconnect(data, &SimpleCameraData::bufferReady); - data->converterBuffers_.clear(); + data->conversionBuffers_.clear(); releasePipeline(data); } @@ -1315,7 +1316,7 @@ int SimplePipelineHandler::queueRequestDevice(Camera *camera, Request *request) * queue, it will be handed to the converter in the capture * completion handler. */ - if (data->useConverter_) { + if (data->useConversion_) { buffers.emplace(data->streamIndex(stream), buffer); } else { ret = data->video_->queueBuffer(buffer); @@ -1324,8 +1325,8 @@ int SimplePipelineHandler::queueRequestDevice(Camera *camera, Request *request) } } - if (data->useConverter_) - data->converterQueue_.push(std::move(buffers)); + if (data->useConversion_) + data->conversionQueue_.push(std::move(buffers)); return 0; } From patchwork Thu Feb 29 18:36:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19583 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B1FA7BD160 for ; Thu, 29 Feb 2024 18:37:22 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5AB0D628AC; Thu, 29 Feb 2024 19:37:22 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="DuLpTJZE"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 20E096295C for ; Thu, 29 Feb 2024 19:37:20 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231839; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZCGGG09266/HfVuCKCr/1r+yeE30VuyotvYb7b55Cwg=; b=DuLpTJZEMebwOh7uZBuoAQ1QvCxvu7M2bDWpkaW8h/+1sGNfFb6mFQV0t5fwEu+hSFU+NR gIdw6JCTsKOpTcsJeAwtiGQNFRLDAH+ilXsmWjejMdBh8qiqKpZsTSR5LdHeEVoHN906Ll yHaKIqhTjo0F2HPmxvJbJ41r5x0wT6U= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-538-Q14wReTEMhGqkuaeZvsu4Q-1; Thu, 29 Feb 2024 13:37:15 -0500 X-MC-Unique: Q14wReTEMhGqkuaeZvsu4Q-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EC318848524; Thu, 29 Feb 2024 18:37:14 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id D9A981C060AF; Thu, 29 Feb 2024 18:37:13 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 12/18] libcamera: pipeline: simple: enable use of Soft ISP and Soft IPA Date: Thu, 29 Feb 2024 19:36:23 +0100 Message-ID: <20240229183654.7206-13-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrey Konovalov To enable the Simple Soft ISP and Soft IPA for simple pipeline handler configure the build with: -Dpipelines=simple -Dipas=simple Also using the Soft ISP for the particular hardware platform must be enabled in the supportedDevices[] table. If the pipeline uses Converter, Soft ISP and Soft IPA aren't available. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Signed-off-by: Andrey Konovalov Signed-off-by: Hans de Goede --- src/libcamera/pipeline/simple/simple.cpp | 138 ++++++++++++++++++----- 1 file changed, 109 insertions(+), 29 deletions(-) diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index f4e56e3c..e491ab62 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -34,6 +34,7 @@ #include "libcamera/internal/device_enumerator.h" #include "libcamera/internal/media_device.h" #include "libcamera/internal/pipeline_handler.h" +#include "libcamera/internal/software_isp/software_isp.h" #include "libcamera/internal/v4l2_subdevice.h" #include "libcamera/internal/v4l2_videodevice.h" @@ -185,18 +186,23 @@ struct SimplePipelineInfo { * and the number of streams it supports. */ std::vector> converters; + /* + * Using Software ISP is to be enabled per driver. + * The Software ISP can't be used together with the converters. + */ + bool swIspEnabled; }; namespace { static const SimplePipelineInfo supportedDevices[] = { - { "dcmipp", {} }, - { "imx7-csi", { { "pxp", 1 } } }, - { "j721e-csi2rx", {} }, - { "mtk-seninf", { { "mtk-mdp", 3 } } }, - { "mxc-isi", {} }, - { "qcom-camss", {} }, - { "sun6i-csi", {} }, + { "dcmipp", {}, false }, + { "imx7-csi", { { "pxp", 1 } }, false }, + { "j721e-csi2rx", {}, false }, + { "mtk-seninf", { { "mtk-mdp", 3 } }, false }, + { "mxc-isi", {}, false }, + { "qcom-camss", {}, true }, + { "sun6i-csi", {}, false }, }; } /* namespace */ @@ -275,6 +281,7 @@ public: bool useConversion_; std::unique_ptr converter_; + std::unique_ptr swIsp_; private: void tryPipeline(unsigned int code, const Size &size); @@ -282,6 +289,9 @@ private: void conversionInputDone(FrameBuffer *buffer); void conversionOutputDone(FrameBuffer *buffer); + + void ispStatsReady(int dummy); + void setSensorControls(const ControlList &sensorControls); }; class SimpleCameraConfiguration : public CameraConfiguration @@ -333,6 +343,7 @@ public: V4L2VideoDevice *video(const MediaEntity *entity); V4L2Subdevice *subdev(const MediaEntity *entity); MediaDevice *converter() { return converter_; } + bool swIspEnabled() { return swIspEnabled_; } protected: int queueRequestDevice(Camera *camera, Request *request) override; @@ -361,6 +372,7 @@ private: std::map entities_; MediaDevice *converter_; + bool swIspEnabled_; }; /* ----------------------------------------------------------------------------- @@ -510,6 +522,29 @@ int SimpleCameraData::init() } } + /* + * Instantiate Soft ISP if this is enabled for the given driver and no converter is used. + */ + if (!converter_ && pipe->swIspEnabled()) { + swIsp_ = std::make_unique(pipe, sensor_->controls()); + if (!swIsp_->isValid()) { + LOG(SimplePipeline, Warning) + << "Failed to create software ISP, disabling software debayering"; + swIsp_.reset(); + } else { + /* + * \todo explain why SimpleCameraData::conversionInputDone() can't be directly + * connected to inputBufferReady signal. + */ + swIsp_->inputBufferReady.connect(pipe, [this](FrameBuffer *buffer) { + this->conversionInputDone(buffer); + }); + swIsp_->outputBufferReady.connect(this, &SimpleCameraData::conversionOutputDone); + swIsp_->ispStatsReady.connect(this, &SimpleCameraData::ispStatsReady); + swIsp_->setSensorControls.connect(this, &SimpleCameraData::setSensorControls); + } + } + video_ = pipe->video(entities_.back().entity); ASSERT(video_); @@ -600,12 +635,20 @@ void SimpleCameraData::tryPipeline(unsigned int code, const Size &size) config.captureFormat = pixelFormat; config.captureSize = format.size; - if (!converter_) { - config.outputFormats = { pixelFormat }; - config.outputSizes = config.captureSize; - } else { + if (converter_) { config.outputFormats = converter_->formats(pixelFormat); config.outputSizes = converter_->sizes(format.size); + } else if (swIsp_) { + config.outputFormats = swIsp_->formats(pixelFormat); + config.outputSizes = swIsp_->sizes(pixelFormat, format.size); + if (config.outputFormats.empty()) { + /* Do not use swIsp for unsupported pixelFormat's: */ + config.outputFormats = { pixelFormat }; + config.outputSizes = config.captureSize; + } + } else { + config.outputFormats = { pixelFormat }; + config.outputSizes = config.captureSize; } configs_.push_back(config); @@ -751,9 +794,9 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) } /* - * The converter is in use. Requeue the internal buffer for - * capture (unless the stream is being stopped), and complete - * the request with all the user-facing buffers. + * The converter or Software ISP is in use. Requeue the internal + * buffer for capture (unless the stream is being stopped), and + * complete the request with all the user-facing buffers. */ if (buffer->metadata().status != FrameMetadata::FrameCancelled) video_->queueBuffer(buffer); @@ -799,9 +842,9 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) buffer->metadata().timestamp); /* - * Queue the captured and the request buffer to the converter if format - * conversion is needed. If there's no queued request, just requeue the - * captured buffer for capture. + * Queue the captured and the request buffer to the converter or Software + * ISP if format conversion is needed. If there's no queued request, just + * requeue the captured buffer for capture. */ if (useConversion_) { if (conversionQueue_.empty()) { @@ -809,7 +852,11 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) return; } - converter_->queueBuffers(buffer, conversionQueue_.front()); + if (converter_) + converter_->queueBuffers(buffer, conversionQueue_.front()); + else + swIsp_->queueBuffers(buffer, conversionQueue_.front()); + conversionQueue_.pop(); return; } @@ -835,6 +882,18 @@ void SimpleCameraData::conversionOutputDone(FrameBuffer *buffer) pipe->completeRequest(request); } +void SimpleCameraData::ispStatsReady([[maybe_unused]] int dummy) +{ + swIsp_->processStats(sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, + V4L2_CID_EXPOSURE })); +} + +void SimpleCameraData::setSensorControls(const ControlList &sensorControls) +{ + ControlList ctrls(sensorControls); + sensor_->setControls(&ctrls); +} + /* Retrieve all source pads connected to a sink pad through active routes. */ std::vector SimpleCameraData::routedSourcePads(MediaPad *sink) { @@ -1047,8 +1106,10 @@ CameraConfiguration::Status SimpleCameraConfiguration::validate() /* Set the stride, frameSize and bufferCount. */ if (needConversion_) { std::tie(cfg.stride, cfg.frameSize) = - data_->converter_->strideAndFrameSize(cfg.pixelFormat, - cfg.size); + (data_->converter_) ? data_->converter_->strideAndFrameSize(cfg.pixelFormat, + cfg.size) + : data_->swIsp_->strideAndFrameSize(cfg.pixelFormat, + cfg.size); if (cfg.stride == 0) return Invalid; } else { @@ -1211,7 +1272,9 @@ int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c) inputCfg.stride = captureFormat.planes[0].bpl; inputCfg.bufferCount = kNumInternalBuffers; - return data->converter_->configure(inputCfg, outputCfgs); + return (data->converter_) ? data->converter_->configure(inputCfg, outputCfgs) + : data->swIsp_->configure(inputCfg, outputCfgs, + data->sensor_->controls()); } int SimplePipelineHandler::exportFrameBuffers(Camera *camera, Stream *stream, @@ -1225,8 +1288,10 @@ int SimplePipelineHandler::exportFrameBuffers(Camera *camera, Stream *stream, * whether the converter is used or not. */ if (data->useConversion_) - return data->converter_->exportBuffers(data->streamIndex(stream), - count, buffers); + return (data->converter_) ? data->converter_->exportBuffers(data->streamIndex(stream), + count, buffers) + : data->swIsp_->exportBuffers(data->streamIndex(stream), + count, buffers); else return data->video_->exportBuffers(count, buffers); } @@ -1271,10 +1336,18 @@ int SimplePipelineHandler::start(Camera *camera, [[maybe_unused]] const ControlL } if (data->useConversion_) { - ret = data->converter_->start(); - if (ret < 0) { - stop(camera); - return ret; + if (data->converter_) { + ret = data->converter_->start(); + if (ret < 0) { + stop(camera); + return ret; + } + } else if (data->swIsp_) { + ret = data->swIsp_->start(); + if (ret < 0) { + stop(camera); + return ret; + } } /* Queue all internal buffers for capture. */ @@ -1290,8 +1363,13 @@ void SimplePipelineHandler::stopDevice(Camera *camera) SimpleCameraData *data = cameraData(camera); V4L2VideoDevice *video = data->video_; - if (data->useConversion_) - data->converter_->stop(); + if (data->useConversion_) { + if (data->converter_) + data->converter_->stop(); + else if (data->swIsp_) { + data->swIsp_->stop(); + } + } video->streamOff(); video->releaseBuffers(); @@ -1453,6 +1531,8 @@ bool SimplePipelineHandler::match(DeviceEnumerator *enumerator) } } + swIspEnabled_ = info->swIspEnabled; + /* Locate the sensors. */ std::vector sensors = locateSensors(); if (sensors.empty()) { From patchwork Thu Feb 29 18:36:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19584 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C9134BD160 for ; Thu, 29 Feb 2024 18:37:25 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 741BC62C81; Thu, 29 Feb 2024 19:37:25 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="KHVDo1/v"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3F43F62878 for ; Thu, 29 Feb 2024 19:37:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231842; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mvFGjF8voLB3VKikKpdqxFCwGlrp1UAUDpjAFYWQZ2I=; b=KHVDo1/vyRjvDmVIqljkAXcyLrzKt9dMYbbhefTr4izG8TwJ4g0CwUYrmPSDQyk3E2Wyw+ nGO3I5kp+tzuU225HsTq84prNJva9sXOEQwmowBGWoHaQsvPSROqgzd+txGee2XpUXPJS4 dnA4pP2RLP2Ju8uu8kHqsIGG69pJC/o= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-582-Tb_oFwIsOg67FVMsKVJVPg-1; Thu, 29 Feb 2024 13:37:17 -0500 X-MC-Unique: Tb_oFwIsOg67FVMsKVJVPg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3B7AE848524; Thu, 29 Feb 2024 18:37:16 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 297B31C060B1; Thu, 29 Feb 2024 18:37:15 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 13/18] libcamera: swstats_cpu: Add support for 8, 10 and 12 bpp unpacked bayer input Date: Thu, 29 Feb 2024 19:36:24 +0100 Message-ID: <20240229183654.7206-14-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add support for 8, 10 and 12 bpp unpacked bayer input for all 4 standard bayer orders. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Signed-off-by: Kieran Bingham Signed-off-by: Hans de Goede Reviewed-by: Milan Zamazal --- src/libcamera/software_isp/swstats_cpu.cpp | 128 +++++++++++++++++++++ src/libcamera/software_isp/swstats_cpu.h | 9 ++ 2 files changed, 137 insertions(+) diff --git a/src/libcamera/software_isp/swstats_cpu.cpp b/src/libcamera/software_isp/swstats_cpu.cpp index 448d0e4c..be310f56 100644 --- a/src/libcamera/software_isp/swstats_cpu.cpp +++ b/src/libcamera/software_isp/swstats_cpu.cpp @@ -71,6 +71,83 @@ static const unsigned int kBlueYMul = 29; /* 0.114 * 256 */ stats_.sumG_ += sumG; \ stats_.sumB_ += sumB; +void SwStatsCpu::statsBGGR8Line0(const uint8_t *src[]) +{ + const uint8_t *src0 = src[1] + window_.x; + const uint8_t *src1 = src[2] + window_.x; + + SWSTATS_START_LINE_STATS(uint8_t) + + if (swapLines_) + std::swap(src0, src1); + + /* x += 4 sample every other 2x2 block */ + for (int x = 0; x < (int)window_.width; x += 4) { + b = src0[x]; + g = src0[x + 1]; + g2 = src1[x]; + r = src1[x + 1]; + + g = (g + g2) / 2; + + SWSTATS_ACCUMULATE_LINE_STATS(1) + } + + SWSTATS_FINISH_LINE_STATS() +} + +void SwStatsCpu::statsBGGR10Line0(const uint8_t *src[]) +{ + const uint16_t *src0 = (const uint16_t *)src[1] + window_.x; + const uint16_t *src1 = (const uint16_t *)src[2] + window_.x; + + SWSTATS_START_LINE_STATS(uint16_t) + + if (swapLines_) + std::swap(src0, src1); + + /* x += 4 sample every other 2x2 block */ + for (int x = 0; x < (int)window_.width; x += 4) { + b = src0[x]; + g = src0[x + 1]; + g2 = src1[x]; + r = src1[x + 1]; + + g = (g + g2) / 2; + + /* divide Y by 4 for 10 -> 8 bpp value */ + SWSTATS_ACCUMULATE_LINE_STATS(4) + } + + SWSTATS_FINISH_LINE_STATS() +} + +void SwStatsCpu::statsBGGR12Line0(const uint8_t *src[]) +{ + const uint16_t *src0 = (const uint16_t *)src[1] + window_.x; + const uint16_t *src1 = (const uint16_t *)src[2] + window_.x; + + SWSTATS_START_LINE_STATS(uint16_t) + + if (swapLines_) + std::swap(src0, src1); + + /* x += 4 sample every other 2x2 block */ + for (int x = 0; x < (int)window_.width; x += 4) { + b = src0[x]; + g = src0[x + 1]; + g2 = src1[x]; + r = src1[x + 1]; + + g = (g + g2) / 2; + + /* divide Y by 16 for 12 -> 8 bpp value */ + SWSTATS_ACCUMULATE_LINE_STATS(16) + } + + SWSTATS_FINISH_LINE_STATS() +} + void SwStatsCpu::statsBGGR10PLine0(const uint8_t *src[]) { const uint8_t *src0 = src[1] + window_.x * 5 / 4; @@ -147,6 +224,42 @@ void SwStatsCpu::finishFrame(void) statsReady.emit(0); } +/** + * \brief Setup SwStatsCpu object for standard Bayer orders + * \param[in] order The Bayer order + * + * Check if order is a standard Bayer order and setup xShift_ and swapLines_ + * so that a single BGGR stats function can be used for all 4 standard orders. + */ +int SwStatsCpu::setupStandardBayerOrder(BayerFormat::Order order) +{ + switch (order) { + case BayerFormat::BGGR: + xShift_ = 0; + swapLines_ = false; + break; + case BayerFormat::GBRG: + xShift_ = 1; /* BGGR -> GBRG */ + swapLines_ = false; + break; + case BayerFormat::GRBG: + xShift_ = 0; + swapLines_ = true; /* BGGR -> GRBG */ + break; + case BayerFormat::RGGB: + xShift_ = 1; /* BGGR -> GBRG */ + swapLines_ = true; /* GBRG -> RGGB */ + break; + default: + return -EINVAL; + } + + patternSize_.height = 2; + patternSize_.width = 2; + ySkipMask_ = 0x02; /* Skip every 3th and 4th line */ + return 0; +} + /** * \brief Configure the statistics object for the passed in input format. * \param[in] inputCfg The input format @@ -158,6 +271,21 @@ int SwStatsCpu::configure(const StreamConfiguration &inputCfg) BayerFormat bayerFormat = BayerFormat::fromPixelFormat(inputCfg.pixelFormat); + if (bayerFormat.packing == BayerFormat::Packing::None && + setupStandardBayerOrder(bayerFormat.order) == 0) { + switch (bayerFormat.bitDepth) { + case 8: + stats0_ = &SwStatsCpu::statsBGGR8Line0; + return 0; + case 10: + stats0_ = &SwStatsCpu::statsBGGR10Line0; + return 0; + case 12: + stats0_ = &SwStatsCpu::statsBGGR12Line0; + return 0; + } + } + if (bayerFormat.bitDepth == 10 && bayerFormat.packing == BayerFormat::Packing::CSI2) { patternSize_.height = 2; diff --git a/src/libcamera/software_isp/swstats_cpu.h b/src/libcamera/software_isp/swstats_cpu.h index df13263b..d956c15a 100644 --- a/src/libcamera/software_isp/swstats_cpu.h +++ b/src/libcamera/software_isp/swstats_cpu.h @@ -17,6 +17,7 @@ #include +#include "libcamera/internal/bayer_format.h" #include "libcamera/internal/shared_mem_object.h" #include "libcamera/internal/software_isp/swisp_stats.h" @@ -120,6 +121,14 @@ private: */ using statsProcessFn = void (SwStatsCpu::*)(const uint8_t *src[]); + int setupStandardBayerOrder(BayerFormat::Order order); + /* Bayer 8 bpp unpacked */ + void statsBGGR8Line0(const uint8_t *src[]); + /* Bayer 10 bpp unpacked */ + void statsBGGR10Line0(const uint8_t *src[]); + /* Bayer 12 bpp unpacked */ + void statsBGGR12Line0(const uint8_t *src[]); + /* Bayer 10 bpp packed */ void statsBGGR10PLine0(const uint8_t *src[]); void statsGBRG10PLine0(const uint8_t *src[]); From patchwork Thu Feb 29 18:36:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19589 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id DAAB5BD160 for ; Thu, 29 Feb 2024 18:37:39 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 960EC62871; Thu, 29 Feb 2024 19:37:39 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="Yiy8/Hw0"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9F8706286C for ; Thu, 29 Feb 2024 19:37:37 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231856; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X9dhoVefx3VKvVFsEfX1bv24fjvJYRu0cJbTg6l1j3s=; b=Yiy8/Hw0HbSVWgcmCWIdSlGz5QQDT6XLBAEgDNejZJmdfwUVUuoWgQxYmwbfzPPJUaxkHR cTWOoDuGc9kASPTK+8jE8gxm9qSCgoyNELCJcK5+i1pCW+WKcFtlulrgiOE4nIa4+6KQte ocGFDVmeAnQXzp091h8dTx0re8NDnNs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-496-2lIOIobdPpCrYs-nWX0u2g-1; Thu, 29 Feb 2024 13:37:33 -0500 X-MC-Unique: 2lIOIobdPpCrYs-nWX0u2g-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8449A84852B; Thu, 29 Feb 2024 18:37:17 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6C7541C060AF; Thu, 29 Feb 2024 18:37:16 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 14/18] libcamera: debayer_cpu: Add support for 8, 10 and 12 bpp unpacked bayer input Date: Thu, 29 Feb 2024 19:36:25 +0100 Message-ID: <20240229183654.7206-15-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add support for 8, 10 and 12 bpp unpacked bayer input for all 4 standard bayer orders. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Reviewed-by: Pavel Machek Signed-off-by: Kieran Bingham Signed-off-by: Hans de Goede Reviewed-by: Milan Zamazal --- Changes in v3: - Also add support for 12bpp (by Kieran Bingham) --- src/libcamera/software_isp/debayer_cpu.cpp | 128 +++++++++++++++++++++ src/libcamera/software_isp/debayer_cpu.h | 13 +++ 2 files changed, 141 insertions(+) diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index 53e90776..22ae99cd 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -56,6 +56,11 @@ DebayerCpu::~DebayerCpu() free(lineBuffers_[i]); } +#define DECLARE_SRC_POINTERS(pixel_t) \ + const pixel_t *prev = (const pixel_t *)src[0] + xShift_; \ + const pixel_t *curr = (const pixel_t *)src[1] + xShift_; \ + const pixel_t *next = (const pixel_t *)src[2] + xShift_; + // RGR // GBG // RGR @@ -92,6 +97,70 @@ DebayerCpu::~DebayerCpu() *dst++ = red_[curr[x] / (div)]; \ x++; +void DebayerCpu::debayer8_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + DECLARE_SRC_POINTERS(uint8_t) + + for (int x = 0; x < (int)window_.width;) { + BGGR_BGR888(1, 1, 1) + GBRG_BGR888(1, 1, 1) + } +} + +void DebayerCpu::debayer8_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + DECLARE_SRC_POINTERS(uint8_t) + + for (int x = 0; x < (int)window_.width;) { + GRBG_BGR888(1, 1, 1) + RGGB_BGR888(1, 1, 1) + } +} + +void DebayerCpu::debayer10_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + DECLARE_SRC_POINTERS(uint16_t) + + for (int x = 0; x < (int)window_.width;) { + /* divide values by 4 for 10 -> 8 bpp value */ + BGGR_BGR888(1, 1, 4) + GBRG_BGR888(1, 1, 4) + } +} + +void DebayerCpu::debayer10_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + DECLARE_SRC_POINTERS(uint16_t) + + for (int x = 0; x < (int)window_.width;) { + /* divide values by 4 for 10 -> 8 bpp value */ + GRBG_BGR888(1, 1, 4) + RGGB_BGR888(1, 1, 4) + } +} + +void DebayerCpu::debayer12_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + DECLARE_SRC_POINTERS(uint16_t) + + for (int x = 0; x < (int)window_.width;) { + /* divide values by 16 for 12 -> 8 bpp value */ + BGGR_BGR888(1, 1, 16) + GBRG_BGR888(1, 1, 16) + } +} + +void DebayerCpu::debayer12_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]) +{ + DECLARE_SRC_POINTERS(uint16_t) + + for (int x = 0; x < (int)window_.width;) { + /* divide values by 16 for 12 -> 8 bpp value */ + GRBG_BGR888(1, 1, 16) + RGGB_BGR888(1, 1, 16) + } +} + void DebayerCpu::debayer10P_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]) { const int width_in_bytes = window_.width * 5 / 4; @@ -186,6 +255,16 @@ int DebayerCpu::getInputConfig(PixelFormat inputFormat, DebayerInputConfig &conf BayerFormat bayerFormat = BayerFormat::fromPixelFormat(inputFormat); + if ((bayerFormat.bitDepth == 8 || bayerFormat.bitDepth == 10 || bayerFormat.bitDepth == 12) && + bayerFormat.packing == BayerFormat::Packing::None && + isStandardBayerOrder(bayerFormat.order)) { + config.bpp = (bayerFormat.bitDepth + 7) & ~7; + config.patternSize.width = 2; + config.patternSize.height = 2; + config.outputFormats = std::vector({ formats::RGB888 }); + return 0; + } + if (bayerFormat.bitDepth == 10 && bayerFormat.packing == BayerFormat::Packing::CSI2 && isStandardBayerOrder(bayerFormat.order)) { @@ -213,12 +292,61 @@ int DebayerCpu::getOutputConfig(PixelFormat outputFormat, DebayerOutputConfig &c return -EINVAL; } +/* + * Check for standard Bayer orders and set xShift_ and swap debayer0/1, so that + * a single pair of BGGR debayer functions can be used for all 4 standard orders. + */ +int DebayerCpu::setupStandardBayerOrder(BayerFormat::Order order) +{ + switch (order) { + case BayerFormat::BGGR: + break; + case BayerFormat::GBRG: + xShift_ = 1; /* BGGR -> GBRG */ + break; + case BayerFormat::GRBG: + std::swap(debayer0_, debayer1_); /* BGGR -> GRBG */ + break; + case BayerFormat::RGGB: + xShift_ = 1; /* BGGR -> GBRG */ + std::swap(debayer0_, debayer1_); /* GBRG -> RGGB */ + break; + default: + return -EINVAL; + } + + return 0; +} + /* TODO: this ignores outputFormat since there is only 1 supported outputFormat for now */ int DebayerCpu::setDebayerFunctions(PixelFormat inputFormat, [[maybe_unused]] PixelFormat outputFormat) { BayerFormat bayerFormat = BayerFormat::fromPixelFormat(inputFormat); + xShift_ = 0; + + if ((bayerFormat.bitDepth == 8 || bayerFormat.bitDepth == 10 || bayerFormat.bitDepth == 12) && + bayerFormat.packing == BayerFormat::Packing::None && + isStandardBayerOrder(bayerFormat.order)) { + switch (bayerFormat.bitDepth) { + case 8: + debayer0_ = &DebayerCpu::debayer8_BGBG_BGR888; + debayer1_ = &DebayerCpu::debayer8_GRGR_BGR888; + break; + case 10: + debayer0_ = &DebayerCpu::debayer10_BGBG_BGR888; + debayer1_ = &DebayerCpu::debayer10_GRGR_BGR888; + break; + case 12: + debayer0_ = &DebayerCpu::debayer12_BGBG_BGR888; + debayer1_ = &DebayerCpu::debayer12_GRGR_BGR888; + break; + } + setupStandardBayerOrder(bayerFormat.order); + return 0; + } + if (bayerFormat.bitDepth == 10 && bayerFormat.packing == BayerFormat::Packing::CSI2) { switch (bayerFormat.order) { diff --git a/src/libcamera/software_isp/debayer_cpu.h b/src/libcamera/software_isp/debayer_cpu.h index 98782838..73ca41f1 100644 --- a/src/libcamera/software_isp/debayer_cpu.h +++ b/src/libcamera/software_isp/debayer_cpu.h @@ -17,6 +17,8 @@ #include +#include "libcamera/internal/bayer_format.h" + #include "debayer.h" #include "swstats_cpu.h" @@ -82,6 +84,15 @@ private: */ using debayerFn = void (DebayerCpu::*)(uint8_t *dst, const uint8_t *src[]); + /* 8-bit raw bayer format */ + void debayer8_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]); + void debayer8_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]); + /* unpacked 10-bit raw bayer format */ + void debayer10_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]); + void debayer10_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]); + /* unpacked 12-bit raw bayer format */ + void debayer12_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]); + void debayer12_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]); /* CSI-2 packed 10-bit raw bayer format (all the 4 orders) */ void debayer10P_BGBG_BGR888(uint8_t *dst, const uint8_t *src[]); void debayer10P_GRGR_BGR888(uint8_t *dst, const uint8_t *src[]); @@ -103,6 +114,7 @@ private: int getInputConfig(PixelFormat inputFormat, DebayerInputConfig &config); int getOutputConfig(PixelFormat outputFormat, DebayerOutputConfig &config); + int setupStandardBayerOrder(BayerFormat::Order order); int setDebayerFunctions(PixelFormat inputFormat, PixelFormat outputFormat); void setupInputMemcpy(const uint8_t *linePointers[]); void shiftLinePointers(const uint8_t *linePointers[], const uint8_t *src); @@ -131,6 +143,7 @@ private: unsigned int lineBufferLength_; unsigned int lineBufferPadding_; unsigned int lineBufferIndex_; + unsigned int xShift_; /* Offset of 0/1 applied to window_.x */ bool enableInputMemcpy_; float gamma_correction_; unsigned int measuredFrames_; From patchwork Thu Feb 29 18:36:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19585 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8D49CC3260 for ; Thu, 29 Feb 2024 18:37:26 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 478C3628AC; Thu, 29 Feb 2024 19:37:26 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="Rcw07sXg"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9F76862867 for ; Thu, 29 Feb 2024 19:37:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231842; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zF+jY1hMJCOKrIyTdrEWjnNF4aEiOnF+X5CTW5L4Lyk=; b=Rcw07sXg7lyKBExwj7pRZGvHppV4iFwSvFVoCTVIcX5G6+l/5mk5KUjvrm3iPZ3DfbW0SK oNF50mEiwl6jjx8/MTovwcqPzFJmJwhlW2/lKwZVOrT+3Z8dPjrA7q2oBeYaPAS9LmbuBS qYEk9kTDVVLqsGJraAgRj/qh9x+25MI= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-418-fghS5g8GPcahZC2axQrBwg-1; Thu, 29 Feb 2024 13:37:19 -0500 X-MC-Unique: fghS5g8GPcahZC2axQrBwg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AF19B1C05EA6; Thu, 29 Feb 2024 18:37:18 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id B569D1C060B1; Thu, 29 Feb 2024 18:37:17 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 15/18] libcamera: debayer_cpu: Add BGR888 output support Date: Thu, 29 Feb 2024 19:36:26 +0100 Message-ID: <20240229183654.7206-16-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" BGR888 is RGB888 with the red and blue pixels swapped, adjust the debayering to swap the red and blue pixels in the bayer pattern to add support for writing formats::BGR888. Tested-by: Bryan O'Donoghue # sc8280xp Lenovo x13s Tested-by: Pavel Machek Signed-off-by: Hans de Goede Reviewed-by: Milan Zamazal --- src/libcamera/software_isp/debayer_cpu.cpp | 42 +++++++++++++++++++--- src/libcamera/software_isp/debayer_cpu.h | 1 + 2 files changed, 38 insertions(+), 5 deletions(-) diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index 22ae99cd..215ca615 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -261,7 +261,7 @@ int DebayerCpu::getInputConfig(PixelFormat inputFormat, DebayerInputConfig &conf config.bpp = (bayerFormat.bitDepth + 7) & ~7; config.patternSize.width = 2; config.patternSize.height = 2; - config.outputFormats = std::vector({ formats::RGB888 }); + config.outputFormats = std::vector({ formats::RGB888, formats::BGR888 }); return 0; } @@ -271,7 +271,7 @@ int DebayerCpu::getInputConfig(PixelFormat inputFormat, DebayerInputConfig &conf config.bpp = 10; config.patternSize.width = 4; /* 5 bytes per *4* pixels */ config.patternSize.height = 2; - config.outputFormats = std::vector({ formats::RGB888 }); + config.outputFormats = std::vector({ formats::RGB888, formats::BGR888 }); return 0; } @@ -282,7 +282,7 @@ int DebayerCpu::getInputConfig(PixelFormat inputFormat, DebayerInputConfig &conf int DebayerCpu::getOutputConfig(PixelFormat outputFormat, DebayerOutputConfig &config) { - if (outputFormat == formats::RGB888) { + if (outputFormat == formats::RGB888 || outputFormat == formats::BGR888) { config.bpp = 24; return 0; } @@ -318,13 +318,41 @@ int DebayerCpu::setupStandardBayerOrder(BayerFormat::Order order) return 0; } -/* TODO: this ignores outputFormat since there is only 1 supported outputFormat for now */ -int DebayerCpu::setDebayerFunctions(PixelFormat inputFormat, [[maybe_unused]] PixelFormat outputFormat) +int DebayerCpu::setDebayerFunctions(PixelFormat inputFormat, PixelFormat outputFormat) { BayerFormat bayerFormat = BayerFormat::fromPixelFormat(inputFormat); xShift_ = 0; + swapRedBlueGains_ = false; + + switch (outputFormat) { + case formats::RGB888: + break; + case formats::BGR888: + /* Swap R and B in bayer order to generate BGR888 instead of RGB888 */ + swapRedBlueGains_ = true; + + switch (bayerFormat.order) { + case BayerFormat::BGGR: + bayerFormat.order = BayerFormat::RGGB; + break; + case BayerFormat::GBRG: + bayerFormat.order = BayerFormat::GRBG; + break; + case BayerFormat::GRBG: + bayerFormat.order = BayerFormat::GBRG; + break; + case BayerFormat::RGGB: + bayerFormat.order = BayerFormat::BGGR; + break; + default: + goto invalid_fmt; + } + break; + default: + goto invalid_fmt; + } if ((bayerFormat.bitDepth == 8 || bayerFormat.bitDepth == 10 || bayerFormat.bitDepth == 12) && bayerFormat.packing == BayerFormat::Packing::None && @@ -371,6 +399,7 @@ int DebayerCpu::setDebayerFunctions(PixelFormat inputFormat, [[maybe_unused]] Pi } } +invalid_fmt: LOG(Debayer, Error) << "Unsupported input output format combination"; return -EINVAL; } @@ -654,6 +683,9 @@ void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams gamma_correction_ = params.gamma; } + if (swapRedBlueGains_) + std::swap(params.gainR, params.gainB); + for (unsigned int i = 0; i < kRGBLookupSize; i++) { constexpr unsigned int div = kRGBLookupSize * DebayerParams::kGain10 / kGammaLookupSize; diff --git a/src/libcamera/software_isp/debayer_cpu.h b/src/libcamera/software_isp/debayer_cpu.h index 73ca41f1..b9e5f3fd 100644 --- a/src/libcamera/software_isp/debayer_cpu.h +++ b/src/libcamera/software_isp/debayer_cpu.h @@ -145,6 +145,7 @@ private: unsigned int lineBufferIndex_; unsigned int xShift_; /* Offset of 0/1 applied to window_.x */ bool enableInputMemcpy_; + bool swapRedBlueGains_; float gamma_correction_; unsigned int measuredFrames_; int64_t frameProcessTime_; From patchwork Thu Feb 29 18:36:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19587 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 96F62C3260 for ; Thu, 29 Feb 2024 18:37:29 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 501B76287A; Thu, 29 Feb 2024 19:37:29 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="Eb9Y+4gX"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 53F7462876 for ; Thu, 29 Feb 2024 19:37:27 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231845; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TEa9IZJSJV8dTI16DwvNpi0Yu5m8bealC9KMbuoPoqM=; b=Eb9Y+4gXoxxOoX8dJcbOr+q1VflBV5oNpFeJlnejEk5VoNsAoVIqPnh7ptnYXRy0R4XJx+ d/z+R/oi5z4bU75z3fgY+H96ksj5NQptdVW2XpTfT1vy/KIsp4szKhPuCUurZ9Y1Guc67w RRyABW+fasK/eAUzWxD6s5Spjl1KhCM= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-267-uD7E_nedMmKspURPuhFPog-1; Thu, 29 Feb 2024 13:37:20 -0500 X-MC-Unique: uD7E_nedMmKspURPuhFPog-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D8F1038212E9; Thu, 29 Feb 2024 18:37:19 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id DFCB41C060AF; Thu, 29 Feb 2024 18:37:18 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 16/18] libcamera: Add "Software ISP benchmarking" documentation Date: Thu, 29 Feb 2024 19:36:27 +0100 Message-ID: <20240229183654.7206-17-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a "Software ISP benchmarking" documentation section which describes the performance/power consumption measurements used during the Software ISP's development. Signed-off-by: Hans de Goede Reviewed-by: Milan Zamazal --- Documentation/index.rst | 1 + Documentation/meson.build | 1 + Documentation/software-isp-benchmarking.rst | 82 +++++++++++++++++++++ 3 files changed, 84 insertions(+) create mode 100644 Documentation/software-isp-benchmarking.rst diff --git a/Documentation/index.rst b/Documentation/index.rst index 63fac72d..5442ae75 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -24,3 +24,4 @@ Lens driver requirements Python Bindings Camera Sensor Model + SoftwareISP Benchmarking diff --git a/Documentation/meson.build b/Documentation/meson.build index 7a58fec8..3872e0a8 100644 --- a/Documentation/meson.build +++ b/Documentation/meson.build @@ -80,6 +80,7 @@ if sphinx.found() 'lens_driver_requirements.rst', 'python-bindings.rst', 'sensor_driver_requirements.rst', + 'software-isp-benchmarking.rst', '../README.rst', ] diff --git a/Documentation/software-isp-benchmarking.rst b/Documentation/software-isp-benchmarking.rst new file mode 100644 index 00000000..d94595d5 --- /dev/null +++ b/Documentation/software-isp-benchmarking.rst @@ -0,0 +1,82 @@ +.. SPDX-License-Identifier: CC-BY-SA-4.0 + +.. _software-isp-benchmarking: + +Software ISP benchmarking +========================= + +The Software ISP is particularly sensitive to performance regressions +therefore it is a good idea to always benchmark the Software ISP +before and after making changes to it and ensure that there are +no performance regressions. + +DebayerCpu class builtin benchmark +---------------------------------- + +The DebayerCpu class has a builtin benchmark. This benchmark +measures the time spent on processing (collecting statistics +and debayering) only, it does not measure the time spent on +capturing or outputting the frames. + +The builtin benchmark always runs. So this can be used by simply +running "cam" or "qcam" with a pipeline using the Software ISP. + +When it runs it will skip measuring the first 30 frames to +allow the caches and the CPU temperature (turbo-ing) to warm-up +and then it measures 30 fps and shows the total and per frame +processing time using an info level log message: + +.. code-block:: + + INFO Debayer debayer_cpu.cpp:907 Processed 30 frames in 244317us, 8143 us/frame + +To get stable measurements it is advised to disable any other processes which +may cause significant CPU usage (e.g. disable wifi, bluetooth and browsers). +When possible it is also advisable to disable CPU turbo-ing and +frequency-scaling. + +For example when benchmarking on a Lenovo ThinkPad X1 Yoga Gen 8, with +the charger plugged in, the CPU can be fixed to run at 2 GHz using: + +.. code-block:: + + sudo x86_energy_perf_policy --turbo-enable 0 + sudo cpupower frequency-set -d 2GHz -u 2GHz + +with these settings the builtin bench reports a processing time of ~7.8ms/frame +on this laptop for FHD SGRBG10 (unpacked) bayer data. + +Measuring power consumption +--------------------------- + +Since the Software ISP is often used on mobile devices it is also +important to measure power consumption and ensure that that does +not regress. + +For example to measure power consumption on a Lenovo ThinkPad X1 Yoga Gen 8 +it needs to be running on battery and it should be configured with its +platform-profile (/sys/firmware/acpi/platform_profile) set to balanced and +with its default turbo and frequency-scaling behavior to match real world usage. + +Then start qcam to capture a FHD picture at 30 fps and position the qcam window +so that it is fully visible. After this run the following command to monitor +the power consumption: + +.. code-block:: + + watch -n 10 cat /sys/class/power_supply/BAT0/power_now /sys/class/hwmon/hwmon6/fan?_input + +Note this not only measures the power consumption in µW it also monitors +the speed of this laptop's 2 fans. This is important because depending on +the ambient temperature the 2 fans may spin up while testing and this +will cause an additional power consumption of approx. 0.5 W messing up +the measurement. + +After starting qcam + the watch command let the laptop sit without using +it for 2 minutes for the readings to stabilize. Then check that the fans +have not turned on and manually take a couple of consecutive power readings +and avarage these. + +On the example Lenovo ThinkPad X1 Yoga Gen 8 laptop this results in +a measured power consumption of approx. 13 W while running qcam versus +approx. 4-5 W while setting idle with its OLED panel on. From patchwork Thu Feb 29 18:36:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19586 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0C6CBBD160 for ; Thu, 29 Feb 2024 18:37:29 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C8911628AC; Thu, 29 Feb 2024 19:37:28 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="ayDrZl/D"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2F87B62867 for ; Thu, 29 Feb 2024 19:37:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231843; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wgPRdehWi6kccte0Z0xpn+NNJauwwVsVp7eyKHNl4Ck=; b=ayDrZl/DFezgHRgVehv1mNe43h+6YgqFpTeK1zRJMk3Rq+KLhsBhGqbqJhPm1byCaXbY3p 4oQT4e9CknmYPoWSa+12N1qFGSL/cRild3RzK/Qzc/cRy0aFEQgFwstrNz8VIkPk/ApsCc R+Zd9NY3YGPfso3a0vMEo/KcVv3ixf4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-632-R-MKXmEMMIKeZs-pPB01xQ-1; Thu, 29 Feb 2024 13:37:21 -0500 X-MC-Unique: R-MKXmEMMIKeZs-pPB01xQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1634E185A784; Thu, 29 Feb 2024 18:37:21 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 190E31C06712; Thu, 29 Feb 2024 18:37:19 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 17/18] libcamera: software_isp: Apply black level compensation Date: Thu, 29 Feb 2024 19:36:28 +0100 Message-ID: <20240229183654.7206-18-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Milan Zamazal Black may not be represented as 0 pixel value for given hardware, it may be higher. If this is not compensated then various problems may occur such as low contrast or suboptimal exposure. The black pixel value can be either retrieved from a tuning file for the given hardware, or automatically on fly. The former is the right and correct method, while the latter can be used when a tuning file is not available for the given hardware. Since there is currently no support for tuning files in software ISP, the automatic, hardware independent way, is always used. Support for tuning files should be added in future but it will require more work than this patch. The patch looks at the image histogram and assumes that black starts when pixel values start occurring on the left. A certain amount of the darkest pixels is ignored; it doesn't matter whether they represent various kinds of noise or are real, they are better to omit in any case to make the image looking better. It also doesn't matter whether the darkest pixels occur around the supposed black level or are spread between 0 and the black level, the difference is not important. An arbitrary threshold of 2% darkest pixels is applied; there is no magic about that value. The patch assumes that the black values for different colors are the same and doesn't attempt any other non-primitive enhancements. It cannot completely replace tuning files and simplicity, while providing visible benefit, is its goal. Anything more sophisticated is left for future patches. A possible cheap enhancement, if needed, could be setting exposure + gain to minimum values temporarily, before setting the black level. In theory, the black level should be fixed but it may not be reached in all images. For this reason, the patch updates black level only if the observed value is lower than the current one; it should be never increased. The purpose of the patch is to compensate for hardware properties. General image contrast enhancements are out of scope of this patch. Stats are still gathered as an uncorrected histogram, to avoid any confusion and to represent the raw image data. Exposure must be determined after the black level correction -- it has no influence on the sub-black area and must be correct after applying the black level correction. The granularity of the histogram is increased from 16 to 64 to provide a better precision (there is no theory behind either of those numbers). Signed-off-by: Milan Zamazal --- .../internal/software_isp/debayer_params.h | 4 + .../internal/software_isp/swisp_stats.h | 10 ++- src/ipa/simple/black_level.cpp | 84 +++++++++++++++++++ src/ipa/simple/black_level.h | 28 +++++++ src/ipa/simple/meson.build | 7 +- src/ipa/simple/soft_simple.cpp | 28 +++++-- src/libcamera/software_isp/debayer_cpu.cpp | 13 ++- src/libcamera/software_isp/debayer_cpu.h | 1 + src/libcamera/software_isp/software_isp.cpp | 2 +- 9 files changed, 160 insertions(+), 17 deletions(-) create mode 100644 src/ipa/simple/black_level.cpp create mode 100644 src/ipa/simple/black_level.h diff --git a/include/libcamera/internal/software_isp/debayer_params.h b/include/libcamera/internal/software_isp/debayer_params.h index 98e7fc92..0bd46add 100644 --- a/include/libcamera/internal/software_isp/debayer_params.h +++ b/include/libcamera/internal/software_isp/debayer_params.h @@ -43,6 +43,10 @@ struct DebayerParams { * \brief Gamma correction, 1.0 is no correction */ float gamma; + /** + * \brief Level of the black point, 0..255, 0 is no correction. + */ + unsigned int blackLevel; }; } /* namespace libcamera */ diff --git a/include/libcamera/internal/software_isp/swisp_stats.h b/include/libcamera/internal/software_isp/swisp_stats.h index afe42c9a..25cd5abd 100644 --- a/include/libcamera/internal/software_isp/swisp_stats.h +++ b/include/libcamera/internal/software_isp/swisp_stats.h @@ -7,6 +7,8 @@ #pragma once +#include + namespace libcamera { /** @@ -28,11 +30,15 @@ struct SwIspStats { /** * \brief Number of bins in the yHistogram. */ - static constexpr unsigned int kYHistogramSize = 16; + static constexpr unsigned int kYHistogramSize = 64; + /** + * \brief Type of the histogram. + */ + using histogram = std::array; /** * \brief A histogram of luminance values. */ - std::array yHistogram; + histogram yHistogram; }; } /* namespace libcamera */ diff --git a/src/ipa/simple/black_level.cpp b/src/ipa/simple/black_level.cpp new file mode 100644 index 00000000..ab45d5f9 --- /dev/null +++ b/src/ipa/simple/black_level.cpp @@ -0,0 +1,84 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * black_level.cpp - black level handling + */ + +#include "black_level.h" + +#include + +#include + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoftBL) + +/** + * \class BlackLevel + * \brief Object providing black point level for software ISP + * + * Black level can be provided in hardware tuning files or, if no tuning file is + * available for the given hardware, guessed automatically, with less accuracy. + * As tuning files are not yet implemented for software ISP, BlackLevel + * currently provides only guessed black levels. + * + * This class serves for tracking black level as a property of the underlying + * hardware, not as means of enhancing a particular scene or image. + * + * The class is supposed to be instantiated for the given camera stream. + * The black level can be retrieved using BlackLevel::get() method. It is + * initially 0 and may change when updated using BlackLevel::update() method. + */ + +BlackLevel::BlackLevel() + : blackLevelHistIdx_(255), blackLevelSet_(false) +{ +} + +/** + * \brief Return the current black level + * + * \return The black level, in the range from 0 (minimum) to 255 (maximum). + * If the black level couldn't be determined yet, return 0. + */ +unsigned int BlackLevel::get() +{ + return blackLevelSet_ ? blackLevelHistIdx_ * (256 / SwIspStats::kYHistogramSize) : 0; +} + +/** + * \brief Update black level from the provided histogram + * \param[in] yHistogram The histogram to be used for updating black level + * + * The black level is property of the given hardware, not image. It is updated + * only if it has not been yet set or if it is lower than the lowest value seen + * so far. + */ +void BlackLevel::update(SwIspStats::histogram &yHistogram) +{ + // The constant is selected to be "good enough", not overly conservative or + // aggressive. There is no magic about the given value. + constexpr float ignoredPercentage_ = 0.02; + const unsigned int total = + std::accumulate(begin(yHistogram), end(yHistogram), 0); + const unsigned int pixelThreshold = ignoredPercentage_ * total; + + for (unsigned int i = 0, seen = 0; + i < blackLevelHistIdx_ && i < SwIspStats::kYHistogramSize; + i++) { + seen += yHistogram[i]; + if (seen >= pixelThreshold) { + blackLevelHistIdx_ = i; + blackLevelSet_ = true; + LOG(IPASoftBL, Debug) + << "Auto-set black level: " + << blackLevelHistIdx_ << "/" << SwIspStats::kYHistogramSize + << " (" << 100 * (seen - yHistogram[i]) / total << "% below, " + << 100 * seen / total << "% at or below)"; + break; + } + }; +} +} // namespace libcamera diff --git a/src/ipa/simple/black_level.h b/src/ipa/simple/black_level.h new file mode 100644 index 00000000..c8f5ec73 --- /dev/null +++ b/src/ipa/simple/black_level.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * black_level.h - black level handling + */ + +#pragma once + +#include + +#include "libcamera/internal/software_isp/swisp_stats.h" + +namespace libcamera { + +class BlackLevel +{ +public: + BlackLevel(); + unsigned int get(); + void update(std::array &yHistogram); + +private: + unsigned int blackLevelHistIdx_; + bool blackLevelSet_; +}; + +} // namespace libcamera diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build index 3e863db7..44b5f1d7 100644 --- a/src/ipa/simple/meson.build +++ b/src/ipa/simple/meson.build @@ -2,8 +2,13 @@ ipa_name = 'ipa_soft_simple' +soft_simple_sources = files([ + 'soft_simple.cpp', + 'black_level.cpp', +]) + mod = shared_module(ipa_name, - ['soft_simple.cpp', libcamera_generated_ipa_headers], + [soft_simple_sources, libcamera_generated_ipa_headers], name_prefix : '', include_directories : [ipa_includes, libipa_includes], dependencies : libcamera_private, diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 312df4ba..ac027568 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -22,6 +22,8 @@ #include "libcamera/internal/software_isp/debayer_params.h" #include "libcamera/internal/software_isp/swisp_stats.h" +#include "black_level.h" + namespace libcamera { LOG_DEFINE_CATEGORY(IPASoft) @@ -33,7 +35,8 @@ class IPASoftSimple : public ipa::soft::IPASoftInterface public: IPASoftSimple() : params_(static_cast(MAP_FAILED)), - stats_(static_cast(MAP_FAILED)), ignore_updates_(0) + stats_(static_cast(MAP_FAILED)), + blackLevel_(BlackLevel()), ignore_updates_(0) { } @@ -63,6 +66,7 @@ private: SharedFD fdParams_; DebayerParams *params_; SwIspStats *stats_; + BlackLevel blackLevel_; int32_t exposure_min_, exposure_max_; int32_t again_min_, again_max_; @@ -196,6 +200,10 @@ void IPASoftSimple::processStats(const ControlList &sensorControls) params_->gainG = 256; params_->gamma = 0.5; + if (ignore_updates_ > 0) + blackLevel_.update(stats_->yHistogram); + params_->blackLevel = blackLevel_.get(); + setIspParams.emit(0); /* @@ -211,18 +219,19 @@ void IPASoftSimple::processStats(const ControlList &sensorControls) * Calculate Mean Sample Value (MSV) according to formula from: * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf */ - constexpr unsigned int yHistValsPerBin = - SwIspStats::kYHistogramSize / kExposureBinsCount; - constexpr unsigned int yHistValsPerBinMod = - SwIspStats::kYHistogramSize / - (SwIspStats::kYHistogramSize % kExposureBinsCount + 1); + const unsigned int blackLevelHistIdx = + params_->blackLevel / (256 / SwIspStats::kYHistogramSize); + const unsigned int histogramSize = SwIspStats::kYHistogramSize - blackLevelHistIdx; + const unsigned int yHistValsPerBin = histogramSize / kExposureBinsCount; + const unsigned int yHistValsPerBinMod = + histogramSize / (histogramSize % kExposureBinsCount + 1); int ExposureBins[kExposureBinsCount] = {}; unsigned int denom = 0; unsigned int num = 0; - for (unsigned int i = 0; i < SwIspStats::kYHistogramSize; i++) { + for (unsigned int i = 0; i < histogramSize; i++) { unsigned int idx = (i - (i / yHistValsPerBinMod)) / yHistValsPerBin; - ExposureBins[idx] += stats_->yHistogram[i]; + ExposureBins[idx] += stats_->yHistogram[blackLevelHistIdx + i]; } for (unsigned int i = 0; i < kExposureBinsCount; i++) { @@ -256,7 +265,8 @@ void IPASoftSimple::processStats(const ControlList &sensorControls) LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV << " exp " << exposure_ << " again " << again_ - << " gain R/B " << params_->gainR << "/" << params_->gainB; + << " gain R/B " << params_->gainR << "/" << params_->gainB + << " black level " << params_->blackLevel; } void IPASoftSimple::updateExposure(double exposureMSV) diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index 215ca615..b221bc07 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -35,7 +35,7 @@ namespace libcamera { * \param[in] stats Pointer to the stats object to use. */ DebayerCpu::DebayerCpu(std::unique_ptr stats) - : stats_(std::move(stats)), gamma_correction_(1.0) + : stats_(std::move(stats)), gamma_correction_(1.0), blackLevel_(0) { #ifdef __x86_64__ enableInputMemcpy_ = false; @@ -676,11 +676,16 @@ void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams } /* Apply DebayerParams */ - if (params.gamma != gamma_correction_) { - for (unsigned int i = 0; i < kGammaLookupSize; i++) - gamma_[i] = UINT8_MAX * powf(i / (kGammaLookupSize - 1.0), params.gamma); + if (params.gamma != gamma_correction_ || params.blackLevel != blackLevel_) { + const unsigned int blackIndex = + params.blackLevel * kGammaLookupSize / 256; + std::fill(gamma_.begin(), gamma_.begin() + blackIndex, 0); + const float divisor = kGammaLookupSize - blackIndex - 1.0; + for (unsigned int i = blackIndex; i < kGammaLookupSize; i++) + gamma_[i] = UINT8_MAX * powf((i - blackIndex) / divisor, params.gamma); gamma_correction_ = params.gamma; + blackLevel_ = params.blackLevel; } if (swapRedBlueGains_) diff --git a/src/libcamera/software_isp/debayer_cpu.h b/src/libcamera/software_isp/debayer_cpu.h index b9e5f3fd..88b15ac7 100644 --- a/src/libcamera/software_isp/debayer_cpu.h +++ b/src/libcamera/software_isp/debayer_cpu.h @@ -147,6 +147,7 @@ private: bool enableInputMemcpy_; bool swapRedBlueGains_; float gamma_correction_; + unsigned int blackLevel_; unsigned int measuredFrames_; int64_t frameProcessTime_; /* Skip 30 frames for things to stabilize then measure 30 frames */ diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 388b4496..9b49be41 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -64,7 +64,7 @@ LOG_DEFINE_CATEGORY(SoftwareIsp) */ SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const ControlInfoMap &sensorControls) : debayer_(nullptr), - debayerParams_{ DebayerParams::kGain10, DebayerParams::kGain10, DebayerParams::kGain10, 0.5f }, + debayerParams_{ DebayerParams::kGain10, DebayerParams::kGain10, DebayerParams::kGain10, 0.5f, 0 }, dmaHeap_(DmaHeap::DmaHeapFlag::Cma | DmaHeap::DmaHeapFlag::System) { if (!dmaHeap_.isValid()) { From patchwork Thu Feb 29 18:36:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 19588 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 57FF7BD160 for ; Thu, 29 Feb 2024 18:37:32 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id EEFF962C81; Thu, 29 Feb 2024 19:37:31 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="UhmqPyON"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1B1ED62C81 for ; Thu, 29 Feb 2024 19:37:29 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709231848; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=55AN3oJoSaC8eH0IkCrv55lvkKdHq0UF2HqNVhbjf8c=; b=UhmqPyONRwBy85BsrrfQ7YDD35w6lyjI8lBddhPhDWx1nZgJKtQxCcyxLGbZNvGxqYBtJh 59FghxVtHRCIibM7RnhIIQZW7baLEcax3RKQrP/4mJYee/NuqeNri3DVXuu1reHH1AnrK3 IFNzqObUFlJKBPCGOesdfFai4qO4J1M= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-622-r9EGxYFhPcGPsP4qdM8IXA-1; Thu, 29 Feb 2024 13:37:22 -0500 X-MC-Unique: r9EGxYFhPcGPsP4qdM8IXA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4352485A588; Thu, 29 Feb 2024 18:37:22 +0000 (UTC) Received: from shalem.redhat.com (unknown [10.39.192.10]) by smtp.corp.redhat.com (Postfix) with ESMTP id 470F11C060AF; Thu, 29 Feb 2024 18:37:21 +0000 (UTC) From: Hans de Goede To: libcamera-devel@lists.libcamera.org Subject: [PATCH v4 18/18] libcamera: Soft IPA: use CameraSensorHelper for analogue gain Date: Thu, 29 Feb 2024 19:36:29 +0100 Message-ID: <20240229183654.7206-19-hdegoede@redhat.com> In-Reply-To: <20240229183654.7206-1-hdegoede@redhat.com> References: <20240229183654.7206-1-hdegoede@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Ripard , Pavel Machek , Bryan O'Donoghue Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Andrei Konovalov Use CameraSensorHelper to convert the analogue gain code read from the camera sensor into real analogue gain value. In the future this makes it possible to use faster AE/AGC algorithm. For now the same AE/AGC algorithm is used, but even then the CameraSensorHelper lets us use the full range of analogue gain values. If there is no CameraSensorHelper for the camera sensor in use, a warning log message is printed, and the AE/AGC works exactly as before this change. Signed-off-by: Andrei Konovalov --- .../internal/software_isp/software_isp.h | 3 +- src/ipa/simple/soft_simple.cpp | 77 ++++++++++++------- src/libcamera/pipeline/simple/simple.cpp | 2 +- src/libcamera/software_isp/software_isp.cpp | 8 +- 4 files changed, 57 insertions(+), 33 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index 8d25e979..2a6db7ba 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -26,6 +26,7 @@ #include #include +#include "libcamera/internal/camera_sensor.h" #include "libcamera/internal/dma_heaps.h" #include "libcamera/internal/pipeline_handler.h" #include "libcamera/internal/shared_mem_object.h" @@ -43,7 +44,7 @@ LOG_DECLARE_CATEGORY(SoftwareIsp) class SoftwareIsp { public: - SoftwareIsp(PipelineHandler *pipe, const ControlInfoMap &sensorControls); + SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor); ~SoftwareIsp(); int loadConfiguration([[maybe_unused]] const std::string &filename) { return 0; } diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index ac027568..e4d64762 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -22,6 +22,8 @@ #include "libcamera/internal/software_isp/debayer_params.h" #include "libcamera/internal/software_isp/swisp_stats.h" +#include "libipa/camera_sensor_helper.h" + #include "black_level.h" namespace libcamera { @@ -67,18 +69,27 @@ private: DebayerParams *params_; SwIspStats *stats_; BlackLevel blackLevel_; + std::unique_ptr camHelper_; int32_t exposure_min_, exposure_max_; - int32_t again_min_, again_max_; - int32_t again_, exposure_; + int32_t exposure_; + double again_min_, again_max_, againMinStep_; + double again_; unsigned int ignore_updates_; }; -int IPASoftSimple::init([[maybe_unused]] const IPASettings &settings, +int IPASoftSimple::init(const IPASettings &settings, const SharedFD &fdStats, const SharedFD &fdParams, const ControlInfoMap &sensorInfoMap) { + camHelper_ = CameraSensorHelperFactoryBase::create(settings.sensorModel); + if (camHelper_ == nullptr) { + LOG(IPASoft, Warning) + << "Failed to create camera sensor helper for " + << settings.sensorModel; + } + fdStats_ = fdStats; if (!fdStats_.isValid()) { LOG(IPASoft, Error) << "Invalid Statistics handle"; @@ -132,25 +143,35 @@ int IPASoftSimple::configure(const ControlInfoMap &sensorInfoMap) exposure_min_ = 1; } - again_min_ = gain_info.min().get(); - again_max_ = gain_info.max().get(); - /* - * The camera sensor gain (g) is usually not equal to the value written - * into the gain register (x). But the way how the AGC algorithm changes - * the gain value to make the total exposure closer to the optimum assumes - * that g(x) is not too far from linear function. If the minimal gain is 0, - * the g(x) is likely to be far from the linear, like g(x) = a / (b * x + c). - * To avoid unexpected changes to the gain by the AGC algorithm (abrupt near - * one edge, and very small near the other) we limit the range of the gain - * values used. - */ - if (!again_min_) { - LOG(IPASoft, Warning) << "Minimum gain is zero, that can't be linear"; - again_min_ = std::min(100, again_min_ / 2 + again_max_ / 2); + int32_t again_min = gain_info.min().get(); + int32_t again_max = gain_info.max().get(); + + if (camHelper_) { + again_min_ = camHelper_->gain(again_min); + again_max_ = camHelper_->gain(again_max); + againMinStep_ = (again_max_ - again_min_) / 100.0; + } else { + /* + * The camera sensor gain (g) is usually not equal to the value written + * into the gain register (x). But the way how the AGC algorithm changes + * the gain value to make the total exposure closer to the optimum assumes + * that g(x) is not too far from linear function. If the minimal gain is 0, + * the g(x) is likely to be far from the linear, like g(x) = a / (b * x + c). + * To avoid unexpected changes to the gain by the AGC algorithm (abrupt near + * one edge, and very small near the other) we limit the range of the gain + * values used. + */ + again_max_ = again_max; + if (!again_min) { + LOG(IPASoft, Warning) << "Minimum gain is zero, that can't be linear"; + again_min_ = std::min(100, again_min / 2 + again_max / 2); + } + againMinStep_ = 1.0; } LOG(IPASoft, Info) << "Exposure " << exposure_min_ << "-" << exposure_max_ - << ", gain " << again_min_ << "-" << again_max_; + << ", gain " << again_min_ << "-" << again_max_ + << " (" << againMinStep_ << ")"; return 0; } @@ -252,12 +273,14 @@ void IPASoftSimple::processStats(const ControlList &sensorControls) ControlList ctrls(sensorControls); exposure_ = ctrls.get(V4L2_CID_EXPOSURE).get(); - again_ = ctrls.get(V4L2_CID_ANALOGUE_GAIN).get(); + int32_t again = ctrls.get(V4L2_CID_ANALOGUE_GAIN).get(); + again_ = camHelper_ ? camHelper_->gain(again) : again; updateExposure(exposureMSV); ctrls.set(V4L2_CID_EXPOSURE, exposure_); - ctrls.set(V4L2_CID_ANALOGUE_GAIN, again_); + ctrls.set(V4L2_CID_ANALOGUE_GAIN, + static_cast(camHelper_ ? camHelper_->gainCode(again_) : again_)); ignore_updates_ = 2; @@ -276,7 +299,7 @@ void IPASoftSimple::updateExposure(double exposureMSV) static constexpr uint8_t kExpNumeratorUp = kExpDenominator + 1; static constexpr uint8_t kExpNumeratorDown = kExpDenominator - 1; - int next; + double next; if (exposureMSV < kExposureOptimal - kExposureSatisfactory) { next = exposure_ * kExpNumeratorUp / kExpDenominator; @@ -286,18 +309,18 @@ void IPASoftSimple::updateExposure(double exposureMSV) exposure_ = next; if (exposure_ >= exposure_max_) { next = again_ * kExpNumeratorUp / kExpDenominator; - if (next - again_ < 1) - again_ += 1; + if (next - again_ < againMinStep_) + again_ += againMinStep_; else again_ = next; } } if (exposureMSV > kExposureOptimal + kExposureSatisfactory) { - if (exposure_ == exposure_max_ && again_ != again_min_) { + if (exposure_ == exposure_max_ && again_ > again_min_) { next = again_ * kExpNumeratorDown / kExpDenominator; - if (again_ - next < 1) - again_ -= 1; + if (again_ - next < againMinStep_) + again_ -= againMinStep_; else again_ = next; } else { diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index e491ab62..32b56c57 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -526,7 +526,7 @@ int SimpleCameraData::init() * Instantiate Soft ISP if this is enabled for the given driver and no converter is used. */ if (!converter_ && pipe->swIspEnabled()) { - swIsp_ = std::make_unique(pipe, sensor_->controls()); + swIsp_ = std::make_unique(pipe, sensor_.get()); if (!swIsp_->isValid()) { LOG(SimplePipeline, Warning) << "Failed to create software ISP, disabling software debayering"; diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 9b49be41..ea4d96e4 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -60,9 +60,9 @@ LOG_DEFINE_CATEGORY(SoftwareIsp) /** * \brief Constructs SoftwareIsp object * \param[in] pipe The pipeline handler in use - * \param[in] sensorControls ControlInfoMap describing the controls supported by the sensor + * \param[in] sensor Pointer to the CameraSensor instance owned by the pipeline handler */ -SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const ControlInfoMap &sensorControls) +SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const CameraSensor *sensor) : debayer_(nullptr), debayerParams_{ DebayerParams::kGain10, DebayerParams::kGain10, DebayerParams::kGain10, 0.5f, 0 }, dmaHeap_(DmaHeap::DmaHeapFlag::Cma | DmaHeap::DmaHeapFlag::System) @@ -97,10 +97,10 @@ SoftwareIsp::SoftwareIsp(PipelineHandler *pipe, const ControlInfoMap &sensorCont return; } - int ret = ipa_->init(IPASettings{ "No cfg file", "No sensor model" }, + int ret = ipa_->init(IPASettings{ "No cfg file", sensor->model() }, debayer_->getStatsFD(), sharedParams_.fd(), - sensorControls); + sensor->controls()); if (ret) { LOG(SoftwareIsp, Error) << "IPA init failed"; debayer_.reset();