From patchwork Fri Sep 6 12:09:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21174 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E904DC324C for ; Fri, 6 Sep 2024 12:09:59 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 429F5634EF; Fri, 6 Sep 2024 14:09:59 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="cnZYjaYS"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 06FD1633CC for ; Fri, 6 Sep 2024 14:09:54 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624593; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rKWQHrO/S9qgxIaMcZCbyPXr9JvRVCJjfvb8264XDuI=; b=cnZYjaYSRYD4NDKjZv1UwGIdO7CARdz/77PQUMBRwULIlzOYf4oIaYorTXevQxUhpq42kY q83C0AC68UIamI8TBN62C3gsfnPoRSA1Hj0DrmsIQ/4o/OhAzHITctvK79Hi7bpsf116ml hntTDqSJ3mlxo3gDMRfb3y6kkSDBWgA= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-604-BD_NGuglM9OjIcO3Z7HQlg-1; Fri, 06 Sep 2024 08:09:50 -0400 X-MC-Unique: BD_NGuglM9OjIcO3Z7HQlg-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8B8621955DC1; Fri, 6 Sep 2024 12:09:49 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 15B891956086; Fri, 6 Sep 2024 12:09:46 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 01/18] libcamera: software_isp: Remove superfluous includes Date: Fri, 6 Sep 2024 14:09:10 +0200 Message-ID: <20240906120927.4071508-2-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Remove unused libcamera internal headers bayer_format.h, framebuffer.h and mapped_frameBuffer.h. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Laurent Pinchart Reviewed-by: Daniel Scally --- src/libcamera/software_isp/software_isp.cpp | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 1140372c..a3ae3e22 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -16,10 +16,7 @@ #include #include -#include "libcamera/internal/bayer_format.h" -#include "libcamera/internal/framebuffer.h" #include "libcamera/internal/ipa_manager.h" -#include "libcamera/internal/mapped_framebuffer.h" #include "libcamera/internal/software_isp/debayer_params.h" #include "debayer_cpu.h" From patchwork Fri Sep 6 12:09:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21175 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 767DCC324C for ; Fri, 6 Sep 2024 12:10:04 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D524D634F4; Fri, 6 Sep 2024 14:10:03 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="Y6cNtbts"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3DA63634E3 for ; Fri, 6 Sep 2024 14:09:57 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624596; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SxcxrWdqckPYJJM31vXC2zzG/WyUmEsjDhhR7dm2j24=; b=Y6cNtbtsgYRtcXEUiDTBWKeHzcXfRoMlWXHMXCx/v8aNKyAtwmO8DfIFUWLWx229I7C44Q 53PWDYnJu2YkZXMHSrUYDhES+cuWKqNeS9qN3yf4RM8cBqdfICxFKTq+7RE6yO11IcGIhk p7E1WnL0lK/UeMYXczWWvbF3NM1gbMI= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-691-COK4TYWZOYiOw4yAIQIzVA-1; Fri, 06 Sep 2024 08:09:53 -0400 X-MC-Unique: COK4TYWZOYiOw4yAIQIzVA-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id DADCB1953956; Fri, 6 Sep 2024 12:09:51 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id E072B1956086; Fri, 6 Sep 2024 12:09:49 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 02/18] libcamera: software_isp: Move BlackLevel to libcamera::ipa::soft Date: Fri, 6 Sep 2024 14:09:11 +0200 Message-ID: <20240906120927.4071508-3-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" IPA modules use custom namespaces for all their internal components to avoid namespace clashes. The simple IPA module for the software ISP uses libcamera::ipa::soft for this purpose. It however defines an internal class named BlackLevel in the root of the libcamera namespace, making it prone to clashes. Move it to the ipa::soft namespace along with the rest of the code. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Laurent Pinchart Reviewed-by: Daniel Scally --- src/ipa/simple/black_level.cpp | 5 +++++ src/ipa/simple/black_level.h | 4 ++++ 2 files changed, 9 insertions(+) diff --git a/src/ipa/simple/black_level.cpp b/src/ipa/simple/black_level.cpp index cc490eb5..37e0109c 100644 --- a/src/ipa/simple/black_level.cpp +++ b/src/ipa/simple/black_level.cpp @@ -15,6 +15,8 @@ namespace libcamera { LOG_DEFINE_CATEGORY(IPASoftBL) +namespace ipa::soft { + /** * \class BlackLevel * \brief Object providing black point level for software ISP @@ -85,4 +87,7 @@ void BlackLevel::update(SwIspStats::Histogram &yHistogram) } }; } + +} /* namespace ipa::soft */ + } /* namespace libcamera */ diff --git a/src/ipa/simple/black_level.h b/src/ipa/simple/black_level.h index 5e032f9f..a04230c9 100644 --- a/src/ipa/simple/black_level.h +++ b/src/ipa/simple/black_level.h @@ -14,6 +14,8 @@ namespace libcamera { +namespace ipa::soft { + class BlackLevel { public: @@ -26,4 +28,6 @@ private: bool blackLevelSet_; }; +} /* namespace ipa::soft */ + } /* namespace libcamera */ From patchwork Fri Sep 6 12:09:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21177 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4778EC324C for ; Fri, 6 Sep 2024 12:10:12 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 75EF0634F0; Fri, 6 Sep 2024 14:10:11 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="QrCgDx9I"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5AA7E634E5 for ; Fri, 6 Sep 2024 14:10:00 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624599; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lw9fjH10smSWz/R80egNMsgp85tvE4lK6vb+/TYL4TA=; b=QrCgDx9IrCQBAKKq0pcfy90/PiEsQSarEnFg/Y9ON0GejhD6wyqSgFgO0C7ZaVjWK6f68v JnQwTfPXAgpAzzbiJYFoU1Xt/JbQdG2iHKhLMhIrunYNSktK9mP+7vsHib8ssZBP8HGQya w6vYAeUbzEJLYwGUg7BDhHU+Xbdk73w= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-318-9rUPKU3iPziZkyuJGS2ilw-1; Fri, 06 Sep 2024 08:09:55 -0400 X-MC-Unique: 9rUPKU3iPziZkyuJGS2ilw-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A4E6A1955D44; Fri, 6 Sep 2024 12:09:54 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 752D61956086; Fri, 6 Sep 2024 12:09:52 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 03/18] libcamera: software_isp: Define skeletons for IPA refactoring Date: Fri, 6 Sep 2024 14:09:12 +0200 Message-ID: <20240906120927.4071508-4-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Software ISP image processing algorithms are currently defined in a simplified way, different from other libcamera pipelines. This is not good for several reasons: - It makes the software ISP code harder to understand due to its different structuring. - Adding more algorithms may make the code harder to understand generally. - Mass libcamera code changes may not be easily applicable to software ISP. - Algorithm sharing with other pipelines is not easily possible. This patch introduces basic software ISP IPA skeletons structured similarly to the other pipelines. The newly added files are currently not used or compiled and the general skeleton structures don't contain anything particular. It is just a preparation step for a larger refactoring and the code will be actually used and extended as needed in followup patches. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Daniel Scally Reviewed-by: Laurent Pinchart --- include/libcamera/ipa/soft.mojom | 4 ++ src/ipa/simple/algorithms/algorithm.h | 22 +++++++++++ src/ipa/simple/ipa_context.cpp | 53 +++++++++++++++++++++++++++ src/ipa/simple/ipa_context.h | 33 +++++++++++++++++ src/ipa/simple/meson.build | 1 + src/ipa/simple/module.h | 30 +++++++++++++++ 6 files changed, 143 insertions(+) create mode 100644 src/ipa/simple/algorithms/algorithm.h create mode 100644 src/ipa/simple/ipa_context.cpp create mode 100644 src/ipa/simple/ipa_context.h create mode 100644 src/ipa/simple/module.h diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index 3aa2066e..0fd47bb0 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -8,6 +8,10 @@ module ipa.soft; import "include/libcamera/ipa/core.mojom"; +struct IPAConfigInfo { + libcamera.ControlInfoMap sensorControls; +}; + interface IPASoftInterface { init(libcamera.IPASettings settings, libcamera.SharedFD fdStats, diff --git a/src/ipa/simple/algorithms/algorithm.h b/src/ipa/simple/algorithms/algorithm.h new file mode 100644 index 00000000..41f63170 --- /dev/null +++ b/src/ipa/simple/algorithms/algorithm.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Red Hat, Inc. + * + * Software ISP control algorithm interface + */ + +#pragma once + +#include + +#include "module.h" + +namespace libcamera { + +namespace ipa::soft { + +using Algorithm = libcamera::ipa::Algorithm; + +} /* namespace ipa::soft */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/ipa_context.cpp b/src/ipa/simple/ipa_context.cpp new file mode 100644 index 00000000..3c1c7262 --- /dev/null +++ b/src/ipa/simple/ipa_context.cpp @@ -0,0 +1,53 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Google Inc. + * Copyright (C) 2024 Red Hat Inc. + * + * Software ISP IPA Context + */ + +#include "ipa_context.h" + +/** + * \file ipa_context.h + * \brief Context and state information shared between the algorithms + */ + +namespace libcamera::ipa::soft { + +/** + * \struct IPASessionConfiguration + * \brief Session configuration for the IPA module + * + * The session configuration contains all IPA configuration parameters that + * remain constant during the capture session, from IPA module start to stop. + * It is typically set during the configure() operation of the IPA module, but + * may also be updated in the start() operation. + */ + +/** + * \struct IPAActiveState + * \brief The active state of the IPA algorithms + * + * The IPA is fed with the statistics generated from the latest frame processed. + * The statistics are then processed by the IPA algorithms to compute parameters + * required for the next frame capture and processing. The current state of the + * algorithms is reflected through the IPAActiveState to store the values most + * recently computed by the IPA algorithms. + */ + +/** + * \struct IPAContext + * \brief Global IPA context data shared between all algorithms + * + * \var IPAContext::configuration + * \brief The IPA session configuration, immutable during the session + * + * \var IPAContext::frameContexts + * \brief Ring buffer of the IPAFrameContext(s) + * + * \var IPAContext::activeState + * \brief The current state of IPA algorithms + */ + +} /* namespace libcamera::ipa::soft */ diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h new file mode 100644 index 00000000..e7d8b8df --- /dev/null +++ b/src/ipa/simple/ipa_context.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Red Hat, Inc. + * + * Simple pipeline IPA Context + */ + +#pragma once + +#include + +namespace libcamera { + +namespace ipa::soft { + +struct IPASessionConfiguration { +}; + +struct IPAActiveState { +}; + +struct IPAFrameContext : public FrameContext { +}; + +struct IPAContext { + IPASessionConfiguration configuration; + IPAActiveState activeState; + FCQueue frameContexts; +}; + +} /* namespace ipa::soft */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build index b297e1d2..7757f99b 100644 --- a/src/ipa/simple/meson.build +++ b/src/ipa/simple/meson.build @@ -3,6 +3,7 @@ ipa_name = 'ipa_soft_simple' soft_simple_sources = files([ + 'ipa_context.cpp', 'soft_simple.cpp', 'black_level.cpp', ]) diff --git a/src/ipa/simple/module.h b/src/ipa/simple/module.h new file mode 100644 index 00000000..8d4d53fb --- /dev/null +++ b/src/ipa/simple/module.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Red Hat, Inc. + * + * Software ISP IPA Module + */ + +#pragma once + +#include + +#include + +#include "libcamera/internal/software_isp/debayer_params.h" +#include "libcamera/internal/software_isp/swisp_stats.h" + +#include + +#include "ipa_context.h" + +namespace libcamera { + +namespace ipa::soft { + +using Module = ipa::Module; + +} /* namespace ipa::soft */ + +} /* namespace libcamera */ From patchwork Fri Sep 6 12:09:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21176 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 27617C324C for ; Fri, 6 Sep 2024 12:10:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 95D13634F6; Fri, 6 Sep 2024 14:10:08 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="JRGr3I0p"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C30DC634ED for ; Fri, 6 Sep 2024 14:10:00 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624599; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MX9NCOwcE1ykHIPz/zU/TVZmkBNqq1TkDXFuf3lnGdo=; b=JRGr3I0pQIp8glTo+UZd1hcA7c8VkXxSvw0aPeCacvGYMMNrIvZj9nxeu6byeHNG2zYdld +saYygpfudYGGCbUSYZAb6ZfqAN1v7co94ssQjIpCnxB+AozHAkeDkHR4xsEuF3f2Yq3lI xhl2aeHUfw4RLrNgyPu0qmg4tBZmMrA= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-653--vt2hDPdNgunxjIeHVL4yA-1; Fri, 06 Sep 2024 08:09:58 -0400 X-MC-Unique: -vt2hDPdNgunxjIeHVL4yA-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 16EA41955F40; Fri, 6 Sep 2024 12:09:57 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 22E591956086; Fri, 6 Sep 2024 12:09:54 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 04/18] libcamera: software_isp: Let IPASoftSimple inherit Module Date: Fri, 6 Sep 2024 14:09:13 +0200 Message-ID: <20240906120927.4071508-5-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The Module class is a base class for all IPA modules. In addition, implement logPrefix() of the module for the softIPA. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Daniel Scally Reviewed-by: Laurent Pinchart --- src/ipa/simple/soft_simple.cpp | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index b7746ce0..72321f44 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -30,6 +30,7 @@ #include "libipa/camera_sensor_helper.h" #include "black_level.h" +#include "module.h" namespace libcamera { LOG_DEFINE_CATEGORY(IPASoft) @@ -54,7 +55,7 @@ static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; */ static constexpr float kExposureSatisfactory = 0.2; -class IPASoftSimple : public ipa::soft::IPASoftInterface +class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() @@ -76,6 +77,9 @@ public: void processStats(const ControlList &sensorControls) override; +protected: + std::string logPrefix() const override; + private: void updateExposure(double exposureMSV); @@ -421,6 +425,11 @@ void IPASoftSimple::updateExposure(double exposureMSV) again_ = std::clamp(again_, againMin_, againMax_); } +std::string IPASoftSimple::logPrefix() const +{ + return "IPASoft"; +} + } /* namespace ipa::soft */ /* From patchwork Fri Sep 6 12:09:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21178 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 543EAC324C for ; Fri, 6 Sep 2024 12:10:14 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5919A634F2; Fri, 6 Sep 2024 14:10:13 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="gu69lmQ3"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6EA5F634F0 for ; Fri, 6 Sep 2024 14:10:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624602; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1LPQvFtc1Dyd3UHwmpOY1rLzAIN8pTcKYSxag7Cn+9Q=; b=gu69lmQ3MTFpSBYevJZ9hc0LL25kZkdc0LiZqhSM2rfoGQANJYQtuJN9vcl9a9w2zVGDh+ mk2wtNrg/98zmUsjWY7HE8Ndt/eoFlyD7KYbtmNKg7qJIOamw1Q7MmR5A0UDCEUPXm8WoQ NRm/raLbqY5HphSHlE7F70dulHiYMgk= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-244-_Fqbbb95NZ6wHWdGOf3-6g-1; Fri, 06 Sep 2024 08:10:00 -0400 X-MC-Unique: _Fqbbb95NZ6wHWdGOf3-6g-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 925B419560AB; Fri, 6 Sep 2024 12:09:59 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 695F41956086; Fri, 6 Sep 2024 12:09:57 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 05/18] libcamera: software_isp: Make stats frame and buffer aware Date: Fri, 6 Sep 2024 14:09:14 +0200 Message-ID: <20240906120927.4071508-6-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds frame and bufferId arguments to stats related calls. Although the parameters are currently unused, because frame ids are not tracked and used and the stats buffer is passed around directly rather than being referred by its id, they bring the internal APIs closer to their counterparts in hardware pipelines. It serves as a preparation for followup patches that will introduce: - Frame number tracking in order to switch to DelayedControls (software ISP TODO #11 + #12). - A ring buffer for stats in order to improve passing the stats (software ISP TODO #2). Frame and buffer ids are unrelated for the given purposes but since they are passed together at the same places, the change is implemented as a single patch rather than two, basically the same, patches. Signed-off-by: Milan Zamazal Reviewed-by: Laurent Pinchart Reviewed-by: Daniel Scally --- .../libcamera/internal/software_isp/software_isp.h | 8 +++++--- include/libcamera/ipa/soft.mojom | 4 +++- src/ipa/simple/soft_simple.cpp | 7 +++++-- src/libcamera/pipeline/simple/simple.cpp | 8 +++++--- src/libcamera/software_isp/debayer_cpu.cpp | 8 +++++++- src/libcamera/software_isp/software_isp.cpp | 11 +++++++---- src/libcamera/software_isp/swstats_cpu.cpp | 6 ++++-- src/libcamera/software_isp/swstats_cpu.h | 4 ++-- 8 files changed, 38 insertions(+), 18 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index f8e00003..3602bce8 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -66,7 +67,8 @@ public: int exportBuffers(const Stream *stream, unsigned int count, std::vector> *buffers); - void processStats(const ControlList &sensorControls); + void processStats(const uint32_t frame, const uint32_t bufferId, + const ControlList &sensorControls); int start(); void stop(); @@ -78,13 +80,13 @@ public: Signal inputBufferReady; Signal outputBufferReady; - Signal<> ispStatsReady; + Signal ispStatsReady; Signal setSensorControls; private: void saveIspParams(); void setSensorCtrls(const ControlList &sensorControls); - void statsReady(); + void statsReady(uint32_t frame, uint32_t bufferId); void inputReady(FrameBuffer *input); void outputReady(FrameBuffer *output); diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index 0fd47bb0..f65b6231 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -23,7 +23,9 @@ interface IPASoftInterface { configure(libcamera.ControlInfoMap sensorCtrlInfoMap) => (int32 ret); - [async] processStats(libcamera.ControlList sensorControls); + [async] processStats(uint32 frame, + uint32 bufferId, + libcamera.ControlList sensorControls); }; interface IPASoftEventInterface { diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 72321f44..12b5245e 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -75,7 +75,8 @@ public: int start() override; void stop() override; - void processStats(const ControlList &sensorControls) override; + void processStats(const uint32_t frame, const uint32_t bufferId, + const ControlList &sensorControls) override; protected: std::string logPrefix() const override; @@ -249,7 +250,9 @@ void IPASoftSimple::stop() { } -void IPASoftSimple::processStats(const ControlList &sensorControls) +void IPASoftSimple::processStats([[maybe_unused]] const uint32_t frame, + [[maybe_unused]] const uint32_t bufferId, + const ControlList &sensorControls) { SwIspStats::Histogram histogram = stats_->yHistogram; if (ignoreUpdates_ > 0) diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 1e7ec7d9..48a568da 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -291,7 +292,7 @@ private: void conversionInputDone(FrameBuffer *buffer); void conversionOutputDone(FrameBuffer *buffer); - void ispStatsReady(); + void ispStatsReady(uint32_t frame, uint32_t bufferId); void setSensorControls(const ControlList &sensorControls); }; @@ -891,10 +892,11 @@ void SimpleCameraData::conversionOutputDone(FrameBuffer *buffer) pipe->completeRequest(request); } -void SimpleCameraData::ispStatsReady() +void SimpleCameraData::ispStatsReady(uint32_t frame, uint32_t bufferId) { /* \todo Use the DelayedControls class */ - swIsp_->processStats(sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, + swIsp_->processStats(frame, bufferId, + sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, V4L2_CID_EXPOSURE })); } diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index 077f7f4b..2a2e7edb 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -777,7 +777,13 @@ void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams } } - stats_->finishFrame(); + /* + * Frame and buffer ids are currently not used, so pass zeros as parameters. + * + * \todo Pass real values once frame is passed here and stats buffer passing + * is changed. + */ + stats_->finishFrame(0, 0); outputBufferReady.emit(output); inputBufferReady.emit(input); } diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index a3ae3e22..a3855568 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -155,15 +155,18 @@ SoftwareIsp::~SoftwareIsp() /** * \brief Process the statistics gathered + * \param[in] frame The frame number + * \param[in] bufferId ID of the statistics buffer * \param[in] sensorControls The sensor controls * * Requests the IPA to calculate new parameters for ISP and new control * values for the sensor. */ -void SoftwareIsp::processStats(const ControlList &sensorControls) +void SoftwareIsp::processStats(const uint32_t frame, const uint32_t bufferId, + const ControlList &sensorControls) { ASSERT(ipa_); - ipa_->processStats(sensorControls); + ipa_->processStats(frame, bufferId, sensorControls); } /** @@ -349,9 +352,9 @@ void SoftwareIsp::setSensorCtrls(const ControlList &sensorControls) setSensorControls.emit(sensorControls); } -void SoftwareIsp::statsReady() +void SoftwareIsp::statsReady(uint32_t frame, uint32_t bufferId) { - ispStatsReady.emit(); + ispStatsReady.emit(frame, bufferId); } void SoftwareIsp::inputReady(FrameBuffer *input) diff --git a/src/libcamera/software_isp/swstats_cpu.cpp b/src/libcamera/software_isp/swstats_cpu.cpp index 815c4d4f..c520c806 100644 --- a/src/libcamera/software_isp/swstats_cpu.cpp +++ b/src/libcamera/software_isp/swstats_cpu.cpp @@ -311,13 +311,15 @@ void SwStatsCpu::startFrame(void) /** * \brief Finish statistics calculation for the current frame + * \param[in] frame The frame number + * \param[in] bufferId ID of the statistics buffer * * This may only be called after a successful setWindow() call. */ -void SwStatsCpu::finishFrame(void) +void SwStatsCpu::finishFrame(uint32_t frame, uint32_t bufferId) { *sharedStats_ = stats_; - statsReady.emit(); + statsReady.emit(frame, bufferId); } /** diff --git a/src/libcamera/software_isp/swstats_cpu.h b/src/libcamera/software_isp/swstats_cpu.h index 363e326f..26a2f462 100644 --- a/src/libcamera/software_isp/swstats_cpu.h +++ b/src/libcamera/software_isp/swstats_cpu.h @@ -41,7 +41,7 @@ public: int configure(const StreamConfiguration &inputCfg); void setWindow(const Rectangle &window); void startFrame(); - void finishFrame(); + void finishFrame(uint32_t frame, uint32_t bufferId); void processLine0(unsigned int y, const uint8_t *src[]) { @@ -61,7 +61,7 @@ public: (this->*stats2_)(src); } - Signal<> statsReady; + Signal statsReady; private: using statsProcessFn = void (SwStatsCpu::*)(const uint8_t *src[]); From patchwork Fri Sep 6 12:09:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21179 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 21BA3C324C for ; Fri, 6 Sep 2024 12:10:17 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D6F1B634FA; Fri, 6 Sep 2024 14:10:15 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="XW2CiDji"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 59053634F0 for ; Fri, 6 Sep 2024 14:10:07 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624606; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1KZmM50ODUWNoDjYHlqfTIxDIdoCNLWj3qYt2cCvtr4=; b=XW2CiDjiRvBMck7ujlNe6h3a2pOtw+CeJejHH1e7rbes1T/W9Y87TL+VJ7qt5R2oz/D+Px lSgwVSFvR6cVa1Lmx7mf6d2oJ2u+bdeZFxqDlLeWZjZ4ffjxeN07u5E2zhnoFsBc+A/cFD gFnCI2UcwJLUxLB2kiqWDY0FbIVyLwE= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-456-akvGVTJ7NdyVNx4tFndzPA-1; Fri, 06 Sep 2024 08:10:03 -0400 X-MC-Unique: akvGVTJ7NdyVNx4tFndzPA-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 12C701953941; Fri, 6 Sep 2024 12:10:02 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id E0D641956086; Fri, 6 Sep 2024 12:09:59 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 06/18] libcamera: software_isp: Remove final dots in debayer.cpp docstrings Date: Fri, 6 Sep 2024 14:09:15 +0200 Message-ID: <20240906120927.4071508-7-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The policy and the style checker require that \brief, \param and \return texts don't finish with a dot. This needs to be fixed in debayer.cpp. Also leading spaces in a \return statement are removed from there. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Daniel Scally Reviewed-by: Laurent Pinchart --- src/libcamera/software_isp/debayer.cpp | 48 +++++++++++++------------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/src/libcamera/software_isp/debayer.cpp b/src/libcamera/software_isp/debayer.cpp index f4a299d5..db26c380 100644 --- a/src/libcamera/software_isp/debayer.cpp +++ b/src/libcamera/software_isp/debayer.cpp @@ -58,47 +58,47 @@ Debayer::~Debayer() /** * \fn int Debayer::configure(const StreamConfiguration &inputCfg, const std::vector> &outputCfgs) - * \brief Configure the debayer object according to the passed in parameters. - * \param[in] inputCfg The input configuration. - * \param[in] outputCfgs The output configurations. + * \brief Configure the debayer object according to the passed in parameters + * \param[in] inputCfg The input configuration + * \param[in] outputCfgs The output configurations * - * \return 0 on success, a negative errno on failure. + * \return 0 on success, a negative errno on failure */ /** * \fn Size Debayer::patternSize(PixelFormat inputFormat) - * \brief Get the width and height at which the bayer pattern repeats. - * \param[in] inputFormat The input format. + * \brief Get the width and height at which the bayer pattern repeats + * \param[in] inputFormat The input format * * Valid sizes are: 2x2, 4x2 or 4x4. * - * \return Pattern size or an empty size for unsupported inputFormats. + * \return Pattern size or an empty size for unsupported inputFormats */ /** * \fn std::vector Debayer::formats(PixelFormat inputFormat) - * \brief Get the supported output formats. - * \param[in] inputFormat The input format. + * \brief Get the supported output formats + * \param[in] inputFormat The input format * - * \return All supported output formats or an empty vector if there are none. + * \return All supported output formats or an empty vector if there are none */ /** * \fn std::tuple Debayer::strideAndFrameSize(const PixelFormat &outputFormat, const Size &size) - * \brief Get the stride and the frame size. - * \param[in] outputFormat The output format. - * \param[in] size The output size. + * \brief Get the stride and the frame size + * \param[in] outputFormat The output format + * \param[in] size The output size * * \return A tuple of the stride and the frame size, or a tuple with 0,0 if - * there is no valid output config. + * there is no valid output config */ /** * \fn void Debayer::process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) - * \brief Process the bayer data into the requested format. - * \param[in] input The input buffer. - * \param[in] output The output buffer. - * \param[in] params The parameters to be used in debayering. + * \brief Process the bayer data into the requested format + * \param[in] input The input buffer + * \param[in] output The output buffer + * \param[in] params The parameters to be used in debayering * * \note DebayerParams is passed by value deliberately so that a copy is passed * when this is run in another thread by invokeMethod(). @@ -106,21 +106,21 @@ Debayer::~Debayer() /** * \fn virtual SizeRange Debayer::sizes(PixelFormat inputFormat, const Size &inputSize) - * \brief Get the supported output sizes for the given input format and size. - * \param[in] inputFormat The input format. - * \param[in] inputSize The input size. + * \brief Get the supported output sizes for the given input format and size + * \param[in] inputFormat The input format + * \param[in] inputSize The input size * - * \return The valid size ranges or an empty range if there are none. + * \return The valid size ranges or an empty range if there are none */ /** * \var Signal Debayer::inputBufferReady - * \brief Signals when the input buffer is ready. + * \brief Signals when the input buffer is ready */ /** * \var Signal Debayer::outputBufferReady - * \brief Signals when the output buffer is ready. + * \brief Signals when the output buffer is ready */ } /* namespace libcamera */ From patchwork Fri Sep 6 12:09:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21180 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4511AC324C for ; Fri, 6 Sep 2024 12:10:19 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C6BD7634FE; Fri, 6 Sep 2024 14:10:17 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="Ej/fQp3S"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C2ECD634EE for ; Fri, 6 Sep 2024 14:10:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624608; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=laVMS9tWTJVwDwJX0wk8aN8Lf7oY7XY3hIONyTbhkQI=; b=Ej/fQp3SQT8HJC9vj+bfvoE8str4+RF22C1sDEgyg3ZfythAFG+V7Xur19mtkEywf4B5zh cYS+d+2i6lfx78RmEQUPkTkqLwnFeaWiXC4+r4iGLDDK7n0DDySoHKZW9MjZW94gDAKdQ4 TVv4kVsjQlQwx02nNrPfgEZY2kDXna8= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-584-AoTWgugGP4-Wwv8sdjNvKA-1; Fri, 06 Sep 2024 08:10:05 -0400 X-MC-Unique: AoTWgugGP4-Wwv8sdjNvKA-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 5DAA11955F10; Fri, 6 Sep 2024 12:10:04 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 7D6E81956094; Fri, 6 Sep 2024 12:10:02 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 07/18] libcamera: software_isp: Track and pass frame ids Date: Fri, 6 Sep 2024 14:09:16 +0200 Message-ID: <20240906120927.4071508-8-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" A previous preparation patch implemented passing frame ids to stats processing but without actual meaningful frame id value passed there. This patch extends that by actually providing the frame id and passing it through to the stats processor. The frame id is taken from the request sequence number, the same as in hardware pipelines. Signed-off-by: Milan Zamazal Reviewed-by: Daniel Scally Reviewed-by: Kieran Bingham --- .../libcamera/internal/software_isp/software_isp.h | 4 ++-- src/libcamera/pipeline/simple/simple.cpp | 8 +++++++- src/libcamera/software_isp/debayer.cpp | 3 ++- src/libcamera/software_isp/debayer.h | 2 +- src/libcamera/software_isp/debayer_cpu.cpp | 9 ++++----- src/libcamera/software_isp/debayer_cpu.h | 2 +- src/libcamera/software_isp/software_isp.cpp | 11 +++++++---- 7 files changed, 24 insertions(+), 15 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index 3602bce8..3a84418e 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -73,10 +73,10 @@ public: int start(); void stop(); - int queueBuffers(FrameBuffer *input, + int queueBuffers(uint32_t frame, FrameBuffer *input, const std::map &outputs); - void process(FrameBuffer *input, FrameBuffer *output); + void process(uint32_t frame, FrameBuffer *input, FrameBuffer *output); Signal inputBufferReady; Signal outputBufferReady; diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 48a568da..ebec592a 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -865,7 +865,13 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) if (converter_) converter_->queueBuffers(buffer, conversionQueue_.front()); else - swIsp_->queueBuffers(buffer, conversionQueue_.front()); + /* + * request->sequence() cannot be retrieved from `buffer' inside + * queueBuffers because unique_ptr's make buffer->request() invalid + * already here. + */ + swIsp_->queueBuffers(request->sequence(), buffer, + conversionQueue_.front()); conversionQueue_.pop(); return; diff --git a/src/libcamera/software_isp/debayer.cpp b/src/libcamera/software_isp/debayer.cpp index db26c380..f0b83261 100644 --- a/src/libcamera/software_isp/debayer.cpp +++ b/src/libcamera/software_isp/debayer.cpp @@ -94,8 +94,9 @@ Debayer::~Debayer() */ /** - * \fn void Debayer::process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) + * \fn void Debayer::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params) * \brief Process the bayer data into the requested format + * \param[in] frame The frame number * \param[in] input The input buffer * \param[in] output The output buffer * \param[in] params The parameters to be used in debayering diff --git a/src/libcamera/software_isp/debayer.h b/src/libcamera/software_isp/debayer.h index c151fe5d..d7ca060d 100644 --- a/src/libcamera/software_isp/debayer.h +++ b/src/libcamera/software_isp/debayer.h @@ -40,7 +40,7 @@ public: virtual std::tuple strideAndFrameSize(const PixelFormat &outputFormat, const Size &size) = 0; - virtual void process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) = 0; + virtual void process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params) = 0; virtual SizeRange sizes(PixelFormat inputFormat, const Size &inputSize) = 0; diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index 2a2e7edb..f7b3a7d1 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -724,7 +724,7 @@ static inline int64_t timeDiff(timespec &after, timespec &before) (int64_t)after.tv_nsec - (int64_t)before.tv_nsec; } -void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) +void DebayerCpu::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params) { timespec frameStartTime; @@ -778,12 +778,11 @@ void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams } /* - * Frame and buffer ids are currently not used, so pass zeros as parameters. + * Buffer ids are currently not used, so pass zeros as its parameter. * - * \todo Pass real values once frame is passed here and stats buffer passing - * is changed. + * \todo Pass real bufferId once stats buffer passing is changed. */ - stats_->finishFrame(0, 0); + stats_->finishFrame(frame, 0); outputBufferReady.emit(output); inputBufferReady.emit(input); } diff --git a/src/libcamera/software_isp/debayer_cpu.h b/src/libcamera/software_isp/debayer_cpu.h index 8237a64b..2c47e7c6 100644 --- a/src/libcamera/software_isp/debayer_cpu.h +++ b/src/libcamera/software_isp/debayer_cpu.h @@ -36,7 +36,7 @@ public: std::vector formats(PixelFormat input); std::tuple strideAndFrameSize(const PixelFormat &outputFormat, const Size &size); - void process(FrameBuffer *input, FrameBuffer *output, DebayerParams params); + void process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params); SizeRange sizes(PixelFormat inputFormat, const Size &inputSize); /** diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index a3855568..c5db45ae 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -14,6 +14,7 @@ #include #include +#include #include #include "libcamera/internal/ipa_manager.h" @@ -278,12 +279,13 @@ int SoftwareIsp::exportBuffers(const Stream *stream, unsigned int count, /** * \brief Queue buffers to Software ISP + * \param[in] frame The frame number * \param[in] input The input framebuffer * \param[in] outputs The container holding the output stream pointers and * their respective frame buffer outputs * \return 0 on success, a negative errno on failure */ -int SoftwareIsp::queueBuffers(FrameBuffer *input, +int SoftwareIsp::queueBuffers(uint32_t frame, FrameBuffer *input, const std::map &outputs) { /* @@ -301,7 +303,7 @@ int SoftwareIsp::queueBuffers(FrameBuffer *input, } for (auto iter = outputs.begin(); iter != outputs.end(); iter++) - process(input, iter->second); + process(frame, input, iter->second); return 0; } @@ -333,13 +335,14 @@ void SoftwareIsp::stop() /** * \brief Passes the input framebuffer to the ISP worker to process + * \param[in] frame The frame number * \param[in] input The input framebuffer * \param[out] output The framebuffer to write the processed frame to */ -void SoftwareIsp::process(FrameBuffer *input, FrameBuffer *output) +void SoftwareIsp::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output) { debayer_->invokeMethod(&DebayerCpu::process, - ConnectionTypeQueued, input, output, debayerParams_); + ConnectionTypeQueued, frame, input, output, debayerParams_); } void SoftwareIsp::saveIspParams() From patchwork Fri Sep 6 12:09:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21181 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9CE25C324C for ; Fri, 6 Sep 2024 12:10:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0162B63502; Fri, 6 Sep 2024 14:10:23 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="Wm3miDHu"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E94DD634F1 for ; Fri, 6 Sep 2024 14:10:10 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624609; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4OojRvR7hV5VSh6rZzOTFwKTXnvVsUvy9KHsSwPlpII=; b=Wm3miDHuqGB5sB00OfI1akcXzp+a49/BBaxS+5e3gG81eklsjv3lvUtOX1uARZQMdn3737 GoIreexst3zEisbcsFVP5Q876hhoyeThEIMQzrcbQRY7o/MuZbzjIxRvH+gIfZxIQuDMuF eZAsIHUhpPXMaaVLUyJSUx1zOb6MaUE= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-608-Gz91CiS6NhyJrkLylVyTyQ-1; Fri, 06 Sep 2024 08:10:08 -0400 X-MC-Unique: Gz91CiS6NhyJrkLylVyTyQ-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 66A1B1955D45; Fri, 6 Sep 2024 12:10:07 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id D316A195608A; Fri, 6 Sep 2024 12:10:04 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 08/18] libcamera: software_isp: Create algorithms Date: Fri, 6 Sep 2024 14:09:17 +0200 Message-ID: <20240906120927.4071508-9-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We are ready to introduce algorithms now. First, let's create algorithms. The algorithms are not called yet, calls to them will be added in followup patches. The maximum number of contexts is set to the same value as in hardware pipelines. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Daniel Scally Reviewed-by: Laurent Pinchart --- src/ipa/simple/algorithms/meson.build | 4 ++++ src/ipa/simple/data/uncalibrated.yaml | 1 + src/ipa/simple/meson.build | 7 +++++-- src/ipa/simple/soft_simple.cpp | 14 ++++++++++++++ 4 files changed, 24 insertions(+), 2 deletions(-) create mode 100644 src/ipa/simple/algorithms/meson.build diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build new file mode 100644 index 00000000..31d26e43 --- /dev/null +++ b/src/ipa/simple/algorithms/meson.build @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: CC0-1.0 + +soft_simple_ipa_algorithms = files([ +]) diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index ff981a1a..2cdc39a8 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -2,4 +2,5 @@ %YAML 1.1 --- version: 1 +algorithms: ... diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build index 7757f99b..dcd7c70a 100644 --- a/src/ipa/simple/meson.build +++ b/src/ipa/simple/meson.build @@ -1,5 +1,8 @@ # SPDX-License-Identifier: CC0-1.0 +subdir('algorithms') +subdir('data') + ipa_name = 'ipa_soft_simple' soft_simple_sources = files([ @@ -8,6 +11,8 @@ soft_simple_sources = files([ 'black_level.cpp', ]) +soft_simple_sources += soft_simple_ipa_algorithms + mod = shared_module(ipa_name, soft_simple_sources, name_prefix : '', include_directories : [ipa_includes], @@ -24,6 +29,4 @@ if ipa_sign_module build_by_default : true) endif -subdir('data') - ipa_names += ipa_name diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 12b5245e..79ed4891 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -54,12 +54,15 @@ static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; * enough to prevent the exposure from wobbling around the optimal value. */ static constexpr float kExposureSatisfactory = 0.2; +/* Maximum number of frame contexts to be held */ +static constexpr uint32_t kMaxFrameContexts = 16; class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() : params_(nullptr), stats_(nullptr), blackLevel_(BlackLevel()), + context_({ {}, {}, { kMaxFrameContexts } }), ignoreUpdates_(0) { } @@ -93,6 +96,8 @@ private: static constexpr unsigned int kGammaLookupSize = 1024; std::array gammaTable_; int lastBlackLevel_ = -1; + /* Local parameter storage */ + struct IPAContext context_; int32_t exposureMin_, exposureMax_; int32_t exposure_; @@ -139,6 +144,15 @@ int IPASoftSimple::init(const IPASettings &settings, unsigned int version = (*data)["version"].get(0); LOG(IPASoft, Debug) << "Tuning file version " << version; + if (!data->contains("algorithms")) { + LOG(IPASoft, Error) << "Tuning file doesn't contain algorithms"; + return -EINVAL; + } + + int ret = createAlgorithms(context_, (*data)["algorithms"]); + if (ret) + return ret; + params_ = nullptr; stats_ = nullptr; From patchwork Fri Sep 6 12:09:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21182 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 04D46C324C for ; Fri, 6 Sep 2024 12:10:29 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8B59D63503; Fri, 6 Sep 2024 14:10:28 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="Id/3Nwwp"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 330F5634EE for ; Fri, 6 Sep 2024 14:10:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624614; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W27xTzwxO2ezkfZYbRIGrA12qEXKi2NN5UzkJw4dKnY=; b=Id/3Nwwp19ZCV6C/bYFbdAbIyILNe1YqTOePvNT3n4sQIersFVozh/nsKehdj6RaCDLeXI wprYmdZmJ3I7gExKJD5UDKTWZjT/JE3t8fxOzulqwSt1RCxXt931QHXWcCrUn82gc0NW39 OzNNEe+Ut/CeTZoYmux/Ogr8BcCrHYM= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-73-7LttjJ0ZOtS01S7licTRSw-1; Fri, 06 Sep 2024 08:10:11 -0400 X-MC-Unique: 7LttjJ0ZOtS01S7licTRSw-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id C72671955F28; Fri, 6 Sep 2024 12:10:09 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 89A49195608A; Fri, 6 Sep 2024 12:10:07 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 09/18] libcamera: software_isp: Call Algorithm::configure Date: Fri, 6 Sep 2024 14:09:18 +0200 Message-ID: <20240906120927.4071508-10-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds Algorithm::configure call for the defined algorithms. This is preparation only since there are currently no Algorithm based algorithms defined. A part of this change is passing IPAConfigInfo instead of ControlInfoMap to configure() calls as this is what Algorithm::configure expects. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Laurent Pinchart Reviewed-by: Daniel Scally --- .../libcamera/internal/software_isp/software_isp.h | 2 +- include/libcamera/ipa/soft.mojom | 2 +- src/ipa/simple/soft_simple.cpp | 12 +++++++++--- src/libcamera/pipeline/simple/simple.cpp | 11 +++++++---- src/libcamera/software_isp/software_isp.cpp | 7 ++++--- 5 files changed, 22 insertions(+), 12 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index 3a84418e..c5d5a46f 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -62,7 +62,7 @@ public: int configure(const StreamConfiguration &inputCfg, const std::vector> &outputCfgs, - const ControlInfoMap &sensorControls); + const ipa::soft::IPAConfigInfo &configInfo); int exportBuffers(const Stream *stream, unsigned int count, std::vector> *buffers); diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index f65b6231..88a35a48 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -20,7 +20,7 @@ interface IPASoftInterface { => (int32 ret); start() => (int32 ret); stop(); - configure(libcamera.ControlInfoMap sensorCtrlInfoMap) + configure(IPAConfigInfo configInfo) => (int32 ret); [async] processStats(uint32 frame, diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 79ed4891..0ea62266 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -73,7 +73,7 @@ public: const SharedFD &fdStats, const SharedFD &fdParams, const ControlInfoMap &sensorInfoMap) override; - int configure(const ControlInfoMap &sensorInfoMap) override; + int configure(const IPAConfigInfo &configInfo) override; int start() override; void stop() override; @@ -207,9 +207,9 @@ int IPASoftSimple::init(const IPASettings &settings, return 0; } -int IPASoftSimple::configure(const ControlInfoMap &sensorInfoMap) +int IPASoftSimple::configure(const IPAConfigInfo &configInfo) { - sensorInfoMap_ = sensorInfoMap; + sensorInfoMap_ = configInfo.sensorControls; const ControlInfo &exposureInfo = sensorInfoMap_.find(V4L2_CID_EXPOSURE)->second; const ControlInfo &gainInfo = sensorInfoMap_.find(V4L2_CID_ANALOGUE_GAIN)->second; @@ -248,6 +248,12 @@ int IPASoftSimple::configure(const ControlInfoMap &sensorInfoMap) againMinStep_ = 1.0; } + for (auto const &algo : algorithms()) { + int ret = algo->configure(context_, configInfo); + if (ret) + return ret; + } + LOG(IPASoft, Info) << "Exposure " << exposureMin_ << "-" << exposureMax_ << ", gain " << againMin_ << "-" << againMax_ << " (" << againMinStep_ << ")"; diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index ebec592a..834b33d9 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -1294,10 +1294,13 @@ int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c) inputCfg.stride = captureFormat.planes[0].bpl; inputCfg.bufferCount = kNumInternalBuffers; - return data->converter_ - ? data->converter_->configure(inputCfg, outputCfgs) - : data->swIsp_->configure(inputCfg, outputCfgs, - data->sensor_->controls()); + if (data->converter_) { + return data->converter_->configure(inputCfg, outputCfgs); + } else { + ipa::soft::IPAConfigInfo configInfo; + configInfo.sensorControls = data->sensor_->controls(); + return data->swIsp_->configure(inputCfg, outputCfgs, configInfo); + } } int SimplePipelineHandler::exportFrameBuffers(Camera *camera, Stream *stream, diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index c5db45ae..8f1c0f65 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -223,16 +223,17 @@ SoftwareIsp::strideAndFrameSize(const PixelFormat &outputFormat, const Size &siz * \brief Configure the SoftwareIsp object according to the passed in parameters * \param[in] inputCfg The input configuration * \param[in] outputCfgs The output configurations - * \param[in] sensorControls ControlInfoMap of the controls supported by the sensor + * \param[in] configInfo The IPA configuration data, received from the pipeline + * handler * \return 0 on success, a negative errno on failure */ int SoftwareIsp::configure(const StreamConfiguration &inputCfg, const std::vector> &outputCfgs, - const ControlInfoMap &sensorControls) + const ipa::soft::IPAConfigInfo &configInfo) { ASSERT(ipa_ && debayer_); - int ret = ipa_->configure(sensorControls); + int ret = ipa_->configure(configInfo); if (ret < 0) return ret; From patchwork Fri Sep 6 12:09:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21184 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 818C3BF415 for ; Fri, 6 Sep 2024 12:10:34 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4C032634F7; Fri, 6 Sep 2024 14:10:33 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="C8hv/B2Q"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 053AD634F4 for ; Fri, 6 Sep 2024 14:10:18 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624617; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n8yMYttvCE7JlrEzkV5RB4XMpYhIKFgsnDBpMz/zA20=; b=C8hv/B2Qn+PdSPdv1sNCIgkFUaDqSwdzLOAXFo7W42BHVkjTG0SVIpVypbLeRTE8WUFN7n eBarFi9NjiTVnjZIJpW5n+YY7YENt7NsK1boysIjRzcSgC0IVpB1+TAvm6DBtzAULMk3Yz fN8f80dvPt5p6a5gXqpbRC3Xoq21JoQ= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-643-Q3pCveHINoCcY3KiLfpEqw-1; Fri, 06 Sep 2024 08:10:16 -0400 X-MC-Unique: Q3pCveHINoCcY3KiLfpEqw-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id ACE721955D4B; Fri, 6 Sep 2024 12:10:12 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 61631195608A; Fri, 6 Sep 2024 12:10:10 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 10/18] libcamera: software_isp: Call Algorithm::queueRequest Date: Fri, 6 Sep 2024 14:09:19 +0200 Message-ID: <20240906120927.4071508-11-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds Algorithm::queueRequest call for the defined algorithms. As there are currently no control knobs in software ISP nor the corresponding queueRequest call chain, the patch also introduces the queueRequest methods and calls from the pipeline to the IPA. This is preparation only since there are currently no Algorithm based algorithms defined and no current software ISP algorithms support control knobs. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Daniel Scally Reviewed-by: Laurent Pinchart --- include/libcamera/internal/software_isp/software_isp.h | 1 + include/libcamera/ipa/soft.mojom | 1 + src/ipa/simple/soft_simple.cpp | 9 +++++++++ src/libcamera/pipeline/simple/simple.cpp | 5 ++++- src/libcamera/software_isp/software_isp.cpp | 10 ++++++++++ 5 files changed, 25 insertions(+), 1 deletion(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index c5d5a46f..a3e3a9da 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -73,6 +73,7 @@ public: int start(); void stop(); + void queueRequest(const uint32_t frame, const ControlList &controls); int queueBuffers(uint32_t frame, FrameBuffer *input, const std::map &outputs); diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index 88a35a48..ddccd154 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -23,6 +23,7 @@ interface IPASoftInterface { configure(IPAConfigInfo configInfo) => (int32 ret); + [async] queueRequest(uint32 frame, libcamera.ControlList sensorControls); [async] processStats(uint32 frame, uint32 bufferId, libcamera.ControlList sensorControls); diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 0ea62266..eb3bbd92 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -78,6 +78,7 @@ public: int start() override; void stop() override; + void queueRequest(const uint32_t frame, const ControlList &controls) override; void processStats(const uint32_t frame, const uint32_t bufferId, const ControlList &sensorControls) override; @@ -270,6 +271,14 @@ void IPASoftSimple::stop() { } +void IPASoftSimple::queueRequest(const uint32_t frame, const ControlList &controls) +{ + IPAFrameContext &frameContext = context_.frameContexts.alloc(frame); + + for (auto const &algo : algorithms()) + algo->queueRequest(context_, frame, frameContext, controls); +} + void IPASoftSimple::processStats([[maybe_unused]] const uint32_t frame, [[maybe_unused]] const uint32_t bufferId, const ControlList &sensorControls) diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 834b33d9..7e412e17 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -1425,8 +1425,11 @@ int SimplePipelineHandler::queueRequestDevice(Camera *camera, Request *request) } } - if (data->useConversion_) + if (data->useConversion_) { data->conversionQueue_.push(std::move(buffers)); + if (data->swIsp_) + data->swIsp_->queueRequest(request->sequence(), request->controls()); + } return 0; } diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 8f1c0f65..eda18947 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -278,6 +278,16 @@ int SoftwareIsp::exportBuffers(const Stream *stream, unsigned int count, return count; } +/** + * \brief Queue a request and process the control list from the application + * \param[in] frame The number of the frame which will be processed next + * \param[in] controls The controls for the \a frame + */ +void SoftwareIsp::queueRequest(const uint32_t frame, const ControlList &controls) +{ + ipa_->queueRequest(frame, controls); +} + /** * \brief Queue buffers to Software ISP * \param[in] frame The frame number From patchwork Fri Sep 6 12:09:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21183 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id EAF5FBF415 for ; Fri, 6 Sep 2024 12:10:31 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BAFE5634FB; Fri, 6 Sep 2024 14:10:30 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="Wx1/owF9"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id BA625634F7 for ; Fri, 6 Sep 2024 14:10:18 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624617; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hEtQxcCLdObCG6r6Ya6gn0k3ep95d0wOWAtE7xk4lLU=; b=Wx1/owF9/bNyt9Trz9fvffsuR64epXkaLHrcG7UcCwThpjH17tZGlCT8uqK2htFnCJ8gml egB51YEJ7znVw12MxaJ02uPClquDgwdBR/QraDOZiqYHoggoY9jKWjEUDqcwZTFRWhAVUh DILZ5acwKbdd3G1I4vhCp1GEzs4BStw= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-590-Sz2KlX6wPHS5HzZ1XF0nHQ-1; Fri, 06 Sep 2024 08:10:16 -0400 X-MC-Unique: Sz2KlX6wPHS5HzZ1XF0nHQ-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 23216195608C; Fri, 6 Sep 2024 12:10:15 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 2810B1956086; Fri, 6 Sep 2024 12:10:12 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 11/18] libcamera: software_isp: Call Algorithm::prepare Date: Fri, 6 Sep 2024 14:09:20 +0200 Message-ID: <20240906120927.4071508-12-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds Algorithm::prepare call for the defined algorithms. This is preparation only since there are currently no Algorithm based algorithms defined. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Daniel Scally Reviewed-by: Kieran Bingham --- include/libcamera/ipa/soft.mojom | 1 + src/ipa/simple/soft_simple.cpp | 8 ++++++++ src/libcamera/software_isp/software_isp.cpp | 1 + 3 files changed, 10 insertions(+) diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index ddccd154..347fd69b 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -24,6 +24,7 @@ interface IPASoftInterface { => (int32 ret); [async] queueRequest(uint32 frame, libcamera.ControlList sensorControls); + [async] fillParamsBuffer(uint32 frame); [async] processStats(uint32 frame, uint32 bufferId, libcamera.ControlList sensorControls); diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index eb3bbd92..3a0cb6e0 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -79,6 +79,7 @@ public: void stop() override; void queueRequest(const uint32_t frame, const ControlList &controls) override; + void fillParamsBuffer(const uint32_t frame) override; void processStats(const uint32_t frame, const uint32_t bufferId, const ControlList &sensorControls) override; @@ -279,6 +280,13 @@ void IPASoftSimple::queueRequest(const uint32_t frame, const ControlList &contro algo->queueRequest(context_, frame, frameContext, controls); } +void IPASoftSimple::fillParamsBuffer(const uint32_t frame) +{ + IPAFrameContext &frameContext = context_.frameContexts.get(frame); + for (auto const &algo : algorithms()) + algo->prepare(context_, frame, frameContext, params_); +} + void IPASoftSimple::processStats([[maybe_unused]] const uint32_t frame, [[maybe_unused]] const uint32_t bufferId, const ControlList &sensorControls) diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index eda18947..564e44e8 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -352,6 +352,7 @@ void SoftwareIsp::stop() */ void SoftwareIsp::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output) { + ipa_->fillParamsBuffer(frame); debayer_->invokeMethod(&DebayerCpu::process, ConnectionTypeQueued, frame, input, output, debayerParams_); } From patchwork Fri Sep 6 12:09:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21185 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 951E7BF415 for ; Fri, 6 Sep 2024 12:10:37 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 85259634FA; Fri, 6 Sep 2024 14:10:36 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="E6uR2zvh"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7E43A634FC for ; Fri, 6 Sep 2024 14:10:22 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624620; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RbyAJlXBktvqX7d9o7V2fel/xt8W2HigLsJ6IKCxoWs=; b=E6uR2zvhvq0hw6CUv2EC/qzYFoOv0vj6jjPTp/2+e3kFzdgbKL73iw7rudpSlTfwCjth85 fMMHWsZq9tnhH//ykQYdaWOSQcL6dAdunQxTHJ4SpEo72spPnFTYhDO01/NBT1gz8n/4YE myj2FbDXoz973IVNejYlK+OtRl79XPc= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-438-3N6ebnBbMDSYOXCgfgBSTA-1; Fri, 06 Sep 2024 08:10:19 -0400 X-MC-Unique: 3N6ebnBbMDSYOXCgfgBSTA-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id AB6231956096; Fri, 6 Sep 2024 12:10:17 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id B08AF1956086; Fri, 6 Sep 2024 12:10:15 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 12/18] libcamera: software_isp: Call Algorithm::process Date: Fri, 6 Sep 2024 14:09:21 +0200 Message-ID: <20240906120927.4071508-13-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds Algorithm::process call for the defined algorithms. This is preparation only since there are currently no Algorithm based algorithms defined. As software ISP currently doesn't produce any metadata, a dummy and unused metadata instance is created to satisfy Algorithm::process API. This should be changed in future. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Daniel Scally Reviewed-by: Laurent Pinchart --- src/ipa/simple/soft_simple.cpp | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 3a0cb6e0..5beec45a 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -287,10 +287,21 @@ void IPASoftSimple::fillParamsBuffer(const uint32_t frame) algo->prepare(context_, frame, frameContext, params_); } -void IPASoftSimple::processStats([[maybe_unused]] const uint32_t frame, +void IPASoftSimple::processStats(const uint32_t frame, [[maybe_unused]] const uint32_t bufferId, const ControlList &sensorControls) { + IPAFrameContext &frameContext = context_.frameContexts.get(frame); + /* + * Software ISP currently does not produce any metadata. Use an empty + * ControlList for now. + * + * \todo Implement proper metadata handling + */ + ControlList metadata(controls::controls); + for (auto const &algo : algorithms()) + algo->process(context_, frame, frameContext, stats_, metadata); + SwIspStats::Histogram histogram = stats_->yHistogram; if (ignoreUpdates_ > 0) blackLevel_.update(histogram); From patchwork Fri Sep 6 12:09:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21186 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7B8E2BF415 for ; Fri, 6 Sep 2024 12:10:44 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id EA394634FD; Fri, 6 Sep 2024 14:10:43 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="NDuPuRBm"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9EE2D634F1 for ; Fri, 6 Sep 2024 14:10:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624626; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v3hbskSj10Sebk+q4Ui2FQh35wFYE0OLZnvkVpq2zc8=; b=NDuPuRBmkBdWHL/Zq7UHw4c3wAaYKy+tVIQSJVN57ItLQPXvZXJzVjGDFCZShxA+OZNkRT mpiuwX6r4kgr71FIaOJG26UnJ51KmbhUINKWXLmYApBHRUg+BrJnUtxgcBjaHW05mEvWx6 A5Dn73jsVNBPzfC0s4SbeE0lPVq8odM= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-678-d4ae-EcZMSC0cFKbCkkbSg-1; Fri, 06 Sep 2024 08:10:21 -0400 X-MC-Unique: d4ae-EcZMSC0cFKbCkkbSg-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 903F41955DDC; Fri, 6 Sep 2024 12:10:20 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 451D11956086; Fri, 6 Sep 2024 12:10:17 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 13/18] libcamera: software_isp: Move black level to an algorithm module Date: Fri, 6 Sep 2024 14:09:22 +0200 Message-ID: <20240906120927.4071508-14-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The black level determination, already present as a separate class, can be moved to the prepared Algorithm processing structure. It is the first of the current software ISP algorithms that is called in the stats processing sequence, which means it is also the first one that should be moved to the new structure. Stats processing starts with calling Algorithm-based processing so the call order of the algorithms is retained. Movement of this algorithm is relatively straightforward because all it does is processing stats. The comment about getting black level from the tuning files is dropped. The black level will be taken from CameraSensorHelper if available and that will be implemented in one of the followup patches. Black level is now recomputed on each stats processing. In a future patch, after DelayedControls are used, this will be changed to recompute the black level only after exposure/gain changes. The black level depends on the sensor used, the computed value can be reused in a followup capture sessions with the same sensor. Thus it is sufficient to (re)set the initial value in BlackLevel::init. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Laurent Pinchart Reviewed-by: Daniel Scally --- src/ipa/simple/algorithms/blc.cpp | 71 ++++++++++++++++++++ src/ipa/simple/algorithms/blc.h | 32 +++++++++ src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/black_level.cpp | 93 --------------------------- src/ipa/simple/black_level.h | 33 ---------- src/ipa/simple/data/uncalibrated.yaml | 1 + src/ipa/simple/ipa_context.cpp | 8 +++ src/ipa/simple/ipa_context.h | 5 ++ src/ipa/simple/meson.build | 1 - src/ipa/simple/soft_simple.cpp | 8 +-- 10 files changed, 120 insertions(+), 133 deletions(-) create mode 100644 src/ipa/simple/algorithms/blc.cpp create mode 100644 src/ipa/simple/algorithms/blc.h delete mode 100644 src/ipa/simple/black_level.cpp delete mode 100644 src/ipa/simple/black_level.h diff --git a/src/ipa/simple/algorithms/blc.cpp b/src/ipa/simple/algorithms/blc.cpp new file mode 100644 index 00000000..08f4345e --- /dev/null +++ b/src/ipa/simple/algorithms/blc.cpp @@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Black level handling + */ + +#include "blc.h" + +#include + +#include + +namespace libcamera { + +namespace ipa::soft::algorithms { + +LOG_DEFINE_CATEGORY(IPASoftBL) + +BlackLevel::BlackLevel() +{ +} + +int BlackLevel::init(IPAContext &context, + [[maybe_unused]] const YamlObject &tuningData) +{ + context.activeState.blc.level = 255; + return 0; +} + +void BlackLevel::process(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + const SwIspStats *stats, + [[maybe_unused]] ControlList &metadata) +{ + const SwIspStats::Histogram &histogram = stats->yHistogram; + + /* + * The constant is selected to be "good enough", not overly + * conservative or aggressive. There is no magic about the given value. + */ + constexpr float ignoredPercentage = 0.02; + const unsigned int total = + std::accumulate(begin(histogram), end(histogram), 0); + const unsigned int pixelThreshold = ignoredPercentage * total; + const unsigned int histogramRatio = 256 / SwIspStats::kYHistogramSize; + const unsigned int currentBlackIdx = + context.activeState.blc.level / histogramRatio; + + for (unsigned int i = 0, seen = 0; + i < currentBlackIdx && i < SwIspStats::kYHistogramSize; + i++) { + seen += histogram[i]; + if (seen >= pixelThreshold) { + context.activeState.blc.level = i * histogramRatio; + LOG(IPASoftBL, Debug) + << "Auto-set black level: " + << i << "/" << SwIspStats::kYHistogramSize + << " (" << 100 * (seen - histogram[i]) / total << "% below, " + << 100 * seen / total << "% at or below)"; + break; + } + }; +} + +REGISTER_IPA_ALGORITHM(BlackLevel, "BlackLevel") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/blc.h b/src/ipa/simple/algorithms/blc.h new file mode 100644 index 00000000..c2140b4b --- /dev/null +++ b/src/ipa/simple/algorithms/blc.h @@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Black level handling + */ + +#pragma once + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class BlackLevel : public Algorithm +{ +public: + BlackLevel(); + ~BlackLevel() = default; + + int init(IPAContext &context, const YamlObject &tuningData) + override; + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const SwIspStats *stats, + ControlList &metadata) override; +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index 31d26e43..1f63c220 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -1,4 +1,5 @@ # SPDX-License-Identifier: CC0-1.0 soft_simple_ipa_algorithms = files([ + 'blc.cpp', ]) diff --git a/src/ipa/simple/black_level.cpp b/src/ipa/simple/black_level.cpp deleted file mode 100644 index 37e0109c..00000000 --- a/src/ipa/simple/black_level.cpp +++ /dev/null @@ -1,93 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1-or-later */ -/* - * Copyright (C) 2024, Red Hat Inc. - * - * black level handling - */ - -#include "black_level.h" - -#include - -#include - -namespace libcamera { - -LOG_DEFINE_CATEGORY(IPASoftBL) - -namespace ipa::soft { - -/** - * \class BlackLevel - * \brief Object providing black point level for software ISP - * - * Black level can be provided in hardware tuning files or, if no tuning file is - * available for the given hardware, guessed automatically, with less accuracy. - * As tuning files are not yet implemented for software ISP, BlackLevel - * currently provides only guessed black levels. - * - * This class serves for tracking black level as a property of the underlying - * hardware, not as means of enhancing a particular scene or image. - * - * The class is supposed to be instantiated for the given camera stream. - * The black level can be retrieved using BlackLevel::get() method. It is - * initially 0 and may change when updated using BlackLevel::update() method. - */ - -BlackLevel::BlackLevel() - : blackLevel_(255), blackLevelSet_(false) -{ -} - -/** - * \brief Return the current black level - * - * \return The black level, in the range from 0 (minimum) to 255 (maximum). - * If the black level couldn't be determined yet, return 0. - */ -uint8_t BlackLevel::get() const -{ - return blackLevelSet_ ? blackLevel_ : 0; -} - -/** - * \brief Update black level from the provided histogram - * \param[in] yHistogram The histogram to be used for updating black level - * - * The black level is property of the given hardware, not image. It is updated - * only if it has not been yet set or if it is lower than the lowest value seen - * so far. - */ -void BlackLevel::update(SwIspStats::Histogram &yHistogram) -{ - /* - * The constant is selected to be "good enough", not overly conservative or - * aggressive. There is no magic about the given value. - */ - constexpr float ignoredPercentage_ = 0.02; - const unsigned int total = - std::accumulate(begin(yHistogram), end(yHistogram), 0); - const unsigned int pixelThreshold = ignoredPercentage_ * total; - const unsigned int histogramRatio = 256 / SwIspStats::kYHistogramSize; - const unsigned int currentBlackIdx = blackLevel_ / histogramRatio; - - for (unsigned int i = 0, seen = 0; - i < currentBlackIdx && i < SwIspStats::kYHistogramSize; - i++) { - seen += yHistogram[i]; - if (seen >= pixelThreshold) { - blackLevel_ = i * histogramRatio; - blackLevelSet_ = true; - LOG(IPASoftBL, Debug) - << "Auto-set black level: " - << i << "/" << SwIspStats::kYHistogramSize - << " (" << 100 * (seen - yHistogram[i]) / total << "% below, " - << 100 * seen / total << "% at or below)"; - break; - } - }; -} - -} /* namespace ipa::soft */ - -} /* namespace libcamera */ diff --git a/src/ipa/simple/black_level.h b/src/ipa/simple/black_level.h deleted file mode 100644 index a04230c9..00000000 --- a/src/ipa/simple/black_level.h +++ /dev/null @@ -1,33 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1-or-later */ -/* - * Copyright (C) 2024, Red Hat Inc. - * - * black level handling - */ - -#pragma once - -#include -#include - -#include "libcamera/internal/software_isp/swisp_stats.h" - -namespace libcamera { - -namespace ipa::soft { - -class BlackLevel -{ -public: - BlackLevel(); - uint8_t get() const; - void update(SwIspStats::Histogram &yHistogram); - -private: - uint8_t blackLevel_; - bool blackLevelSet_; -}; - -} /* namespace ipa::soft */ - -} /* namespace libcamera */ diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index 2cdc39a8..e0d03d96 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -3,4 +3,5 @@ --- version: 1 algorithms: + - BlackLevel: ... diff --git a/src/ipa/simple/ipa_context.cpp b/src/ipa/simple/ipa_context.cpp index 3c1c7262..268cff32 100644 --- a/src/ipa/simple/ipa_context.cpp +++ b/src/ipa/simple/ipa_context.cpp @@ -50,4 +50,12 @@ namespace libcamera::ipa::soft { * \brief The current state of IPA algorithms */ +/** + * \var IPAActiveState::black + * \brief Context for the Black Level algorithm + * + * \var IPAActiveState::black.level + * \brief Current determined black level + */ + } /* namespace libcamera::ipa::soft */ diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index e7d8b8df..ac2a59d7 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -7,6 +7,8 @@ #pragma once +#include + #include namespace libcamera { @@ -17,6 +19,9 @@ struct IPASessionConfiguration { }; struct IPAActiveState { + struct { + uint8_t level; + } blc; }; struct IPAFrameContext : public FrameContext { diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build index dcd7c70a..2f9f15f4 100644 --- a/src/ipa/simple/meson.build +++ b/src/ipa/simple/meson.build @@ -8,7 +8,6 @@ ipa_name = 'ipa_soft_simple' soft_simple_sources = files([ 'ipa_context.cpp', 'soft_simple.cpp', - 'black_level.cpp', ]) soft_simple_sources += soft_simple_ipa_algorithms diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 5beec45a..3332e2b1 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -29,7 +29,6 @@ #include "libipa/camera_sensor_helper.h" -#include "black_level.h" #include "module.h" namespace libcamera { @@ -61,7 +60,7 @@ class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() - : params_(nullptr), stats_(nullptr), blackLevel_(BlackLevel()), + : params_(nullptr), stats_(nullptr), context_({ {}, {}, { kMaxFrameContexts } }), ignoreUpdates_(0) { @@ -93,7 +92,6 @@ private: SwIspStats *stats_; std::unique_ptr camHelper_; ControlInfoMap sensorInfoMap_; - BlackLevel blackLevel_; static constexpr unsigned int kGammaLookupSize = 1024; std::array gammaTable_; @@ -303,9 +301,7 @@ void IPASoftSimple::processStats(const uint32_t frame, algo->process(context_, frame, frameContext, stats_, metadata); SwIspStats::Histogram histogram = stats_->yHistogram; - if (ignoreUpdates_ > 0) - blackLevel_.update(histogram); - const uint8_t blackLevel = blackLevel_.get(); + const uint8_t blackLevel = context_.activeState.blc.level; /* * Black level must be subtracted to get the correct AWB ratios, they From patchwork Fri Sep 6 12:09:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21187 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8DC34BF415 for ; Fri, 6 Sep 2024 12:10:46 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id EC74D63500; Fri, 6 Sep 2024 14:10:45 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="DO90ovhl"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0FDC663502 for ; Fri, 6 Sep 2024 14:10:28 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624628; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9RkOfsWz3YnM9bkPbXe1D/saK6M+VpbupXf2+4tVu4g=; b=DO90ovhlmeVdPQRP118ay8ZPjyzaTDxFdG9p8NSkhbbrxq4dLtS4v1EMNIY+tHaB3gvVWT cu2zXSNgnUiiDZsW40sxssfcr5GYCLgZU45TawiCo/H0gmGul5HLwJ76nk9/BfudcvThOk jW9jJmMEGz2sZ7UC9TUojXWCPBVbysU= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-214-7x5otMNNP5iqILZiTuALog-1; Fri, 06 Sep 2024 08:10:24 -0400 X-MC-Unique: 7x5otMNNP5iqILZiTuALog-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 09839195608C; Fri, 6 Sep 2024 12:10:23 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 0BBFC1956086; Fri, 6 Sep 2024 12:10:20 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 14/18] libcamera: software_isp: Move color handling to an algorithm module Date: Fri, 6 Sep 2024 14:09:23 +0200 Message-ID: <20240906120927.4071508-15-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" After black level handling has been moved to an algorithm module, white balance and the construction of color tables can be moved to algorithm modules too. This time, the moved code is split between stats processing and parameter construction methods. It is also split to two algorithm modules: - White balance computation. - Gamma table computation and color lookup tables construction. While this applies the color gains computed by the white balance algorithm, it is not directly related to white balance. And we may want to modify the color lookup tables in future according to other parameters than just gamma and white balance gains. Gamma table computation and color lookup tables construction could be split to separate algorithms too. But there is no big need for that now so they are kept together for simplicity. This is the only part of the software ISP algorithms that sets the parameters so emitting setIspParams can be moved to prepare() method. A more natural representation of the gains (and black level) would be floating point numbers. This is not done here in order to minimize changes in code movements. It will be addressed in a followup patch. Signed-off-by: Milan Zamazal Reviewed-by: Kieran Bingham --- src/ipa/simple/algorithms/awb.cpp | 70 ++++++++++++++++++++++++ src/ipa/simple/algorithms/awb.h | 32 +++++++++++ src/ipa/simple/algorithms/lut.cpp | 78 +++++++++++++++++++++++++++ src/ipa/simple/algorithms/lut.h | 35 ++++++++++++ src/ipa/simple/algorithms/meson.build | 2 + src/ipa/simple/data/uncalibrated.yaml | 2 + src/ipa/simple/ipa_context.cpp | 30 +++++++++++ src/ipa/simple/ipa_context.h | 14 +++++ src/ipa/simple/soft_simple.cpp | 64 +--------------------- 9 files changed, 265 insertions(+), 62 deletions(-) create mode 100644 src/ipa/simple/algorithms/awb.cpp create mode 100644 src/ipa/simple/algorithms/awb.h create mode 100644 src/ipa/simple/algorithms/lut.cpp create mode 100644 src/ipa/simple/algorithms/lut.h diff --git a/src/ipa/simple/algorithms/awb.cpp b/src/ipa/simple/algorithms/awb.cpp new file mode 100644 index 00000000..f89f1d3d --- /dev/null +++ b/src/ipa/simple/algorithms/awb.cpp @@ -0,0 +1,70 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Auto white balance + */ + +#include "awb.h" + +#include +#include + +#include + +#include "simple/ipa_context.h" + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoftAwb) + +namespace ipa::soft::algorithms { + +int Awb::init(IPAContext &context, + [[maybe_unused]] const YamlObject &tuningData) +{ + auto &gains = context.activeState.gains; + gains.red = gains.green = gains.blue = 256; + + return 0; +} + +void Awb::process(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + const SwIspStats *stats, + [[maybe_unused]] ControlList &metadata) +{ + const SwIspStats::Histogram &histogram = stats->yHistogram; + const uint8_t blackLevel = context.activeState.blc.level; + + /* + * Black level must be subtracted to get the correct AWB ratios, they + * would be off if they were computed from the whole brightness range + * rather than from the sensor range. + */ + const uint64_t nPixels = std::accumulate( + histogram.begin(), histogram.end(), 0); + const uint64_t offset = blackLevel * nPixels; + const uint64_t sumR = stats->sumR_ - offset / 4; + const uint64_t sumG = stats->sumG_ - offset / 2; + const uint64_t sumB = stats->sumB_ - offset / 4; + + /* + * Calculate red and blue gains for AWB. + * Clamp max gain at 4.0, this also avoids 0 division. + * Gain: 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. + */ + auto &gains = context.activeState.gains; + gains.red = sumR <= sumG / 4 ? 1024 : 256 * sumG / sumR; + gains.blue = sumB <= sumG / 4 ? 1024 : 256 * sumG / sumB; + /* Green gain is fixed to 256 */ + + LOG(IPASoftAwb, Debug) << "gain R/B " << gains.red << "/" << gains.blue; +} + +REGISTER_IPA_ALGORITHM(Awb, "Awb") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/awb.h b/src/ipa/simple/algorithms/awb.h new file mode 100644 index 00000000..235249b6 --- /dev/null +++ b/src/ipa/simple/algorithms/awb.h @@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Auto white balance + */ + +#pragma once + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class Awb : public Algorithm +{ +public: + Awb() = default; + ~Awb() = default; + + int init(IPAContext &context, const YamlObject &tuningData) override; + void process(IPAContext &context, + const uint32_t frame, + IPAFrameContext &frameContext, + const SwIspStats *stats, + ControlList &metadata) override; +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/lut.cpp b/src/ipa/simple/algorithms/lut.cpp new file mode 100644 index 00000000..6781f34e --- /dev/null +++ b/src/ipa/simple/algorithms/lut.cpp @@ -0,0 +1,78 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Color lookup tables construction + */ + +#include "lut.h" + +#include +#include +#include + +#include + +#include "simple/ipa_context.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +int Lut::init(IPAContext &context, [[maybe_unused]] const YamlObject &tuningData) +{ + /* Gamma value is fixed */ + context.configuration.gamma = 0.5; + updateGammaTable(context); + + return 0; +} + +void Lut::updateGammaTable(IPAContext &context) +{ + auto &gammaTable = context.activeState.gamma.gammaTable; + auto blackLevel = context.activeState.blc.level; + const unsigned int blackIndex = blackLevel * gammaTable.size() / 256; + std::fill(gammaTable.begin(), gammaTable.begin() + blackIndex, 0); + const float divisor = gammaTable.size() - blackIndex - 1.0; + for (unsigned int i = blackIndex; i < gammaTable.size(); i++) + gammaTable[i] = UINT8_MAX * std::pow((i - blackIndex) / divisor, + context.configuration.gamma); + context.activeState.gamma.blackLevel = blackLevel; +} + +void Lut::prepare(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + [[maybe_unused]] DebayerParams *params) +{ + /* + * Update the gamma table if needed. This means if black level changes + * and since the black level gets updated only if a lower value is + * observed, it's not permanently prone to minor fluctuations or + * rounding errors. + */ + if (context.activeState.gamma.blackLevel != context.activeState.blc.level) + updateGammaTable(context); + auto &gains = context.activeState.gains; + auto &gammaTable = context.activeState.gamma.gammaTable; + const unsigned int gammaTableSize = gammaTable.size(); + for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { + const unsigned int div = static_cast(DebayerParams::kRGBLookupSize) * + 256 / gammaTableSize; + /* Apply gamma after gain! */ + unsigned int idx; + idx = std::min({ i * gains.red / div, gammaTableSize - 1 }); + params->red[i] = gammaTable[idx]; + idx = std::min({ i * gains.green / div, gammaTableSize - 1 }); + params->green[i] = gammaTable[idx]; + idx = std::min({ i * gains.blue / div, gammaTableSize - 1 }); + params->blue[i] = gammaTable[idx]; + } +} + +REGISTER_IPA_ALGORITHM(Lut, "Lut") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/lut.h b/src/ipa/simple/algorithms/lut.h new file mode 100644 index 00000000..2c034e9f --- /dev/null +++ b/src/ipa/simple/algorithms/lut.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Color lookup tables construction + */ + +#pragma once + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class Lut : public Algorithm +{ +public: + Lut() = default; + ~Lut() = default; + + int init(IPAContext &context, const YamlObject &tuningData) + override; + void prepare(IPAContext &context, + const uint32_t frame, + IPAFrameContext &frameContext, + DebayerParams *params) override; + +private: + void updateGammaTable(IPAContext &context); +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index 1f63c220..f575611e 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -1,5 +1,7 @@ # SPDX-License-Identifier: CC0-1.0 soft_simple_ipa_algorithms = files([ + 'awb.cpp', 'blc.cpp', + 'lut.cpp', ]) diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index e0d03d96..893a0b92 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -4,4 +4,6 @@ version: 1 algorithms: - BlackLevel: + - Awb: + - Lut: ... diff --git a/src/ipa/simple/ipa_context.cpp b/src/ipa/simple/ipa_context.cpp index 268cff32..5fa492cb 100644 --- a/src/ipa/simple/ipa_context.cpp +++ b/src/ipa/simple/ipa_context.cpp @@ -50,6 +50,11 @@ namespace libcamera::ipa::soft { * \brief The current state of IPA algorithms */ +/** + * \var IPASessionConfiguration::gamma + * \brief Gamma value to be used in the raw image processing + */ + /** * \var IPAActiveState::black * \brief Context for the Black Level algorithm @@ -58,4 +63,29 @@ namespace libcamera::ipa::soft { * \brief Current determined black level */ +/** + * \var IPAActiveState::gains + * \brief Context for gains in the Colors algorithm + * + * \var IPAActiveState::gains.red + * \brief Gain of red color + * + * \var IPAActiveState::gains.green + * \brief Gain of green color + * + * \var IPAActiveState::gains.blue + * \brief Gain of blue color + */ + +/** + * \var IPAActiveState::gamma + * \brief Context for gamma in the Colors algorithm + * + * \var IPAActiveState::gamma.gammaTable + * \brief Computed gamma table + * + * \var IPAActiveState::gamma.blackLevel + * \brief Black level used for the gamma table computation + */ + } /* namespace libcamera::ipa::soft */ diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index ac2a59d7..737bbbc3 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -7,6 +7,7 @@ #pragma once +#include #include #include @@ -16,12 +17,25 @@ namespace libcamera { namespace ipa::soft { struct IPASessionConfiguration { + float gamma; }; struct IPAActiveState { struct { uint8_t level; } blc; + + struct { + unsigned int red; + unsigned int green; + unsigned int blue; + } gains; + + static constexpr unsigned int kGammaLookupSize = 1024; + struct { + std::array gammaTable; + uint8_t blackLevel; + } gamma; }; struct IPAFrameContext : public FrameContext { diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 3332e2b1..f8d923c5 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -5,8 +5,6 @@ * Simple Software Image Processing Algorithm module */ -#include -#include #include #include @@ -93,9 +91,6 @@ private: std::unique_ptr camHelper_; ControlInfoMap sensorInfoMap_; - static constexpr unsigned int kGammaLookupSize = 1024; - std::array gammaTable_; - int lastBlackLevel_ = -1; /* Local parameter storage */ struct IPAContext context_; @@ -283,6 +278,7 @@ void IPASoftSimple::fillParamsBuffer(const uint32_t frame) IPAFrameContext &frameContext = context_.frameContexts.get(frame); for (auto const &algo : algorithms()) algo->prepare(context_, frame, frameContext, params_); + setIspParams.emit(); } void IPASoftSimple::processStats(const uint32_t frame, @@ -300,62 +296,6 @@ void IPASoftSimple::processStats(const uint32_t frame, for (auto const &algo : algorithms()) algo->process(context_, frame, frameContext, stats_, metadata); - SwIspStats::Histogram histogram = stats_->yHistogram; - const uint8_t blackLevel = context_.activeState.blc.level; - - /* - * Black level must be subtracted to get the correct AWB ratios, they - * would be off if they were computed from the whole brightness range - * rather than from the sensor range. - */ - const uint64_t nPixels = std::accumulate( - histogram.begin(), histogram.end(), 0); - const uint64_t offset = blackLevel * nPixels; - const uint64_t sumR = stats_->sumR_ - offset / 4; - const uint64_t sumG = stats_->sumG_ - offset / 2; - const uint64_t sumB = stats_->sumB_ - offset / 4; - - /* - * Calculate red and blue gains for AWB. - * Clamp max gain at 4.0, this also avoids 0 division. - * Gain: 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. - */ - const unsigned int gainR = sumR <= sumG / 4 ? 1024 : 256 * sumG / sumR; - const unsigned int gainB = sumB <= sumG / 4 ? 1024 : 256 * sumG / sumB; - /* Green gain and gamma values are fixed */ - constexpr unsigned int gainG = 256; - - /* Update the gamma table if needed */ - if (blackLevel != lastBlackLevel_) { - constexpr float gamma = 0.5; - const unsigned int blackIndex = blackLevel * kGammaLookupSize / 256; - std::fill(gammaTable_.begin(), gammaTable_.begin() + blackIndex, 0); - const float divisor = kGammaLookupSize - blackIndex - 1.0; - for (unsigned int i = blackIndex; i < kGammaLookupSize; i++) - gammaTable_[i] = UINT8_MAX * - std::pow((i - blackIndex) / divisor, gamma); - - lastBlackLevel_ = blackLevel; - } - - for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { - constexpr unsigned int div = - DebayerParams::kRGBLookupSize * 256 / kGammaLookupSize; - unsigned int idx; - - /* Apply gamma after gain! */ - idx = std::min({ i * gainR / div, (kGammaLookupSize - 1) }); - params_->red[i] = gammaTable_[idx]; - - idx = std::min({ i * gainG / div, (kGammaLookupSize - 1) }); - params_->green[i] = gammaTable_[idx]; - - idx = std::min({ i * gainB / div, (kGammaLookupSize - 1) }); - params_->blue[i] = gammaTable_[idx]; - } - - setIspParams.emit(); - /* \todo Switch to the libipa/algorithm.h API someday. */ /* @@ -372,6 +312,7 @@ void IPASoftSimple::processStats(const uint32_t frame, * Calculate Mean Sample Value (MSV) according to formula from: * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf */ + const uint8_t blackLevel = context_.activeState.blc.level; const unsigned int blackLevelHistIdx = blackLevel / (256 / SwIspStats::kYHistogramSize); const unsigned int histogramSize = @@ -421,7 +362,6 @@ void IPASoftSimple::processStats(const uint32_t frame, LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV << " exp " << exposure_ << " again " << again_ - << " gain R/B " << gainR << "/" << gainB << " black level " << static_cast(blackLevel); } From patchwork Fri Sep 6 12:09:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21188 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8B3B1BF415 for ; Fri, 6 Sep 2024 12:10:50 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D2CBE634F7; Fri, 6 Sep 2024 14:10:48 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="NXJaox4F"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 20994634FD for ; Fri, 6 Sep 2024 14:10:31 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624629; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LhPYahubLgT9TeEaeFrjxpAUFrfwXDyiNQQjCtfn0Tc=; b=NXJaox4FjjH0XygBRJ5gzVYeHqvZ83ks/fk0m1avvUSFwIgujasBZCZqwUZbcaTrrFnNGT 5QWiTevaGqRrjdUJDdqNhbDWNWCX+HL9koUzc+D3jdRkQexiIMmZrTI44qQWyUJQqwMg+/ I5ddUXLVEnapfO6TruYifcCndifECTQ= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-288-51D37E-6NS-j2iyGdqHnLg-1; Fri, 06 Sep 2024 08:10:26 -0400 X-MC-Unique: 51D37E-6NS-j2iyGdqHnLg-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id AD5CD1955F40; Fri, 6 Sep 2024 12:10:25 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 946AB1956086; Fri, 6 Sep 2024 12:10:23 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 15/18] libcamera: software_isp: Use floating point for color parameters Date: Fri, 6 Sep 2024 14:09:24 +0200 Message-ID: <20240906120927.4071508-16-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" It's more natural to represent color gains and black level as floating point numbers rather than using a particular pixel-related representation. double is used rather than float because it's a more common floating point type in libcamera algorithms. Otherwise there is no obvious reason to select one over the other here. The constructed color tables still use integer representation for efficiency. Signed-off-by: Milan Zamazal Reviewed-by: Kieran Bingham --- src/ipa/simple/algorithms/awb.cpp | 11 +++++------ src/ipa/simple/algorithms/blc.cpp | 8 ++++---- src/ipa/simple/algorithms/lut.cpp | 15 +++++++++------ src/ipa/simple/ipa_context.h | 11 +++++------ 4 files changed, 23 insertions(+), 22 deletions(-) diff --git a/src/ipa/simple/algorithms/awb.cpp b/src/ipa/simple/algorithms/awb.cpp index f89f1d3d..61d9fc73 100644 --- a/src/ipa/simple/algorithms/awb.cpp +++ b/src/ipa/simple/algorithms/awb.cpp @@ -24,7 +24,7 @@ int Awb::init(IPAContext &context, [[maybe_unused]] const YamlObject &tuningData) { auto &gains = context.activeState.gains; - gains.red = gains.green = gains.blue = 256; + gains.red = gains.green = gains.blue = 1.0; return 0; } @@ -36,7 +36,7 @@ void Awb::process(IPAContext &context, [[maybe_unused]] ControlList &metadata) { const SwIspStats::Histogram &histogram = stats->yHistogram; - const uint8_t blackLevel = context.activeState.blc.level; + const double blackLevel = context.activeState.blc.level; /* * Black level must be subtracted to get the correct AWB ratios, they @@ -53,12 +53,11 @@ void Awb::process(IPAContext &context, /* * Calculate red and blue gains for AWB. * Clamp max gain at 4.0, this also avoids 0 division. - * Gain: 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. */ auto &gains = context.activeState.gains; - gains.red = sumR <= sumG / 4 ? 1024 : 256 * sumG / sumR; - gains.blue = sumB <= sumG / 4 ? 1024 : 256 * sumG / sumB; - /* Green gain is fixed to 256 */ + gains.red = sumR <= sumG / 4 ? 4.0 : static_cast(sumG) / sumR; + gains.blue = sumB <= sumG / 4 ? 4.0 : static_cast(sumG) / sumB; + /* Green gain is fixed to 1.0 */ LOG(IPASoftAwb, Debug) << "gain R/B " << gains.red << "/" << gains.blue; } diff --git a/src/ipa/simple/algorithms/blc.cpp b/src/ipa/simple/algorithms/blc.cpp index 08f4345e..1ceae85d 100644 --- a/src/ipa/simple/algorithms/blc.cpp +++ b/src/ipa/simple/algorithms/blc.cpp @@ -24,7 +24,7 @@ BlackLevel::BlackLevel() int BlackLevel::init(IPAContext &context, [[maybe_unused]] const YamlObject &tuningData) { - context.activeState.blc.level = 255; + context.activeState.blc.level = 1.0; return 0; } @@ -44,16 +44,16 @@ void BlackLevel::process(IPAContext &context, const unsigned int total = std::accumulate(begin(histogram), end(histogram), 0); const unsigned int pixelThreshold = ignoredPercentage * total; - const unsigned int histogramRatio = 256 / SwIspStats::kYHistogramSize; const unsigned int currentBlackIdx = - context.activeState.blc.level / histogramRatio; + context.activeState.blc.level * SwIspStats::kYHistogramSize; for (unsigned int i = 0, seen = 0; i < currentBlackIdx && i < SwIspStats::kYHistogramSize; i++) { seen += histogram[i]; if (seen >= pixelThreshold) { - context.activeState.blc.level = i * histogramRatio; + context.activeState.blc.level = + static_cast(i) / SwIspStats::kYHistogramSize; LOG(IPASoftBL, Debug) << "Auto-set black level: " << i << "/" << SwIspStats::kYHistogramSize diff --git a/src/ipa/simple/algorithms/lut.cpp b/src/ipa/simple/algorithms/lut.cpp index 6781f34e..d3b33710 100644 --- a/src/ipa/simple/algorithms/lut.cpp +++ b/src/ipa/simple/algorithms/lut.cpp @@ -32,7 +32,7 @@ void Lut::updateGammaTable(IPAContext &context) { auto &gammaTable = context.activeState.gamma.gammaTable; auto blackLevel = context.activeState.blc.level; - const unsigned int blackIndex = blackLevel * gammaTable.size() / 256; + const unsigned int blackIndex = blackLevel * gammaTable.size(); std::fill(gammaTable.begin(), gammaTable.begin() + blackIndex, 0); const float divisor = gammaTable.size() - blackIndex - 1.0; for (unsigned int i = blackIndex; i < gammaTable.size(); i++) @@ -58,15 +58,18 @@ void Lut::prepare(IPAContext &context, auto &gammaTable = context.activeState.gamma.gammaTable; const unsigned int gammaTableSize = gammaTable.size(); for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { - const unsigned int div = static_cast(DebayerParams::kRGBLookupSize) * - 256 / gammaTableSize; + const double div = static_cast(DebayerParams::kRGBLookupSize) / + gammaTableSize; /* Apply gamma after gain! */ unsigned int idx; - idx = std::min({ i * gains.red / div, gammaTableSize - 1 }); + idx = std::min({ static_cast(i * gains.red / div), + gammaTableSize - 1 }); params->red[i] = gammaTable[idx]; - idx = std::min({ i * gains.green / div, gammaTableSize - 1 }); + idx = std::min({ static_cast(i * gains.green / div), + gammaTableSize - 1 }); params->green[i] = gammaTable[idx]; - idx = std::min({ i * gains.blue / div, gammaTableSize - 1 }); + idx = std::min({ static_cast(i * gains.blue / div), + gammaTableSize - 1 }); params->blue[i] = gammaTable[idx]; } } diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index 737bbbc3..d09de9a9 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -8,7 +8,6 @@ #pragma once #include -#include #include @@ -22,19 +21,19 @@ struct IPASessionConfiguration { struct IPAActiveState { struct { - uint8_t level; + double level; } blc; struct { - unsigned int red; - unsigned int green; - unsigned int blue; + double red; + double green; + double blue; } gains; static constexpr unsigned int kGammaLookupSize = 1024; struct { std::array gammaTable; - uint8_t blackLevel; + double blackLevel; } gamma; }; From patchwork Fri Sep 6 12:09:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21189 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 654C9BF415 for ; Fri, 6 Sep 2024 12:10:52 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0159D63502; Fri, 6 Sep 2024 14:10:50 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="hWBTKFB8"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 612D7634F9 for ; Fri, 6 Sep 2024 14:10:35 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624634; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rUZBSrc9x0MDIj7tneXWnjebz4LEt8Ci8iM5xCWOJIg=; b=hWBTKFB8BRR7FwgM8cTLKWHT1oiScP+LpqCJ2EF6Ic5hTk5rZX7PXarufJbpxFE1do7XZ3 Xl4AzMEWg2lHJKJdGz5sDrBRBXzbZVf2fbqnoKth4j4wY4huGn9iHXkpJ8YqIx4PORnJ8g CfOW6SbZlnWUh8YIfVtFORIvhQVtc6M= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-408-5T049qx3PESyr1GJVt2qNQ-1; Fri, 06 Sep 2024 08:10:29 -0400 X-MC-Unique: 5T049qx3PESyr1GJVt2qNQ-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 49A171955F65; Fri, 6 Sep 2024 12:10:28 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 292DC1956086; Fri, 6 Sep 2024 12:10:25 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 16/18] libcamera: software_isp: Use DelayedControls Date: Fri, 6 Sep 2024 14:09:25 +0200 Message-ID: <20240906120927.4071508-17-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Use the standard libcamera mechanism to report the "current" controls rather than delaying updates by counting from the last update. MY SPECULATION -- valid or not?: A problem is that with software ISP we cannot be sure about the sensor delay. The original implementation simply skips exposure updates for 2 frames, which should be enough in most cases. After this change, we assume the delay being exactly 2 frames, which may or may not be correct and may work with outdated values if the delay is shorter. This patch also prepares moving exposure+gain to an algorithm module where the original delay mechanism would be a (possibly unnecessary) complication. Resolves software ISP TODO #11 + #12. Signed-off-by: Milan Zamazal Reviewed-by: Kieran Bingham --- src/ipa/simple/soft_simple.cpp | 16 +------------ src/libcamera/pipeline/simple/simple.cpp | 18 ++++++++++++--- src/libcamera/software_isp/TODO | 29 ------------------------ 3 files changed, 16 insertions(+), 47 deletions(-) diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index f8d923c5..60310a8e 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -59,8 +59,7 @@ class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module public: IPASoftSimple() : params_(nullptr), stats_(nullptr), - context_({ {}, {}, { kMaxFrameContexts } }), - ignoreUpdates_(0) + context_({ {}, {}, { kMaxFrameContexts } }) { } @@ -98,7 +97,6 @@ private: int32_t exposure_; double againMin_, againMax_, againMinStep_; double again_; - unsigned int ignoreUpdates_; }; IPASoftSimple::~IPASoftSimple() @@ -298,16 +296,6 @@ void IPASoftSimple::processStats(const uint32_t frame, /* \todo Switch to the libipa/algorithm.h API someday. */ - /* - * AE / AGC, use 2 frames delay to make sure that the exposure and - * the gain set have applied to the camera sensor. - * \todo This could be handled better with DelayedControls. - */ - if (ignoreUpdates_ > 0) { - --ignoreUpdates_; - return; - } - /* * Calculate Mean Sample Value (MSV) according to formula from: * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf @@ -356,8 +344,6 @@ void IPASoftSimple::processStats(const uint32_t frame, ctrls.set(V4L2_CID_ANALOGUE_GAIN, static_cast(camHelper_ ? camHelper_->gainCode(again_) : again_)); - ignoreUpdates_ = 2; - setSensorControls.emit(ctrls); LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 7e412e17..40a7b1da 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -32,6 +32,7 @@ #include "libcamera/internal/camera.h" #include "libcamera/internal/camera_sensor.h" #include "libcamera/internal/converter.h" +#include "libcamera/internal/delayed_controls.h" #include "libcamera/internal/device_enumerator.h" #include "libcamera/internal/media_device.h" #include "libcamera/internal/pipeline_handler.h" @@ -278,6 +279,8 @@ public: std::vector configs_; std::map> formats_; + std::unique_ptr delayedCtrls_; + std::vector> conversionBuffers_; std::queue> conversionQueue_; bool useConversion_; @@ -900,14 +903,13 @@ void SimpleCameraData::conversionOutputDone(FrameBuffer *buffer) void SimpleCameraData::ispStatsReady(uint32_t frame, uint32_t bufferId) { - /* \todo Use the DelayedControls class */ swIsp_->processStats(frame, bufferId, - sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, - V4L2_CID_EXPOSURE })); + delayedCtrls_->get(frame)); } void SimpleCameraData::setSensorControls(const ControlList &sensorControls) { + delayedCtrls_->push(sensorControls); ControlList ctrls(sensorControls); sensor_->setControls(&ctrls); } @@ -1288,6 +1290,16 @@ int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c) if (outputCfgs.empty()) return 0; + std::unordered_map params = { + { V4L2_CID_ANALOGUE_GAIN, { 2, false } }, + { V4L2_CID_EXPOSURE, { 2, false } }, + }; + data->delayedCtrls_ = + std::make_unique(data->sensor_->device(), + params); + data->video_->frameStart.connect(data->delayedCtrls_.get(), + &DelayedControls::applyControls); + StreamConfiguration inputCfg; inputCfg.pixelFormat = pipeConfig->captureFormat; inputCfg.size = pipeConfig->captureSize; diff --git a/src/libcamera/software_isp/TODO b/src/libcamera/software_isp/TODO index 9978afc0..8eeede46 100644 --- a/src/libcamera/software_isp/TODO +++ b/src/libcamera/software_isp/TODO @@ -209,35 +209,6 @@ At some point, yes. --- -11. Improve handling the sensor controls which take effect with a delay - -> void IPASoftSimple::processStats(const ControlList &sensorControls) -> { -> ... -> /* -> * AE / AGC, use 2 frames delay to make sure that the exposure and -> * the gain set have applied to the camera sensor. -> */ -> if (ignore_updates_ > 0) { -> --ignore_updates_; -> return; -> } - -This could be handled better with DelayedControls. - ---- - -12. Use DelayedControls class in ispStatsReady() - -> void SimpleCameraData::ispStatsReady() -> { -> swIsp_->processStats(sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, -> V4L2_CID_EXPOSURE })); - -You should use the DelayedControls class. - ---- - 13. Improve black level and colour gains application I think the black level should eventually be moved before debayering, and From patchwork Fri Sep 6 12:09:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21190 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E373EBF415 for ; Fri, 6 Sep 2024 12:10:55 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 200B463508; Fri, 6 Sep 2024 14:10:54 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="U+tBmYQe"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5E0EA634F2 for ; Fri, 6 Sep 2024 14:10:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624633; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UtWW8gYIye/v7M4umWI8owvCfaPQ64KoDOp52JRvbR8=; b=U+tBmYQeRC0t5cWrlxdqOHrxXtTGgTHf0Ku1tPWVY0p7RqsfdU8cD52D1QlMyIdi4DxKcZ zijxeFu4OZrnjsKNwa3FafqmZsTwyHxL/DAbTB6V2pGkJuOC/9/tCU9uH/bWEQHNKQ7UnU 9+GfL6pZirQuudoVUiFUNqfES2mJPJY= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-368-aEzC773UPjyl__zDUtdJyA-1; Fri, 06 Sep 2024 08:10:32 -0400 X-MC-Unique: aEzC773UPjyl__zDUtdJyA-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1BC4E1955E99; Fri, 6 Sep 2024 12:10:31 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id B26851956086; Fri, 6 Sep 2024 12:10:28 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 17/18] libcamera: software_isp: Move exposure+gain to an algorithm module Date: Fri, 6 Sep 2024 14:09:26 +0200 Message-ID: <20240906120927.4071508-18-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This is the last step to fully convert software ISP to Algorithm-based processing. The newly introduced frameContext.sensor properties are set, and the updated code moved, before calling Algorithm::process() to have the values up-to-date in stats processing. Resolves software ISP TODO #10. Signed-off-by: Milan Zamazal Reviewed-by: Kieran Bingham --- src/ipa/simple/algorithms/agc.cpp | 139 +++++++++++++++++++++++ src/ipa/simple/algorithms/agc.h | 33 ++++++ src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/data/uncalibrated.yaml | 1 + src/ipa/simple/ipa_context.cpp | 11 ++ src/ipa/simple/ipa_context.h | 18 +++ src/ipa/simple/soft_simple.cpp | 157 +++++--------------------- src/libcamera/software_isp/TODO | 10 -- 8 files changed, 233 insertions(+), 137 deletions(-) create mode 100644 src/ipa/simple/algorithms/agc.cpp create mode 100644 src/ipa/simple/algorithms/agc.h diff --git a/src/ipa/simple/algorithms/agc.cpp b/src/ipa/simple/algorithms/agc.cpp new file mode 100644 index 00000000..becfb99f --- /dev/null +++ b/src/ipa/simple/algorithms/agc.cpp @@ -0,0 +1,139 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Exposure and gain + */ + +#include "agc.h" + +#include + +#include + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoftExposure) + +namespace ipa::soft::algorithms { + +/* + * The number of bins to use for the optimal exposure calculations. + */ +static constexpr unsigned int kExposureBinsCount = 5; + +/* + * The exposure is optimal when the mean sample value of the histogram is + * in the middle of the range. + */ +static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; + +/* + * The below value implements the hysteresis for the exposure adjustment. + * It is small enough to have the exposure close to the optimal, and is big + * enough to prevent the exposure from wobbling around the optimal value. + */ +static constexpr float kExposureSatisfactory = 0.2; + +Agc::Agc() +{ +} + +void Agc::updateExposure(IPAContext &context, double exposureMSV) +{ + /* + * kExpDenominator of 10 gives ~10% increment/decrement; + * kExpDenominator of 5 - about ~20% + */ + static constexpr uint8_t kExpDenominator = 10; + static constexpr uint8_t kExpNumeratorUp = kExpDenominator + 1; + static constexpr uint8_t kExpNumeratorDown = kExpDenominator - 1; + + double next; + int32_t &exposure = context.activeState.agc.exposure; + double &again = context.activeState.agc.again; + + if (exposureMSV < kExposureOptimal - kExposureSatisfactory) { + next = exposure * kExpNumeratorUp / kExpDenominator; + if (next - exposure < 1) + exposure += 1; + else + exposure = next; + if (exposure >= context.configuration.agc.exposureMax) { + next = again * kExpNumeratorUp / kExpDenominator; + if (next - again < context.configuration.agc.againMinStep) + again += context.configuration.agc.againMinStep; + else + again = next; + } + } + + if (exposureMSV > kExposureOptimal + kExposureSatisfactory) { + if (exposure == context.configuration.agc.exposureMax && + again > context.configuration.agc.againMin) { + next = again * kExpNumeratorDown / kExpDenominator; + if (again - next < context.configuration.agc.againMinStep) + again -= context.configuration.agc.againMinStep; + else + again = next; + } else { + next = exposure * kExpNumeratorDown / kExpDenominator; + if (exposure - next < 1) + exposure -= 1; + else + exposure = next; + } + } + + exposure = std::clamp(exposure, context.configuration.agc.exposureMin, + context.configuration.agc.exposureMax); + again = std::clamp(again, context.configuration.agc.againMin, + context.configuration.agc.againMax); + + LOG(IPASoftExposure, Debug) + << "exposureMSV " << exposureMSV + << " exp " << exposure << " again " << again; +} + +void Agc::process(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + const SwIspStats *stats, + [[maybe_unused]] ControlList &metadata) +{ + /* + * Calculate Mean Sample Value (MSV) according to formula from: + * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf + */ + const auto &histogram = stats->yHistogram; + const unsigned int blackLevelHistIdx = + context.activeState.blc.level * SwIspStats::kYHistogramSize; + const unsigned int histogramSize = + SwIspStats::kYHistogramSize - blackLevelHistIdx; + const unsigned int yHistValsPerBin = histogramSize / kExposureBinsCount; + const unsigned int yHistValsPerBinMod = + histogramSize / (histogramSize % kExposureBinsCount + 1); + int exposureBins[kExposureBinsCount] = {}; + unsigned int denom = 0; + unsigned int num = 0; + + for (unsigned int i = 0; i < histogramSize; i++) { + unsigned int idx = (i - (i / yHistValsPerBinMod)) / yHistValsPerBin; + exposureBins[idx] += histogram[blackLevelHistIdx + i]; + } + + for (unsigned int i = 0; i < kExposureBinsCount; i++) { + LOG(IPASoftExposure, Debug) << i << ": " << exposureBins[i]; + denom += exposureBins[i]; + num += exposureBins[i] * (i + 1); + } + + float exposureMSV = (denom == 0 ? 0 : static_cast(num) / denom); + updateExposure(context, exposureMSV); +} + +REGISTER_IPA_ALGORITHM(Agc, "Agc") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/agc.h b/src/ipa/simple/algorithms/agc.h new file mode 100644 index 00000000..ad5fca9f --- /dev/null +++ b/src/ipa/simple/algorithms/agc.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Exposure and gain + */ + +#pragma once + +#include "algorithm.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class Agc : public Algorithm +{ +public: + Agc(); + ~Agc() = default; + + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const SwIspStats *stats, + ControlList &metadata) override; + +private: + void updateExposure(IPAContext &context, double exposureMSV); +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index f575611e..37a2eb53 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -2,6 +2,7 @@ soft_simple_ipa_algorithms = files([ 'awb.cpp', + 'agc.cpp', 'blc.cpp', 'lut.cpp', ]) diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index 893a0b92..3f147112 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -6,4 +6,5 @@ algorithms: - BlackLevel: - Awb: - Lut: + - Agc: ... diff --git a/src/ipa/simple/ipa_context.cpp b/src/ipa/simple/ipa_context.cpp index 5fa492cb..3f94bbeb 100644 --- a/src/ipa/simple/ipa_context.cpp +++ b/src/ipa/simple/ipa_context.cpp @@ -77,6 +77,17 @@ namespace libcamera::ipa::soft { * \brief Gain of blue color */ +/** + * \var IPAActiveState::agc + * \brief Context for the AGC algorithm + * + * \var IPAActiveState::agc.exposure + * \brief Current exposure value + * + * \var IPAActiveState::agc.again + * \brief Current analog gain value + */ + /** * \var IPAActiveState::gamma * \brief Context for gamma in the Colors algorithm diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index d09de9a9..ad869f64 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -9,6 +9,8 @@ #include +#include + #include namespace libcamera { @@ -17,6 +19,13 @@ namespace ipa::soft { struct IPASessionConfiguration { float gamma; + struct { + int32_t exposureMin, exposureMax; + double againMin, againMax, againMinStep; + } agc; + struct { + double level; + } black; }; struct IPAActiveState { @@ -30,6 +39,11 @@ struct IPAActiveState { double blue; } gains; + struct { + int32_t exposure; + double again; + } agc; + static constexpr unsigned int kGammaLookupSize = 1024; struct { std::array gammaTable; @@ -38,6 +52,10 @@ struct IPAActiveState { }; struct IPAFrameContext : public FrameContext { + struct { + uint32_t exposure; + double gain; + } sensor; }; struct IPAContext { diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 60310a8e..b28c7039 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -34,23 +34,6 @@ LOG_DEFINE_CATEGORY(IPASoft) namespace ipa::soft { -/* - * The number of bins to use for the optimal exposure calculations. - */ -static constexpr unsigned int kExposureBinsCount = 5; - -/* - * The exposure is optimal when the mean sample value of the histogram is - * in the middle of the range. - */ -static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; - -/* - * The below value implements the hysteresis for the exposure adjustment. - * It is small enough to have the exposure close to the optimal, and is big - * enough to prevent the exposure from wobbling around the optimal value. - */ -static constexpr float kExposureSatisfactory = 0.2; /* Maximum number of frame contexts to be held */ static constexpr uint32_t kMaxFrameContexts = 16; @@ -58,8 +41,7 @@ class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() - : params_(nullptr), stats_(nullptr), - context_({ {}, {}, { kMaxFrameContexts } }) + : context_({ {}, {}, { kMaxFrameContexts } }) { } @@ -92,11 +74,6 @@ private: /* Local parameter storage */ struct IPAContext context_; - - int32_t exposureMin_, exposureMax_; - int32_t exposure_; - double againMin_, againMax_, againMinStep_; - double again_; }; IPASoftSimple::~IPASoftSimple() @@ -207,20 +184,23 @@ int IPASoftSimple::configure(const IPAConfigInfo &configInfo) const ControlInfo &exposureInfo = sensorInfoMap_.find(V4L2_CID_EXPOSURE)->second; const ControlInfo &gainInfo = sensorInfoMap_.find(V4L2_CID_ANALOGUE_GAIN)->second; - exposureMin_ = exposureInfo.min().get(); - exposureMax_ = exposureInfo.max().get(); - if (!exposureMin_) { + context_.configuration.agc.exposureMin = exposureInfo.min().get(); + context_.configuration.agc.exposureMax = exposureInfo.max().get(); + if (!context_.configuration.agc.exposureMin) { LOG(IPASoft, Warning) << "Minimum exposure is zero, that can't be linear"; - exposureMin_ = 1; + context_.configuration.agc.exposureMin = 1; } int32_t againMin = gainInfo.min().get(); int32_t againMax = gainInfo.max().get(); if (camHelper_) { - againMin_ = camHelper_->gain(againMin); - againMax_ = camHelper_->gain(againMax); - againMinStep_ = (againMax_ - againMin_) / 100.0; + context_.configuration.agc.againMin = camHelper_->gain(againMin); + context_.configuration.agc.againMax = camHelper_->gain(againMax); + context_.configuration.agc.againMinStep = + (context_.configuration.agc.againMax - + context_.configuration.agc.againMin) / + 100.0; } else { /* * The camera sensor gain (g) is usually not equal to the value written @@ -232,13 +212,14 @@ int IPASoftSimple::configure(const IPAConfigInfo &configInfo) * the AGC algorithm (abrupt near one edge, and very small near the * other) we limit the range of the gain values used. */ - againMax_ = againMax; + context_.configuration.agc.againMax = againMax; if (!againMin) { LOG(IPASoft, Warning) << "Minimum gain is zero, that can't be linear"; - againMin_ = std::min(100, againMin / 2 + againMax / 2); + context_.configuration.agc.againMin = + std::min(100, againMin / 2 + againMax / 2); } - againMinStep_ = 1.0; + context_.configuration.agc.againMinStep = 1.0; } for (auto const &algo : algorithms()) { @@ -247,9 +228,12 @@ int IPASoftSimple::configure(const IPAConfigInfo &configInfo) return ret; } - LOG(IPASoft, Info) << "Exposure " << exposureMin_ << "-" << exposureMax_ - << ", gain " << againMin_ << "-" << againMax_ - << " (" << againMinStep_ << ")"; + LOG(IPASoft, Info) + << "Exposure " << context_.configuration.agc.exposureMin << "-" + << context_.configuration.agc.exposureMax + << ", gain " << context_.configuration.agc.againMin << "-" + << context_.configuration.agc.againMax + << " (" << context_.configuration.agc.againMinStep << ")"; return 0; } @@ -284,6 +268,12 @@ void IPASoftSimple::processStats(const uint32_t frame, const ControlList &sensorControls) { IPAFrameContext &frameContext = context_.frameContexts.get(frame); + + frameContext.sensor.exposure = + sensorControls.get(V4L2_CID_EXPOSURE).get(); + int32_t again = sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get(); + frameContext.sensor.gain = camHelper_ ? camHelper_->gain(again) : again; + /* * Software ISP currently does not produce any metadata. Use an empty * ControlList for now. @@ -294,37 +284,6 @@ void IPASoftSimple::processStats(const uint32_t frame, for (auto const &algo : algorithms()) algo->process(context_, frame, frameContext, stats_, metadata); - /* \todo Switch to the libipa/algorithm.h API someday. */ - - /* - * Calculate Mean Sample Value (MSV) according to formula from: - * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf - */ - const uint8_t blackLevel = context_.activeState.blc.level; - const unsigned int blackLevelHistIdx = - blackLevel / (256 / SwIspStats::kYHistogramSize); - const unsigned int histogramSize = - SwIspStats::kYHistogramSize - blackLevelHistIdx; - const unsigned int yHistValsPerBin = histogramSize / kExposureBinsCount; - const unsigned int yHistValsPerBinMod = - histogramSize / (histogramSize % kExposureBinsCount + 1); - int exposureBins[kExposureBinsCount] = {}; - unsigned int denom = 0; - unsigned int num = 0; - - for (unsigned int i = 0; i < histogramSize; i++) { - unsigned int idx = (i - (i / yHistValsPerBinMod)) / yHistValsPerBin; - exposureBins[idx] += stats_->yHistogram[blackLevelHistIdx + i]; - } - - for (unsigned int i = 0; i < kExposureBinsCount; i++) { - LOG(IPASoft, Debug) << i << ": " << exposureBins[i]; - denom += exposureBins[i]; - num += exposureBins[i] * (i + 1); - } - - float exposureMSV = static_cast(num) / denom; - /* Sanity check */ if (!sensorControls.contains(V4L2_CID_EXPOSURE) || !sensorControls.contains(V4L2_CID_ANALOGUE_GAIN)) { @@ -332,70 +291,14 @@ void IPASoftSimple::processStats(const uint32_t frame, return; } - exposure_ = sensorControls.get(V4L2_CID_EXPOSURE).get(); - int32_t again = sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get(); - again_ = camHelper_ ? camHelper_->gain(again) : again; - - updateExposure(exposureMSV); - ControlList ctrls(sensorInfoMap_); - ctrls.set(V4L2_CID_EXPOSURE, exposure_); + auto &againNew = context_.activeState.agc.again; + ctrls.set(V4L2_CID_EXPOSURE, context_.activeState.agc.exposure); ctrls.set(V4L2_CID_ANALOGUE_GAIN, - static_cast(camHelper_ ? camHelper_->gainCode(again_) : again_)); + static_cast(camHelper_ ? camHelper_->gainCode(againNew) : againNew)); setSensorControls.emit(ctrls); - - LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV - << " exp " << exposure_ << " again " << again_ - << " black level " << static_cast(blackLevel); -} - -void IPASoftSimple::updateExposure(double exposureMSV) -{ - /* - * kExpDenominator of 10 gives ~10% increment/decrement; - * kExpDenominator of 5 - about ~20% - */ - static constexpr uint8_t kExpDenominator = 10; - static constexpr uint8_t kExpNumeratorUp = kExpDenominator + 1; - static constexpr uint8_t kExpNumeratorDown = kExpDenominator - 1; - - double next; - - if (exposureMSV < kExposureOptimal - kExposureSatisfactory) { - next = exposure_ * kExpNumeratorUp / kExpDenominator; - if (next - exposure_ < 1) - exposure_ += 1; - else - exposure_ = next; - if (exposure_ >= exposureMax_) { - next = again_ * kExpNumeratorUp / kExpDenominator; - if (next - again_ < againMinStep_) - again_ += againMinStep_; - else - again_ = next; - } - } - - if (exposureMSV > kExposureOptimal + kExposureSatisfactory) { - if (exposure_ == exposureMax_ && again_ > againMin_) { - next = again_ * kExpNumeratorDown / kExpDenominator; - if (again_ - next < againMinStep_) - again_ -= againMinStep_; - else - again_ = next; - } else { - next = exposure_ * kExpNumeratorDown / kExpDenominator; - if (exposure_ - next < 1) - exposure_ -= 1; - else - exposure_ = next; - } - } - - exposure_ = std::clamp(exposure_, exposureMin_, exposureMax_); - again_ = std::clamp(again_, againMin_, againMax_); } std::string IPASoftSimple::logPrefix() const diff --git a/src/libcamera/software_isp/TODO b/src/libcamera/software_isp/TODO index 8eeede46..a50db668 100644 --- a/src/libcamera/software_isp/TODO +++ b/src/libcamera/software_isp/TODO @@ -199,16 +199,6 @@ Yes, because, well... all the other IPAs were doing that... --- -10. Switch to libipa/algorithm.h API in processStats - ->> void IPASoftSimple::processStats(const ControlList &sensorControls) ->> -> Do you envision switching to the libipa/algorithm.h API at some point ? - -At some point, yes. - ---- - 13. Improve black level and colour gains application I think the black level should eventually be moved before debayering, and From patchwork Fri Sep 6 12:09:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 21191 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 92B33C324C for ; Fri, 6 Sep 2024 12:10:57 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 66D1363509; Fri, 6 Sep 2024 14:10:56 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="BIIa1Rx6"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D859F634F7 for ; Fri, 6 Sep 2024 14:10:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725624636; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S4zg3vXRb1p9ZyQSNmXLGi6uQFR4qpnCnu1rzU25Tow=; b=BIIa1Rx6zTnYx8eH+26EOjqd3h3s8WXvrGTrdV/zIZVmX8Qyzy3aFnOSKTtv8Z+WSLyoB5 2w5z+cLVvw2xpUskGdcf/QBGx+Bdrf+QXJM6Y8PSjs6mbQ+egielC+U0BqEAsT623RNiMs 4nYWxjY2hflOa381q7pSDMsjhoBTuec= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-52-C62arktWNY-nVRFYXA0VQw-1; Fri, 06 Sep 2024 08:10:34 -0400 X-MC-Unique: C62arktWNY-nVRFYXA0VQw-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 9C6C7195395C; Fri, 6 Sep 2024 12:10:33 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.65]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id AA534195608A; Fri, 6 Sep 2024 12:10:31 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v6 18/18] libcamera: software_isp: Update black level only on exposure changes Date: Fri, 6 Sep 2024 14:09:27 +0200 Message-ID: <20240906120927.4071508-19-mzamazal@redhat.com> In-Reply-To: <20240906120927.4071508-1-mzamazal@redhat.com> References: <20240906120927.4071508-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The black level is likely to get updated, if ever, only after exposure or gain changes. Don't compute its possible updates if exposure and gain are unchanged. It's probably not worth trying to implement something more sophisticated. Better to spend the effort on supporting tuning files. Signed-off-by: Milan Zamazal Reviewed-by: Kieran Bingham --- src/ipa/simple/algorithms/blc.cpp | 9 ++++++++- src/ipa/simple/algorithms/blc.h | 4 ++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/src/ipa/simple/algorithms/blc.cpp b/src/ipa/simple/algorithms/blc.cpp index 1ceae85d..5e0a36d9 100644 --- a/src/ipa/simple/algorithms/blc.cpp +++ b/src/ipa/simple/algorithms/blc.cpp @@ -30,10 +30,15 @@ int BlackLevel::init(IPAContext &context, void BlackLevel::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, - [[maybe_unused]] IPAFrameContext &frameContext, + IPAFrameContext &frameContext, const SwIspStats *stats, [[maybe_unused]] ControlList &metadata) { + if (frameContext.sensor.exposure == exposure_ && + frameContext.sensor.gain == gain_) { + return; + } + const SwIspStats::Histogram &histogram = stats->yHistogram; /* @@ -54,6 +59,8 @@ void BlackLevel::process(IPAContext &context, if (seen >= pixelThreshold) { context.activeState.blc.level = static_cast(i) / SwIspStats::kYHistogramSize; + exposure_ = frameContext.sensor.exposure; + gain_ = frameContext.sensor.gain; LOG(IPASoftBL, Debug) << "Auto-set black level: " << i << "/" << SwIspStats::kYHistogramSize diff --git a/src/ipa/simple/algorithms/blc.h b/src/ipa/simple/algorithms/blc.h index c2140b4b..fc38375e 100644 --- a/src/ipa/simple/algorithms/blc.h +++ b/src/ipa/simple/algorithms/blc.h @@ -25,6 +25,10 @@ public: IPAFrameContext &frameContext, const SwIspStats *stats, ControlList &metadata) override; + +private: + uint32_t exposure_; + double gain_; }; } /* namespace ipa::soft::algorithms */