From patchwork Thu Aug 15 08:16:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20914 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 32FE4C323E for ; Thu, 15 Aug 2024 08:16:56 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DA7BA633BD; Thu, 15 Aug 2024 10:16:55 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="SXK+Tz2b"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3730E633B4 for ; Thu, 15 Aug 2024 10:16:54 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709813; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rKWQHrO/S9qgxIaMcZCbyPXr9JvRVCJjfvb8264XDuI=; b=SXK+Tz2b5VNXmeWSlXOAM/Ff8JxKqg5jyKaVLEsQizYIwBMj0WgY+i5gTuGX1BY1HVm8wP 8QeC1xfz9MPX2p42DWjQlQl8vF9gRI+M/WDRIUWcsAH6L6Ig3sPxIKJs+y0Fn6WYtWp1Cc GpAia5I2tGrnMLe5m7HKfZCugDuGFdk= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-662-j_bWVlwoNkeTUJ3oXpajRg-1; Thu, 15 Aug 2024 04:16:49 -0400 X-MC-Unique: j_bWVlwoNkeTUJ3oXpajRg-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 37A9D1955F65; Thu, 15 Aug 2024 08:16:48 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id EB32219560A3; Thu, 15 Aug 2024 08:16:45 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 01/18] libcamera: software_isp: Remove superfluous includes Date: Thu, 15 Aug 2024 10:16:05 +0200 Message-ID: <20240815081622.174210-2-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Remove unused libcamera internal headers bayer_format.h, framebuffer.h and mapped_frameBuffer.h. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Laurent Pinchart Reviewed-by: Daniel Scally --- src/libcamera/software_isp/software_isp.cpp | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 1140372c..a3ae3e22 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -16,10 +16,7 @@ #include #include -#include "libcamera/internal/bayer_format.h" -#include "libcamera/internal/framebuffer.h" #include "libcamera/internal/ipa_manager.h" -#include "libcamera/internal/mapped_framebuffer.h" #include "libcamera/internal/software_isp/debayer_params.h" #include "debayer_cpu.h" From patchwork Thu Aug 15 08:16:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20916 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 442E9C32A9 for ; Thu, 15 Aug 2024 08:17:02 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 80E92633C1; Thu, 15 Aug 2024 10:17:01 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="bpnQlI7u"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 00AC0633BA for ; Thu, 15 Aug 2024 10:16:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709817; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SxcxrWdqckPYJJM31vXC2zzG/WyUmEsjDhhR7dm2j24=; b=bpnQlI7udhlHa+5u9j4xfLpCCoJo9MEKT1hebovK19O53Pc6s3U44mCfLtkxsIkxQtRVwv Hhs7zVDi15utIlSxv2aw2+wW7fLlClYy8fzr2UL2ejEy3F5QXJBdMVaj08NozrXfvSBe3A O3WPRwi68zsTuzLJFAV1Op0raExy4Lc= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-665-I71CndHXPQSffD655FgmJg-1; Thu, 15 Aug 2024 04:16:51 -0400 X-MC-Unique: I71CndHXPQSffD655FgmJg-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id BA7EF19560B4; Thu, 15 Aug 2024 08:16:50 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id B1F3619560AE; Thu, 15 Aug 2024 08:16:48 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 02/18] libcamera: software_isp: Move BlackLevel to libcamera::ipa::soft Date: Thu, 15 Aug 2024 10:16:06 +0200 Message-ID: <20240815081622.174210-3-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" IPA modules use custom namespaces for all their internal components to avoid namespace clashes. The simple IPA module for the software ISP uses libcamera::ipa::soft for this purpose. It however defines an internal class named BlackLevel in the root of the libcamera namespace, making it prone to clashes. Move it to the ipa::soft namespace along with the rest of the code. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Laurent Pinchart Reviewed-by: Daniel Scally --- src/ipa/simple/black_level.cpp | 5 +++++ src/ipa/simple/black_level.h | 4 ++++ 2 files changed, 9 insertions(+) diff --git a/src/ipa/simple/black_level.cpp b/src/ipa/simple/black_level.cpp index cc490eb5..37e0109c 100644 --- a/src/ipa/simple/black_level.cpp +++ b/src/ipa/simple/black_level.cpp @@ -15,6 +15,8 @@ namespace libcamera { LOG_DEFINE_CATEGORY(IPASoftBL) +namespace ipa::soft { + /** * \class BlackLevel * \brief Object providing black point level for software ISP @@ -85,4 +87,7 @@ void BlackLevel::update(SwIspStats::Histogram &yHistogram) } }; } + +} /* namespace ipa::soft */ + } /* namespace libcamera */ diff --git a/src/ipa/simple/black_level.h b/src/ipa/simple/black_level.h index 5e032f9f..a04230c9 100644 --- a/src/ipa/simple/black_level.h +++ b/src/ipa/simple/black_level.h @@ -14,6 +14,8 @@ namespace libcamera { +namespace ipa::soft { + class BlackLevel { public: @@ -26,4 +28,6 @@ private: bool blackLevelSet_; }; +} /* namespace ipa::soft */ + } /* namespace libcamera */ From patchwork Thu Aug 15 08:16:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20915 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 13C64C323E for ; Thu, 15 Aug 2024 08:17:01 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BFF08633BD; Thu, 15 Aug 2024 10:17:00 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="iMw75nEi"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7112C63393 for ; Thu, 15 Aug 2024 10:16:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709817; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K5sz67NkMkKrwfn/yYMG0bC9SxrqWTZ2/kfYntqLcX8=; b=iMw75nEi0RgDi39/SNkF+0suPvGHv+HxmlNkxUlft7OK38llyg3kVmyNaETryRIViiC3A/ evvb8d4M5PyPSsk4lT70OTvLvbrWa/JMgnyKXMWzV69wVfq8rQSw/jg7wse7iXSemur3CL jgkVRZKZah0Lhx0mwLC2gf8C5hwm3p8= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-696-tTL3JmVsNCOyj1x0b7UkGw-1; Thu, 15 Aug 2024 04:16:54 -0400 X-MC-Unique: tTL3JmVsNCOyj1x0b7UkGw-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 686A61956080; Thu, 15 Aug 2024 08:16:53 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 4D8BA19560A3; Thu, 15 Aug 2024 08:16:51 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 03/18] libcamera: software_isp: Define skeletons for IPA refactoring Date: Thu, 15 Aug 2024 10:16:07 +0200 Message-ID: <20240815081622.174210-4-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Software ISP image processing algorithms are currently defined in a simplified way, different from other libcamera pipelines. This is not good for several reasons: - It makes the software ISP code harder to understand due to its different structuring. - Adding more algorithms may make the code harder to understand generally. - Mass libcamera code changes may not be easily applicable to software ISP. - Algorithm sharing with other pipelines is not easily possible. This patch introduces basic software ISP IPA skeletons structured similarly to the other pipelines. The newly added files are currently not used or compiled and the general skeleton structures don't contain anything particular. It is just a preparation step for a larger refactoring and the code will be actually used and extended as needed in followup patches. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Daniel Scally Reviewed-by: Laurent Pinchart --- include/libcamera/ipa/soft.mojom | 4 ++ src/ipa/simple/algorithms/algorithm.h | 22 +++++++++++ src/ipa/simple/ipa_context.cpp | 53 +++++++++++++++++++++++++++ src/ipa/simple/ipa_context.h | 33 +++++++++++++++++ src/ipa/simple/meson.build | 1 + src/ipa/simple/module.h | 29 +++++++++++++++ 6 files changed, 142 insertions(+) create mode 100644 src/ipa/simple/algorithms/algorithm.h create mode 100644 src/ipa/simple/ipa_context.cpp create mode 100644 src/ipa/simple/ipa_context.h create mode 100644 src/ipa/simple/module.h diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index 3aa2066e..0fd47bb0 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -8,6 +8,10 @@ module ipa.soft; import "include/libcamera/ipa/core.mojom"; +struct IPAConfigInfo { + libcamera.ControlInfoMap sensorControls; +}; + interface IPASoftInterface { init(libcamera.IPASettings settings, libcamera.SharedFD fdStats, diff --git a/src/ipa/simple/algorithms/algorithm.h b/src/ipa/simple/algorithms/algorithm.h new file mode 100644 index 00000000..41f63170 --- /dev/null +++ b/src/ipa/simple/algorithms/algorithm.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Red Hat, Inc. + * + * Software ISP control algorithm interface + */ + +#pragma once + +#include + +#include "module.h" + +namespace libcamera { + +namespace ipa::soft { + +using Algorithm = libcamera::ipa::Algorithm; + +} /* namespace ipa::soft */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/ipa_context.cpp b/src/ipa/simple/ipa_context.cpp new file mode 100644 index 00000000..3c1c7262 --- /dev/null +++ b/src/ipa/simple/ipa_context.cpp @@ -0,0 +1,53 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Google Inc. + * Copyright (C) 2024 Red Hat Inc. + * + * Software ISP IPA Context + */ + +#include "ipa_context.h" + +/** + * \file ipa_context.h + * \brief Context and state information shared between the algorithms + */ + +namespace libcamera::ipa::soft { + +/** + * \struct IPASessionConfiguration + * \brief Session configuration for the IPA module + * + * The session configuration contains all IPA configuration parameters that + * remain constant during the capture session, from IPA module start to stop. + * It is typically set during the configure() operation of the IPA module, but + * may also be updated in the start() operation. + */ + +/** + * \struct IPAActiveState + * \brief The active state of the IPA algorithms + * + * The IPA is fed with the statistics generated from the latest frame processed. + * The statistics are then processed by the IPA algorithms to compute parameters + * required for the next frame capture and processing. The current state of the + * algorithms is reflected through the IPAActiveState to store the values most + * recently computed by the IPA algorithms. + */ + +/** + * \struct IPAContext + * \brief Global IPA context data shared between all algorithms + * + * \var IPAContext::configuration + * \brief The IPA session configuration, immutable during the session + * + * \var IPAContext::frameContexts + * \brief Ring buffer of the IPAFrameContext(s) + * + * \var IPAContext::activeState + * \brief The current state of IPA algorithms + */ + +} /* namespace libcamera::ipa::soft */ diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h new file mode 100644 index 00000000..e7d8b8df --- /dev/null +++ b/src/ipa/simple/ipa_context.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Red Hat, Inc. + * + * Simple pipeline IPA Context + */ + +#pragma once + +#include + +namespace libcamera { + +namespace ipa::soft { + +struct IPASessionConfiguration { +}; + +struct IPAActiveState { +}; + +struct IPAFrameContext : public FrameContext { +}; + +struct IPAContext { + IPASessionConfiguration configuration; + IPAActiveState activeState; + FCQueue frameContexts; +}; + +} /* namespace ipa::soft */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build index b297e1d2..7757f99b 100644 --- a/src/ipa/simple/meson.build +++ b/src/ipa/simple/meson.build @@ -3,6 +3,7 @@ ipa_name = 'ipa_soft_simple' soft_simple_sources = files([ + 'ipa_context.cpp', 'soft_simple.cpp', 'black_level.cpp', ]) diff --git a/src/ipa/simple/module.h b/src/ipa/simple/module.h new file mode 100644 index 00000000..5aaf8eb0 --- /dev/null +++ b/src/ipa/simple/module.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Red Hat, Inc. + * + * Software ISP IPA Module + */ + +#pragma once + +#include + +#include "libcamera/internal/software_isp/debayer_params.h" +#include "libcamera/internal/software_isp/swisp_stats.h" + +#include + +#include "ipa_context.h" +#include "soft_ipa_interface.h" + +namespace libcamera { + +namespace ipa::soft { + +using Module = ipa::Module; + +} /* namespace ipa::soft */ + +} /* namespace libcamera */ From patchwork Thu Aug 15 08:16:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20917 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 12344C323E for ; Thu, 15 Aug 2024 08:17:05 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 87ED2633BF; Thu, 15 Aug 2024 10:17:04 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="M9dETOSm"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1AC6A633BA for ; Thu, 15 Aug 2024 10:17:02 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709821; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MX9NCOwcE1ykHIPz/zU/TVZmkBNqq1TkDXFuf3lnGdo=; b=M9dETOSm25Ez8Q+8w9dmjRS7Dghp6Nt3IB1nTd3RfWfPAgNrKmAYmi3jUprv4y9K8Fk6a3 SO6ZjttzGeHCZvs4uX4wswvwqANXveIxt0VJa5ntptEwDwvEGVr+r1zvOAw/rmNP/4j7pe owetEp5t00wnDc/72O6OWf9U4j0mYOw= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-677-U6DqzKpyN-WGZPCn2xbj4Q-1; Thu, 15 Aug 2024 04:16:57 -0400 X-MC-Unique: U6DqzKpyN-WGZPCn2xbj4Q-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 6A2211956064; Thu, 15 Aug 2024 08:16:56 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 0C73E19560A3; Thu, 15 Aug 2024 08:16:53 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 04/18] libcamera: software_isp: Let IPASoftSimple inherit Module Date: Thu, 15 Aug 2024 10:16:08 +0200 Message-ID: <20240815081622.174210-5-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The Module class is a base class for all IPA modules. In addition, implement logPrefix() of the module for the softIPA. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Daniel Scally Reviewed-by: Laurent Pinchart --- src/ipa/simple/soft_simple.cpp | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index b7746ce0..72321f44 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -30,6 +30,7 @@ #include "libipa/camera_sensor_helper.h" #include "black_level.h" +#include "module.h" namespace libcamera { LOG_DEFINE_CATEGORY(IPASoft) @@ -54,7 +55,7 @@ static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; */ static constexpr float kExposureSatisfactory = 0.2; -class IPASoftSimple : public ipa::soft::IPASoftInterface +class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() @@ -76,6 +77,9 @@ public: void processStats(const ControlList &sensorControls) override; +protected: + std::string logPrefix() const override; + private: void updateExposure(double exposureMSV); @@ -421,6 +425,11 @@ void IPASoftSimple::updateExposure(double exposureMSV) again_ = std::clamp(again_, againMin_, againMax_); } +std::string IPASoftSimple::logPrefix() const +{ + return "IPASoft"; +} + } /* namespace ipa::soft */ /* From patchwork Thu Aug 15 08:16:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20918 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 6DD9DC32A9 for ; Thu, 15 Aug 2024 08:17:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 1329D633BD; Thu, 15 Aug 2024 10:17:09 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="EqF4jVar"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 76669633C7 for ; Thu, 15 Aug 2024 10:17:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709824; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q2v30WFtc+QLkbchyPhtbVVN0nHMzx8KfR3ygJ58HBw=; b=EqF4jVarsm0e7nEjekme8W03tbzlft1gIsD4FbBpRlXbTBklUd22hvq0yssnea2xrM73M9 VorONJDEGU1xmErOCFfM6Xj/uaiOkEf2VPr1GX5UlaKeGcm+5WuJRhM0Xbj41VnQOQon9I i2XP4SBIJSi2uxRLkVXGQff+ikf/ndI= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-496-u_1oPtmuMEKCOZzs2PWBQQ-1; Thu, 15 Aug 2024 04:17:00 -0400 X-MC-Unique: u_1oPtmuMEKCOZzs2PWBQQ-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EAE011955D48; Thu, 15 Aug 2024 08:16:58 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id BFF3919560A3; Thu, 15 Aug 2024 08:16:56 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 05/18] libcamera: software_isp: Make stats frame and buffer aware Date: Thu, 15 Aug 2024 10:16:09 +0200 Message-ID: <20240815081622.174210-6-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds frame and bufferId arguments to stats related calls. Although the parameters are currently unused, because frame ids are not tracked and used and the stats buffer is passed around directly rather than being referred by its id, they bring the internal APIs closer to their counterparts in hardware pipelines. It serves as a preparation for followup patches that will introduce: - Frame number tracking in order to switch to DelayedControls (software ISP TODO #11 + #12). - A ring buffer for stats in order to improve passing the stats (software ISP TODO #2). Frame and buffer ids are unrelated for the given purposes but since they are passed together at the same places, the change is implemented as a single patch rather than two, basically the same, patches. Signed-off-by: Milan Zamazal Reviewed-by: Laurent Pinchart --- .../libcamera/internal/software_isp/software_isp.h | 8 +++++--- include/libcamera/ipa/soft.mojom | 4 +++- src/ipa/simple/soft_simple.cpp | 7 +++++-- src/libcamera/pipeline/simple/simple.cpp | 8 +++++--- src/libcamera/software_isp/debayer_cpu.cpp | 8 +++++++- src/libcamera/software_isp/software_isp.cpp | 11 +++++++---- src/libcamera/software_isp/swstats_cpu.cpp | 6 ++++-- src/libcamera/software_isp/swstats_cpu.h | 4 ++-- 8 files changed, 38 insertions(+), 18 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index f8e00003..3602bce8 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -66,7 +67,8 @@ public: int exportBuffers(const Stream *stream, unsigned int count, std::vector> *buffers); - void processStats(const ControlList &sensorControls); + void processStats(const uint32_t frame, const uint32_t bufferId, + const ControlList &sensorControls); int start(); void stop(); @@ -78,13 +80,13 @@ public: Signal inputBufferReady; Signal outputBufferReady; - Signal<> ispStatsReady; + Signal ispStatsReady; Signal setSensorControls; private: void saveIspParams(); void setSensorCtrls(const ControlList &sensorControls); - void statsReady(); + void statsReady(uint32_t frame, uint32_t bufferId); void inputReady(FrameBuffer *input); void outputReady(FrameBuffer *output); diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index 0fd47bb0..cc5c37b2 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -23,7 +23,9 @@ interface IPASoftInterface { configure(libcamera.ControlInfoMap sensorCtrlInfoMap) => (int32 ret); - [async] processStats(libcamera.ControlList sensorControls); + [async] processStats(uint32 frame, + uint32 bufferId, + libcamera.ControlList sensorControls); }; interface IPASoftEventInterface { diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 72321f44..12b5245e 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -75,7 +75,8 @@ public: int start() override; void stop() override; - void processStats(const ControlList &sensorControls) override; + void processStats(const uint32_t frame, const uint32_t bufferId, + const ControlList &sensorControls) override; protected: std::string logPrefix() const override; @@ -249,7 +250,9 @@ void IPASoftSimple::stop() { } -void IPASoftSimple::processStats(const ControlList &sensorControls) +void IPASoftSimple::processStats([[maybe_unused]] const uint32_t frame, + [[maybe_unused]] const uint32_t bufferId, + const ControlList &sensorControls) { SwIspStats::Histogram histogram = stats_->yHistogram; if (ignoreUpdates_ > 0) diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 1e7ec7d9..48a568da 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -291,7 +292,7 @@ private: void conversionInputDone(FrameBuffer *buffer); void conversionOutputDone(FrameBuffer *buffer); - void ispStatsReady(); + void ispStatsReady(uint32_t frame, uint32_t bufferId); void setSensorControls(const ControlList &sensorControls); }; @@ -891,10 +892,11 @@ void SimpleCameraData::conversionOutputDone(FrameBuffer *buffer) pipe->completeRequest(request); } -void SimpleCameraData::ispStatsReady() +void SimpleCameraData::ispStatsReady(uint32_t frame, uint32_t bufferId) { /* \todo Use the DelayedControls class */ - swIsp_->processStats(sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, + swIsp_->processStats(frame, bufferId, + sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, V4L2_CID_EXPOSURE })); } diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index 077f7f4b..2a2e7edb 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -777,7 +777,13 @@ void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams } } - stats_->finishFrame(); + /* + * Frame and buffer ids are currently not used, so pass zeros as parameters. + * + * \todo Pass real values once frame is passed here and stats buffer passing + * is changed. + */ + stats_->finishFrame(0, 0); outputBufferReady.emit(output); inputBufferReady.emit(input); } diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index a3ae3e22..a3855568 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -155,15 +155,18 @@ SoftwareIsp::~SoftwareIsp() /** * \brief Process the statistics gathered + * \param[in] frame The frame number + * \param[in] bufferId ID of the statistics buffer * \param[in] sensorControls The sensor controls * * Requests the IPA to calculate new parameters for ISP and new control * values for the sensor. */ -void SoftwareIsp::processStats(const ControlList &sensorControls) +void SoftwareIsp::processStats(const uint32_t frame, const uint32_t bufferId, + const ControlList &sensorControls) { ASSERT(ipa_); - ipa_->processStats(sensorControls); + ipa_->processStats(frame, bufferId, sensorControls); } /** @@ -349,9 +352,9 @@ void SoftwareIsp::setSensorCtrls(const ControlList &sensorControls) setSensorControls.emit(sensorControls); } -void SoftwareIsp::statsReady() +void SoftwareIsp::statsReady(uint32_t frame, uint32_t bufferId) { - ispStatsReady.emit(); + ispStatsReady.emit(frame, bufferId); } void SoftwareIsp::inputReady(FrameBuffer *input) diff --git a/src/libcamera/software_isp/swstats_cpu.cpp b/src/libcamera/software_isp/swstats_cpu.cpp index 815c4d4f..c520c806 100644 --- a/src/libcamera/software_isp/swstats_cpu.cpp +++ b/src/libcamera/software_isp/swstats_cpu.cpp @@ -311,13 +311,15 @@ void SwStatsCpu::startFrame(void) /** * \brief Finish statistics calculation for the current frame + * \param[in] frame The frame number + * \param[in] bufferId ID of the statistics buffer * * This may only be called after a successful setWindow() call. */ -void SwStatsCpu::finishFrame(void) +void SwStatsCpu::finishFrame(uint32_t frame, uint32_t bufferId) { *sharedStats_ = stats_; - statsReady.emit(); + statsReady.emit(frame, bufferId); } /** diff --git a/src/libcamera/software_isp/swstats_cpu.h b/src/libcamera/software_isp/swstats_cpu.h index 363e326f..26a2f462 100644 --- a/src/libcamera/software_isp/swstats_cpu.h +++ b/src/libcamera/software_isp/swstats_cpu.h @@ -41,7 +41,7 @@ public: int configure(const StreamConfiguration &inputCfg); void setWindow(const Rectangle &window); void startFrame(); - void finishFrame(); + void finishFrame(uint32_t frame, uint32_t bufferId); void processLine0(unsigned int y, const uint8_t *src[]) { @@ -61,7 +61,7 @@ public: (this->*stats2_)(src); } - Signal<> statsReady; + Signal statsReady; private: using statsProcessFn = void (SwStatsCpu::*)(const uint8_t *src[]); From patchwork Thu Aug 15 08:16:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20919 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4C20FC323E for ; Thu, 15 Aug 2024 08:17:11 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DC62A633C8; Thu, 15 Aug 2024 10:17:10 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="TJ7sMLGx"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E0145633BF for ; Thu, 15 Aug 2024 10:17:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709827; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1KZmM50ODUWNoDjYHlqfTIxDIdoCNLWj3qYt2cCvtr4=; b=TJ7sMLGxaQfVQ6qdYkrDTEmXZR9ji5wUL8oMbQrdjpbkAT1xP+8Gnw+YV+0c8jtWVaHgdt Gq4nIddWjC2yxcsPz0ZT/i0CYzGONkL9ja3cu+vn3EB/PLai6XFCqyg3A9L489zC92lBjF m7VhyKRZ3retiWAHUKsF7e5YEVk21i4= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-433-NCCOkUlQMmS0ohUEu9Sl6g-1; Thu, 15 Aug 2024 04:17:02 -0400 X-MC-Unique: NCCOkUlQMmS0ohUEu9Sl6g-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id CDB381956095; Thu, 15 Aug 2024 08:17:01 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 851E819560A3; Thu, 15 Aug 2024 08:16:59 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 06/18] libcamera: software_isp: Remove final dots in debayer.cpp docstrings Date: Thu, 15 Aug 2024 10:16:10 +0200 Message-ID: <20240815081622.174210-7-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The policy and the style checker require that \brief, \param and \return texts don't finish with a dot. This needs to be fixed in debayer.cpp. Also leading spaces in a \return statement are removed from there. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Daniel Scally Reviewed-by: Laurent Pinchart --- src/libcamera/software_isp/debayer.cpp | 48 +++++++++++++------------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/src/libcamera/software_isp/debayer.cpp b/src/libcamera/software_isp/debayer.cpp index f4a299d5..db26c380 100644 --- a/src/libcamera/software_isp/debayer.cpp +++ b/src/libcamera/software_isp/debayer.cpp @@ -58,47 +58,47 @@ Debayer::~Debayer() /** * \fn int Debayer::configure(const StreamConfiguration &inputCfg, const std::vector> &outputCfgs) - * \brief Configure the debayer object according to the passed in parameters. - * \param[in] inputCfg The input configuration. - * \param[in] outputCfgs The output configurations. + * \brief Configure the debayer object according to the passed in parameters + * \param[in] inputCfg The input configuration + * \param[in] outputCfgs The output configurations * - * \return 0 on success, a negative errno on failure. + * \return 0 on success, a negative errno on failure */ /** * \fn Size Debayer::patternSize(PixelFormat inputFormat) - * \brief Get the width and height at which the bayer pattern repeats. - * \param[in] inputFormat The input format. + * \brief Get the width and height at which the bayer pattern repeats + * \param[in] inputFormat The input format * * Valid sizes are: 2x2, 4x2 or 4x4. * - * \return Pattern size or an empty size for unsupported inputFormats. + * \return Pattern size or an empty size for unsupported inputFormats */ /** * \fn std::vector Debayer::formats(PixelFormat inputFormat) - * \brief Get the supported output formats. - * \param[in] inputFormat The input format. + * \brief Get the supported output formats + * \param[in] inputFormat The input format * - * \return All supported output formats or an empty vector if there are none. + * \return All supported output formats or an empty vector if there are none */ /** * \fn std::tuple Debayer::strideAndFrameSize(const PixelFormat &outputFormat, const Size &size) - * \brief Get the stride and the frame size. - * \param[in] outputFormat The output format. - * \param[in] size The output size. + * \brief Get the stride and the frame size + * \param[in] outputFormat The output format + * \param[in] size The output size * * \return A tuple of the stride and the frame size, or a tuple with 0,0 if - * there is no valid output config. + * there is no valid output config */ /** * \fn void Debayer::process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) - * \brief Process the bayer data into the requested format. - * \param[in] input The input buffer. - * \param[in] output The output buffer. - * \param[in] params The parameters to be used in debayering. + * \brief Process the bayer data into the requested format + * \param[in] input The input buffer + * \param[in] output The output buffer + * \param[in] params The parameters to be used in debayering * * \note DebayerParams is passed by value deliberately so that a copy is passed * when this is run in another thread by invokeMethod(). @@ -106,21 +106,21 @@ Debayer::~Debayer() /** * \fn virtual SizeRange Debayer::sizes(PixelFormat inputFormat, const Size &inputSize) - * \brief Get the supported output sizes for the given input format and size. - * \param[in] inputFormat The input format. - * \param[in] inputSize The input size. + * \brief Get the supported output sizes for the given input format and size + * \param[in] inputFormat The input format + * \param[in] inputSize The input size * - * \return The valid size ranges or an empty range if there are none. + * \return The valid size ranges or an empty range if there are none */ /** * \var Signal Debayer::inputBufferReady - * \brief Signals when the input buffer is ready. + * \brief Signals when the input buffer is ready */ /** * \var Signal Debayer::outputBufferReady - * \brief Signals when the output buffer is ready. + * \brief Signals when the output buffer is ready */ } /* namespace libcamera */ From patchwork Thu Aug 15 08:16:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20920 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A52D0C32D5 for ; Thu, 15 Aug 2024 08:17:12 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4CE3E633C6; Thu, 15 Aug 2024 10:17:12 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="FEu5wk7v"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 72519633C1 for ; Thu, 15 Aug 2024 10:17:10 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709829; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=laVMS9tWTJVwDwJX0wk8aN8Lf7oY7XY3hIONyTbhkQI=; b=FEu5wk7vwjEdQ6BQwJQ7F/Cq9oUykoVOKfc2BLnBLyjADhYwe7SB1DHSNp9nHJUgnv8be8 KQfi6lZ+gFDZ0WIHP5KMUZsFQ/0V71E839za4Sreuu9xHhOYcDBMEiSB7D3AFcI0ay46G4 XZPRWb6YZHsbcBOT5VmUdsbopvY+XeY= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-280-UkvZ-CDBPvCk226WqT5aNw-1; Thu, 15 Aug 2024 04:17:05 -0400 X-MC-Unique: UkvZ-CDBPvCk226WqT5aNw-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 76A9F1955D58; Thu, 15 Aug 2024 08:17:04 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 4BAF5195605F; Thu, 15 Aug 2024 08:17:02 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 07/18] libcamera: software_isp: Track and pass frame ids Date: Thu, 15 Aug 2024 10:16:11 +0200 Message-ID: <20240815081622.174210-8-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" A previous preparation patch implemented passing frame ids to stats processing but without actual meaningful frame id value passed there. This patch extends that by actually providing the frame id and passing it through to the stats processor. The frame id is taken from the request sequence number, the same as in hardware pipelines. Signed-off-by: Milan Zamazal --- .../libcamera/internal/software_isp/software_isp.h | 4 ++-- src/libcamera/pipeline/simple/simple.cpp | 8 +++++++- src/libcamera/software_isp/debayer.cpp | 3 ++- src/libcamera/software_isp/debayer.h | 2 +- src/libcamera/software_isp/debayer_cpu.cpp | 9 ++++----- src/libcamera/software_isp/debayer_cpu.h | 2 +- src/libcamera/software_isp/software_isp.cpp | 11 +++++++---- 7 files changed, 24 insertions(+), 15 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index 3602bce8..3a84418e 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -73,10 +73,10 @@ public: int start(); void stop(); - int queueBuffers(FrameBuffer *input, + int queueBuffers(uint32_t frame, FrameBuffer *input, const std::map &outputs); - void process(FrameBuffer *input, FrameBuffer *output); + void process(uint32_t frame, FrameBuffer *input, FrameBuffer *output); Signal inputBufferReady; Signal outputBufferReady; diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 48a568da..ebec592a 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -865,7 +865,13 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) if (converter_) converter_->queueBuffers(buffer, conversionQueue_.front()); else - swIsp_->queueBuffers(buffer, conversionQueue_.front()); + /* + * request->sequence() cannot be retrieved from `buffer' inside + * queueBuffers because unique_ptr's make buffer->request() invalid + * already here. + */ + swIsp_->queueBuffers(request->sequence(), buffer, + conversionQueue_.front()); conversionQueue_.pop(); return; diff --git a/src/libcamera/software_isp/debayer.cpp b/src/libcamera/software_isp/debayer.cpp index db26c380..f0b83261 100644 --- a/src/libcamera/software_isp/debayer.cpp +++ b/src/libcamera/software_isp/debayer.cpp @@ -94,8 +94,9 @@ Debayer::~Debayer() */ /** - * \fn void Debayer::process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) + * \fn void Debayer::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params) * \brief Process the bayer data into the requested format + * \param[in] frame The frame number * \param[in] input The input buffer * \param[in] output The output buffer * \param[in] params The parameters to be used in debayering diff --git a/src/libcamera/software_isp/debayer.h b/src/libcamera/software_isp/debayer.h index c151fe5d..d7ca060d 100644 --- a/src/libcamera/software_isp/debayer.h +++ b/src/libcamera/software_isp/debayer.h @@ -40,7 +40,7 @@ public: virtual std::tuple strideAndFrameSize(const PixelFormat &outputFormat, const Size &size) = 0; - virtual void process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) = 0; + virtual void process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params) = 0; virtual SizeRange sizes(PixelFormat inputFormat, const Size &inputSize) = 0; diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index 2a2e7edb..f7b3a7d1 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -724,7 +724,7 @@ static inline int64_t timeDiff(timespec &after, timespec &before) (int64_t)after.tv_nsec - (int64_t)before.tv_nsec; } -void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) +void DebayerCpu::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params) { timespec frameStartTime; @@ -778,12 +778,11 @@ void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams } /* - * Frame and buffer ids are currently not used, so pass zeros as parameters. + * Buffer ids are currently not used, so pass zeros as its parameter. * - * \todo Pass real values once frame is passed here and stats buffer passing - * is changed. + * \todo Pass real bufferId once stats buffer passing is changed. */ - stats_->finishFrame(0, 0); + stats_->finishFrame(frame, 0); outputBufferReady.emit(output); inputBufferReady.emit(input); } diff --git a/src/libcamera/software_isp/debayer_cpu.h b/src/libcamera/software_isp/debayer_cpu.h index 8237a64b..2c47e7c6 100644 --- a/src/libcamera/software_isp/debayer_cpu.h +++ b/src/libcamera/software_isp/debayer_cpu.h @@ -36,7 +36,7 @@ public: std::vector formats(PixelFormat input); std::tuple strideAndFrameSize(const PixelFormat &outputFormat, const Size &size); - void process(FrameBuffer *input, FrameBuffer *output, DebayerParams params); + void process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params); SizeRange sizes(PixelFormat inputFormat, const Size &inputSize); /** diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index a3855568..c5db45ae 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -14,6 +14,7 @@ #include #include +#include #include #include "libcamera/internal/ipa_manager.h" @@ -278,12 +279,13 @@ int SoftwareIsp::exportBuffers(const Stream *stream, unsigned int count, /** * \brief Queue buffers to Software ISP + * \param[in] frame The frame number * \param[in] input The input framebuffer * \param[in] outputs The container holding the output stream pointers and * their respective frame buffer outputs * \return 0 on success, a negative errno on failure */ -int SoftwareIsp::queueBuffers(FrameBuffer *input, +int SoftwareIsp::queueBuffers(uint32_t frame, FrameBuffer *input, const std::map &outputs) { /* @@ -301,7 +303,7 @@ int SoftwareIsp::queueBuffers(FrameBuffer *input, } for (auto iter = outputs.begin(); iter != outputs.end(); iter++) - process(input, iter->second); + process(frame, input, iter->second); return 0; } @@ -333,13 +335,14 @@ void SoftwareIsp::stop() /** * \brief Passes the input framebuffer to the ISP worker to process + * \param[in] frame The frame number * \param[in] input The input framebuffer * \param[out] output The framebuffer to write the processed frame to */ -void SoftwareIsp::process(FrameBuffer *input, FrameBuffer *output) +void SoftwareIsp::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output) { debayer_->invokeMethod(&DebayerCpu::process, - ConnectionTypeQueued, input, output, debayerParams_); + ConnectionTypeQueued, frame, input, output, debayerParams_); } void SoftwareIsp::saveIspParams() From patchwork Thu Aug 15 08:16:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20921 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 6504DC32A9 for ; Thu, 15 Aug 2024 08:17:17 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id F3AE2633C8; Thu, 15 Aug 2024 10:17:16 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="Cobbd8x4"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 700D8633C1 for ; Thu, 15 Aug 2024 10:17:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709833; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4OojRvR7hV5VSh6rZzOTFwKTXnvVsUvy9KHsSwPlpII=; b=Cobbd8x4PscBpfKKuITrskLmlXlrdujg/xHjmGn8f4e/kx/vR+j2ejBN+/MMH+57C7gX5+ R9UQmr7b7S6hifHxnp7CrvmhN67QGZ7O4tZrjSztnxJazyIaVlT9z7e3GMELcs8WyJFnzV W9zpSKvqBpRE7OWsI3ctVZEdb27Ccio= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-52-NPU6JGNTOWa22maZPMOPng-1; Thu, 15 Aug 2024 04:17:08 -0400 X-MC-Unique: NPU6JGNTOWa22maZPMOPng-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 5C14F1955F06; Thu, 15 Aug 2024 08:17:07 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 1322B19560A3; Thu, 15 Aug 2024 08:17:04 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 08/18] libcamera: software_isp: Create algorithms Date: Thu, 15 Aug 2024 10:16:12 +0200 Message-ID: <20240815081622.174210-9-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We are ready to introduce algorithms now. First, let's create algorithms. The algorithms are not called yet, calls to them will be added in followup patches. The maximum number of contexts is set to the same value as in hardware pipelines. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Daniel Scally Reviewed-by: Laurent Pinchart --- src/ipa/simple/algorithms/meson.build | 4 ++++ src/ipa/simple/data/uncalibrated.yaml | 1 + src/ipa/simple/meson.build | 7 +++++-- src/ipa/simple/soft_simple.cpp | 14 ++++++++++++++ 4 files changed, 24 insertions(+), 2 deletions(-) create mode 100644 src/ipa/simple/algorithms/meson.build diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build new file mode 100644 index 00000000..31d26e43 --- /dev/null +++ b/src/ipa/simple/algorithms/meson.build @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: CC0-1.0 + +soft_simple_ipa_algorithms = files([ +]) diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index ff981a1a..2cdc39a8 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -2,4 +2,5 @@ %YAML 1.1 --- version: 1 +algorithms: ... diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build index 7757f99b..dcd7c70a 100644 --- a/src/ipa/simple/meson.build +++ b/src/ipa/simple/meson.build @@ -1,5 +1,8 @@ # SPDX-License-Identifier: CC0-1.0 +subdir('algorithms') +subdir('data') + ipa_name = 'ipa_soft_simple' soft_simple_sources = files([ @@ -8,6 +11,8 @@ soft_simple_sources = files([ 'black_level.cpp', ]) +soft_simple_sources += soft_simple_ipa_algorithms + mod = shared_module(ipa_name, soft_simple_sources, name_prefix : '', include_directories : [ipa_includes], @@ -24,6 +29,4 @@ if ipa_sign_module build_by_default : true) endif -subdir('data') - ipa_names += ipa_name diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 12b5245e..79ed4891 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -54,12 +54,15 @@ static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; * enough to prevent the exposure from wobbling around the optimal value. */ static constexpr float kExposureSatisfactory = 0.2; +/* Maximum number of frame contexts to be held */ +static constexpr uint32_t kMaxFrameContexts = 16; class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() : params_(nullptr), stats_(nullptr), blackLevel_(BlackLevel()), + context_({ {}, {}, { kMaxFrameContexts } }), ignoreUpdates_(0) { } @@ -93,6 +96,8 @@ private: static constexpr unsigned int kGammaLookupSize = 1024; std::array gammaTable_; int lastBlackLevel_ = -1; + /* Local parameter storage */ + struct IPAContext context_; int32_t exposureMin_, exposureMax_; int32_t exposure_; @@ -139,6 +144,15 @@ int IPASoftSimple::init(const IPASettings &settings, unsigned int version = (*data)["version"].get(0); LOG(IPASoft, Debug) << "Tuning file version " << version; + if (!data->contains("algorithms")) { + LOG(IPASoft, Error) << "Tuning file doesn't contain algorithms"; + return -EINVAL; + } + + int ret = createAlgorithms(context_, (*data)["algorithms"]); + if (ret) + return ret; + params_ = nullptr; stats_ = nullptr; From patchwork Thu Aug 15 08:16:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20922 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 526D2C323E for ; Thu, 15 Aug 2024 08:17:19 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A0FAA633C4; Thu, 15 Aug 2024 10:17:18 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="BFJBOahC"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 55EEC633C2 for ; Thu, 15 Aug 2024 10:17:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709834; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0BO/w5QMYz6bbvwUpiY8EPkIgmGoWmsNkoodaDvNTIU=; b=BFJBOahCjlH5ieLeAuNRP//+qvaA5XMYPQK24yojTi0hWQjYnI3fucFF59h6Vbfn35j9sQ tkpquu2/kBUbZ+DU23K5jbQsgiBOgzDKTLV6HgWhngNwNqV2J/B03Qinnn8Zdk5wKIlc4v rcnlj5InAdjpbMqav+NARmiejzPtYAo= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-8-ZFbp3yhINwOF1Xk_LoqVMw-1; Thu, 15 Aug 2024 04:17:10 -0400 X-MC-Unique: ZFbp3yhINwOF1Xk_LoqVMw-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id CC7731954B1C; Thu, 15 Aug 2024 08:17:09 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id CD14D19560A3; Thu, 15 Aug 2024 08:17:07 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 09/18] libcamera: software_isp: Call Algorithm::configure Date: Thu, 15 Aug 2024 10:16:13 +0200 Message-ID: <20240815081622.174210-10-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds Algorithm::configure call for the defined algorithms. This is preparation only since there are currently no Algorithm based algorithms defined. A part of this change is passing IPAConfigInfo instead of ControlInfoMap to configure() calls as this is what Algorithm::configure expects. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- .../libcamera/internal/software_isp/software_isp.h | 2 +- include/libcamera/ipa/soft.mojom | 2 +- src/ipa/simple/module.h | 2 ++ src/ipa/simple/soft_simple.cpp | 12 +++++++++--- src/libcamera/pipeline/simple/simple.cpp | 11 +++++++---- src/libcamera/software_isp/software_isp.cpp | 7 ++++--- 6 files changed, 24 insertions(+), 12 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index 3a84418e..c5d5a46f 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -62,7 +62,7 @@ public: int configure(const StreamConfiguration &inputCfg, const std::vector> &outputCfgs, - const ControlInfoMap &sensorControls); + const ipa::soft::IPAConfigInfo &configInfo); int exportBuffers(const Stream *stream, unsigned int count, std::vector> *buffers); diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index cc5c37b2..6b049d20 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -20,7 +20,7 @@ interface IPASoftInterface { => (int32 ret); start() => (int32 ret); stop(); - configure(libcamera.ControlInfoMap sensorCtrlInfoMap) + configure(IPAConfigInfo configInfo) => (int32 ret); [async] processStats(uint32 frame, diff --git a/src/ipa/simple/module.h b/src/ipa/simple/module.h index 5aaf8eb0..d1321f59 100644 --- a/src/ipa/simple/module.h +++ b/src/ipa/simple/module.h @@ -9,6 +9,8 @@ #include +#include + #include "libcamera/internal/software_isp/debayer_params.h" #include "libcamera/internal/software_isp/swisp_stats.h" diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 79ed4891..0ea62266 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -73,7 +73,7 @@ public: const SharedFD &fdStats, const SharedFD &fdParams, const ControlInfoMap &sensorInfoMap) override; - int configure(const ControlInfoMap &sensorInfoMap) override; + int configure(const IPAConfigInfo &configInfo) override; int start() override; void stop() override; @@ -207,9 +207,9 @@ int IPASoftSimple::init(const IPASettings &settings, return 0; } -int IPASoftSimple::configure(const ControlInfoMap &sensorInfoMap) +int IPASoftSimple::configure(const IPAConfigInfo &configInfo) { - sensorInfoMap_ = sensorInfoMap; + sensorInfoMap_ = configInfo.sensorControls; const ControlInfo &exposureInfo = sensorInfoMap_.find(V4L2_CID_EXPOSURE)->second; const ControlInfo &gainInfo = sensorInfoMap_.find(V4L2_CID_ANALOGUE_GAIN)->second; @@ -248,6 +248,12 @@ int IPASoftSimple::configure(const ControlInfoMap &sensorInfoMap) againMinStep_ = 1.0; } + for (auto const &algo : algorithms()) { + int ret = algo->configure(context_, configInfo); + if (ret) + return ret; + } + LOG(IPASoft, Info) << "Exposure " << exposureMin_ << "-" << exposureMax_ << ", gain " << againMin_ << "-" << againMax_ << " (" << againMinStep_ << ")"; diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index ebec592a..834b33d9 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -1294,10 +1294,13 @@ int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c) inputCfg.stride = captureFormat.planes[0].bpl; inputCfg.bufferCount = kNumInternalBuffers; - return data->converter_ - ? data->converter_->configure(inputCfg, outputCfgs) - : data->swIsp_->configure(inputCfg, outputCfgs, - data->sensor_->controls()); + if (data->converter_) { + return data->converter_->configure(inputCfg, outputCfgs); + } else { + ipa::soft::IPAConfigInfo configInfo; + configInfo.sensorControls = data->sensor_->controls(); + return data->swIsp_->configure(inputCfg, outputCfgs, configInfo); + } } int SimplePipelineHandler::exportFrameBuffers(Camera *camera, Stream *stream, diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index c5db45ae..8f1c0f65 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -223,16 +223,17 @@ SoftwareIsp::strideAndFrameSize(const PixelFormat &outputFormat, const Size &siz * \brief Configure the SoftwareIsp object according to the passed in parameters * \param[in] inputCfg The input configuration * \param[in] outputCfgs The output configurations - * \param[in] sensorControls ControlInfoMap of the controls supported by the sensor + * \param[in] configInfo The IPA configuration data, received from the pipeline + * handler * \return 0 on success, a negative errno on failure */ int SoftwareIsp::configure(const StreamConfiguration &inputCfg, const std::vector> &outputCfgs, - const ControlInfoMap &sensorControls) + const ipa::soft::IPAConfigInfo &configInfo) { ASSERT(ipa_ && debayer_); - int ret = ipa_->configure(sensorControls); + int ret = ipa_->configure(configInfo); if (ret < 0) return ret; From patchwork Thu Aug 15 08:16:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20923 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 38B0EC32D5 for ; Thu, 15 Aug 2024 08:17:22 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C1ED9633CD; Thu, 15 Aug 2024 10:17:21 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="QI0+zk5I"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A9CE3633CD for ; Thu, 15 Aug 2024 10:17:18 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709837; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QhxqDGmaspBPYM04rM9qZqhO81940aSHi9UFpNeLeDM=; b=QI0+zk5IwW7v83jR8IcTteet5QzCVH/NyQpKyyv5DX+Jql8jaAr6Pk2O0iG1tvEkFbS6DF Jv0FD15p8gvXsLsY6soQvWRPEo/VqdZgy6j53eACKG/IUwzuSNDYuYSMnld2v1eiD4ihzM aO58X9sITTmKu8grBF/FWjsdpOvQPNk= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-241-VNcl70kJMX2JmcCKE-ohZg-1; Thu, 15 Aug 2024 04:17:13 -0400 X-MC-Unique: VNcl70kJMX2JmcCKE-ohZg-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 937021955D4A; Thu, 15 Aug 2024 08:17:12 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 6298019560A3; Thu, 15 Aug 2024 08:17:10 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 10/18] libcamera: software_isp: Call Algorithm::queueRequest Date: Thu, 15 Aug 2024 10:16:14 +0200 Message-ID: <20240815081622.174210-11-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds Algorithm::queueRequest call for the defined algorithms. As there are currently no control knobs in software ISP nor the corresponding queueRequest call chain, the patch also introduces the queueRequest methods and calls from the pipeline to the IPA. This is preparation only since there are currently no Algorithm based algorithms defined and no current software ISP algorithms support control knobs. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Daniel Scally --- include/libcamera/internal/software_isp/software_isp.h | 1 + include/libcamera/ipa/soft.mojom | 1 + src/ipa/simple/soft_simple.cpp | 9 +++++++++ src/libcamera/pipeline/simple/simple.cpp | 2 ++ src/libcamera/software_isp/software_isp.cpp | 10 ++++++++++ 5 files changed, 23 insertions(+) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index c5d5a46f..a3e3a9da 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -73,6 +73,7 @@ public: int start(); void stop(); + void queueRequest(const uint32_t frame, const ControlList &controls); int queueBuffers(uint32_t frame, FrameBuffer *input, const std::map &outputs); diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index 6b049d20..0ca2869c 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -23,6 +23,7 @@ interface IPASoftInterface { configure(IPAConfigInfo configInfo) => (int32 ret); + [async] queueRequest(uint32 frame, libcamera.ControlList sensorControls); [async] processStats(uint32 frame, uint32 bufferId, libcamera.ControlList sensorControls); diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 0ea62266..eb3bbd92 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -78,6 +78,7 @@ public: int start() override; void stop() override; + void queueRequest(const uint32_t frame, const ControlList &controls) override; void processStats(const uint32_t frame, const uint32_t bufferId, const ControlList &sensorControls) override; @@ -270,6 +271,14 @@ void IPASoftSimple::stop() { } +void IPASoftSimple::queueRequest(const uint32_t frame, const ControlList &controls) +{ + IPAFrameContext &frameContext = context_.frameContexts.alloc(frame); + + for (auto const &algo : algorithms()) + algo->queueRequest(context_, frame, frameContext, controls); +} + void IPASoftSimple::processStats([[maybe_unused]] const uint32_t frame, [[maybe_unused]] const uint32_t bufferId, const ControlList &sensorControls) diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 834b33d9..ed50ab52 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -1428,6 +1428,8 @@ int SimplePipelineHandler::queueRequestDevice(Camera *camera, Request *request) if (data->useConversion_) data->conversionQueue_.push(std::move(buffers)); + data->swIsp_->queueRequest(request->sequence(), request->controls()); + return 0; } diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 8f1c0f65..eda18947 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -278,6 +278,16 @@ int SoftwareIsp::exportBuffers(const Stream *stream, unsigned int count, return count; } +/** + * \brief Queue a request and process the control list from the application + * \param[in] frame The number of the frame which will be processed next + * \param[in] controls The controls for the \a frame + */ +void SoftwareIsp::queueRequest(const uint32_t frame, const ControlList &controls) +{ + ipa_->queueRequest(frame, controls); +} + /** * \brief Queue buffers to Software ISP * \param[in] frame The frame number From patchwork Thu Aug 15 08:16:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20924 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 107B7C32D6 for ; Thu, 15 Aug 2024 08:17:23 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 60809633C2; Thu, 15 Aug 2024 10:17:23 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="hN41r7QZ"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id EAD55633C1 for ; Thu, 15 Aug 2024 10:17:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709839; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LzExjFmOMWaQshd+wHdnRR6XuJZ3FZ5MPMv6UWL7yfs=; b=hN41r7QZuyYAhKn5VLBsUPo5hcGgzqrgnLEyGWIVHdAYW2Vk1ckXMfmpU1hprwyFLPYL8z EPyEAQ1ikC3iNVSkaIy3r7ChHMXKf67ibbsKt1dP/UOFSXoCum1uGQSSRm4KwM+eSi9wEH RvRhdpMGnMVsYbt13tJpoLxF1om+U4s= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-224-LZn_eucvNmuwLM4kxBc0IQ-1; Thu, 15 Aug 2024 04:17:16 -0400 X-MC-Unique: LZn_eucvNmuwLM4kxBc0IQ-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 565E91955F3F; Thu, 15 Aug 2024 08:17:15 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 2835219560A3; Thu, 15 Aug 2024 08:17:12 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 11/18] libcamera: software_isp: Call Algorithm::prepare Date: Thu, 15 Aug 2024 10:16:15 +0200 Message-ID: <20240815081622.174210-12-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds Algorithm::prepare call for the defined algorithms. This is preparation only since there are currently no Algorithm based algorithms defined. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- include/libcamera/ipa/soft.mojom | 1 + src/ipa/simple/soft_simple.cpp | 8 ++++++++ src/libcamera/software_isp/software_isp.cpp | 1 + 3 files changed, 10 insertions(+) diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index 0ca2869c..bc70d4e4 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -24,6 +24,7 @@ interface IPASoftInterface { => (int32 ret); [async] queueRequest(uint32 frame, libcamera.ControlList sensorControls); + [async] prepare(uint32 frame); [async] processStats(uint32 frame, uint32 bufferId, libcamera.ControlList sensorControls); diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index eb3bbd92..d52242be 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -79,6 +79,7 @@ public: void stop() override; void queueRequest(const uint32_t frame, const ControlList &controls) override; + void prepare(const uint32_t frame) override; void processStats(const uint32_t frame, const uint32_t bufferId, const ControlList &sensorControls) override; @@ -279,6 +280,13 @@ void IPASoftSimple::queueRequest(const uint32_t frame, const ControlList &contro algo->queueRequest(context_, frame, frameContext, controls); } +void IPASoftSimple::prepare(const uint32_t frame) +{ + IPAFrameContext &frameContext = context_.frameContexts.get(frame); + for (auto const &algo : algorithms()) + algo->prepare(context_, frame, frameContext, params_); +} + void IPASoftSimple::processStats([[maybe_unused]] const uint32_t frame, [[maybe_unused]] const uint32_t bufferId, const ControlList &sensorControls) diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index eda18947..dbf27f31 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -352,6 +352,7 @@ void SoftwareIsp::stop() */ void SoftwareIsp::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output) { + ipa_->prepare(frame); debayer_->invokeMethod(&DebayerCpu::process, ConnectionTypeQueued, frame, input, output, debayerParams_); } From patchwork Thu Aug 15 08:16:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20925 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D060DC32A9 for ; Thu, 15 Aug 2024 08:17:27 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 68D73633C0; Thu, 15 Aug 2024 10:17:27 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="O0gfkMNH"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D452A633C8 for ; Thu, 15 Aug 2024 10:17:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709843; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=R6pipysZ/qYOzyonkafacIN3VlyHa05TRmcXWXBfMSU=; b=O0gfkMNHSc5WHZFIJQ5vFD3sxaTc4LiHMJ7NhD3i3Q88pSjediKnceZ6lM0R+20T8lAjsy to7DOPLsQGfTlAxsRxvEuwe6ZQj51JqNuAD/Ubfocyulm0bQkLWsjmSquCXsvOiG6aKsuZ QbQSkUN/W8iWvcqE3ImpeUjAZTGDgr4= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-589-ukDjdYymMben8ptYRlmy2Q-1; Thu, 15 Aug 2024 04:17:19 -0400 X-MC-Unique: ukDjdYymMben8ptYRlmy2Q-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1F6B91954B1D; Thu, 15 Aug 2024 08:17:18 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id D1B7119560A3; Thu, 15 Aug 2024 08:17:15 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 12/18] libcamera: software_isp: Call Algorithm::process Date: Thu, 15 Aug 2024 10:16:16 +0200 Message-ID: <20240815081622.174210-13-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds Algorithm::process call for the defined algorithms. This is preparation only since there are currently no Algorithm based algorithms defined. As software ISP currently doesn't produce any metadata, a dummy and unused metadata instance is created to satisfy Algorithm::process API. This should be changed in future. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain Reviewed-by: Daniel Scally --- src/ipa/simple/soft_simple.cpp | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index d52242be..4b9201dc 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -287,10 +287,21 @@ void IPASoftSimple::prepare(const uint32_t frame) algo->prepare(context_, frame, frameContext, params_); } -void IPASoftSimple::processStats([[maybe_unused]] const uint32_t frame, +void IPASoftSimple::processStats(const uint32_t frame, [[maybe_unused]] const uint32_t bufferId, const ControlList &sensorControls) { + IPAFrameContext &frameContext = context_.frameContexts.get(frame); + /* + * Software ISP currently does not produce any metadata. Use an empty + * ControlList for now. + * + * \todo Implement proper metadata handling + */ + ControlList metadata(controls::controls); + for (auto const &algo : algorithms()) + algo->process(context_, frame, frameContext, stats_, metadata); + SwIspStats::Histogram histogram = stats_->yHistogram; if (ignoreUpdates_ > 0) blackLevel_.update(histogram); From patchwork Thu Aug 15 08:16:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20926 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8CC5DBDB13 for ; Thu, 15 Aug 2024 08:17:30 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 232E0633D0; Thu, 15 Aug 2024 10:17:30 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="c4BgL1P9"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 22DE4633CD for ; Thu, 15 Aug 2024 10:17:26 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709845; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VqIsveihM9RNHJG1JC7Ty0xirratIuyEXSrqRU2uvVw=; b=c4BgL1P9Y9tcP6JWC/xABzC3OCOyzBfhtdKOErXqHehcpJCG2uDP7yDoOLkLIISI2gy0Nz TFDOAWCqJaltw51aM+DLscd1v+i1Bbzd7v7m5TI7B3Y8zO1jMbmR87qy/b2BOC6v1wYESX 5ORNbJGKColE/nbsKldJ3ad38CTMIA0= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-570-JPnPjjmiNfe578gA3AdJnQ-1; Thu, 15 Aug 2024 04:17:21 -0400 X-MC-Unique: JPnPjjmiNfe578gA3AdJnQ-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8BCAF1956048; Thu, 15 Aug 2024 08:17:20 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 90A8D19560A3; Thu, 15 Aug 2024 08:17:18 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 13/18] libcamera: software_isp: Move black level to an algorithm module Date: Thu, 15 Aug 2024 10:16:17 +0200 Message-ID: <20240815081622.174210-14-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The black level determination, already present as a separate class, can be moved to the prepared Algorithm processing structure. It is the first of the current software ISP algorithms that is called in the stats processing sequence, which means it is also the first one that should be moved to the new structure. Stats processing starts with calling Algorithm-based processing so the call order of the algorithms is retained. Movement of this algorithm is relatively straightforward because all it does is processing stats. The comment about getting black level from the tuning files is dropped. The black level will be taken from CameraSensorHelper if available and that will be implemented in one of the followup patches. Black level is now recomputed on each stats processing. In a future patch, after DelayedControls are used, this will be changed to recompute the black level only after exposure/gain changes. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- src/ipa/simple/algorithms/blc.cpp | 71 ++++++++++++++++++++ src/ipa/simple/algorithms/blc.h | 37 +++++++++++ src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/black_level.cpp | 93 --------------------------- src/ipa/simple/black_level.h | 33 ---------- src/ipa/simple/data/uncalibrated.yaml | 1 + src/ipa/simple/ipa_context.cpp | 8 +++ src/ipa/simple/ipa_context.h | 5 ++ src/ipa/simple/meson.build | 1 - src/ipa/simple/soft_simple.cpp | 8 +-- 10 files changed, 125 insertions(+), 133 deletions(-) create mode 100644 src/ipa/simple/algorithms/blc.cpp create mode 100644 src/ipa/simple/algorithms/blc.h delete mode 100644 src/ipa/simple/black_level.cpp delete mode 100644 src/ipa/simple/black_level.h diff --git a/src/ipa/simple/algorithms/blc.cpp b/src/ipa/simple/algorithms/blc.cpp new file mode 100644 index 00000000..3b73d830 --- /dev/null +++ b/src/ipa/simple/algorithms/blc.cpp @@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Black level handling + */ + +#include "blc.h" + +#include + +#include + +namespace libcamera { + +namespace ipa::soft::algorithms { + +LOG_DEFINE_CATEGORY(IPASoftBL) + +BlackLevel::BlackLevel() +{ +} + +int BlackLevel::init(IPAContext &context, + [[maybe_unused]] const YamlObject &tuningData) +{ + context.activeState.black.level = 255; + return 0; +} + +void BlackLevel::process(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + const SwIspStats *stats, + [[maybe_unused]] ControlList &metadata) +{ + const SwIspStats::Histogram &histogram = stats->yHistogram; + + /* + * The constant is selected to be "good enough", not overly conservative or + * aggressive. There is no magic about the given value. + */ + constexpr float ignoredPercentage_ = 0.02; + const unsigned int total = + std::accumulate(begin(histogram), end(histogram), 0); + const unsigned int pixelThreshold = ignoredPercentage_ * total; + const unsigned int histogramRatio = 256 / SwIspStats::kYHistogramSize; + const unsigned int currentBlackIdx = + context.activeState.black.level / histogramRatio; + + for (unsigned int i = 0, seen = 0; + i < currentBlackIdx && i < SwIspStats::kYHistogramSize; + i++) { + seen += histogram[i]; + if (seen >= pixelThreshold) { + context.activeState.black.level = i * histogramRatio; + LOG(IPASoftBL, Debug) + << "Auto-set black level: " + << i << "/" << SwIspStats::kYHistogramSize + << " (" << 100 * (seen - histogram[i]) / total << "% below, " + << 100 * seen / total << "% at or below)"; + break; + } + }; +} + +REGISTER_IPA_ALGORITHM(BlackLevel, "BlackLevel") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/blc.h b/src/ipa/simple/algorithms/blc.h new file mode 100644 index 00000000..2f73db52 --- /dev/null +++ b/src/ipa/simple/algorithms/blc.h @@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Black level handling + */ + +#pragma once + +#include + +#include "libcamera/internal/software_isp/swisp_stats.h" + +#include "algorithm.h" +#include "ipa_context.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class BlackLevel : public Algorithm +{ +public: + BlackLevel(); + ~BlackLevel() = default; + + int init(IPAContext &context, const YamlObject &tuningData) + override; + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const SwIspStats *stats, + ControlList &metadata) override; +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index 31d26e43..1f63c220 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -1,4 +1,5 @@ # SPDX-License-Identifier: CC0-1.0 soft_simple_ipa_algorithms = files([ + 'blc.cpp', ]) diff --git a/src/ipa/simple/black_level.cpp b/src/ipa/simple/black_level.cpp deleted file mode 100644 index 37e0109c..00000000 --- a/src/ipa/simple/black_level.cpp +++ /dev/null @@ -1,93 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1-or-later */ -/* - * Copyright (C) 2024, Red Hat Inc. - * - * black level handling - */ - -#include "black_level.h" - -#include - -#include - -namespace libcamera { - -LOG_DEFINE_CATEGORY(IPASoftBL) - -namespace ipa::soft { - -/** - * \class BlackLevel - * \brief Object providing black point level for software ISP - * - * Black level can be provided in hardware tuning files or, if no tuning file is - * available for the given hardware, guessed automatically, with less accuracy. - * As tuning files are not yet implemented for software ISP, BlackLevel - * currently provides only guessed black levels. - * - * This class serves for tracking black level as a property of the underlying - * hardware, not as means of enhancing a particular scene or image. - * - * The class is supposed to be instantiated for the given camera stream. - * The black level can be retrieved using BlackLevel::get() method. It is - * initially 0 and may change when updated using BlackLevel::update() method. - */ - -BlackLevel::BlackLevel() - : blackLevel_(255), blackLevelSet_(false) -{ -} - -/** - * \brief Return the current black level - * - * \return The black level, in the range from 0 (minimum) to 255 (maximum). - * If the black level couldn't be determined yet, return 0. - */ -uint8_t BlackLevel::get() const -{ - return blackLevelSet_ ? blackLevel_ : 0; -} - -/** - * \brief Update black level from the provided histogram - * \param[in] yHistogram The histogram to be used for updating black level - * - * The black level is property of the given hardware, not image. It is updated - * only if it has not been yet set or if it is lower than the lowest value seen - * so far. - */ -void BlackLevel::update(SwIspStats::Histogram &yHistogram) -{ - /* - * The constant is selected to be "good enough", not overly conservative or - * aggressive. There is no magic about the given value. - */ - constexpr float ignoredPercentage_ = 0.02; - const unsigned int total = - std::accumulate(begin(yHistogram), end(yHistogram), 0); - const unsigned int pixelThreshold = ignoredPercentage_ * total; - const unsigned int histogramRatio = 256 / SwIspStats::kYHistogramSize; - const unsigned int currentBlackIdx = blackLevel_ / histogramRatio; - - for (unsigned int i = 0, seen = 0; - i < currentBlackIdx && i < SwIspStats::kYHistogramSize; - i++) { - seen += yHistogram[i]; - if (seen >= pixelThreshold) { - blackLevel_ = i * histogramRatio; - blackLevelSet_ = true; - LOG(IPASoftBL, Debug) - << "Auto-set black level: " - << i << "/" << SwIspStats::kYHistogramSize - << " (" << 100 * (seen - yHistogram[i]) / total << "% below, " - << 100 * seen / total << "% at or below)"; - break; - } - }; -} - -} /* namespace ipa::soft */ - -} /* namespace libcamera */ diff --git a/src/ipa/simple/black_level.h b/src/ipa/simple/black_level.h deleted file mode 100644 index a04230c9..00000000 --- a/src/ipa/simple/black_level.h +++ /dev/null @@ -1,33 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1-or-later */ -/* - * Copyright (C) 2024, Red Hat Inc. - * - * black level handling - */ - -#pragma once - -#include -#include - -#include "libcamera/internal/software_isp/swisp_stats.h" - -namespace libcamera { - -namespace ipa::soft { - -class BlackLevel -{ -public: - BlackLevel(); - uint8_t get() const; - void update(SwIspStats::Histogram &yHistogram); - -private: - uint8_t blackLevel_; - bool blackLevelSet_; -}; - -} /* namespace ipa::soft */ - -} /* namespace libcamera */ diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index 2cdc39a8..e0d03d96 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -3,4 +3,5 @@ --- version: 1 algorithms: + - BlackLevel: ... diff --git a/src/ipa/simple/ipa_context.cpp b/src/ipa/simple/ipa_context.cpp index 3c1c7262..268cff32 100644 --- a/src/ipa/simple/ipa_context.cpp +++ b/src/ipa/simple/ipa_context.cpp @@ -50,4 +50,12 @@ namespace libcamera::ipa::soft { * \brief The current state of IPA algorithms */ +/** + * \var IPAActiveState::black + * \brief Context for the Black Level algorithm + * + * \var IPAActiveState::black.level + * \brief Current determined black level + */ + } /* namespace libcamera::ipa::soft */ diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index e7d8b8df..22156569 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -7,6 +7,8 @@ #pragma once +#include + #include namespace libcamera { @@ -17,6 +19,9 @@ struct IPASessionConfiguration { }; struct IPAActiveState { + struct { + uint8_t level; + } black; }; struct IPAFrameContext : public FrameContext { diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build index dcd7c70a..2f9f15f4 100644 --- a/src/ipa/simple/meson.build +++ b/src/ipa/simple/meson.build @@ -8,7 +8,6 @@ ipa_name = 'ipa_soft_simple' soft_simple_sources = files([ 'ipa_context.cpp', 'soft_simple.cpp', - 'black_level.cpp', ]) soft_simple_sources += soft_simple_ipa_algorithms diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 4b9201dc..a1bd2f0c 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -29,7 +29,6 @@ #include "libipa/camera_sensor_helper.h" -#include "black_level.h" #include "module.h" namespace libcamera { @@ -61,7 +60,7 @@ class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() - : params_(nullptr), stats_(nullptr), blackLevel_(BlackLevel()), + : params_(nullptr), stats_(nullptr), context_({ {}, {}, { kMaxFrameContexts } }), ignoreUpdates_(0) { @@ -93,7 +92,6 @@ private: SwIspStats *stats_; std::unique_ptr camHelper_; ControlInfoMap sensorInfoMap_; - BlackLevel blackLevel_; static constexpr unsigned int kGammaLookupSize = 1024; std::array gammaTable_; @@ -303,9 +301,7 @@ void IPASoftSimple::processStats(const uint32_t frame, algo->process(context_, frame, frameContext, stats_, metadata); SwIspStats::Histogram histogram = stats_->yHistogram; - if (ignoreUpdates_ > 0) - blackLevel_.update(histogram); - const uint8_t blackLevel = blackLevel_.get(); + const uint8_t blackLevel = context_.activeState.black.level; /* * Black level must be subtracted to get the correct AWB ratios, they From patchwork Thu Aug 15 08:16:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20927 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C4A38C323E for ; Thu, 15 Aug 2024 08:17:32 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 25F2E633D5; Thu, 15 Aug 2024 10:17:32 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="aSs3qqOM"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 56A3B633CD for ; Thu, 15 Aug 2024 10:17:29 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709848; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oxirCQjNxoqxmDjTI3TuJQSYHax1xd4GSFmDW79/D8E=; b=aSs3qqOMNcqUvdQtFfzVmEakIGseL2N+BvaZoJ9F7cQC+L8da7MUr7YaSF7h5hH2X/+O7d gPgsaDpMjVr/nQfcaTAaF0bHKt+EB0zmcxG9I6kBnuJ+a/qAooY1pd331J8mZc4lL6tFVe 4WUJl9gw9dD3xFan4bltfBhLMuo1SMo= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-629-i-sAx6FePdi3XZ6IOMBPKA-1; Thu, 15 Aug 2024 04:17:24 -0400 X-MC-Unique: i-sAx6FePdi3XZ6IOMBPKA-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 53FAF19560B7; Thu, 15 Aug 2024 08:17:23 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 2593219560A3; Thu, 15 Aug 2024 08:17:20 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 14/18] libcamera: software_isp: Move color handling to an algorithm module Date: Thu, 15 Aug 2024 10:16:18 +0200 Message-ID: <20240815081622.174210-15-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" After black level handling has been moved to an algorithm module, white balance and the construction of color tables can be moved to an algorithm module too. This time, the moved code is split between stats processing and parameter construction methods. This is the only part of the software ISP algorithms that sets the parameters so emitting setIspParams can be moved to prepare() method. A more natural representation of the gains (and black level) would be floating point numbers. This is not done here in order to minimize changes in code movements. It will be addressed in a followup patch. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/awb.cpp | 94 +++++++++++++++++++++++++++ src/ipa/simple/algorithms/awb.h | 43 ++++++++++++ src/ipa/simple/algorithms/gamma.cpp | 64 ++++++++++++++++++ src/ipa/simple/algorithms/gamma.h | 42 ++++++++++++ src/ipa/simple/algorithms/meson.build | 2 + src/ipa/simple/data/uncalibrated.yaml | 2 + src/ipa/simple/ipa_context.cpp | 30 +++++++++ src/ipa/simple/ipa_context.h | 12 ++++ src/ipa/simple/soft_simple.cpp | 66 ++----------------- 9 files changed, 293 insertions(+), 62 deletions(-) create mode 100644 src/ipa/simple/algorithms/awb.cpp create mode 100644 src/ipa/simple/algorithms/awb.h create mode 100644 src/ipa/simple/algorithms/gamma.cpp create mode 100644 src/ipa/simple/algorithms/gamma.h diff --git a/src/ipa/simple/algorithms/awb.cpp b/src/ipa/simple/algorithms/awb.cpp new file mode 100644 index 00000000..1e43f3fd --- /dev/null +++ b/src/ipa/simple/algorithms/awb.cpp @@ -0,0 +1,94 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Auto white balance + */ + +#include "awb.h" + +#include +#include +#include + +#include + +#include "simple/ipa_context.h" + +#include "gamma.h" + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoftAwb) + +namespace ipa::soft::algorithms { + +int Awb::init(IPAContext &context, + [[maybe_unused]] const YamlObject &tuningData) +{ + auto &gains = context.activeState.gains; + gains.red = gains.green = gains.blue = 256; + + return 0; +} + +void Awb::prepare(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + DebayerParams *params) +{ + auto &gains = context.activeState.gains; + auto &gammaTable = context.activeState.gamma.gammaTable; + for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { + constexpr unsigned int div = + static_cast(DebayerParams::kRGBLookupSize) * 256 / kGammaLookupSize; + /* Apply gamma after gain! */ + unsigned int idx; + idx = std::min({ i * gains.red / div, kGammaLookupSize - 1 }); + params->red[i] = gammaTable[idx]; + idx = std::min({ i * gains.green / div, kGammaLookupSize - 1 }); + params->green[i] = gammaTable[idx]; + idx = std::min({ i * gains.blue / div, kGammaLookupSize - 1 }); + params->blue[i] = gammaTable[idx]; + } +} + +void Awb::process(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + const SwIspStats *stats, + [[maybe_unused]] ControlList &metadata) +{ + const SwIspStats::Histogram &histogram = stats->yHistogram; + const uint8_t blackLevel = context.activeState.black.level; + + /* + * Black level must be subtracted to get the correct AWB ratios, they + * would be off if they were computed from the whole brightness range + * rather than from the sensor range. + */ + const uint64_t nPixels = std::accumulate( + histogram.begin(), histogram.end(), 0); + const uint64_t offset = blackLevel * nPixels; + const uint64_t sumR = stats->sumR_ - offset / 4; + const uint64_t sumG = stats->sumG_ - offset / 2; + const uint64_t sumB = stats->sumB_ - offset / 4; + + /* + * Calculate red and blue gains for AWB. + * Clamp max gain at 4.0, this also avoids 0 division. + * Gain: 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. + */ + auto &gains = context.activeState.gains; + gains.red = sumR <= sumG / 4 ? 1024 : 256 * sumG / sumR; + gains.blue = sumB <= sumG / 4 ? 1024 : 256 * sumG / sumB; + /* Green gain is fixed to 256 */ + + LOG(IPASoftAwb, Debug) << "gain R/B " << gains.red << "/" << gains.blue; +} + +REGISTER_IPA_ALGORITHM(Awb, "Awb") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/awb.h b/src/ipa/simple/algorithms/awb.h new file mode 100644 index 00000000..f94c1119 --- /dev/null +++ b/src/ipa/simple/algorithms/awb.h @@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Auto white balance + */ + +#pragma once + +#include + +#include "libcamera/internal/software_isp/debayer_params.h" +#include "libcamera/internal/software_isp/swisp_stats.h" + +#include "algorithm.h" +#include "ipa_context.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class Awb : public Algorithm +{ +public: + Awb() = default; + ~Awb() = default; + + int init(IPAContext &context, const YamlObject &tuningData) + override; + void prepare(IPAContext &context, + const uint32_t frame, + IPAFrameContext &frameContext, + DebayerParams *params) override; + void process(IPAContext &context, + const uint32_t frame, + IPAFrameContext &frameContext, + const SwIspStats *stats, + ControlList &metadata) override; +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/gamma.cpp b/src/ipa/simple/algorithms/gamma.cpp new file mode 100644 index 00000000..0b8ec5ee --- /dev/null +++ b/src/ipa/simple/algorithms/gamma.cpp @@ -0,0 +1,64 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Gamma table construction + */ + +#include "gamma.h" + +#include +#include +#include + +#include + +#include "simple/ipa_context.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +int Gamma::init(IPAContext &context, + [[maybe_unused]] const YamlObject &tuningData) +{ + /* Gamma value is fixed */ + context.configuration.gamma = 0.5; + updateGammaTable(context); + + return 0; +} + +void Gamma::updateGammaTable(IPAContext &context) +{ + auto &gammaTable = context.activeState.gamma.gammaTable; + auto blackLevel = context.activeState.black.level; + const unsigned int blackIndex = + blackLevel * IPAActiveState::kGammaLookupSize / 256; + std::fill(gammaTable.begin(), gammaTable.begin() + blackIndex, 0); + const float divisor = kGammaLookupSize - blackIndex - 1.0; + for (unsigned int i = blackIndex; i < kGammaLookupSize; i++) + gammaTable[i] = UINT8_MAX * powf((i - blackIndex) / divisor, + context.configuration.gamma); + context.activeState.gamma.blackLevel = blackLevel; +} + +void Gamma::prepare(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + [[maybe_unused]] DebayerParams *params) +{ + /* + * Update the gamma table if needed. This means if black level changes and + * since the black level gets updated only if a lower value is observed, + * it's not permanently prone to minor fluctuations or rounding errors. + */ + if (context.activeState.gamma.blackLevel != context.activeState.black.level) + updateGammaTable(context); +} + +REGISTER_IPA_ALGORITHM(Gamma, "Gamma") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/gamma.h b/src/ipa/simple/algorithms/gamma.h new file mode 100644 index 00000000..43e88e87 --- /dev/null +++ b/src/ipa/simple/algorithms/gamma.h @@ -0,0 +1,42 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Gamma table construction + */ + +#pragma once + +#include + +#include "libcamera/internal/software_isp/debayer_params.h" + +#include "algorithm.h" +#include "ipa_context.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +static constexpr unsigned int kGammaLookupSize = 1024; + +class Gamma : public Algorithm +{ +public: + Gamma() = default; + ~Gamma() = default; + + int init(IPAContext &context, const YamlObject &tuningData) + override; + void prepare(IPAContext &context, + const uint32_t frame, + IPAFrameContext &frameContext, + DebayerParams *params) override; + +private: + void updateGammaTable(IPAContext &context); +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index 1f63c220..de50fe6d 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -2,4 +2,6 @@ soft_simple_ipa_algorithms = files([ 'blc.cpp', + 'gamma.cpp', + 'awb.cpp', ]) diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index e0d03d96..3ca0a55f 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -4,4 +4,6 @@ version: 1 algorithms: - BlackLevel: + - Gamma: + - Awb: ... diff --git a/src/ipa/simple/ipa_context.cpp b/src/ipa/simple/ipa_context.cpp index 268cff32..5fa492cb 100644 --- a/src/ipa/simple/ipa_context.cpp +++ b/src/ipa/simple/ipa_context.cpp @@ -50,6 +50,11 @@ namespace libcamera::ipa::soft { * \brief The current state of IPA algorithms */ +/** + * \var IPASessionConfiguration::gamma + * \brief Gamma value to be used in the raw image processing + */ + /** * \var IPAActiveState::black * \brief Context for the Black Level algorithm @@ -58,4 +63,29 @@ namespace libcamera::ipa::soft { * \brief Current determined black level */ +/** + * \var IPAActiveState::gains + * \brief Context for gains in the Colors algorithm + * + * \var IPAActiveState::gains.red + * \brief Gain of red color + * + * \var IPAActiveState::gains.green + * \brief Gain of green color + * + * \var IPAActiveState::gains.blue + * \brief Gain of blue color + */ + +/** + * \var IPAActiveState::gamma + * \brief Context for gamma in the Colors algorithm + * + * \var IPAActiveState::gamma.gammaTable + * \brief Computed gamma table + * + * \var IPAActiveState::gamma.blackLevel + * \brief Black level used for the gamma table computation + */ + } /* namespace libcamera::ipa::soft */ diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index 22156569..f8b8834d 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -7,6 +7,7 @@ #pragma once +#include #include #include @@ -16,12 +17,23 @@ namespace libcamera { namespace ipa::soft { struct IPASessionConfiguration { + float gamma; }; struct IPAActiveState { struct { uint8_t level; } black; + struct { + unsigned int red; + unsigned int green; + unsigned int blue; + } gains; + static constexpr unsigned int kGammaLookupSize = 1024; + struct { + std::array gammaTable; + uint8_t blackLevel; + } gamma; }; struct IPAFrameContext : public FrameContext { diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index a1bd2f0c..8f01bf7d 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -5,8 +5,6 @@ * Simple Software Image Processing Algorithm module */ -#include -#include #include #include @@ -93,9 +91,6 @@ private: std::unique_ptr camHelper_; ControlInfoMap sensorInfoMap_; - static constexpr unsigned int kGammaLookupSize = 1024; - std::array gammaTable_; - int lastBlackLevel_ = -1; /* Local parameter storage */ struct IPAContext context_; @@ -283,6 +278,7 @@ void IPASoftSimple::prepare(const uint32_t frame) IPAFrameContext &frameContext = context_.frameContexts.get(frame); for (auto const &algo : algorithms()) algo->prepare(context_, frame, frameContext, params_); + setIspParams.emit(); } void IPASoftSimple::processStats(const uint32_t frame, @@ -300,62 +296,6 @@ void IPASoftSimple::processStats(const uint32_t frame, for (auto const &algo : algorithms()) algo->process(context_, frame, frameContext, stats_, metadata); - SwIspStats::Histogram histogram = stats_->yHistogram; - const uint8_t blackLevel = context_.activeState.black.level; - - /* - * Black level must be subtracted to get the correct AWB ratios, they - * would be off if they were computed from the whole brightness range - * rather than from the sensor range. - */ - const uint64_t nPixels = std::accumulate( - histogram.begin(), histogram.end(), 0); - const uint64_t offset = blackLevel * nPixels; - const uint64_t sumR = stats_->sumR_ - offset / 4; - const uint64_t sumG = stats_->sumG_ - offset / 2; - const uint64_t sumB = stats_->sumB_ - offset / 4; - - /* - * Calculate red and blue gains for AWB. - * Clamp max gain at 4.0, this also avoids 0 division. - * Gain: 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. - */ - const unsigned int gainR = sumR <= sumG / 4 ? 1024 : 256 * sumG / sumR; - const unsigned int gainB = sumB <= sumG / 4 ? 1024 : 256 * sumG / sumB; - /* Green gain and gamma values are fixed */ - constexpr unsigned int gainG = 256; - - /* Update the gamma table if needed */ - if (blackLevel != lastBlackLevel_) { - constexpr float gamma = 0.5; - const unsigned int blackIndex = blackLevel * kGammaLookupSize / 256; - std::fill(gammaTable_.begin(), gammaTable_.begin() + blackIndex, 0); - const float divisor = kGammaLookupSize - blackIndex - 1.0; - for (unsigned int i = blackIndex; i < kGammaLookupSize; i++) - gammaTable_[i] = UINT8_MAX * - std::pow((i - blackIndex) / divisor, gamma); - - lastBlackLevel_ = blackLevel; - } - - for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { - constexpr unsigned int div = - DebayerParams::kRGBLookupSize * 256 / kGammaLookupSize; - unsigned int idx; - - /* Apply gamma after gain! */ - idx = std::min({ i * gainR / div, (kGammaLookupSize - 1) }); - params_->red[i] = gammaTable_[idx]; - - idx = std::min({ i * gainG / div, (kGammaLookupSize - 1) }); - params_->green[i] = gammaTable_[idx]; - - idx = std::min({ i * gainB / div, (kGammaLookupSize - 1) }); - params_->blue[i] = gammaTable_[idx]; - } - - setIspParams.emit(); - /* \todo Switch to the libipa/algorithm.h API someday. */ /* @@ -372,6 +312,7 @@ void IPASoftSimple::processStats(const uint32_t frame, * Calculate Mean Sample Value (MSV) according to formula from: * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf */ + const uint8_t blackLevel = context_.activeState.black.level; const unsigned int blackLevelHistIdx = blackLevel / (256 / SwIspStats::kYHistogramSize); const unsigned int histogramSize = @@ -421,7 +362,8 @@ void IPASoftSimple::processStats(const uint32_t frame, LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV << " exp " << exposure_ << " again " << again_ - << " gain R/B " << gainR << "/" << gainB + << " gain R/B " << context_.activeState.gains.red + << "/" << context_.activeState.gains.blue << " black level " << static_cast(blackLevel); } From patchwork Thu Aug 15 08:16:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20928 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D5778C32D5 for ; Thu, 15 Aug 2024 08:17:34 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 53388633CD; Thu, 15 Aug 2024 10:17:34 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="J836sC9/"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9FA55633D2 for ; Thu, 15 Aug 2024 10:17:31 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709850; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=feNHMmQPRk0OZHvCFmmY2InaFfyBBPiS8Tk9yGS11dY=; b=J836sC9/FjDUew5tnTAaXmZxW7eeaWkk+k6LR5etp+CiZlJyM69b4AmiJErD0wYXTmOszw Up08e03MzNlQakhb+EHDBM8WyPMXmlq3mz9ndhUP6KmkwfUnGb2hVJpdoOS6SKOwQxrGHz uvE7GUb0NjewfxYEu7rWQqvm0nxxybI= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-610-88ig3-J4NRCFnuZ2C1P4XQ-1; Thu, 15 Aug 2024 04:17:27 -0400 X-MC-Unique: 88ig3-J4NRCFnuZ2C1P4XQ-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1CDBE1955D48; Thu, 15 Aug 2024 08:17:26 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id C6EBC19560A3; Thu, 15 Aug 2024 08:17:23 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 15/18] libcamera: software_isp: Use floating point for color parameters Date: Thu, 15 Aug 2024 10:16:19 +0200 Message-ID: <20240815081622.174210-16-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" It's more natural to represent color gains and black level as floating point numbers rather than using a particular pixel-related representation. double is used rather than float because it's a more common floating point type in libcamera algorithms. Otherwise there is no obvious reason to select one over the other here. The constructed color tables still use integer representation for efficiency. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/awb.cpp | 24 +++++++++++++----------- src/ipa/simple/algorithms/blc.cpp | 8 ++++---- src/ipa/simple/algorithms/gamma.cpp | 2 +- src/ipa/simple/ipa_context.h | 11 +++++------ 4 files changed, 23 insertions(+), 22 deletions(-) diff --git a/src/ipa/simple/algorithms/awb.cpp b/src/ipa/simple/algorithms/awb.cpp index 1e43f3fd..ff6591de 100644 --- a/src/ipa/simple/algorithms/awb.cpp +++ b/src/ipa/simple/algorithms/awb.cpp @@ -27,7 +27,7 @@ int Awb::init(IPAContext &context, [[maybe_unused]] const YamlObject &tuningData) { auto &gains = context.activeState.gains; - gains.red = gains.green = gains.blue = 256; + gains.red = gains.green = gains.blue = 1.0; return 0; } @@ -40,15 +40,18 @@ void Awb::prepare(IPAContext &context, auto &gains = context.activeState.gains; auto &gammaTable = context.activeState.gamma.gammaTable; for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { - constexpr unsigned int div = - static_cast(DebayerParams::kRGBLookupSize) * 256 / kGammaLookupSize; + constexpr double div = + static_cast(DebayerParams::kRGBLookupSize) / kGammaLookupSize; /* Apply gamma after gain! */ unsigned int idx; - idx = std::min({ i * gains.red / div, kGammaLookupSize - 1 }); + idx = std::min({ static_cast(i * gains.red / div), + kGammaLookupSize - 1 }); params->red[i] = gammaTable[idx]; - idx = std::min({ i * gains.green / div, kGammaLookupSize - 1 }); + idx = std::min({ static_cast(i * gains.green / div), + kGammaLookupSize - 1 }); params->green[i] = gammaTable[idx]; - idx = std::min({ i * gains.blue / div, kGammaLookupSize - 1 }); + idx = std::min({ static_cast(i * gains.blue / div), + kGammaLookupSize - 1 }); params->blue[i] = gammaTable[idx]; } } @@ -60,7 +63,7 @@ void Awb::process(IPAContext &context, [[maybe_unused]] ControlList &metadata) { const SwIspStats::Histogram &histogram = stats->yHistogram; - const uint8_t blackLevel = context.activeState.black.level; + const double blackLevel = context.activeState.black.level; /* * Black level must be subtracted to get the correct AWB ratios, they @@ -77,12 +80,11 @@ void Awb::process(IPAContext &context, /* * Calculate red and blue gains for AWB. * Clamp max gain at 4.0, this also avoids 0 division. - * Gain: 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. */ auto &gains = context.activeState.gains; - gains.red = sumR <= sumG / 4 ? 1024 : 256 * sumG / sumR; - gains.blue = sumB <= sumG / 4 ? 1024 : 256 * sumG / sumB; - /* Green gain is fixed to 256 */ + gains.red = sumR <= sumG / 4 ? 4.0 : static_cast(sumG) / sumR; + gains.blue = sumB <= sumG / 4 ? 4.0 : static_cast(sumG) / sumB; + /* Green gain is fixed to 1.0 */ LOG(IPASoftAwb, Debug) << "gain R/B " << gains.red << "/" << gains.blue; } diff --git a/src/ipa/simple/algorithms/blc.cpp b/src/ipa/simple/algorithms/blc.cpp index 3b73d830..ab7e7dd9 100644 --- a/src/ipa/simple/algorithms/blc.cpp +++ b/src/ipa/simple/algorithms/blc.cpp @@ -24,7 +24,7 @@ BlackLevel::BlackLevel() int BlackLevel::init(IPAContext &context, [[maybe_unused]] const YamlObject &tuningData) { - context.activeState.black.level = 255; + context.activeState.black.level = 1.0; return 0; } @@ -44,16 +44,16 @@ void BlackLevel::process(IPAContext &context, const unsigned int total = std::accumulate(begin(histogram), end(histogram), 0); const unsigned int pixelThreshold = ignoredPercentage_ * total; - const unsigned int histogramRatio = 256 / SwIspStats::kYHistogramSize; const unsigned int currentBlackIdx = - context.activeState.black.level / histogramRatio; + context.activeState.black.level * SwIspStats::kYHistogramSize; for (unsigned int i = 0, seen = 0; i < currentBlackIdx && i < SwIspStats::kYHistogramSize; i++) { seen += histogram[i]; if (seen >= pixelThreshold) { - context.activeState.black.level = i * histogramRatio; + context.activeState.black.level = + static_cast(i) / SwIspStats::kYHistogramSize; LOG(IPASoftBL, Debug) << "Auto-set black level: " << i << "/" << SwIspStats::kYHistogramSize diff --git a/src/ipa/simple/algorithms/gamma.cpp b/src/ipa/simple/algorithms/gamma.cpp index 0b8ec5ee..b03dff25 100644 --- a/src/ipa/simple/algorithms/gamma.cpp +++ b/src/ipa/simple/algorithms/gamma.cpp @@ -34,7 +34,7 @@ void Gamma::updateGammaTable(IPAContext &context) auto &gammaTable = context.activeState.gamma.gammaTable; auto blackLevel = context.activeState.black.level; const unsigned int blackIndex = - blackLevel * IPAActiveState::kGammaLookupSize / 256; + blackLevel * IPAActiveState::kGammaLookupSize; std::fill(gammaTable.begin(), gammaTable.begin() + blackIndex, 0); const float divisor = kGammaLookupSize - blackIndex - 1.0; for (unsigned int i = blackIndex; i < kGammaLookupSize; i++) diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index f8b8834d..28a02475 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -8,7 +8,6 @@ #pragma once #include -#include #include @@ -22,17 +21,17 @@ struct IPASessionConfiguration { struct IPAActiveState { struct { - uint8_t level; + double level; } black; struct { - unsigned int red; - unsigned int green; - unsigned int blue; + double red; + double green; + double blue; } gains; static constexpr unsigned int kGammaLookupSize = 1024; struct { std::array gammaTable; - uint8_t blackLevel; + double blackLevel; } gamma; }; From patchwork Thu Aug 15 08:16:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20929 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1287FC32D6 for ; Thu, 15 Aug 2024 08:17:37 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8010E633D9; Thu, 15 Aug 2024 10:17:37 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="RgTQpAXb"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C6439633D6 for ; Thu, 15 Aug 2024 10:17:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709851; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zqi62KONqOYAkH9I2+EWHCyIrENNhFJ60R7QakIJ3pA=; b=RgTQpAXbF6AINcOdDoN3/BlU/7i/WDD1qkHDUq6/Bn0jSMYmJZqvohLMZao5wMFQ26+suC QJb80b4+64KlF5Diz3H2IV8GfA9KyFrm3H32UmeoAZFZzpe1OKcKmxf3s2aSMbP6iL/Zjj QCxsHMAhd7tV6lYPG+TLWKF6sbBs+lg= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-637-0ORnjQvNOEeeNJEy6uPSXw-1; Thu, 15 Aug 2024 04:17:30 -0400 X-MC-Unique: 0ORnjQvNOEeeNJEy6uPSXw-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 29B1D1955D48; Thu, 15 Aug 2024 08:17:29 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id F179819560A3; Thu, 15 Aug 2024 08:17:26 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 16/18] libcamera: software_isp: Use DelayedControls Date: Thu, 15 Aug 2024 10:16:20 +0200 Message-ID: <20240815081622.174210-17-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Use the standard libcamera mechanism to report the "current" controls rather than delaying updates by counting from the last update. MY SPECULATION -- valid or not?: A problem is that with software ISP we cannot be sure about the sensor delay. The original implementation simply skips exposure updates for 2 frames, which should be enough in most cases. After this change, we assume the delay being exactly 2 frames, which may or may not be correct and may work with outdated values if the delay is shorter. This patch also prepares moving exposure+gain to an algorithm module where the original delay mechanism would be a (possibly unnecessary) complication. Resolves software ISP TODO #11 + #12. Signed-off-by: Milan Zamazal --- src/ipa/simple/soft_simple.cpp | 16 +------------ src/libcamera/pipeline/simple/simple.cpp | 18 ++++++++++++--- src/libcamera/software_isp/TODO | 29 ------------------------ 3 files changed, 16 insertions(+), 47 deletions(-) diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 8f01bf7d..0e47b9c5 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -59,8 +59,7 @@ class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module public: IPASoftSimple() : params_(nullptr), stats_(nullptr), - context_({ {}, {}, { kMaxFrameContexts } }), - ignoreUpdates_(0) + context_({ {}, {}, { kMaxFrameContexts } }) { } @@ -98,7 +97,6 @@ private: int32_t exposure_; double againMin_, againMax_, againMinStep_; double again_; - unsigned int ignoreUpdates_; }; IPASoftSimple::~IPASoftSimple() @@ -298,16 +296,6 @@ void IPASoftSimple::processStats(const uint32_t frame, /* \todo Switch to the libipa/algorithm.h API someday. */ - /* - * AE / AGC, use 2 frames delay to make sure that the exposure and - * the gain set have applied to the camera sensor. - * \todo This could be handled better with DelayedControls. - */ - if (ignoreUpdates_ > 0) { - --ignoreUpdates_; - return; - } - /* * Calculate Mean Sample Value (MSV) according to formula from: * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf @@ -356,8 +344,6 @@ void IPASoftSimple::processStats(const uint32_t frame, ctrls.set(V4L2_CID_ANALOGUE_GAIN, static_cast(camHelper_ ? camHelper_->gainCode(again_) : again_)); - ignoreUpdates_ = 2; - setSensorControls.emit(ctrls); LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index ed50ab52..9a778994 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -32,6 +32,7 @@ #include "libcamera/internal/camera.h" #include "libcamera/internal/camera_sensor.h" #include "libcamera/internal/converter.h" +#include "libcamera/internal/delayed_controls.h" #include "libcamera/internal/device_enumerator.h" #include "libcamera/internal/media_device.h" #include "libcamera/internal/pipeline_handler.h" @@ -278,6 +279,8 @@ public: std::vector configs_; std::map> formats_; + std::unique_ptr delayedCtrls_; + std::vector> conversionBuffers_; std::queue> conversionQueue_; bool useConversion_; @@ -900,14 +903,13 @@ void SimpleCameraData::conversionOutputDone(FrameBuffer *buffer) void SimpleCameraData::ispStatsReady(uint32_t frame, uint32_t bufferId) { - /* \todo Use the DelayedControls class */ swIsp_->processStats(frame, bufferId, - sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, - V4L2_CID_EXPOSURE })); + delayedCtrls_->get(frame)); } void SimpleCameraData::setSensorControls(const ControlList &sensorControls) { + delayedCtrls_->push(sensorControls); ControlList ctrls(sensorControls); sensor_->setControls(&ctrls); } @@ -1288,6 +1290,16 @@ int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c) if (outputCfgs.empty()) return 0; + std::unordered_map params = { + { V4L2_CID_ANALOGUE_GAIN, { 2, false } }, + { V4L2_CID_EXPOSURE, { 2, false } }, + }; + data->delayedCtrls_ = + std::make_unique(data->sensor_->device(), + params); + data->video_->frameStart.connect(data->delayedCtrls_.get(), + &DelayedControls::applyControls); + StreamConfiguration inputCfg; inputCfg.pixelFormat = pipeConfig->captureFormat; inputCfg.size = pipeConfig->captureSize; diff --git a/src/libcamera/software_isp/TODO b/src/libcamera/software_isp/TODO index 9978afc0..8eeede46 100644 --- a/src/libcamera/software_isp/TODO +++ b/src/libcamera/software_isp/TODO @@ -209,35 +209,6 @@ At some point, yes. --- -11. Improve handling the sensor controls which take effect with a delay - -> void IPASoftSimple::processStats(const ControlList &sensorControls) -> { -> ... -> /* -> * AE / AGC, use 2 frames delay to make sure that the exposure and -> * the gain set have applied to the camera sensor. -> */ -> if (ignore_updates_ > 0) { -> --ignore_updates_; -> return; -> } - -This could be handled better with DelayedControls. - ---- - -12. Use DelayedControls class in ispStatsReady() - -> void SimpleCameraData::ispStatsReady() -> { -> swIsp_->processStats(sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, -> V4L2_CID_EXPOSURE })); - -You should use the DelayedControls class. - ---- - 13. Improve black level and colour gains application I think the black level should eventually be moved before debayering, and From patchwork Thu Aug 15 08:16:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20930 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E983CC32A9 for ; Thu, 15 Aug 2024 08:17:42 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 85E9C633DC; Thu, 15 Aug 2024 10:17:42 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="MEuK8zjt"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DE03E633C8 for ; Thu, 15 Aug 2024 10:17:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709859; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=m5zpvJwt194Ad3QslEhu0qF25oheKPdoKb1F8d0Jjkg=; b=MEuK8zjtrUc6NstB9e/qJ7DYh8uXIq6uWqpvEJmnQoeVoMvtSVkLDESINDf2YhbSikP1f6 BAj4wp0ASZzlbX9faV1D9I7/DhmmzPPsj2RbllOuiDmE937mpiWTBSB/LWho7HFscTQZ98 o02C5sOqcp88MYeM3qDN42GOv+ftJQA= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-172-MEyvTbOCN9yiY4n-eXdv3A-1; Thu, 15 Aug 2024 04:17:33 -0400 X-MC-Unique: MEyvTbOCN9yiY4n-eXdv3A-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 27FFA19560AB; Thu, 15 Aug 2024 08:17:32 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id B810719560AE; Thu, 15 Aug 2024 08:17:29 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 17/18] libcamera: software_isp: Move exposure+gain to an algorithm module Date: Thu, 15 Aug 2024 10:16:21 +0200 Message-ID: <20240815081622.174210-18-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This is the last step to fully convert software ISP to Algorithm-based processing. The newly introduced frameContext.sensor properties are set, and the updated code moved, before calling Algorithm::process() to have the values up-to-date in stats processing. Resolves software ISP TODO #10. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/agc.cpp | 139 ++++++++++++++++++++++ src/ipa/simple/algorithms/agc.h | 36 ++++++ src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/data/uncalibrated.yaml | 1 + src/ipa/simple/ipa_context.cpp | 11 ++ src/ipa/simple/ipa_context.h | 17 +++ src/ipa/simple/soft_simple.cpp | 159 +++++--------------------- src/libcamera/software_isp/TODO | 10 -- 8 files changed, 235 insertions(+), 139 deletions(-) create mode 100644 src/ipa/simple/algorithms/agc.cpp create mode 100644 src/ipa/simple/algorithms/agc.h diff --git a/src/ipa/simple/algorithms/agc.cpp b/src/ipa/simple/algorithms/agc.cpp new file mode 100644 index 00000000..821b2971 --- /dev/null +++ b/src/ipa/simple/algorithms/agc.cpp @@ -0,0 +1,139 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Exposure and gain + */ + +#include "agc.h" + +#include + +#include + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoftExposure) + +namespace ipa::soft::algorithms { + +/* + * The number of bins to use for the optimal exposure calculations. + */ +static constexpr unsigned int kExposureBinsCount = 5; + +/* + * The exposure is optimal when the mean sample value of the histogram is + * in the middle of the range. + */ +static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; + +/* + * The below value implements the hysteresis for the exposure adjustment. + * It is small enough to have the exposure close to the optimal, and is big + * enough to prevent the exposure from wobbling around the optimal value. + */ +static constexpr float kExposureSatisfactory = 0.2; + +Agc::Agc() +{ +} + +void Agc::updateExposure(IPAContext &context, double exposureMSV) +{ + /* + * kExpDenominator of 10 gives ~10% increment/decrement; + * kExpDenominator of 5 - about ~20% + */ + static constexpr uint8_t kExpDenominator = 10; + static constexpr uint8_t kExpNumeratorUp = kExpDenominator + 1; + static constexpr uint8_t kExpNumeratorDown = kExpDenominator - 1; + + double next; + int32_t &exposure = context.activeState.agc.exposure; + double &again = context.activeState.agc.again; + + if (exposureMSV < kExposureOptimal - kExposureSatisfactory) { + next = exposure * kExpNumeratorUp / kExpDenominator; + if (next - exposure < 1) + exposure += 1; + else + exposure = next; + if (exposure >= context.configuration.agc.exposureMax) { + next = again * kExpNumeratorUp / kExpDenominator; + if (next - again < context.configuration.agc.againMinStep) + again += context.configuration.agc.againMinStep; + else + again = next; + } + } + + if (exposureMSV > kExposureOptimal + kExposureSatisfactory) { + if (exposure == context.configuration.agc.exposureMax && + again > context.configuration.agc.againMin) { + next = again * kExpNumeratorDown / kExpDenominator; + if (again - next < context.configuration.agc.againMinStep) + again -= context.configuration.agc.againMinStep; + else + again = next; + } else { + next = exposure * kExpNumeratorDown / kExpDenominator; + if (exposure - next < 1) + exposure -= 1; + else + exposure = next; + } + } + + exposure = std::clamp(exposure, context.configuration.agc.exposureMin, + context.configuration.agc.exposureMax); + again = std::clamp(again, context.configuration.agc.againMin, + context.configuration.agc.againMax); + + LOG(IPASoftExposure, Debug) + << "exposureMSV " << exposureMSV + << " exp " << exposure << " again " << again; +} + +void Agc::process(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + const SwIspStats *stats, + [[maybe_unused]] ControlList &metadata) +{ + /* + * Calculate Mean Sample Value (MSV) according to formula from: + * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf + */ + const auto &histogram = stats->yHistogram; + const unsigned int blackLevelHistIdx = + context.activeState.black.level * SwIspStats::kYHistogramSize; + const unsigned int histogramSize = + SwIspStats::kYHistogramSize - blackLevelHistIdx; + const unsigned int yHistValsPerBin = histogramSize / kExposureBinsCount; + const unsigned int yHistValsPerBinMod = + histogramSize / (histogramSize % kExposureBinsCount + 1); + int exposureBins[kExposureBinsCount] = {}; + unsigned int denom = 0; + unsigned int num = 0; + + for (unsigned int i = 0; i < histogramSize; i++) { + unsigned int idx = (i - (i / yHistValsPerBinMod)) / yHistValsPerBin; + exposureBins[idx] += histogram[blackLevelHistIdx + i]; + } + + for (unsigned int i = 0; i < kExposureBinsCount; i++) { + LOG(IPASoftExposure, Debug) << i << ": " << exposureBins[i]; + denom += exposureBins[i]; + num += exposureBins[i] * (i + 1); + } + + float exposureMSV = (denom == 0 ? 0 : static_cast(num) / denom); + updateExposure(context, exposureMSV); +} + +REGISTER_IPA_ALGORITHM(Agc, "Agc") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/agc.h b/src/ipa/simple/algorithms/agc.h new file mode 100644 index 00000000..a4aa656c --- /dev/null +++ b/src/ipa/simple/algorithms/agc.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Exposure and gain + */ + +#pragma once + +#include "libcamera/internal/software_isp/swisp_stats.h" + +#include "algorithm.h" +#include "ipa_context.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class Agc : public Algorithm +{ +public: + Agc(); + ~Agc() = default; + + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const SwIspStats *stats, + ControlList &metadata) override; + +private: + void updateExposure(IPAContext &context, double exposureMSV); +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index de50fe6d..3afee4f9 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -1,6 +1,7 @@ # SPDX-License-Identifier: CC0-1.0 soft_simple_ipa_algorithms = files([ + 'agc.cpp', 'blc.cpp', 'gamma.cpp', 'awb.cpp', diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index 3ca0a55f..a62a6e31 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -6,4 +6,5 @@ algorithms: - BlackLevel: - Gamma: - Awb: + - Agc: ... diff --git a/src/ipa/simple/ipa_context.cpp b/src/ipa/simple/ipa_context.cpp index 5fa492cb..3f94bbeb 100644 --- a/src/ipa/simple/ipa_context.cpp +++ b/src/ipa/simple/ipa_context.cpp @@ -77,6 +77,17 @@ namespace libcamera::ipa::soft { * \brief Gain of blue color */ +/** + * \var IPAActiveState::agc + * \brief Context for the AGC algorithm + * + * \var IPAActiveState::agc.exposure + * \brief Current exposure value + * + * \var IPAActiveState::agc.again + * \brief Current analog gain value + */ + /** * \var IPAActiveState::gamma * \brief Context for gamma in the Colors algorithm diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index 28a02475..0e2096f8 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -9,6 +9,8 @@ #include +#include + #include namespace libcamera { @@ -17,6 +19,13 @@ namespace ipa::soft { struct IPASessionConfiguration { float gamma; + struct { + int32_t exposureMin, exposureMax; + double againMin, againMax, againMinStep; + } agc; + struct { + double level; + } black; }; struct IPAActiveState { @@ -28,6 +37,10 @@ struct IPAActiveState { double green; double blue; } gains; + struct { + int32_t exposure; + double again; + } agc; static constexpr unsigned int kGammaLookupSize = 1024; struct { std::array gammaTable; @@ -36,6 +49,10 @@ struct IPAActiveState { }; struct IPAFrameContext : public FrameContext { + struct { + uint32_t exposure; + double gain; + } sensor; }; struct IPAContext { diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 0e47b9c5..aef5f6e5 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -34,23 +34,6 @@ LOG_DEFINE_CATEGORY(IPASoft) namespace ipa::soft { -/* - * The number of bins to use for the optimal exposure calculations. - */ -static constexpr unsigned int kExposureBinsCount = 5; - -/* - * The exposure is optimal when the mean sample value of the histogram is - * in the middle of the range. - */ -static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; - -/* - * The below value implements the hysteresis for the exposure adjustment. - * It is small enough to have the exposure close to the optimal, and is big - * enough to prevent the exposure from wobbling around the optimal value. - */ -static constexpr float kExposureSatisfactory = 0.2; /* Maximum number of frame contexts to be held */ static constexpr uint32_t kMaxFrameContexts = 16; @@ -58,8 +41,7 @@ class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() - : params_(nullptr), stats_(nullptr), - context_({ {}, {}, { kMaxFrameContexts } }) + : context_({ {}, {}, { kMaxFrameContexts } }) { } @@ -92,11 +74,6 @@ private: /* Local parameter storage */ struct IPAContext context_; - - int32_t exposureMin_, exposureMax_; - int32_t exposure_; - double againMin_, againMax_, againMinStep_; - double again_; }; IPASoftSimple::~IPASoftSimple() @@ -207,20 +184,23 @@ int IPASoftSimple::configure(const IPAConfigInfo &configInfo) const ControlInfo &exposureInfo = sensorInfoMap_.find(V4L2_CID_EXPOSURE)->second; const ControlInfo &gainInfo = sensorInfoMap_.find(V4L2_CID_ANALOGUE_GAIN)->second; - exposureMin_ = exposureInfo.min().get(); - exposureMax_ = exposureInfo.max().get(); - if (!exposureMin_) { + context_.configuration.agc.exposureMin = exposureInfo.min().get(); + context_.configuration.agc.exposureMax = exposureInfo.max().get(); + if (!context_.configuration.agc.exposureMin) { LOG(IPASoft, Warning) << "Minimum exposure is zero, that can't be linear"; - exposureMin_ = 1; + context_.configuration.agc.exposureMin = 1; } int32_t againMin = gainInfo.min().get(); int32_t againMax = gainInfo.max().get(); if (camHelper_) { - againMin_ = camHelper_->gain(againMin); - againMax_ = camHelper_->gain(againMax); - againMinStep_ = (againMax_ - againMin_) / 100.0; + context_.configuration.agc.againMin = camHelper_->gain(againMin); + context_.configuration.agc.againMax = camHelper_->gain(againMax); + context_.configuration.agc.againMinStep = + (context_.configuration.agc.againMax - + context_.configuration.agc.againMin) / + 100.0; } else { /* * The camera sensor gain (g) is usually not equal to the value written @@ -232,13 +212,14 @@ int IPASoftSimple::configure(const IPAConfigInfo &configInfo) * the AGC algorithm (abrupt near one edge, and very small near the * other) we limit the range of the gain values used. */ - againMax_ = againMax; + context_.configuration.agc.againMax = againMax; if (!againMin) { LOG(IPASoft, Warning) << "Minimum gain is zero, that can't be linear"; - againMin_ = std::min(100, againMin / 2 + againMax / 2); + context_.configuration.agc.againMin = + std::min(100, againMin / 2 + againMax / 2); } - againMinStep_ = 1.0; + context_.configuration.agc.againMinStep = 1.0; } for (auto const &algo : algorithms()) { @@ -247,9 +228,12 @@ int IPASoftSimple::configure(const IPAConfigInfo &configInfo) return ret; } - LOG(IPASoft, Info) << "Exposure " << exposureMin_ << "-" << exposureMax_ - << ", gain " << againMin_ << "-" << againMax_ - << " (" << againMinStep_ << ")"; + LOG(IPASoft, Info) + << "Exposure " << context_.configuration.agc.exposureMin << "-" + << context_.configuration.agc.exposureMax + << ", gain " << context_.configuration.agc.againMin << "-" + << context_.configuration.agc.againMax + << " (" << context_.configuration.agc.againMinStep << ")"; return 0; } @@ -284,6 +268,12 @@ void IPASoftSimple::processStats(const uint32_t frame, const ControlList &sensorControls) { IPAFrameContext &frameContext = context_.frameContexts.get(frame); + + frameContext.sensor.exposure = + sensorControls.get(V4L2_CID_EXPOSURE).get(); + int32_t again = sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get(); + frameContext.sensor.gain = camHelper_ ? camHelper_->gain(again) : again; + /* * Software ISP currently does not produce any metadata. Use an empty * ControlList for now. @@ -294,37 +284,6 @@ void IPASoftSimple::processStats(const uint32_t frame, for (auto const &algo : algorithms()) algo->process(context_, frame, frameContext, stats_, metadata); - /* \todo Switch to the libipa/algorithm.h API someday. */ - - /* - * Calculate Mean Sample Value (MSV) according to formula from: - * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf - */ - const uint8_t blackLevel = context_.activeState.black.level; - const unsigned int blackLevelHistIdx = - blackLevel / (256 / SwIspStats::kYHistogramSize); - const unsigned int histogramSize = - SwIspStats::kYHistogramSize - blackLevelHistIdx; - const unsigned int yHistValsPerBin = histogramSize / kExposureBinsCount; - const unsigned int yHistValsPerBinMod = - histogramSize / (histogramSize % kExposureBinsCount + 1); - int exposureBins[kExposureBinsCount] = {}; - unsigned int denom = 0; - unsigned int num = 0; - - for (unsigned int i = 0; i < histogramSize; i++) { - unsigned int idx = (i - (i / yHistValsPerBinMod)) / yHistValsPerBin; - exposureBins[idx] += stats_->yHistogram[blackLevelHistIdx + i]; - } - - for (unsigned int i = 0; i < kExposureBinsCount; i++) { - LOG(IPASoft, Debug) << i << ": " << exposureBins[i]; - denom += exposureBins[i]; - num += exposureBins[i] * (i + 1); - } - - float exposureMSV = static_cast(num) / denom; - /* Sanity check */ if (!sensorControls.contains(V4L2_CID_EXPOSURE) || !sensorControls.contains(V4L2_CID_ANALOGUE_GAIN)) { @@ -332,72 +291,14 @@ void IPASoftSimple::processStats(const uint32_t frame, return; } - exposure_ = sensorControls.get(V4L2_CID_EXPOSURE).get(); - int32_t again = sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get(); - again_ = camHelper_ ? camHelper_->gain(again) : again; - - updateExposure(exposureMSV); - ControlList ctrls(sensorInfoMap_); - ctrls.set(V4L2_CID_EXPOSURE, exposure_); + auto &againNew = context_.activeState.agc.again; + ctrls.set(V4L2_CID_EXPOSURE, context_.activeState.agc.exposure); ctrls.set(V4L2_CID_ANALOGUE_GAIN, - static_cast(camHelper_ ? camHelper_->gainCode(again_) : again_)); + static_cast(camHelper_ ? camHelper_->gainCode(againNew) : againNew)); setSensorControls.emit(ctrls); - - LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV - << " exp " << exposure_ << " again " << again_ - << " gain R/B " << context_.activeState.gains.red - << "/" << context_.activeState.gains.blue - << " black level " << static_cast(blackLevel); -} - -void IPASoftSimple::updateExposure(double exposureMSV) -{ - /* - * kExpDenominator of 10 gives ~10% increment/decrement; - * kExpDenominator of 5 - about ~20% - */ - static constexpr uint8_t kExpDenominator = 10; - static constexpr uint8_t kExpNumeratorUp = kExpDenominator + 1; - static constexpr uint8_t kExpNumeratorDown = kExpDenominator - 1; - - double next; - - if (exposureMSV < kExposureOptimal - kExposureSatisfactory) { - next = exposure_ * kExpNumeratorUp / kExpDenominator; - if (next - exposure_ < 1) - exposure_ += 1; - else - exposure_ = next; - if (exposure_ >= exposureMax_) { - next = again_ * kExpNumeratorUp / kExpDenominator; - if (next - again_ < againMinStep_) - again_ += againMinStep_; - else - again_ = next; - } - } - - if (exposureMSV > kExposureOptimal + kExposureSatisfactory) { - if (exposure_ == exposureMax_ && again_ > againMin_) { - next = again_ * kExpNumeratorDown / kExpDenominator; - if (again_ - next < againMinStep_) - again_ -= againMinStep_; - else - again_ = next; - } else { - next = exposure_ * kExpNumeratorDown / kExpDenominator; - if (exposure_ - next < 1) - exposure_ -= 1; - else - exposure_ = next; - } - } - - exposure_ = std::clamp(exposure_, exposureMin_, exposureMax_); - again_ = std::clamp(again_, againMin_, againMax_); } std::string IPASoftSimple::logPrefix() const diff --git a/src/libcamera/software_isp/TODO b/src/libcamera/software_isp/TODO index 8eeede46..a50db668 100644 --- a/src/libcamera/software_isp/TODO +++ b/src/libcamera/software_isp/TODO @@ -199,16 +199,6 @@ Yes, because, well... all the other IPAs were doing that... --- -10. Switch to libipa/algorithm.h API in processStats - ->> void IPASoftSimple::processStats(const ControlList &sensorControls) ->> -> Do you envision switching to the libipa/algorithm.h API at some point ? - -At some point, yes. - ---- - 13. Improve black level and colour gains application I think the black level should eventually be moved before debayering, and From patchwork Thu Aug 15 08:16:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20931 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 39D13C32D8 for ; Thu, 15 Aug 2024 08:17:44 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AC1B8633D8; Thu, 15 Aug 2024 10:17:43 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="B63RWCtY"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 234D6633D4 for ; Thu, 15 Aug 2024 10:17:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723709859; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=G6Rb7ZNQHHJHzus5nxcQH64kx4ElvfE8MPFWA9uBZ+E=; b=B63RWCtYQ/lhc4toVsy7XQabFF0qe3f6ZwnUDjHY7Q2drCBoHohkYIegsh5OL5ydFGGTxk ztTjs0bVZ1aYz1L9Owwh+b3t+XeHXgTIJdCkPqPsjRryJ9+/1CbeuO1azfJxxPh8kCfYji Py2ecpQQSPPo02QBWf8RseRtyEQ4PyI= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-308-sJGT8pB1M-iRG174z0TLeg-1; Thu, 15 Aug 2024 04:17:36 -0400 X-MC-Unique: sJGT8pB1M-iRG174z0TLeg-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 4D4081955D4D; Thu, 15 Aug 2024 08:17:34 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.225.128]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 868DA19560A3; Thu, 15 Aug 2024 08:17:32 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain , Laurent Pinchart , Daniel Scally Subject: [PATCH v4 18/18] libcamera: software_isp: Update black level only on exposure changes Date: Thu, 15 Aug 2024 10:16:22 +0200 Message-ID: <20240815081622.174210-19-mzamazal@redhat.com> In-Reply-To: <20240815081622.174210-1-mzamazal@redhat.com> References: <20240815081622.174210-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The black level is likely to get updated, if ever, only after exposure or gain changes. Don't compute its possible updates if exposure and gain are unchanged. It's probably not worth trying to implement something more sophisticated. Better to spend the effort on supporting tuning files. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/blc.cpp | 9 ++++++++- src/ipa/simple/algorithms/blc.h | 4 ++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/src/ipa/simple/algorithms/blc.cpp b/src/ipa/simple/algorithms/blc.cpp index ab7e7dd9..d01c2c14 100644 --- a/src/ipa/simple/algorithms/blc.cpp +++ b/src/ipa/simple/algorithms/blc.cpp @@ -30,10 +30,15 @@ int BlackLevel::init(IPAContext &context, void BlackLevel::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, - [[maybe_unused]] IPAFrameContext &frameContext, + IPAFrameContext &frameContext, const SwIspStats *stats, [[maybe_unused]] ControlList &metadata) { + if (frameContext.sensor.exposure == exposure_ && + frameContext.sensor.gain == gain_) { + return; + } + const SwIspStats::Histogram &histogram = stats->yHistogram; /* @@ -54,6 +59,8 @@ void BlackLevel::process(IPAContext &context, if (seen >= pixelThreshold) { context.activeState.black.level = static_cast(i) / SwIspStats::kYHistogramSize; + exposure_ = frameContext.sensor.exposure; + gain_ = frameContext.sensor.gain; LOG(IPASoftBL, Debug) << "Auto-set black level: " << i << "/" << SwIspStats::kYHistogramSize diff --git a/src/ipa/simple/algorithms/blc.h b/src/ipa/simple/algorithms/blc.h index 2f73db52..9b9daab1 100644 --- a/src/ipa/simple/algorithms/blc.h +++ b/src/ipa/simple/algorithms/blc.h @@ -30,6 +30,10 @@ public: IPAFrameContext &frameContext, const SwIspStats *stats, ControlList &metadata) override; + +private: + uint32_t exposure_; + double gain_; }; } /* namespace ipa::soft::algorithms */