From patchwork Wed Jun 26 07:20:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20394 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E2A1EBD87C for ; Wed, 26 Jun 2024 07:21:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A07D5654B0; Wed, 26 Jun 2024 09:21:24 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="P13oi8bp"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D6882619E5 for ; Wed, 26 Jun 2024 09:21:23 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386482; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ANVZ7pJaKCh5PeIWi7zZlAcZo8vYlqlc8BAr2fJUD7A=; b=P13oi8bpp+t1dNiuePAYLDDtamN7zU1g3T1gfaOtxp09sTXmajtxN8GIJN5knMejYC/I1K 6tWud35jt1If4g0L24qtifgL0ulAlKEPnNfT3ECOQRDP2Ftc9Bdjy4fviImlyb26MEJdrZ G8p9VV97Ll+racjWR9dC8VFUSs+QC4E= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-423-gdpvgPPwP9KY8S8NByD0qA-1; Wed, 26 Jun 2024 03:21:20 -0400 X-MC-Unique: gdpvgPPwP9KY8S8NByD0qA-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 2AEFA19560BE for ; Wed, 26 Jun 2024 07:21:20 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 281223000601; Wed, 26 Jun 2024 07:21:18 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 01/19] libcamera: software_isp: Remove superfluous includes Date: Wed, 26 Jun 2024 09:20:42 +0200 Message-ID: <20240626072100.55497-2-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- src/libcamera/software_isp/software_isp.cpp | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 20fb6f48..08a1afcc 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -16,10 +16,7 @@ #include #include -#include "libcamera/internal/bayer_format.h" -#include "libcamera/internal/framebuffer.h" #include "libcamera/internal/ipa_manager.h" -#include "libcamera/internal/mapped_framebuffer.h" #include "libcamera/internal/software_isp/debayer_params.h" #include "debayer_cpu.h" From patchwork Wed Jun 26 07:20:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20395 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D2F58BD87C for ; Wed, 26 Jun 2024 07:21:27 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8D120654B3; Wed, 26 Jun 2024 09:21:27 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="a4cMuETQ"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 37432654A9 for ; Wed, 26 Jun 2024 09:21:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386484; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Wo9wL35931Dv+IhTa8G6NKhtJl/yG2SRo7LBSfEpjkg=; b=a4cMuETQNceaDQ6VdxqnpVjqbXZQcBrgIu7KytwprepQHCPZwKFnhSViIN2aMmMUdOPGrL CQvMMq8MMvuPTZG0aZWAEk2/c5hn8517FoxHNctsV1Dzp07oa5tPKn6SgaZbTKDNbh7/H5 J+K2kIMFJtH/LN3KFIw1OO1XvAGLId8= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-78-yvLbj4kZPG6pm0hkQ74ahA-1; Wed, 26 Jun 2024 03:21:22 -0400 X-MC-Unique: yvLbj4kZPG6pm0hkQ74ahA-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 951B619560B4 for ; Wed, 26 Jun 2024 07:21:21 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 937103000221; Wed, 26 Jun 2024 07:21:20 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 02/19] libcamera: software_isp: Move BlackLevel to libcamera::ipa::soft Date: Wed, 26 Jun 2024 09:20:43 +0200 Message-ID: <20240626072100.55497-3-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" To be in the same namespace as the other software ISP IPA stuff. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- src/ipa/simple/black_level.cpp | 5 +++++ src/ipa/simple/black_level.h | 4 ++++ 2 files changed, 9 insertions(+) diff --git a/src/ipa/simple/black_level.cpp b/src/ipa/simple/black_level.cpp index cc490eb5..37e0109c 100644 --- a/src/ipa/simple/black_level.cpp +++ b/src/ipa/simple/black_level.cpp @@ -15,6 +15,8 @@ namespace libcamera { LOG_DEFINE_CATEGORY(IPASoftBL) +namespace ipa::soft { + /** * \class BlackLevel * \brief Object providing black point level for software ISP @@ -85,4 +87,7 @@ void BlackLevel::update(SwIspStats::Histogram &yHistogram) } }; } + +} /* namespace ipa::soft */ + } /* namespace libcamera */ diff --git a/src/ipa/simple/black_level.h b/src/ipa/simple/black_level.h index 5e032f9f..a04230c9 100644 --- a/src/ipa/simple/black_level.h +++ b/src/ipa/simple/black_level.h @@ -14,6 +14,8 @@ namespace libcamera { +namespace ipa::soft { + class BlackLevel { public: @@ -26,4 +28,6 @@ private: bool blackLevelSet_; }; +} /* namespace ipa::soft */ + } /* namespace libcamera */ From patchwork Wed Jun 26 07:20:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20396 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D16B3BD87C for ; Wed, 26 Jun 2024 07:21:30 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 65D39654B8; Wed, 26 Jun 2024 09:21:30 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="S+eFcCwE"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 52108654B0 for ; Wed, 26 Jun 2024 09:21:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386486; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a4MWQXFMH1QP3z42aZ8wtB5motMpnCP4rSwWYTsETDs=; b=S+eFcCwEG9TV7pnEujpCVk2xqTMpqaDJUgV0PZEI3/NT/74+twYRkMJe8eC3Y4INSj/vMC 46Dme/URMalTgWfr0XtuCwc0xEvARbjClImrWYBTHRcOFucd8axQU+hGEmROt6IyagYv1j tnTfbp3suvxgvU3mOgOU/XJbJMi9u8Q= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-177-jh2pd0-bO_Oie2hwR0aMXA-1; Wed, 26 Jun 2024 03:21:24 -0400 X-MC-Unique: jh2pd0-bO_Oie2hwR0aMXA-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A6D7B1956050 for ; Wed, 26 Jun 2024 07:21:23 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 09E6F300021A; Wed, 26 Jun 2024 07:21:21 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 03/19] libcamera: simple: Fix a typo in a doc string Date: Wed, 26 Jun 2024 09:20:44 +0200 Message-ID: <20240626072100.55497-4-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- src/libcamera/pipeline/simple/simple.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index eb36578e..7abe1d1d 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -163,7 +163,7 @@ LOG_DEFINE_CATEGORY(SimplePipeline) * handler has no a priori knowledge of. The pipeline handler thus implements a * heuristic to handle sharing of hardware resources in a generic fashion. * - * Two cameras are considered to be mutually exclusive if their share common + * Two cameras are considered to be mutually exclusive if they share common * pads along the pipeline from the camera sensor to the video node. An entity * can thus be used concurrently by multiple cameras, as long as pads are * distinct. From patchwork Wed Jun 26 07:20:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20397 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id BFC61BD87C for ; Wed, 26 Jun 2024 07:21:32 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 74913654B5; Wed, 26 Jun 2024 09:21:32 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="irdm7sXj"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9F547654B0 for ; Wed, 26 Jun 2024 09:21:29 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386488; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FY4YGKe2XLjYu51/Fvh118lP9SnKKS9jWt+LEw5TySg=; b=irdm7sXjKvrYcNNIeoo0EFkB86A5roctGw7CMsc+j8geB8E1/GGty2j+BpW4vF+PxgNe49 uHjlM0573Qr6i4J3eDLKuVMCguXvplpD3udQQpQmtZbloPwB6YUD+i9A9WbAsO6FqWQuqC 2zbNUNubzq9GijbZsyeCO3H/m//PZCw= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-365-W937HiMRND2tt_0gPgA3KQ-1; Wed, 26 Jun 2024 03:21:26 -0400 X-MC-Unique: W937HiMRND2tt_0gPgA3KQ-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 36C5E19560B0 for ; Wed, 26 Jun 2024 07:21:25 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 10939300021A; Wed, 26 Jun 2024 07:21:23 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 04/19] libcamera: software_isp: Define skeletons for IPA refactoring Date: Wed, 26 Jun 2024 09:20:45 +0200 Message-ID: <20240626072100.55497-5-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Software ISP image processing algorithms are currently defined in a simplified way, different from other libcamera pipelines. This is not good for several reasons: - It makes the software ISP code harder to understand due to its different structuring. - Adding more algorithms may make the code harder to understand generally. - Mass libcamera code changes may not be easily applicable to software ISP. - Algorithm sharing with other pipelines is not easily possible. This patch introduces basic software ISP IPA skeletons structured similarly to the other pipelines. The newly added files are currently not used or compiled and the general skeleton structures don't contain anything particular. It is just a preparation step for a larger refactoring and the code will be actually used and extended as needed in followup patches. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/algorithm.h | 22 +++++++++++++++++ src/ipa/simple/ipa_context.h | 34 +++++++++++++++++++++++++++ src/ipa/simple/module.h | 30 +++++++++++++++++++++++ 3 files changed, 86 insertions(+) create mode 100644 src/ipa/simple/algorithms/algorithm.h create mode 100644 src/ipa/simple/ipa_context.h create mode 100644 src/ipa/simple/module.h diff --git a/src/ipa/simple/algorithms/algorithm.h b/src/ipa/simple/algorithms/algorithm.h new file mode 100644 index 00000000..41f63170 --- /dev/null +++ b/src/ipa/simple/algorithms/algorithm.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Red Hat, Inc. + * + * Software ISP control algorithm interface + */ + +#pragma once + +#include + +#include "module.h" + +namespace libcamera { + +namespace ipa::soft { + +using Algorithm = libcamera::ipa::Algorithm; + +} /* namespace ipa::soft */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h new file mode 100644 index 00000000..bc1235b6 --- /dev/null +++ b/src/ipa/simple/ipa_context.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Red Hat, Inc. + * + * Simple pipeline IPA Context + * + */ + +#pragma once + +#include + +namespace libcamera { + +namespace ipa::soft { + +struct IPASessionConfiguration { +}; + +struct IPAActiveState { +}; + +struct IPAFrameContext : public FrameContext { +}; + +struct IPAContext { + IPASessionConfiguration configuration; + IPAActiveState activeState; + FCQueue frameContexts; +}; + +} /* namespace ipa::soft */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/module.h b/src/ipa/simple/module.h new file mode 100644 index 00000000..21328ecd --- /dev/null +++ b/src/ipa/simple/module.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Red Hat, Inc. + * + * Software ISP IPA Module + */ + +#pragma once + +#include + +#include + +#include "libcamera/internal/software_isp/debayer_params.h" +#include "libcamera/internal/software_isp/swisp_stats.h" + +#include + +#include "ipa_context.h" + +namespace libcamera { + +namespace ipa::soft { + +using Module = ipa::Module; + +} /* namespace ipa::soft */ + +} /* namespace libcamera */ From patchwork Wed Jun 26 07:20:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20406 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 51176BD87C for ; Wed, 26 Jun 2024 07:21:49 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DB3E2654C6; Wed, 26 Jun 2024 09:21:48 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="BSZqAIjF"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 11D15654BC for ; Wed, 26 Jun 2024 09:21:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386499; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VvEyK/qVqGxt+TCu6gXfp62XKhC1qzqSprwy9WaFUQY=; b=BSZqAIjFHtTe4siSHAaYpvRPaNCrwOUk6MRa78rQXQC0Zvmk3BgLJ4DKow4xu6DGmxC4sC 64OCOQX4rLnENkiUfXiSkcHvQLSt14kq+n+qL4pLHJ1PAPSMRaIiqRvfg2nVBi41ZCV5oC 87W+zC2mnWiF/hSQhz+djDTtOBqWDFo= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-477-3uoBxFwoN_eD7glJaq91Vw-1; Wed, 26 Jun 2024 03:21:27 -0400 X-MC-Unique: 3uoBxFwoN_eD7glJaq91Vw-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 6BFFC19560AB for ; Wed, 26 Jun 2024 07:21:26 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 76086300021A; Wed, 26 Jun 2024 07:21:25 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 05/19] libcamera: software_isp: Let IPASoftSimple inherit Module Date: Wed, 26 Jun 2024 09:20:46 +0200 Message-ID: <20240626072100.55497-6-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The Module class is a base class for All IPA modules. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- src/ipa/simple/soft_simple.cpp | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index b7746ce0..04f283c8 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -30,6 +30,7 @@ #include "libipa/camera_sensor_helper.h" #include "black_level.h" +#include "module.h" namespace libcamera { LOG_DEFINE_CATEGORY(IPASoft) @@ -54,7 +55,7 @@ static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; */ static constexpr float kExposureSatisfactory = 0.2; -class IPASoftSimple : public ipa::soft::IPASoftInterface +class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() @@ -76,6 +77,9 @@ public: void processStats(const ControlList &sensorControls) override; +protected: + std::string logPrefix() const override; + private: void updateExposure(double exposureMSV); @@ -421,6 +425,11 @@ void IPASoftSimple::updateExposure(double exposureMSV) again_ = std::clamp(again_, againMin_, againMax_); } +std::string IPASoftSimple::logPrefix() const +{ + return "soft"; +} + } /* namespace ipa::soft */ /* From patchwork Wed Jun 26 07:20:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20402 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id CF546BD87C for ; Wed, 26 Jun 2024 07:21:42 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 54941654C6; Wed, 26 Jun 2024 09:21:42 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="fCf8P2df"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 757DF654C0 for ; Wed, 26 Jun 2024 09:21:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386495; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=D4pSq7tMpHcX1lCi3PtXJtcXS62NH6oVeljGEM+sOpk=; b=fCf8P2dfnKBOr6/YTWLaL0qv6G5wazwvkuRTmychloEbGq6Vbx0f0NNJ24lOZCX9BpKfpu wdq6O4bbyXc6cffmfLaoShJIDtn2QWfFJvWyYS07ZU9gev8W6ukqyN9BVT9hIiOIE/r1VH KOoLkj2Zhm3Eby0DY2aKmLPYVlBePQg= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-106-MDKwNRN7P4OKK0SELdrpfA-1; Wed, 26 Jun 2024 03:21:28 -0400 X-MC-Unique: MDKwNRN7P4OKK0SELdrpfA-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id C8D661955F0C for ; Wed, 26 Jun 2024 07:21:27 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id CEE3D300021A; Wed, 26 Jun 2024 07:21:26 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 06/19] libcamera: software_isp: Make stats frame and buffer aware Date: Wed, 26 Jun 2024 09:20:47 +0200 Message-ID: <20240626072100.55497-7-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds frame and bufferId arguments to stats related calls. Although the parameters are currently unused, because frame ids are not tracked and used and the stats buffer is passed around directly rather than being referred by its id, they bring the internal APIs closer to their counterparts in hardware pipelines. It serves as a preparation for followup patches that will introduce: - Frame number tracking in order to switch to DelayedControls (software ISP TODO #11 + #12). - A ring buffer for stats in order to improve passing the stats (software ISP TODO #2). Frame and buffer ids are unrelated for the given purposes but since they are passed together at the same places, the change is implemented as a single patch rather than two, basically the same, patches. Signed-off-by: Milan Zamazal --- .../libcamera/internal/software_isp/software_isp.h | 8 +++++--- include/libcamera/ipa/soft.mojom | 2 +- src/ipa/simple/soft_simple.cpp | 8 ++++++-- src/libcamera/pipeline/simple/simple.cpp | 11 +++++++---- src/libcamera/software_isp/debayer_cpu.cpp | 8 +++++++- src/libcamera/software_isp/software_isp.cpp | 11 +++++++---- src/libcamera/software_isp/swstats_cpu.cpp | 4 ++-- src/libcamera/software_isp/swstats_cpu.h | 4 ++-- 8 files changed, 37 insertions(+), 19 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index c5338c05..6a03b17f 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -65,7 +66,8 @@ public: int exportBuffers(unsigned int output, unsigned int count, std::vector> *buffers); - void processStats(const ControlList &sensorControls); + void processStats(const uint32_t frame, const uint32_t bufferId, + const ControlList &sensorControls); int start(); void stop(); @@ -77,13 +79,13 @@ public: Signal inputBufferReady; Signal outputBufferReady; - Signal<> ispStatsReady; + Signal ispStatsReady; Signal setSensorControls; private: void saveIspParams(); void setSensorCtrls(const ControlList &sensorControls); - void statsReady(); + void statsReady(uint32_t frame, uint32_t bufferId); void inputReady(FrameBuffer *input); void outputReady(FrameBuffer *output); diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index 3aa2066e..bd48ece9 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -19,7 +19,7 @@ interface IPASoftInterface { configure(libcamera.ControlInfoMap sensorCtrlInfoMap) => (int32 ret); - [async] processStats(libcamera.ControlList sensorControls); + [async] processStats(uint32 frame, uint32 bufferId, libcamera.ControlList sensorControls); }; interface IPASoftEventInterface { diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 04f283c8..f2cfb2dc 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -75,7 +75,8 @@ public: int start() override; void stop() override; - void processStats(const ControlList &sensorControls) override; + void processStats(const uint32_t frame, const uint32_t bufferId, + const ControlList &sensorControls) override; protected: std::string logPrefix() const override; @@ -249,7 +250,10 @@ void IPASoftSimple::stop() { } -void IPASoftSimple::processStats(const ControlList &sensorControls) +void IPASoftSimple::processStats( + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] const uint32_t bufferId, + const ControlList &sensorControls) { SwIspStats::Histogram histogram = stats_->yHistogram; if (ignoreUpdates_ > 0) diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 7abe1d1d..b1bf0d16 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -290,7 +291,7 @@ private: void conversionInputDone(FrameBuffer *buffer); void conversionOutputDone(FrameBuffer *buffer); - void ispStatsReady(); + void ispStatsReady(uint32_t frame, uint32_t bufferId); void setSensorControls(const ControlList &sensorControls); }; @@ -890,11 +891,13 @@ void SimpleCameraData::conversionOutputDone(FrameBuffer *buffer) pipe->completeRequest(request); } -void SimpleCameraData::ispStatsReady() +void SimpleCameraData::ispStatsReady(uint32_t frame, uint32_t bufferId) { /* \todo Use the DelayedControls class */ - swIsp_->processStats(sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, - V4L2_CID_EXPOSURE })); + swIsp_->processStats( + frame, bufferId, + sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, + V4L2_CID_EXPOSURE })); } void SimpleCameraData::setSensorControls(const ControlList &sensorControls) diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index f8d2677d..1575cedb 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -784,7 +784,13 @@ void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams } } - stats_->finishFrame(); + /* + * Frame and buffer ids are currently not used, so pass zeros as parameters. + * + * \todo Pass real values once frame is passed here and stats buffer passing + * is changed. + */ + stats_->finishFrame(0, 0); outputBufferReady.emit(output); inputBufferReady.emit(input); } diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 08a1afcc..3fc4a64b 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -156,15 +156,18 @@ SoftwareIsp::~SoftwareIsp() /** * \brief Process the statistics gathered + * \param[in] frame The frame number + * \param[in] bufferId ID of the statistics buffer * \param[in] sensorControls The sensor controls * * Requests the IPA to calculate new parameters for ISP and new control * values for the sensor. */ -void SoftwareIsp::processStats(const ControlList &sensorControls) +void SoftwareIsp::processStats(const uint32_t frame, const uint32_t bufferId, + const ControlList &sensorControls) { ASSERT(ipa_); - ipa_->processStats(sensorControls); + ipa_->processStats(frame, bufferId, sensorControls); } /** @@ -356,9 +359,9 @@ void SoftwareIsp::setSensorCtrls(const ControlList &sensorControls) setSensorControls.emit(sensorControls); } -void SoftwareIsp::statsReady() +void SoftwareIsp::statsReady(uint32_t frame, uint32_t bufferId) { - ispStatsReady.emit(); + ispStatsReady.emit(frame, bufferId); } void SoftwareIsp::inputReady(FrameBuffer *input) diff --git a/src/libcamera/software_isp/swstats_cpu.cpp b/src/libcamera/software_isp/swstats_cpu.cpp index 815c4d4f..a9b1f35a 100644 --- a/src/libcamera/software_isp/swstats_cpu.cpp +++ b/src/libcamera/software_isp/swstats_cpu.cpp @@ -314,10 +314,10 @@ void SwStatsCpu::startFrame(void) * * This may only be called after a successful setWindow() call. */ -void SwStatsCpu::finishFrame(void) +void SwStatsCpu::finishFrame(uint32_t frame, uint32_t bufferId) { *sharedStats_ = stats_; - statsReady.emit(); + statsReady.emit(frame, bufferId); } /** diff --git a/src/libcamera/software_isp/swstats_cpu.h b/src/libcamera/software_isp/swstats_cpu.h index 363e326f..26a2f462 100644 --- a/src/libcamera/software_isp/swstats_cpu.h +++ b/src/libcamera/software_isp/swstats_cpu.h @@ -41,7 +41,7 @@ public: int configure(const StreamConfiguration &inputCfg); void setWindow(const Rectangle &window); void startFrame(); - void finishFrame(); + void finishFrame(uint32_t frame, uint32_t bufferId); void processLine0(unsigned int y, const uint8_t *src[]) { @@ -61,7 +61,7 @@ public: (this->*stats2_)(src); } - Signal<> statsReady; + Signal statsReady; private: using statsProcessFn = void (SwStatsCpu::*)(const uint8_t *src[]); From patchwork Wed Jun 26 07:20:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20399 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 634EBBD87C for ; Wed, 26 Jun 2024 07:21:36 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E870A654B8; Wed, 26 Jun 2024 09:21:35 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="RIjkSYQP"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0FAA2654B8 for ; Wed, 26 Jun 2024 09:21:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386492; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ClCGua6EhluTWn6ibEtw5XIq40bvwek+ysI+C2VhWss=; b=RIjkSYQPdTe4T+Y9SXTmcagsA7QlqcVzg3Xt8ernGSuOm4K0QtQ5NC3wj3LXCWBL46vvOI mXr65kVNxjJj/2qBuXO0cNvxjB2QD4pZBp7sBKASFQVnXxiz/yLCvJk0e79hw4kyKduKhi OU4x2njqKdaZ0orEx/7NXfavgkuWKts= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-90-qQM7xtYPNRKQpO0E3p8MWQ-1; Wed, 26 Jun 2024 03:21:30 -0400 X-MC-Unique: qQM7xtYPNRKQpO0E3p8MWQ-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 342261956060 for ; Wed, 26 Jun 2024 07:21:29 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 33E0A3000225; Wed, 26 Jun 2024 07:21:27 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 07/19] libcamera: software_isp: Remove dots in Debayer::process docstring Date: Wed, 26 Jun 2024 09:20:48 +0200 Message-ID: <20240626072100.55497-8-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Signed-off-by: Milan Zamazal --- src/libcamera/software_isp/debayer.cpp | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/libcamera/software_isp/debayer.cpp b/src/libcamera/software_isp/debayer.cpp index f4a299d5..65fc4efc 100644 --- a/src/libcamera/software_isp/debayer.cpp +++ b/src/libcamera/software_isp/debayer.cpp @@ -94,11 +94,12 @@ Debayer::~Debayer() */ /** - * \fn void Debayer::process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) - * \brief Process the bayer data into the requested format. - * \param[in] input The input buffer. - * \param[in] output The output buffer. - * \param[in] params The parameters to be used in debayering. + * \fn void Debayer::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params) + * \brief Process the bayer data into the requested format + * \param[in] frame The frame number + * \param[in] input The input buffer + * \param[in] output The output buffer + * \param[in] params The parameters to be used in debayering * * \note DebayerParams is passed by value deliberately so that a copy is passed * when this is run in another thread by invokeMethod(). From patchwork Wed Jun 26 07:20:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20398 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 947C2BD87C for ; Wed, 26 Jun 2024 07:21:35 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 46B42654BB; Wed, 26 Jun 2024 09:21:35 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="MrRngyBn"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 08484654B0 for ; Wed, 26 Jun 2024 09:21:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386492; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DhjRDTFUZDS7ob74gjd7Bw08NowGjNvfxws4RLmwThM=; b=MrRngyBnD+33P3pp+BQiy9t6JPqpr0SeZ96LukC3vfC7JV08l20Zn6PSoPKxQlrE00XcQb gV9EFMI1BQz6OJfoFAIo5WZ3sxnB3ChZuMIWcvnpT74VyI5IhZmMJdYJtBcIIcAFx6bm9D h/RxAMXNcTMgfSQFG0T6JQWLiJdAtoc= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-460-yrgs4RuPNCGmOpM2YrKNXg-1; Wed, 26 Jun 2024 03:21:31 -0400 X-MC-Unique: yrgs4RuPNCGmOpM2YrKNXg-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8D83A19560B5 for ; Wed, 26 Jun 2024 07:21:30 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 971393000221; Wed, 26 Jun 2024 07:21:29 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 08/19] libcamera: software_isp: Track and pass frame ids Date: Wed, 26 Jun 2024 09:20:49 +0200 Message-ID: <20240626072100.55497-9-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" A previous preparation patch implemented passing frame ids to stats processing but without actual meaningful frame id value passed there. This patch extends that by actually providing the frame id and passing it through to the stats processor. The frame id is taken from the request sequence number, the same as in hardware pipelines. Dear reviewers: I'm confused even after looking at commit 6084217cd3b52ba5677e3ca2de0e21008fdaa735. What's the relationship between requests, buffers and frames? It's not 1:1:1 or is it? Could you please provide some explanation that could be put here? Signed-off-by: Milan Zamazal --- include/libcamera/internal/software_isp/software_isp.h | 4 ++-- src/libcamera/pipeline/simple/simple.cpp | 2 +- src/libcamera/software_isp/debayer.h | 2 +- src/libcamera/software_isp/debayer_cpu.cpp | 9 ++++----- src/libcamera/software_isp/debayer_cpu.h | 2 +- src/libcamera/software_isp/software_isp.cpp | 10 ++++++---- 6 files changed, 15 insertions(+), 14 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index 6a03b17f..7365b49a 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -72,10 +72,10 @@ public: int start(); void stop(); - int queueBuffers(FrameBuffer *input, + int queueBuffers(uint32_t frame, FrameBuffer *input, const std::map &outputs); - void process(FrameBuffer *input, FrameBuffer *output); + void process(uint32_t frame, FrameBuffer *input, FrameBuffer *output); Signal inputBufferReady; Signal outputBufferReady; diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index b1bf0d16..5cca94c3 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -864,7 +864,7 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) if (converter_) converter_->queueBuffers(buffer, conversionQueue_.front()); else - swIsp_->queueBuffers(buffer, conversionQueue_.front()); + swIsp_->queueBuffers(request->sequence(), buffer, conversionQueue_.front()); conversionQueue_.pop(); return; diff --git a/src/libcamera/software_isp/debayer.h b/src/libcamera/software_isp/debayer.h index c151fe5d..d7ca060d 100644 --- a/src/libcamera/software_isp/debayer.h +++ b/src/libcamera/software_isp/debayer.h @@ -40,7 +40,7 @@ public: virtual std::tuple strideAndFrameSize(const PixelFormat &outputFormat, const Size &size) = 0; - virtual void process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) = 0; + virtual void process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params) = 0; virtual SizeRange sizes(PixelFormat inputFormat, const Size &inputSize) = 0; diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index 1575cedb..c75b8967 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -731,7 +731,7 @@ static inline int64_t timeDiff(timespec &after, timespec &before) (int64_t)after.tv_nsec - (int64_t)before.tv_nsec; } -void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) +void DebayerCpu::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params) { timespec frameStartTime; @@ -785,12 +785,11 @@ void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams } /* - * Frame and buffer ids are currently not used, so pass zeros as parameters. + * Buffer ids are currently not used, so pass zeros as its parameter. * - * \todo Pass real values once frame is passed here and stats buffer passing - * is changed. + * \todo Pass real bufferId once stats buffer passing is changed. */ - stats_->finishFrame(0, 0); + stats_->finishFrame(frame, 0); outputBufferReady.emit(output); inputBufferReady.emit(input); } diff --git a/src/libcamera/software_isp/debayer_cpu.h b/src/libcamera/software_isp/debayer_cpu.h index 1dac6435..6a9cb4c7 100644 --- a/src/libcamera/software_isp/debayer_cpu.h +++ b/src/libcamera/software_isp/debayer_cpu.h @@ -36,7 +36,7 @@ public: std::vector formats(PixelFormat input); std::tuple strideAndFrameSize(const PixelFormat &outputFormat, const Size &size); - void process(FrameBuffer *input, FrameBuffer *output, DebayerParams params); + void process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params); SizeRange sizes(PixelFormat inputFormat, const Size &inputSize); /** diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 3fc4a64b..aa60fb5f 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -279,12 +279,13 @@ int SoftwareIsp::exportBuffers(unsigned int output, unsigned int count, /** * \brief Queue buffers to Software ISP + * \param[in] frame The frame number * \param[in] input The input framebuffer * \param[in] outputs The container holding the output stream indexes and * their respective frame buffer outputs * \return 0 on success, a negative errno on failure */ -int SoftwareIsp::queueBuffers(FrameBuffer *input, +int SoftwareIsp::queueBuffers(uint32_t frame, FrameBuffer *input, const std::map &outputs) { unsigned int mask = 0; @@ -308,7 +309,7 @@ int SoftwareIsp::queueBuffers(FrameBuffer *input, mask |= 1 << index; } - process(input, outputs.at(0)); + process(frame, input, outputs.at(0)); return 0; } @@ -340,13 +341,14 @@ void SoftwareIsp::stop() /** * \brief Passes the input framebuffer to the ISP worker to process + * \param[in] frame The frame number * \param[in] input The input framebuffer * \param[out] output The framebuffer to write the processed frame to */ -void SoftwareIsp::process(FrameBuffer *input, FrameBuffer *output) +void SoftwareIsp::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output) { debayer_->invokeMethod(&DebayerCpu::process, - ConnectionTypeQueued, input, output, debayerParams_); + ConnectionTypeQueued, frame, input, output, debayerParams_); } void SoftwareIsp::saveIspParams() From patchwork Wed Jun 26 07:20:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20400 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id BE7D6BD87C for ; Wed, 26 Jun 2024 07:21:39 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 694C0654B9; Wed, 26 Jun 2024 09:21:39 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="SqhjYefy"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3E0D1654BA for ; Wed, 26 Jun 2024 09:21:35 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386494; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CfCIX1MFEeogTDdO/I4EqAyh2EpusuWHO0/GoR+VBvQ=; b=SqhjYefyO/dYM6xk1nhTKvdugvdj5JqlbrzzbSUbMayMu5a/KB7YoR2u9HswUXjTIR/GBx ZgRyx4v45LhDb/S3jjk0Wc39SEtqc7G/y+oJQ3OlpCMjMTGfa9zzUwGjM/yjw6NNK+MFxX 37nWaI92OF/3lUhI1c7aNIpeewPxODA= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-103-2neEqG59PGKPzM7IPmuqkA-1; Wed, 26 Jun 2024 03:21:32 -0400 X-MC-Unique: 2neEqG59PGKPzM7IPmuqkA-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EBCD5195608D for ; Wed, 26 Jun 2024 07:21:31 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id F0DDC300021A; Wed, 26 Jun 2024 07:21:30 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 09/19] libcamera: software_isp: Create algorithms Date: Wed, 26 Jun 2024 09:20:50 +0200 Message-ID: <20240626072100.55497-10-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We are ready to introduce algorithms now. First, let's create algorithms. The algorithms are not called yet, calls to them will be added in followup patches. The maximum number of contexts is set to the same value as in hardware pipelines. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- src/ipa/simple/algorithms/meson.build | 4 ++++ src/ipa/simple/data/uncalibrated.yaml | 1 + src/ipa/simple/meson.build | 7 +++++-- src/ipa/simple/soft_simple.cpp | 14 ++++++++++++++ 4 files changed, 24 insertions(+), 2 deletions(-) create mode 100644 src/ipa/simple/algorithms/meson.build diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build new file mode 100644 index 00000000..31d26e43 --- /dev/null +++ b/src/ipa/simple/algorithms/meson.build @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: CC0-1.0 + +soft_simple_ipa_algorithms = files([ +]) diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index ff981a1a..2cdc39a8 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -2,4 +2,5 @@ %YAML 1.1 --- version: 1 +algorithms: ... diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build index 33d1c96a..08d877bc 100644 --- a/src/ipa/simple/meson.build +++ b/src/ipa/simple/meson.build @@ -1,5 +1,8 @@ # SPDX-License-Identifier: CC0-1.0 +subdir('algorithms') +subdir('data') + ipa_name = 'ipa_soft_simple' soft_simple_sources = files([ @@ -7,6 +10,8 @@ soft_simple_sources = files([ 'black_level.cpp', ]) +soft_simple_sources += soft_simple_ipa_algorithms + mod = shared_module(ipa_name, [soft_simple_sources, libcamera_generated_ipa_headers], name_prefix : '', @@ -24,6 +29,4 @@ if ipa_sign_module build_by_default : true) endif -subdir('data') - ipa_names += ipa_name diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index f2cfb2dc..6ddb4004 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -54,12 +54,15 @@ static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; * enough to prevent the exposure from wobbling around the optimal value. */ static constexpr float kExposureSatisfactory = 0.2; +/* Maximum number of frame contexts to be held */ +static constexpr uint32_t kMaxFrameContexts = 16; class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() : params_(nullptr), stats_(nullptr), blackLevel_(BlackLevel()), + context_({ {}, {}, { kMaxFrameContexts } }), ignoreUpdates_(0) { } @@ -93,6 +96,8 @@ private: static constexpr unsigned int kGammaLookupSize = 1024; std::array gammaTable_; int lastBlackLevel_ = -1; + /* Local parameter storage */ + struct IPAContext context_; int32_t exposureMin_, exposureMax_; int32_t exposure_; @@ -139,6 +144,15 @@ int IPASoftSimple::init(const IPASettings &settings, unsigned int version = (*data)["version"].get(0); LOG(IPASoft, Debug) << "Tuning file version " << version; + if (!data->contains("algorithms")) { + LOG(IPASoft, Error) << "Tuning file doesn't contain algorithms"; + return -EINVAL; + } + + int ret = createAlgorithms(context_, (*data)["algorithms"]); + if (ret) + return ret; + params_ = nullptr; stats_ = nullptr; From patchwork Wed Jun 26 07:20:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20401 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0930BBD87C for ; Wed, 26 Jun 2024 07:21:41 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B4237654C0; Wed, 26 Jun 2024 09:21:40 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="iYBTawpd"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3167C654B0 for ; Wed, 26 Jun 2024 09:21:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386496; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y0xU0TlysKVGcjxmMg9zrKPE7hMGBM9gZKvywnN8zf0=; b=iYBTawpd7R9fyKHXmQKmTlINT37em5kvbtwLqCYp9336pZ2f2/iGhnBShsgrzgwmydfXFA nEQRuN1+A9VRUxkuQh+sUNUL8I+2GfzumhFJPNvkkaeKYQt8FydIMUhB94S554oUM9pvAs BlpwnTgkKBb/xFf1SNsfuty9LjOOj+k= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-55-STZL4v2UNjCiwtGr84Lh7w-1; Wed, 26 Jun 2024 03:21:34 -0400 X-MC-Unique: STZL4v2UNjCiwtGr84Lh7w-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 54A5119560A7 for ; Wed, 26 Jun 2024 07:21:33 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 588A3300021A; Wed, 26 Jun 2024 07:21:32 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 10/19] libcamera: software_isp: Call Algorithm::process Date: Wed, 26 Jun 2024 09:20:51 +0200 Message-ID: <20240626072100.55497-11-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds Algorithm::process call for the defined algorithms. This is preparation only since there are currently no Algorithm based algorithms defined. As software ISP currently doesn't produce any metadata, a dummy and unused metadata instance is created to satisfy Algorithm::process API. This should be changed in future. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- src/ipa/simple/soft_simple.cpp | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 6ddb4004..c0cb6769 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -265,10 +265,21 @@ void IPASoftSimple::stop() } void IPASoftSimple::processStats( - [[maybe_unused]] const uint32_t frame, + const uint32_t frame, [[maybe_unused]] const uint32_t bufferId, const ControlList &sensorControls) { + IPAFrameContext &frameContext = context_.frameContexts.get(frame); + /* + * \todo Software ISP currently doesn't produce any metadata so it is + * possible to use a dummy metadata instance here. But metadata should be + * properly handled in future. + */ + ControlList metadata(controls::controls); + for (auto const &algo : algorithms()) { + algo->process(context_, frame, frameContext, stats_, metadata); + } + SwIspStats::Histogram histogram = stats_->yHistogram; if (ignoreUpdates_ > 0) blackLevel_.update(histogram); From patchwork Wed Jun 26 07:20:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20403 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B9C8CBD87C for ; Wed, 26 Jun 2024 07:21:44 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 53078654BC; Wed, 26 Jun 2024 09:21:44 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="An74cPAk"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id AE6CC654B9 for ; Wed, 26 Jun 2024 09:21:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386497; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MdYs0NFmu5oAj/KDVMjf6VLN3FCqpujQ3Q5ILyAK2AU=; b=An74cPAkpuvcd+aH23LxZFZSxGiXreSxvYqHfq9p448LSByiNQRjAn5Lx50tI9m7Y/zdZT gJ1VdEx19StjKyVJIqljTT6O4uHbynWgYap6cOEeNmbsUXfbWfM4N/9eF4G83oB6B+RbJK NafoNVF7J49YLIpc06HQuw+ExNclcfU= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-202-eLjROdJHNy-LUuqB6gxeVw-1; Wed, 26 Jun 2024 03:21:35 -0400 X-MC-Unique: eLjROdJHNy-LUuqB6gxeVw-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id B22C819560B1 for ; Wed, 26 Jun 2024 07:21:34 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id B30C33000221; Wed, 26 Jun 2024 07:21:33 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 11/19] libcamera: software_isp: Call Algorithm::prepare Date: Wed, 26 Jun 2024 09:20:52 +0200 Message-ID: <20240626072100.55497-12-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds Algorithm::prepare call for the defined algorithms. This is preparation only since there are currently no Algorithm based algorithms defined. Signed-off-by: Milan Zamazal --- include/libcamera/ipa/soft.mojom | 1 + src/ipa/simple/soft_simple.cpp | 8 ++++++++ src/libcamera/software_isp/software_isp.cpp | 1 + 3 files changed, 10 insertions(+) diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index bd48ece9..4975b251 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -19,6 +19,7 @@ interface IPASoftInterface { configure(libcamera.ControlInfoMap sensorCtrlInfoMap) => (int32 ret); + prepare(uint32 frame); [async] processStats(uint32 frame, uint32 bufferId, libcamera.ControlList sensorControls); }; diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index c0cb6769..9387508e 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -78,6 +78,7 @@ public: int start() override; void stop() override; + void prepare(const uint32_t frame) override; void processStats(const uint32_t frame, const uint32_t bufferId, const ControlList &sensorControls) override; @@ -264,6 +265,13 @@ void IPASoftSimple::stop() { } +void IPASoftSimple::prepare(const uint32_t frame) +{ + IPAFrameContext &frameContext = context_.frameContexts.get(frame); + for (auto const &algo : algorithms()) + algo->prepare(context_, frame, frameContext, params_); +} + void IPASoftSimple::processStats( const uint32_t frame, [[maybe_unused]] const uint32_t bufferId, diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index aa60fb5f..812bc910 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -347,6 +347,7 @@ void SoftwareIsp::stop() */ void SoftwareIsp::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output) { + ipa_->prepare(frame); debayer_->invokeMethod(&DebayerCpu::process, ConnectionTypeQueued, frame, input, output, debayerParams_); } From patchwork Wed Jun 26 07:20:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20404 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2131DBD87C for ; Wed, 26 Jun 2024 07:21:46 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D39E2654BB; Wed, 26 Jun 2024 09:21:45 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="QwgfGySH"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7C9C4654B0 for ; Wed, 26 Jun 2024 09:21:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386498; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lAdFWGmfnZVRoq0HAfOS2raUybD5jz4pjzhshDXSwNs=; b=QwgfGySH1CSGFbS8igLTlqyio4uaexZnMpXac03210AX4Ci5wiXyioWdYLWV19zuS4XbB9 SLtd5hAUKycsse26U/hLxwexxfLrU6NxQeLqc1Rg1YdvYtCFjCplj1UmEuHdm4TcAlMyTr DxV8/FbQ0johCd85gtOANjkte7YHVH0= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-130-Xuvf5Fb7ObiaY3Y3Jy7iOQ-1; Wed, 26 Jun 2024 03:21:36 -0400 X-MC-Unique: Xuvf5Fb7ObiaY3Y3Jy7iOQ-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1099F19560AA for ; Wed, 26 Jun 2024 07:21:36 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 181C73000221; Wed, 26 Jun 2024 07:21:34 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 12/19] libcamera: software_isp: Call Algorithm::queueRequest Date: Wed, 26 Jun 2024 09:20:53 +0200 Message-ID: <20240626072100.55497-13-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds Algorithm::queueRequest call for the defined algorithms. This is preparation only since there are currently no Algorithm based algorithms defined. Signed-off-by: Milan Zamazal --- include/libcamera/internal/software_isp/software_isp.h | 1 + include/libcamera/ipa/soft.mojom | 1 + src/ipa/simple/soft_simple.cpp | 10 ++++++++++ src/libcamera/pipeline/simple/simple.cpp | 2 ++ src/libcamera/software_isp/software_isp.cpp | 10 ++++++++++ 5 files changed, 24 insertions(+) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index 7365b49a..8753c2fd 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -72,6 +72,7 @@ public: int start(); void stop(); + void queueRequest(const uint32_t frame, const ControlList &controls); int queueBuffers(uint32_t frame, FrameBuffer *input, const std::map &outputs); diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index 4975b251..dad352ba 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -20,6 +20,7 @@ interface IPASoftInterface { => (int32 ret); prepare(uint32 frame); + queueRequest(uint32 frame, libcamera.ControlList sensorControls); [async] processStats(uint32 frame, uint32 bufferId, libcamera.ControlList sensorControls); }; diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 9387508e..514a9db5 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -78,6 +78,7 @@ public: int start() override; void stop() override; + void queueRequest(const uint32_t frame, const ControlList &controls) override; void prepare(const uint32_t frame) override; void processStats(const uint32_t frame, const uint32_t bufferId, const ControlList &sensorControls) override; @@ -272,6 +273,15 @@ void IPASoftSimple::prepare(const uint32_t frame) algo->prepare(context_, frame, frameContext, params_); } +void IPASoftSimple::queueRequest(const uint32_t frame, const ControlList &controls) +{ + IPAFrameContext &frameContext = context_.frameContexts.alloc(frame); + + for (auto const &algo : algorithms()) { + algo->queueRequest(context_, frame, frameContext, controls); + } +} + void IPASoftSimple::processStats( const uint32_t frame, [[maybe_unused]] const uint32_t bufferId, diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 5cca94c3..0fa9db38 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -1421,6 +1421,8 @@ int SimplePipelineHandler::queueRequestDevice(Camera *camera, Request *request) if (data->useConversion_) data->conversionQueue_.push(std::move(buffers)); + data->swIsp_->queueRequest(request->sequence(), request->controls()); + return 0; } diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 812bc910..ba3f1bae 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -277,6 +277,16 @@ int SoftwareIsp::exportBuffers(unsigned int output, unsigned int count, return count; } +/** + * \brief Queue a request and process the control list from the application + * \param[in] frame The number of the frame which will be processed next + * \param[in] controls The controls for the \a frame + */ +void SoftwareIsp::queueRequest(const uint32_t frame, const ControlList &controls) +{ + ipa_->queueRequest(frame, controls); +} + /** * \brief Queue buffers to Software ISP * \param[in] frame The frame number From patchwork Wed Jun 26 07:20:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20405 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id DDA29BD87C for ; Wed, 26 Jun 2024 07:21:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 940CB654C2; Wed, 26 Jun 2024 09:21:47 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="YmtzHP06"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 15993654C5 for ; Wed, 26 Jun 2024 09:21:41 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386500; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ijfhu1Sx9MTPRpcZECyApzN20pMtjagsC/459JJqGB4=; b=YmtzHP062KwWFJuTtht7reow4t9xXK1dXVn7DCMIif07AWHxblFz8Jr7a1PpdzINxV/7YW j67d/aLckmWuwx+rbAPgaJTQ3BAajgerEmq09xSU6BqhKn3Y2x7yW3S8oJ3IZoSo8LudCN L0Km0IsIhj+H7WdIdxHUG5Wv0miOuNk= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-663-Pz8WxmCgNC6bruACNb9fMw-1; Wed, 26 Jun 2024 03:21:38 -0400 X-MC-Unique: Pz8WxmCgNC6bruACNb9fMw-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 6C83E1956051 for ; Wed, 26 Jun 2024 07:21:37 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 706EB300022A; Wed, 26 Jun 2024 07:21:36 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 13/19] libcamera: software_isp: Call Algorithm::configure Date: Wed, 26 Jun 2024 09:20:54 +0200 Message-ID: <20240626072100.55497-14-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds Algorithm::configure call for the defined algorithms. This is preparation only since there are currently no Algorithm based algorithms defined. A part of this change is passing IPAConfigInfo instead of ControlInfoMap to configure() calls as this is what Algorithm::configure expects. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- .../libcamera/internal/software_isp/software_isp.h | 2 +- include/libcamera/ipa/soft.mojom | 6 +++++- src/ipa/simple/module.h | 3 ++- src/ipa/simple/soft_simple.cpp | 12 +++++++++--- src/libcamera/pipeline/simple/simple.cpp | 11 +++++++---- src/libcamera/software_isp/software_isp.cpp | 7 ++++--- 6 files changed, 28 insertions(+), 13 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index 8753c2fd..1ba9eb62 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -61,7 +61,7 @@ public: int configure(const StreamConfiguration &inputCfg, const std::vector> &outputCfgs, - const ControlInfoMap &sensorControls); + const ipa::soft::IPAConfigInfo &configInfo); int exportBuffers(unsigned int output, unsigned int count, std::vector> *buffers); diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index dad352ba..65d90bfa 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -8,6 +8,10 @@ module ipa.soft; import "include/libcamera/ipa/core.mojom"; +struct IPAConfigInfo { + libcamera.ControlInfoMap sensorControls; +}; + interface IPASoftInterface { init(libcamera.IPASettings settings, libcamera.SharedFD fdStats, @@ -16,7 +20,7 @@ interface IPASoftInterface { => (int32 ret); start() => (int32 ret); stop(); - configure(libcamera.ControlInfoMap sensorCtrlInfoMap) + configure(IPAConfigInfo configInfo) => (int32 ret); prepare(uint32 frame); diff --git a/src/ipa/simple/module.h b/src/ipa/simple/module.h index 21328ecd..e518e742 100644 --- a/src/ipa/simple/module.h +++ b/src/ipa/simple/module.h @@ -10,6 +10,7 @@ #include #include +#include #include "libcamera/internal/software_isp/debayer_params.h" #include "libcamera/internal/software_isp/swisp_stats.h" @@ -22,7 +23,7 @@ namespace libcamera { namespace ipa::soft { -using Module = ipa::Module; } /* namespace ipa::soft */ diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 514a9db5..7200abaf 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -73,7 +73,7 @@ public: const SharedFD &fdStats, const SharedFD &fdParams, const ControlInfoMap &sensorInfoMap) override; - int configure(const ControlInfoMap &sensorInfoMap) override; + int configure(const IPAConfigInfo &configInfo) override; int start() override; void stop() override; @@ -209,9 +209,9 @@ int IPASoftSimple::init(const IPASettings &settings, return 0; } -int IPASoftSimple::configure(const ControlInfoMap &sensorInfoMap) +int IPASoftSimple::configure(const IPAConfigInfo &configInfo) { - sensorInfoMap_ = sensorInfoMap; + sensorInfoMap_ = configInfo.sensorControls; const ControlInfo &exposureInfo = sensorInfoMap_.find(V4L2_CID_EXPOSURE)->second; const ControlInfo &gainInfo = sensorInfoMap_.find(V4L2_CID_ANALOGUE_GAIN)->second; @@ -250,6 +250,12 @@ int IPASoftSimple::configure(const ControlInfoMap &sensorInfoMap) againMinStep_ = 1.0; } + for (auto const &algo : algorithms()) { + int ret = algo->configure(context_, configInfo); + if (ret) + return ret; + } + LOG(IPASoft, Info) << "Exposure " << exposureMin_ << "-" << exposureMax_ << ", gain " << againMin_ << "-" << againMax_ << " (" << againMinStep_ << ")"; diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 0fa9db38..a82d05f4 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -1288,10 +1288,13 @@ int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c) inputCfg.stride = captureFormat.planes[0].bpl; inputCfg.bufferCount = kNumInternalBuffers; - return data->converter_ - ? data->converter_->configure(inputCfg, outputCfgs) - : data->swIsp_->configure(inputCfg, outputCfgs, - data->sensor_->controls()); + if (data->converter_) + return data->converter_->configure(inputCfg, outputCfgs); + else { + ipa::soft::IPAConfigInfo configInfo; + configInfo.sensorControls = data->sensor_->controls(); + return data->swIsp_->configure(inputCfg, outputCfgs, configInfo); + } } int SimplePipelineHandler::exportFrameBuffers(Camera *camera, Stream *stream, diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index ba3f1bae..1f176214 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -223,16 +223,17 @@ SoftwareIsp::strideAndFrameSize(const PixelFormat &outputFormat, const Size &siz * \brief Configure the SoftwareIsp object according to the passed in parameters * \param[in] inputCfg The input configuration * \param[in] outputCfgs The output configurations - * \param[in] sensorControls ControlInfoMap of the controls supported by the sensor + * \param[in] configInfo The IPA configuration data, received from the pipeline + * handler * \return 0 on success, a negative errno on failure */ int SoftwareIsp::configure(const StreamConfiguration &inputCfg, const std::vector> &outputCfgs, - const ControlInfoMap &sensorControls) + const ipa::soft::IPAConfigInfo &configInfo) { ASSERT(ipa_ && debayer_); - int ret = ipa_->configure(sensorControls); + int ret = ipa_->configure(configInfo); if (ret < 0) return ret; From patchwork Wed Jun 26 07:20:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20407 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4948EBD87C for ; Wed, 26 Jun 2024 07:21:50 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D9574654C2; Wed, 26 Jun 2024 09:21:49 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="NkARoiMx"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2074E654B9 for ; Wed, 26 Jun 2024 09:21:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386501; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=njwYqMMfmCvcKV9hfXlhpOQnuYgx5MVCRScwhh5c5+U=; b=NkARoiMxhGTNeSzKPC8IkfLmfUYSsWXbKvaVLWhfdiyEiPNFsKQzFTP/Hr5fKDM2aWDpcP EEQidQWaHwLOyWJqoHL3iuLdtvMV0+1e02LJ1Z5V0CycLro+iPmkqh5WMvZN7J8K3toqiZ zs2/eIbAUlLmIaaF7+289UEGEgsTjbk= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-401--8UZjbPbP5qUOalilbtJzg-1; Wed, 26 Jun 2024 03:21:39 -0400 X-MC-Unique: -8UZjbPbP5qUOalilbtJzg-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id C3EC219560A1 for ; Wed, 26 Jun 2024 07:21:38 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id CA8B1300022B; Wed, 26 Jun 2024 07:21:37 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 14/19] libcamera: software_isp: Move black level to an algorithm module Date: Wed, 26 Jun 2024 09:20:55 +0200 Message-ID: <20240626072100.55497-15-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The black level determination, already present as a separate class, can be moved to the prepared Algorithm processing structure. It is the first of the current software ISP algorithms that is called in the stats processing sequence, which means it is also the first one that should be moved to the new structure. Stats processing starts with calling Algorithm-based processing so the call order of the algorithms is retained. Movement of this algorithm is relatively straightforward because all it does is processing stats. Black level is now recomputed on each stats processing. In a future patch, after DelayedControls are used, this will be changed to recompute the black level only after exposure/gain changes. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/blc.cpp | 79 +++++++++++++++++++++++ src/ipa/simple/algorithms/blc.h | 37 +++++++++++ src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/black_level.cpp | 93 --------------------------- src/ipa/simple/black_level.h | 33 ---------- src/ipa/simple/data/uncalibrated.yaml | 1 + src/ipa/simple/ipa_context.h | 7 ++ src/ipa/simple/meson.build | 1 - src/ipa/simple/soft_simple.cpp | 8 +-- 9 files changed, 127 insertions(+), 133 deletions(-) create mode 100644 src/ipa/simple/algorithms/blc.cpp create mode 100644 src/ipa/simple/algorithms/blc.h delete mode 100644 src/ipa/simple/black_level.cpp delete mode 100644 src/ipa/simple/black_level.h diff --git a/src/ipa/simple/algorithms/blc.cpp b/src/ipa/simple/algorithms/blc.cpp new file mode 100644 index 00000000..e216138d --- /dev/null +++ b/src/ipa/simple/algorithms/blc.cpp @@ -0,0 +1,79 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * black level handling + */ + +#include "blc.h" + +#include + +#include + +namespace libcamera { + +namespace ipa::soft::algorithms { + +LOG_DEFINE_CATEGORY(IPASoftBL) + +BlackLevel::BlackLevel() +{ +} + +int BlackLevel::init(IPAContext &context, + [[maybe_unused]] const YamlObject &tuningData) +{ + context.configuration.black.set = false; + context.configuration.black.changed = true; + context.configuration.black.level = 255; + return 0; +} + +void BlackLevel::process(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + const SwIspStats *stats, + [[maybe_unused]] ControlList &metadata) +{ + if (context.configuration.black.set) { + context.configuration.black.changed = false; + return; + } + + const SwIspStats::Histogram &histogram = stats->yHistogram; + + /* + * The constant is selected to be "good enough", not overly conservative or + * aggressive. There is no magic about the given value. + */ + constexpr float ignoredPercentage_ = 0.02; + const unsigned int total = + std::accumulate(begin(histogram), end(histogram), 0); + const unsigned int pixelThreshold = ignoredPercentage_ * total; + const unsigned int histogramRatio = 256 / SwIspStats::kYHistogramSize; + const unsigned int currentBlackIdx = + context.configuration.black.level / histogramRatio; + + for (unsigned int i = 0, seen = 0; + i < currentBlackIdx && i < SwIspStats::kYHistogramSize; + i++) { + seen += histogram[i]; + if (seen >= pixelThreshold) { + context.configuration.black.level = i * histogramRatio; + context.configuration.black.changed = true; + LOG(IPASoftBL, Debug) + << "Auto-set black level: " + << i << "/" << SwIspStats::kYHistogramSize + << " (" << 100 * (seen - histogram[i]) / total << "% below, " + << 100 * seen / total << "% at or below)"; + break; + } + }; +} + +REGISTER_IPA_ALGORITHM(BlackLevel, "BlackLevel") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/blc.h b/src/ipa/simple/algorithms/blc.h new file mode 100644 index 00000000..c8b8d92d --- /dev/null +++ b/src/ipa/simple/algorithms/blc.h @@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * black level handling + */ + +#pragma once + +#include + +#include "libcamera/internal/software_isp/swisp_stats.h" + +#include "algorithm.h" +#include "ipa_context.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class BlackLevel : public Algorithm +{ +public: + BlackLevel(); + ~BlackLevel() = default; + + int init(IPAContext &context, const YamlObject &tuningData) + override; + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const SwIspStats *stats, + ControlList &metadata) override; +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index 31d26e43..1f63c220 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -1,4 +1,5 @@ # SPDX-License-Identifier: CC0-1.0 soft_simple_ipa_algorithms = files([ + 'blc.cpp', ]) diff --git a/src/ipa/simple/black_level.cpp b/src/ipa/simple/black_level.cpp deleted file mode 100644 index 37e0109c..00000000 --- a/src/ipa/simple/black_level.cpp +++ /dev/null @@ -1,93 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1-or-later */ -/* - * Copyright (C) 2024, Red Hat Inc. - * - * black level handling - */ - -#include "black_level.h" - -#include - -#include - -namespace libcamera { - -LOG_DEFINE_CATEGORY(IPASoftBL) - -namespace ipa::soft { - -/** - * \class BlackLevel - * \brief Object providing black point level for software ISP - * - * Black level can be provided in hardware tuning files or, if no tuning file is - * available for the given hardware, guessed automatically, with less accuracy. - * As tuning files are not yet implemented for software ISP, BlackLevel - * currently provides only guessed black levels. - * - * This class serves for tracking black level as a property of the underlying - * hardware, not as means of enhancing a particular scene or image. - * - * The class is supposed to be instantiated for the given camera stream. - * The black level can be retrieved using BlackLevel::get() method. It is - * initially 0 and may change when updated using BlackLevel::update() method. - */ - -BlackLevel::BlackLevel() - : blackLevel_(255), blackLevelSet_(false) -{ -} - -/** - * \brief Return the current black level - * - * \return The black level, in the range from 0 (minimum) to 255 (maximum). - * If the black level couldn't be determined yet, return 0. - */ -uint8_t BlackLevel::get() const -{ - return blackLevelSet_ ? blackLevel_ : 0; -} - -/** - * \brief Update black level from the provided histogram - * \param[in] yHistogram The histogram to be used for updating black level - * - * The black level is property of the given hardware, not image. It is updated - * only if it has not been yet set or if it is lower than the lowest value seen - * so far. - */ -void BlackLevel::update(SwIspStats::Histogram &yHistogram) -{ - /* - * The constant is selected to be "good enough", not overly conservative or - * aggressive. There is no magic about the given value. - */ - constexpr float ignoredPercentage_ = 0.02; - const unsigned int total = - std::accumulate(begin(yHistogram), end(yHistogram), 0); - const unsigned int pixelThreshold = ignoredPercentage_ * total; - const unsigned int histogramRatio = 256 / SwIspStats::kYHistogramSize; - const unsigned int currentBlackIdx = blackLevel_ / histogramRatio; - - for (unsigned int i = 0, seen = 0; - i < currentBlackIdx && i < SwIspStats::kYHistogramSize; - i++) { - seen += yHistogram[i]; - if (seen >= pixelThreshold) { - blackLevel_ = i * histogramRatio; - blackLevelSet_ = true; - LOG(IPASoftBL, Debug) - << "Auto-set black level: " - << i << "/" << SwIspStats::kYHistogramSize - << " (" << 100 * (seen - yHistogram[i]) / total << "% below, " - << 100 * seen / total << "% at or below)"; - break; - } - }; -} - -} /* namespace ipa::soft */ - -} /* namespace libcamera */ diff --git a/src/ipa/simple/black_level.h b/src/ipa/simple/black_level.h deleted file mode 100644 index a04230c9..00000000 --- a/src/ipa/simple/black_level.h +++ /dev/null @@ -1,33 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1-or-later */ -/* - * Copyright (C) 2024, Red Hat Inc. - * - * black level handling - */ - -#pragma once - -#include -#include - -#include "libcamera/internal/software_isp/swisp_stats.h" - -namespace libcamera { - -namespace ipa::soft { - -class BlackLevel -{ -public: - BlackLevel(); - uint8_t get() const; - void update(SwIspStats::Histogram &yHistogram); - -private: - uint8_t blackLevel_; - bool blackLevelSet_; -}; - -} /* namespace ipa::soft */ - -} /* namespace libcamera */ diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index 2cdc39a8..e0d03d96 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -3,4 +3,5 @@ --- version: 1 algorithms: + - BlackLevel: ... diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index bc1235b6..c79bf6f8 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -8,6 +8,8 @@ #pragma once +#include + #include namespace libcamera { @@ -15,6 +17,11 @@ namespace libcamera { namespace ipa::soft { struct IPASessionConfiguration { + struct { + uint8_t level; + bool set; + bool changed; + } black; }; struct IPAActiveState { diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build index 08d877bc..cbc928c5 100644 --- a/src/ipa/simple/meson.build +++ b/src/ipa/simple/meson.build @@ -7,7 +7,6 @@ ipa_name = 'ipa_soft_simple' soft_simple_sources = files([ 'soft_simple.cpp', - 'black_level.cpp', ]) soft_simple_sources += soft_simple_ipa_algorithms diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 7200abaf..99fb953c 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -29,7 +29,6 @@ #include "libipa/camera_sensor_helper.h" -#include "black_level.h" #include "module.h" namespace libcamera { @@ -61,7 +60,7 @@ class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() - : params_(nullptr), stats_(nullptr), blackLevel_(BlackLevel()), + : params_(nullptr), stats_(nullptr), context_({ {}, {}, { kMaxFrameContexts } }), ignoreUpdates_(0) { @@ -93,7 +92,6 @@ private: SwIspStats *stats_; std::unique_ptr camHelper_; ControlInfoMap sensorInfoMap_; - BlackLevel blackLevel_; static constexpr unsigned int kGammaLookupSize = 1024; std::array gammaTable_; @@ -305,9 +303,7 @@ void IPASoftSimple::processStats( } SwIspStats::Histogram histogram = stats_->yHistogram; - if (ignoreUpdates_ > 0) - blackLevel_.update(histogram); - const uint8_t blackLevel = blackLevel_.get(); + const uint8_t blackLevel = context_.configuration.black.level; /* * Black level must be subtracted to get the correct AWB ratios, they From patchwork Wed Jun 26 07:20:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20408 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C5319BD87C for ; Wed, 26 Jun 2024 07:21:51 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 65A61654CF; Wed, 26 Jun 2024 09:21:51 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="GKM+nWnk"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E99FA654C8 for ; Wed, 26 Jun 2024 09:21:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386502; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J/EA4jzEGZRuRNBPwTgQuUBWxUfDJMLcBCli0UEO8sg=; b=GKM+nWnkK7Yxfq+xX5aDDQrFcHIoUGf9bx2kKuSRMTfZ/LrraOUezG+uCLzDdJPVE5mtuQ XoblnXoIXWijdASPAJDgogd8DwisDWBJEAyUuUySHy4h2O9DtRcyve0NoHblqHIkWFVt/9 Z348TciEmg+H2X4vFAeUw7EdpAv044M= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-583-sgN4rE0DMcadE-Dy5GWclw-1; Wed, 26 Jun 2024 03:21:41 -0400 X-MC-Unique: sgN4rE0DMcadE-Dy5GWclw-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 4957E19560AA for ; Wed, 26 Jun 2024 07:21:40 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 30800300021A; Wed, 26 Jun 2024 07:21:38 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 15/19] libcamera: software_isp: Move color handling to an algorithm module Date: Wed, 26 Jun 2024 09:20:56 +0200 Message-ID: <20240626072100.55497-16-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" After black level handling has been moved to an algorithm module, white balance and the construction of color tables can be moved to an algorithm module too. This time, the moved code is split between stats processing and parameter construction methods. This is the only part of the software ISP algorithms that sets the parameters so emitting setIspParams can be moved to prepare() method. A more natural representation of the gains (and black level) would be floating point numbers. This is not done here in order to minimize changes in code movements. It will be addressed in a followup patch. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/colors.cpp | 119 ++++++++++++++++++++++++++ src/ipa/simple/algorithms/colors.h | 46 ++++++++++ src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/data/uncalibrated.yaml | 1 + src/ipa/simple/ipa_context.h | 9 ++ src/ipa/simple/soft_simple.cpp | 66 +------------- 6 files changed, 180 insertions(+), 62 deletions(-) create mode 100644 src/ipa/simple/algorithms/colors.cpp create mode 100644 src/ipa/simple/algorithms/colors.h diff --git a/src/ipa/simple/algorithms/colors.cpp b/src/ipa/simple/algorithms/colors.cpp new file mode 100644 index 00000000..d4ab409c --- /dev/null +++ b/src/ipa/simple/algorithms/colors.cpp @@ -0,0 +1,119 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Color gains (AWB + gamma) + */ + +#include "colors.h" + +#include +#include +#include +#include + +#include + +#include "simple/ipa_context.h" + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoftColors) + +namespace ipa::soft::algorithms { + +static constexpr unsigned int kGammaLookupSize = 1024; + +Colors::Colors() +{ +} + +int Colors::init(IPAContext &context, + [[maybe_unused]] const YamlObject &tuningData) +{ + /* Gamma value is fixed */ + context.configuration.gamma = 0.5; + updateGammaTable(context); + + auto &gains = context.activeState.gains; + gains.red = gains.green = gains.blue = 256; + + return 0; +} + +void Colors::updateGammaTable(IPAContext &context) +{ + auto &gammaTable = context.activeState.gammaTable; + const unsigned int blackIndex = + context.configuration.black.level * IPAActiveState::kGammaLookupSize / 256; + std::fill(gammaTable.begin(), gammaTable.begin() + blackIndex, 0); + const float divisor = kGammaLookupSize - blackIndex - 1.0; + for (unsigned int i = blackIndex; i < kGammaLookupSize; i++) + gammaTable[i] = UINT8_MAX * powf((i - blackIndex) / divisor, + context.configuration.gamma); +} + +void Colors::prepare(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + DebayerParams *params) +{ + /* Update the gamma table if needed */ + if (context.configuration.black.changed) + updateGammaTable(context); + + auto &gains = context.activeState.gains; + auto &gammaTable = context.activeState.gammaTable; + for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { + constexpr unsigned int div = + static_cast(DebayerParams::kRGBLookupSize) * 256 / kGammaLookupSize; + /* Apply gamma after gain! */ + unsigned int idx; + idx = std::min({ i * gains.red / div, kGammaLookupSize - 1 }); + params->red[i] = gammaTable[idx]; + idx = std::min({ i * gains.green / div, kGammaLookupSize - 1 }); + params->green[i] = gammaTable[idx]; + idx = std::min({ i * gains.blue / div, kGammaLookupSize - 1 }); + params->blue[i] = gammaTable[idx]; + } +} + +void Colors::process(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + const SwIspStats *stats, + [[maybe_unused]] ControlList &metadata) +{ + const SwIspStats::Histogram &histogram = stats->yHistogram; + const uint8_t blackLevel = context.configuration.black.level; + + /* + * Black level must be subtracted to get the correct AWB ratios, they + * would be off if they were computed from the whole brightness range + * rather than from the sensor range. + */ + const uint64_t nPixels = std::accumulate( + histogram.begin(), histogram.end(), 0); + const uint64_t offset = blackLevel * nPixels; + const uint64_t sumR = stats->sumR_ - offset / 4; + const uint64_t sumG = stats->sumG_ - offset / 2; + const uint64_t sumB = stats->sumB_ - offset / 4; + + /* + * Calculate red and blue gains for AWB. + * Clamp max gain at 4.0, this also avoids 0 division. + * Gain: 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. + */ + auto &gains = context.activeState.gains; + gains.red = sumR <= sumG / 4 ? 1024 : 256 * sumG / sumR; + gains.blue = sumB <= sumG / 4 ? 1024 : 256 * sumG / sumB; + /* Green gain is fixed to 256 */ + + LOG(IPASoftColors, Debug) << "gain R/B " << gains.red << "/" << gains.blue; +} + +REGISTER_IPA_ALGORITHM(Colors, "Colors") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/colors.h b/src/ipa/simple/algorithms/colors.h new file mode 100644 index 00000000..2c2f7752 --- /dev/null +++ b/src/ipa/simple/algorithms/colors.h @@ -0,0 +1,46 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Color gains (AWB + gamma) + */ + +#pragma once + +#include + +#include "libcamera/internal/software_isp/debayer_params.h" +#include "libcamera/internal/software_isp/swisp_stats.h" + +#include "algorithm.h" +#include "ipa_context.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class Colors : public Algorithm +{ +public: + Colors(); + ~Colors() = default; + + int init(IPAContext &context, const YamlObject &tuningData) + override; + void prepare(IPAContext &context, + const uint32_t frame, + IPAFrameContext &frameContext, + DebayerParams *params) override; + void process(IPAContext &context, + const uint32_t frame, + IPAFrameContext &frameContext, + const SwIspStats *stats, + ControlList &metadata) override; + +private: + void updateGammaTable(IPAContext &context); +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index 1f63c220..324c7b4e 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -2,4 +2,5 @@ soft_simple_ipa_algorithms = files([ 'blc.cpp', + 'colors.cpp', ]) diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index e0d03d96..7e5149e5 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -4,4 +4,5 @@ version: 1 algorithms: - BlackLevel: + - Colors: ... diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index c79bf6f8..991a45b1 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -8,6 +8,7 @@ #pragma once +#include #include #include @@ -17,6 +18,7 @@ namespace libcamera { namespace ipa::soft { struct IPASessionConfiguration { + float gamma; struct { uint8_t level; bool set; @@ -25,6 +27,13 @@ struct IPASessionConfiguration { }; struct IPAActiveState { + struct { + unsigned int red; + unsigned int green; + unsigned int blue; + } gains; + static constexpr unsigned int kGammaLookupSize = 1024; + std::array gammaTable; }; struct IPAFrameContext : public FrameContext { diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 99fb953c..25f6a71e 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -5,8 +5,6 @@ * Simple Software Image Processing Algorithm module */ -#include -#include #include #include @@ -93,9 +91,6 @@ private: std::unique_ptr camHelper_; ControlInfoMap sensorInfoMap_; - static constexpr unsigned int kGammaLookupSize = 1024; - std::array gammaTable_; - int lastBlackLevel_ = -1; /* Local parameter storage */ struct IPAContext context_; @@ -275,6 +270,7 @@ void IPASoftSimple::prepare(const uint32_t frame) IPAFrameContext &frameContext = context_.frameContexts.get(frame); for (auto const &algo : algorithms()) algo->prepare(context_, frame, frameContext, params_); + setIspParams.emit(); } void IPASoftSimple::queueRequest(const uint32_t frame, const ControlList &controls) @@ -302,62 +298,6 @@ void IPASoftSimple::processStats( algo->process(context_, frame, frameContext, stats_, metadata); } - SwIspStats::Histogram histogram = stats_->yHistogram; - const uint8_t blackLevel = context_.configuration.black.level; - - /* - * Black level must be subtracted to get the correct AWB ratios, they - * would be off if they were computed from the whole brightness range - * rather than from the sensor range. - */ - const uint64_t nPixels = std::accumulate( - histogram.begin(), histogram.end(), 0); - const uint64_t offset = blackLevel * nPixels; - const uint64_t sumR = stats_->sumR_ - offset / 4; - const uint64_t sumG = stats_->sumG_ - offset / 2; - const uint64_t sumB = stats_->sumB_ - offset / 4; - - /* - * Calculate red and blue gains for AWB. - * Clamp max gain at 4.0, this also avoids 0 division. - * Gain: 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. - */ - const unsigned int gainR = sumR <= sumG / 4 ? 1024 : 256 * sumG / sumR; - const unsigned int gainB = sumB <= sumG / 4 ? 1024 : 256 * sumG / sumB; - /* Green gain and gamma values are fixed */ - constexpr unsigned int gainG = 256; - - /* Update the gamma table if needed */ - if (blackLevel != lastBlackLevel_) { - constexpr float gamma = 0.5; - const unsigned int blackIndex = blackLevel * kGammaLookupSize / 256; - std::fill(gammaTable_.begin(), gammaTable_.begin() + blackIndex, 0); - const float divisor = kGammaLookupSize - blackIndex - 1.0; - for (unsigned int i = blackIndex; i < kGammaLookupSize; i++) - gammaTable_[i] = UINT8_MAX * - std::pow((i - blackIndex) / divisor, gamma); - - lastBlackLevel_ = blackLevel; - } - - for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { - constexpr unsigned int div = - DebayerParams::kRGBLookupSize * 256 / kGammaLookupSize; - unsigned int idx; - - /* Apply gamma after gain! */ - idx = std::min({ i * gainR / div, (kGammaLookupSize - 1) }); - params_->red[i] = gammaTable_[idx]; - - idx = std::min({ i * gainG / div, (kGammaLookupSize - 1) }); - params_->green[i] = gammaTable_[idx]; - - idx = std::min({ i * gainB / div, (kGammaLookupSize - 1) }); - params_->blue[i] = gammaTable_[idx]; - } - - setIspParams.emit(); - /* \todo Switch to the libipa/algorithm.h API someday. */ /* @@ -374,6 +314,7 @@ void IPASoftSimple::processStats( * Calculate Mean Sample Value (MSV) according to formula from: * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf */ + const uint8_t blackLevel = context_.configuration.black.level; const unsigned int blackLevelHistIdx = blackLevel / (256 / SwIspStats::kYHistogramSize); const unsigned int histogramSize = @@ -423,7 +364,8 @@ void IPASoftSimple::processStats( LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV << " exp " << exposure_ << " again " << again_ - << " gain R/B " << gainR << "/" << gainB + << " gain R/B " << context_.activeState.gains.red + << "/" << context_.activeState.gains.blue << " black level " << static_cast(blackLevel); } From patchwork Wed Jun 26 07:20:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20409 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D101FBD87C for ; Wed, 26 Jun 2024 07:21:52 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 51458654C5; Wed, 26 Jun 2024 09:21:52 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="J4c1C0JO"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8D9B4654CC for ; Wed, 26 Jun 2024 09:21:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386503; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b+NEdK6aGpbhkihQwAGrm2tKWTaIlM7LK7uVIFk5v1E=; b=J4c1C0JOajC6hqj2M29I7aY6j/ghEz4KCQFhu3aDXIFSEvu517kgL1HZBadnMyMnEBV1wE L/15utdmKuG751C369sv9R5Gd0oKmrxuwO+IK2hLfXfsejkVIfCBoEp2/AuvdHRI+8p1Oq a4hwphhtDxa0PIkkDA7Agao0M5x529I= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-428-_lY57jnEMNu6ERzkSqH24Q-1; Wed, 26 Jun 2024 03:21:42 -0400 X-MC-Unique: _lY57jnEMNu6ERzkSqH24Q-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8113A1956060 for ; Wed, 26 Jun 2024 07:21:41 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 89D953000601; Wed, 26 Jun 2024 07:21:40 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 16/19] libcamera: software_isp: Use floating point for color parameters Date: Wed, 26 Jun 2024 09:20:57 +0200 Message-ID: <20240626072100.55497-17-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" It's more natural to represent color gains and black level as floating point numbers rather than using a particular pixel-related representation. double is used rather than float because it's a more common floating point type in libcamera algorithms. Otherwise there is no obvious reason to select one over the other here. The constructed color tables still use integer representation for efficiency. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/blc.cpp | 8 ++++---- src/ipa/simple/algorithms/colors.cpp | 26 ++++++++++++++------------ src/ipa/simple/ipa_context.h | 8 ++++---- 3 files changed, 22 insertions(+), 20 deletions(-) diff --git a/src/ipa/simple/algorithms/blc.cpp b/src/ipa/simple/algorithms/blc.cpp index e216138d..82c89dbb 100644 --- a/src/ipa/simple/algorithms/blc.cpp +++ b/src/ipa/simple/algorithms/blc.cpp @@ -26,7 +26,7 @@ int BlackLevel::init(IPAContext &context, { context.configuration.black.set = false; context.configuration.black.changed = true; - context.configuration.black.level = 255; + context.configuration.black.level = 1.0; return 0; } @@ -51,16 +51,16 @@ void BlackLevel::process(IPAContext &context, const unsigned int total = std::accumulate(begin(histogram), end(histogram), 0); const unsigned int pixelThreshold = ignoredPercentage_ * total; - const unsigned int histogramRatio = 256 / SwIspStats::kYHistogramSize; const unsigned int currentBlackIdx = - context.configuration.black.level / histogramRatio; + context.configuration.black.level * SwIspStats::kYHistogramSize; for (unsigned int i = 0, seen = 0; i < currentBlackIdx && i < SwIspStats::kYHistogramSize; i++) { seen += histogram[i]; if (seen >= pixelThreshold) { - context.configuration.black.level = i * histogramRatio; + context.configuration.black.level = + static_cast(i) / SwIspStats::kYHistogramSize; context.configuration.black.changed = true; LOG(IPASoftBL, Debug) << "Auto-set black level: " diff --git a/src/ipa/simple/algorithms/colors.cpp b/src/ipa/simple/algorithms/colors.cpp index d4ab409c..726835d9 100644 --- a/src/ipa/simple/algorithms/colors.cpp +++ b/src/ipa/simple/algorithms/colors.cpp @@ -36,7 +36,7 @@ int Colors::init(IPAContext &context, updateGammaTable(context); auto &gains = context.activeState.gains; - gains.red = gains.green = gains.blue = 256; + gains.red = gains.green = gains.blue = 1.0; return 0; } @@ -45,7 +45,7 @@ void Colors::updateGammaTable(IPAContext &context) { auto &gammaTable = context.activeState.gammaTable; const unsigned int blackIndex = - context.configuration.black.level * IPAActiveState::kGammaLookupSize / 256; + context.configuration.black.level * IPAActiveState::kGammaLookupSize; std::fill(gammaTable.begin(), gammaTable.begin() + blackIndex, 0); const float divisor = kGammaLookupSize - blackIndex - 1.0; for (unsigned int i = blackIndex; i < kGammaLookupSize; i++) @@ -65,15 +65,18 @@ void Colors::prepare(IPAContext &context, auto &gains = context.activeState.gains; auto &gammaTable = context.activeState.gammaTable; for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { - constexpr unsigned int div = - static_cast(DebayerParams::kRGBLookupSize) * 256 / kGammaLookupSize; + constexpr double div = + static_cast(DebayerParams::kRGBLookupSize) / kGammaLookupSize; /* Apply gamma after gain! */ unsigned int idx; - idx = std::min({ i * gains.red / div, kGammaLookupSize - 1 }); + idx = std::min({ static_cast(i * gains.red / div), + kGammaLookupSize - 1 }); params->red[i] = gammaTable[idx]; - idx = std::min({ i * gains.green / div, kGammaLookupSize - 1 }); + idx = std::min({ static_cast(i * gains.green / div), + kGammaLookupSize - 1 }); params->green[i] = gammaTable[idx]; - idx = std::min({ i * gains.blue / div, kGammaLookupSize - 1 }); + idx = std::min({ static_cast(i * gains.blue / div), + kGammaLookupSize - 1 }); params->blue[i] = gammaTable[idx]; } } @@ -85,7 +88,7 @@ void Colors::process(IPAContext &context, [[maybe_unused]] ControlList &metadata) { const SwIspStats::Histogram &histogram = stats->yHistogram; - const uint8_t blackLevel = context.configuration.black.level; + const double blackLevel = context.configuration.black.level; /* * Black level must be subtracted to get the correct AWB ratios, they @@ -102,12 +105,11 @@ void Colors::process(IPAContext &context, /* * Calculate red and blue gains for AWB. * Clamp max gain at 4.0, this also avoids 0 division. - * Gain: 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. */ auto &gains = context.activeState.gains; - gains.red = sumR <= sumG / 4 ? 1024 : 256 * sumG / sumR; - gains.blue = sumB <= sumG / 4 ? 1024 : 256 * sumG / sumB; - /* Green gain is fixed to 256 */ + gains.red = sumR <= sumG / 4 ? 4.0 : static_cast(sumG) / sumR; + gains.blue = sumB <= sumG / 4 ? 4.0 : static_cast(sumG) / sumB; + /* Green gain is fixed to 1.0 */ LOG(IPASoftColors, Debug) << "gain R/B " << gains.red << "/" << gains.blue; } diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index 991a45b1..3e0d9b51 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -20,7 +20,7 @@ namespace ipa::soft { struct IPASessionConfiguration { float gamma; struct { - uint8_t level; + double level; bool set; bool changed; } black; @@ -28,9 +28,9 @@ struct IPASessionConfiguration { struct IPAActiveState { struct { - unsigned int red; - unsigned int green; - unsigned int blue; + double red; + double green; + double blue; } gains; static constexpr unsigned int kGammaLookupSize = 1024; std::array gammaTable; From patchwork Wed Jun 26 07:20:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20412 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 79238BD87C for ; Wed, 26 Jun 2024 07:21:59 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 01EF7654C2; Wed, 26 Jun 2024 09:21:59 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="OjsSMTN/"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4DD59654C6 for ; Wed, 26 Jun 2024 09:21:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386515; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wD8iiFHe98rl6wXwuDebVUALRRkeJSufmD9aCZg0Z5A=; b=OjsSMTN/+UaAinccbeLdGUt4EfqQRg8c27RxvdL33LBJw7cdFAMragic8RkdzkJQpSZszA xUbv5VwJh1qdvVhoMT2r/hmRcJe2kbi71dYticxRTTTTz4qzhvYTpM//rJWwiRay/d7CwU GiaqtVdvRx4leCy4zJAoncroQIRP6X0= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-543-6-WR7245P2avA6mNwpobag-1; Wed, 26 Jun 2024 03:21:43 -0400 X-MC-Unique: 6-WR7245P2avA6mNwpobag-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id DF39619560AD for ; Wed, 26 Jun 2024 07:21:42 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id E31ED300022B; Wed, 26 Jun 2024 07:21:41 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 17/19] libcamera: software_isp: Use DelayedControls Date: Wed, 26 Jun 2024 09:20:58 +0200 Message-ID: <20240626072100.55497-18-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Use the standard libcamera mechanism to report the "current" controls rather than delaying updates by counting from the last update. MY SPECULATION -- valid or not?: A problem is that with software ISP we cannot be sure about the sensor delay. The original implementation simply skips exposure updates for 2 frames, which should be enough in most cases. After this change, we assume the delay being exactly 2 frames, which may or may not be correct and may work with outdated values if the delay is shorter. This patch also prepares moving exposure+gain to an algorithm module where the original delay mechanism would be a (possibly unnecessary) complication. Resolves software ISP TODO #11 + #12. Signed-off-by: Milan Zamazal --- src/ipa/simple/soft_simple.cpp | 16 +------------ src/libcamera/pipeline/simple/simple.cpp | 17 ++++++++++++-- src/libcamera/software_isp/TODO | 29 ------------------------ 3 files changed, 16 insertions(+), 46 deletions(-) diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 25f6a71e..6b31fd08 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -59,8 +59,7 @@ class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module public: IPASoftSimple() : params_(nullptr), stats_(nullptr), - context_({ {}, {}, { kMaxFrameContexts } }), - ignoreUpdates_(0) + context_({ {}, {}, { kMaxFrameContexts } }) { } @@ -98,7 +97,6 @@ private: int32_t exposure_; double againMin_, againMax_, againMinStep_; double again_; - unsigned int ignoreUpdates_; }; IPASoftSimple::~IPASoftSimple() @@ -300,16 +298,6 @@ void IPASoftSimple::processStats( /* \todo Switch to the libipa/algorithm.h API someday. */ - /* - * AE / AGC, use 2 frames delay to make sure that the exposure and - * the gain set have applied to the camera sensor. - * \todo This could be handled better with DelayedControls. - */ - if (ignoreUpdates_ > 0) { - --ignoreUpdates_; - return; - } - /* * Calculate Mean Sample Value (MSV) according to formula from: * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf @@ -358,8 +346,6 @@ void IPASoftSimple::processStats( ctrls.set(V4L2_CID_ANALOGUE_GAIN, static_cast(camHelper_ ? camHelper_->gainCode(again_) : again_)); - ignoreUpdates_ = 2; - setSensorControls.emit(ctrls); LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index a82d05f4..7208de86 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -32,6 +32,7 @@ #include "libcamera/internal/camera.h" #include "libcamera/internal/camera_sensor.h" #include "libcamera/internal/converter.h" +#include "libcamera/internal/delayed_controls.h" #include "libcamera/internal/device_enumerator.h" #include "libcamera/internal/media_device.h" #include "libcamera/internal/pipeline_handler.h" @@ -277,6 +278,8 @@ public: std::vector configs_; std::map> formats_; + std::unique_ptr delayedCtrls_; + std::vector> conversionBuffers_; std::queue> conversionQueue_; bool useConversion_; @@ -896,12 +899,12 @@ void SimpleCameraData::ispStatsReady(uint32_t frame, uint32_t bufferId) /* \todo Use the DelayedControls class */ swIsp_->processStats( frame, bufferId, - sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, - V4L2_CID_EXPOSURE })); + delayedCtrls_->get(frame)); } void SimpleCameraData::setSensorControls(const ControlList &sensorControls) { + delayedCtrls_->push(sensorControls); ControlList ctrls(sensorControls); sensor_->setControls(&ctrls); } @@ -1282,6 +1285,16 @@ int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c) if (outputCfgs.empty()) return 0; + std::unordered_map params = { + { V4L2_CID_ANALOGUE_GAIN, { 2, false } }, + { V4L2_CID_EXPOSURE, { 2, false } }, + }; + data->delayedCtrls_ = + std::make_unique(data->sensor_->device(), + params); + data->video_->frameStart.connect(data->delayedCtrls_.get(), + &DelayedControls::applyControls); + StreamConfiguration inputCfg; inputCfg.pixelFormat = pipeConfig->captureFormat; inputCfg.size = pipeConfig->captureSize; diff --git a/src/libcamera/software_isp/TODO b/src/libcamera/software_isp/TODO index 6bdc5905..6b1cb57a 100644 --- a/src/libcamera/software_isp/TODO +++ b/src/libcamera/software_isp/TODO @@ -228,35 +228,6 @@ At some point, yes. --- -11. Improve handling the sensor controls which take effect with a delay - -> void IPASoftSimple::processStats(const ControlList &sensorControls) -> { -> ... -> /* -> * AE / AGC, use 2 frames delay to make sure that the exposure and -> * the gain set have applied to the camera sensor. -> */ -> if (ignore_updates_ > 0) { -> --ignore_updates_; -> return; -> } - -This could be handled better with DelayedControls. - ---- - -12. Use DelayedControls class in ispStatsReady() - -> void SimpleCameraData::ispStatsReady() -> { -> swIsp_->processStats(sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, -> V4L2_CID_EXPOSURE })); - -You should use the DelayedControls class. - ---- - 13. Improve black level and colour gains application I think the black level should eventually be moved before debayering, and From patchwork Wed Jun 26 07:20:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20410 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A3406BD87C for ; Wed, 26 Jun 2024 07:21:55 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5CE6E654C9; Wed, 26 Jun 2024 09:21:55 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="ChkvC7Dc"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6C89F654BB for ; Wed, 26 Jun 2024 09:21:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386507; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Uwi7lLl5bT4ZxRYZKuc2L4vWI+iOa7piBLMlFzSdL0o=; b=ChkvC7Dcaxhj3+VeSK8ixMCrMSqPz+K+uuAvzTFAnEYZtMgQK9M6PHe4JfgnsJqHuTWA86 ynj2XTy8olTQpo1/AQsmiD7usk2e23bBYJoSmTqPjjsu00QD+xnWxFeHvUacQt2xI/Zis0 jXjQFyBmatH5ACf9bwUHYpvc/gogUqc= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-679-LdVSWntWP6iUgYAVM9dU_A-1; Wed, 26 Jun 2024 03:21:45 -0400 X-MC-Unique: LdVSWntWP6iUgYAVM9dU_A-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8870119560AF for ; Wed, 26 Jun 2024 07:21:44 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 4FA053000221; Wed, 26 Jun 2024 07:21:43 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 18/19] libcamera: software_isp: Move exposure+gain to an algorithm module Date: Wed, 26 Jun 2024 09:20:59 +0200 Message-ID: <20240626072100.55497-19-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This is the last step to fully convert software ISP to Algorithm-based processing. The newly introduced frameContext.sensor properties are set, and the updated code moved, before calling Algorithm::process() to have the values up-to-date in stats processing. Resolves software ISP TODO #10. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/agc.cpp | 139 ++++++++++++++++++++++ src/ipa/simple/algorithms/agc.h | 36 ++++++ src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/data/uncalibrated.yaml | 1 + src/ipa/simple/ipa_context.h | 12 ++ src/ipa/simple/soft_simple.cpp | 159 +++++--------------------- src/libcamera/software_isp/TODO | 10 -- 7 files changed, 219 insertions(+), 139 deletions(-) create mode 100644 src/ipa/simple/algorithms/agc.cpp create mode 100644 src/ipa/simple/algorithms/agc.h diff --git a/src/ipa/simple/algorithms/agc.cpp b/src/ipa/simple/algorithms/agc.cpp new file mode 100644 index 00000000..c43c1965 --- /dev/null +++ b/src/ipa/simple/algorithms/agc.cpp @@ -0,0 +1,139 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Exposure and gain + */ + +#include "agc.h" + +#include + +#include + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoftExposure) + +namespace ipa::soft::algorithms { + +/* + * The number of bins to use for the optimal exposure calculations. + */ +static constexpr unsigned int kExposureBinsCount = 5; + +/* + * The exposure is optimal when the mean sample value of the histogram is + * in the middle of the range. + */ +static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; + +/* + * The below value implements the hysteresis for the exposure adjustment. + * It is small enough to have the exposure close to the optimal, and is big + * enough to prevent the exposure from wobbling around the optimal value. + */ +static constexpr float kExposureSatisfactory = 0.2; + +Agc::Agc() +{ +} + +void Agc::updateExposure(IPAContext &context, double exposureMSV) +{ + /* + * kExpDenominator of 10 gives ~10% increment/decrement; + * kExpDenominator of 5 - about ~20% + */ + static constexpr uint8_t kExpDenominator = 10; + static constexpr uint8_t kExpNumeratorUp = kExpDenominator + 1; + static constexpr uint8_t kExpNumeratorDown = kExpDenominator - 1; + + double next; + int32_t &exposure = context.activeState.exposure; + double &again = context.activeState.again; + + if (exposureMSV < kExposureOptimal - kExposureSatisfactory) { + next = exposure * kExpNumeratorUp / kExpDenominator; + if (next - exposure < 1) + exposure += 1; + else + exposure = next; + if (exposure >= context.configuration.agc.exposureMax) { + next = again * kExpNumeratorUp / kExpDenominator; + if (next - again < context.configuration.agc.againMinStep) + again += context.configuration.agc.againMinStep; + else + again = next; + } + } + + if (exposureMSV > kExposureOptimal + kExposureSatisfactory) { + if (exposure == context.configuration.agc.exposureMax && + again > context.configuration.agc.againMin) { + next = again * kExpNumeratorDown / kExpDenominator; + if (again - next < context.configuration.agc.againMinStep) + again -= context.configuration.agc.againMinStep; + else + again = next; + } else { + next = exposure * kExpNumeratorDown / kExpDenominator; + if (exposure - next < 1) + exposure -= 1; + else + exposure = next; + } + } + + exposure = std::clamp(exposure, context.configuration.agc.exposureMin, + context.configuration.agc.exposureMax); + again = std::clamp(again, context.configuration.agc.againMin, + context.configuration.agc.againMax); + + LOG(IPASoftExposure, Debug) + << "exposureMSV " << exposureMSV + << " exp " << exposure << " again " << again; +} + +void Agc::process(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + const SwIspStats *stats, + [[maybe_unused]] ControlList &metadata) +{ + /* + * Calculate Mean Sample Value (MSV) according to formula from: + * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf + */ + const auto &histogram = stats->yHistogram; + const unsigned int blackLevelHistIdx = + context.configuration.black.level * SwIspStats::kYHistogramSize; + const unsigned int histogramSize = + SwIspStats::kYHistogramSize - blackLevelHistIdx; + const unsigned int yHistValsPerBin = histogramSize / kExposureBinsCount; + const unsigned int yHistValsPerBinMod = + histogramSize / (histogramSize % kExposureBinsCount + 1); + int exposureBins[kExposureBinsCount] = {}; + unsigned int denom = 0; + unsigned int num = 0; + + for (unsigned int i = 0; i < histogramSize; i++) { + unsigned int idx = (i - (i / yHistValsPerBinMod)) / yHistValsPerBin; + exposureBins[idx] += histogram[blackLevelHistIdx + i]; + } + + for (unsigned int i = 0; i < kExposureBinsCount; i++) { + LOG(IPASoftExposure, Debug) << i << ": " << exposureBins[i]; + denom += exposureBins[i]; + num += exposureBins[i] * (i + 1); + } + + float exposureMSV = (denom == 0 ? 0 : static_cast(num) / denom); + updateExposure(context, exposureMSV); +} + +REGISTER_IPA_ALGORITHM(Agc, "Agc") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/agc.h b/src/ipa/simple/algorithms/agc.h new file mode 100644 index 00000000..a4aa656c --- /dev/null +++ b/src/ipa/simple/algorithms/agc.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Exposure and gain + */ + +#pragma once + +#include "libcamera/internal/software_isp/swisp_stats.h" + +#include "algorithm.h" +#include "ipa_context.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class Agc : public Algorithm +{ +public: + Agc(); + ~Agc() = default; + + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const SwIspStats *stats, + ControlList &metadata) override; + +private: + void updateExposure(IPAContext &context, double exposureMSV); +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index 324c7b4e..0b178e17 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -1,6 +1,7 @@ # SPDX-License-Identifier: CC0-1.0 soft_simple_ipa_algorithms = files([ + 'agc.cpp', 'blc.cpp', 'colors.cpp', ]) diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index 7e5149e5..7489c81f 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -5,4 +5,5 @@ version: 1 algorithms: - BlackLevel: - Colors: + - Agc: ... diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index 3e0d9b51..0819b1d2 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -11,6 +11,8 @@ #include #include +#include + #include namespace libcamera { @@ -19,6 +21,10 @@ namespace ipa::soft { struct IPASessionConfiguration { float gamma; + struct { + int32_t exposureMin, exposureMax; + double againMin, againMax, againMinStep; + } agc; struct { double level; bool set; @@ -32,11 +38,17 @@ struct IPAActiveState { double green; double blue; } gains; + int32_t exposure; + double again; static constexpr unsigned int kGammaLookupSize = 1024; std::array gammaTable; }; struct IPAFrameContext : public FrameContext { + struct { + uint32_t exposure; + double gain; + } sensor; }; struct IPAContext { diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 6b31fd08..6d7d90b6 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -34,23 +34,6 @@ LOG_DEFINE_CATEGORY(IPASoft) namespace ipa::soft { -/* - * The number of bins to use for the optimal exposure calculations. - */ -static constexpr unsigned int kExposureBinsCount = 5; - -/* - * The exposure is optimal when the mean sample value of the histogram is - * in the middle of the range. - */ -static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; - -/* - * The below value implements the hysteresis for the exposure adjustment. - * It is small enough to have the exposure close to the optimal, and is big - * enough to prevent the exposure from wobbling around the optimal value. - */ -static constexpr float kExposureSatisfactory = 0.2; /* Maximum number of frame contexts to be held */ static constexpr uint32_t kMaxFrameContexts = 16; @@ -58,8 +41,7 @@ class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() - : params_(nullptr), stats_(nullptr), - context_({ {}, {}, { kMaxFrameContexts } }) + : context_({ {}, {}, { kMaxFrameContexts } }) { } @@ -92,11 +74,6 @@ private: /* Local parameter storage */ struct IPAContext context_; - - int32_t exposureMin_, exposureMax_; - int32_t exposure_; - double againMin_, againMax_, againMinStep_; - double again_; }; IPASoftSimple::~IPASoftSimple() @@ -207,20 +184,23 @@ int IPASoftSimple::configure(const IPAConfigInfo &configInfo) const ControlInfo &exposureInfo = sensorInfoMap_.find(V4L2_CID_EXPOSURE)->second; const ControlInfo &gainInfo = sensorInfoMap_.find(V4L2_CID_ANALOGUE_GAIN)->second; - exposureMin_ = exposureInfo.min().get(); - exposureMax_ = exposureInfo.max().get(); - if (!exposureMin_) { + context_.configuration.agc.exposureMin = exposureInfo.min().get(); + context_.configuration.agc.exposureMax = exposureInfo.max().get(); + if (!context_.configuration.agc.exposureMin) { LOG(IPASoft, Warning) << "Minimum exposure is zero, that can't be linear"; - exposureMin_ = 1; + context_.configuration.agc.exposureMin = 1; } int32_t againMin = gainInfo.min().get(); int32_t againMax = gainInfo.max().get(); if (camHelper_) { - againMin_ = camHelper_->gain(againMin); - againMax_ = camHelper_->gain(againMax); - againMinStep_ = (againMax_ - againMin_) / 100.0; + context_.configuration.agc.againMin = camHelper_->gain(againMin); + context_.configuration.agc.againMax = camHelper_->gain(againMax); + context_.configuration.agc.againMinStep = + (context_.configuration.agc.againMax - + context_.configuration.agc.againMin) / + 100.0; } else { /* * The camera sensor gain (g) is usually not equal to the value written @@ -232,13 +212,14 @@ int IPASoftSimple::configure(const IPAConfigInfo &configInfo) * the AGC algorithm (abrupt near one edge, and very small near the * other) we limit the range of the gain values used. */ - againMax_ = againMax; + context_.configuration.agc.againMax = againMax; if (!againMin) { LOG(IPASoft, Warning) << "Minimum gain is zero, that can't be linear"; - againMin_ = std::min(100, againMin / 2 + againMax / 2); + context_.configuration.agc.againMin = + std::min(100, againMin / 2 + againMax / 2); } - againMinStep_ = 1.0; + context_.configuration.agc.againMinStep = 1.0; } for (auto const &algo : algorithms()) { @@ -247,9 +228,12 @@ int IPASoftSimple::configure(const IPAConfigInfo &configInfo) return ret; } - LOG(IPASoft, Info) << "Exposure " << exposureMin_ << "-" << exposureMax_ - << ", gain " << againMin_ << "-" << againMax_ - << " (" << againMinStep_ << ")"; + LOG(IPASoft, Info) + << "Exposure " << context_.configuration.agc.exposureMin << "-" + << context_.configuration.agc.exposureMax + << ", gain " << context_.configuration.agc.againMin << "-" + << context_.configuration.agc.againMax + << " (" << context_.configuration.agc.againMinStep << ")"; return 0; } @@ -286,6 +270,12 @@ void IPASoftSimple::processStats( const ControlList &sensorControls) { IPAFrameContext &frameContext = context_.frameContexts.get(frame); + + frameContext.sensor.exposure = + sensorControls.get(V4L2_CID_EXPOSURE).get(); + int32_t again = sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get(); + frameContext.sensor.gain = camHelper_ ? camHelper_->gain(again) : again; + /* * \todo Software ISP currently doesn't produce any metadata so it is * possible to use a dummy metadata instance here. But metadata should be @@ -296,37 +286,6 @@ void IPASoftSimple::processStats( algo->process(context_, frame, frameContext, stats_, metadata); } - /* \todo Switch to the libipa/algorithm.h API someday. */ - - /* - * Calculate Mean Sample Value (MSV) according to formula from: - * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf - */ - const uint8_t blackLevel = context_.configuration.black.level; - const unsigned int blackLevelHistIdx = - blackLevel / (256 / SwIspStats::kYHistogramSize); - const unsigned int histogramSize = - SwIspStats::kYHistogramSize - blackLevelHistIdx; - const unsigned int yHistValsPerBin = histogramSize / kExposureBinsCount; - const unsigned int yHistValsPerBinMod = - histogramSize / (histogramSize % kExposureBinsCount + 1); - int exposureBins[kExposureBinsCount] = {}; - unsigned int denom = 0; - unsigned int num = 0; - - for (unsigned int i = 0; i < histogramSize; i++) { - unsigned int idx = (i - (i / yHistValsPerBinMod)) / yHistValsPerBin; - exposureBins[idx] += stats_->yHistogram[blackLevelHistIdx + i]; - } - - for (unsigned int i = 0; i < kExposureBinsCount; i++) { - LOG(IPASoft, Debug) << i << ": " << exposureBins[i]; - denom += exposureBins[i]; - num += exposureBins[i] * (i + 1); - } - - float exposureMSV = static_cast(num) / denom; - /* Sanity check */ if (!sensorControls.contains(V4L2_CID_EXPOSURE) || !sensorControls.contains(V4L2_CID_ANALOGUE_GAIN)) { @@ -334,72 +293,14 @@ void IPASoftSimple::processStats( return; } - exposure_ = sensorControls.get(V4L2_CID_EXPOSURE).get(); - int32_t again = sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get(); - again_ = camHelper_ ? camHelper_->gain(again) : again; - - updateExposure(exposureMSV); - ControlList ctrls(sensorInfoMap_); - ctrls.set(V4L2_CID_EXPOSURE, exposure_); + auto &againNew = context_.activeState.again; + ctrls.set(V4L2_CID_EXPOSURE, context_.activeState.exposure); ctrls.set(V4L2_CID_ANALOGUE_GAIN, - static_cast(camHelper_ ? camHelper_->gainCode(again_) : again_)); + static_cast(camHelper_ ? camHelper_->gainCode(againNew) : againNew)); setSensorControls.emit(ctrls); - - LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV - << " exp " << exposure_ << " again " << again_ - << " gain R/B " << context_.activeState.gains.red - << "/" << context_.activeState.gains.blue - << " black level " << static_cast(blackLevel); -} - -void IPASoftSimple::updateExposure(double exposureMSV) -{ - /* - * kExpDenominator of 10 gives ~10% increment/decrement; - * kExpDenominator of 5 - about ~20% - */ - static constexpr uint8_t kExpDenominator = 10; - static constexpr uint8_t kExpNumeratorUp = kExpDenominator + 1; - static constexpr uint8_t kExpNumeratorDown = kExpDenominator - 1; - - double next; - - if (exposureMSV < kExposureOptimal - kExposureSatisfactory) { - next = exposure_ * kExpNumeratorUp / kExpDenominator; - if (next - exposure_ < 1) - exposure_ += 1; - else - exposure_ = next; - if (exposure_ >= exposureMax_) { - next = again_ * kExpNumeratorUp / kExpDenominator; - if (next - again_ < againMinStep_) - again_ += againMinStep_; - else - again_ = next; - } - } - - if (exposureMSV > kExposureOptimal + kExposureSatisfactory) { - if (exposure_ == exposureMax_ && again_ > againMin_) { - next = again_ * kExpNumeratorDown / kExpDenominator; - if (again_ - next < againMinStep_) - again_ -= againMinStep_; - else - again_ = next; - } else { - next = exposure_ * kExpNumeratorDown / kExpDenominator; - if (exposure_ - next < 1) - exposure_ -= 1; - else - exposure_ = next; - } - } - - exposure_ = std::clamp(exposure_, exposureMin_, exposureMax_); - again_ = std::clamp(again_, againMin_, againMax_); } std::string IPASoftSimple::logPrefix() const diff --git a/src/libcamera/software_isp/TODO b/src/libcamera/software_isp/TODO index 6b1cb57a..42828895 100644 --- a/src/libcamera/software_isp/TODO +++ b/src/libcamera/software_isp/TODO @@ -218,16 +218,6 @@ Yes, because, well... all the other IPAs were doing that... --- -10. Switch to libipa/algorithm.h API in processStats - ->> void IPASoftSimple::processStats(const ControlList &sensorControls) ->> -> Do you envision switching to the libipa/algorithm.h API at some point ? - -At some point, yes. - ---- - 13. Improve black level and colour gains application I think the black level should eventually be moved before debayering, and From patchwork Wed Jun 26 07:21:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20411 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 968E1BD87C for ; Wed, 26 Jun 2024 07:21:56 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 43D73654C2; Wed, 26 Jun 2024 09:21:56 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="AxIsdkTZ"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 25483654C9 for ; Wed, 26 Jun 2024 09:21:49 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719386508; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Dw+Xm6qnLKyAnt+RsGrdnpTqaaF6AqT57ojMpeKNQnk=; b=AxIsdkTZ2Sf+tx0pmz0lyMzYV7IJBnbdyERbfzACBfBPT/eUPyZ22ZeB87ilouMKDSRBCI de5ElU4FNe801rbdPm+pntZ4dXWKxqyEeDt2OC3IzcYySftU0F6BS9s3VeVRAP8HGPOeIr gqKeXxld/3KhpJGtKzfFcaVz1ohuFEI= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-26-vhaZQbWBN2Whx4iwfCkEdQ-1; Wed, 26 Jun 2024 03:21:46 -0400 X-MC-Unique: vhaZQbWBN2Whx4iwfCkEdQ-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id E293F19560AB for ; Wed, 26 Jun 2024 07:21:45 +0000 (UTC) Received: from nuthatch.brq.redhat.com (unknown [10.43.17.159]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id E9DEB3000225; Wed, 26 Jun 2024 07:21:44 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal Subject: [PATCH 19/19] libcamera: software_isp: Update black level only on exposure changes Date: Wed, 26 Jun 2024 09:21:00 +0200 Message-ID: <20240626072100.55497-20-mzamazal@redhat.com> In-Reply-To: <20240626072100.55497-1-mzamazal@redhat.com> References: <20240626072100.55497-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The black level is likely to get updated, if ever, only after exposure or gain changes. Don't compute its possible updates if exposure and gain are unchanged. It's probably not worth trying to implement something more sophisticated. Better to spend the effort on supporting tuning files. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/blc.cpp | 9 +++++++-- src/ipa/simple/algorithms/blc.h | 4 ++++ 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/src/ipa/simple/algorithms/blc.cpp b/src/ipa/simple/algorithms/blc.cpp index 82c89dbb..9025cb3a 100644 --- a/src/ipa/simple/algorithms/blc.cpp +++ b/src/ipa/simple/algorithms/blc.cpp @@ -32,11 +32,13 @@ int BlackLevel::init(IPAContext &context, void BlackLevel::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, - [[maybe_unused]] IPAFrameContext &frameContext, + IPAFrameContext &frameContext, const SwIspStats *stats, [[maybe_unused]] ControlList &metadata) { - if (context.configuration.black.set) { + if (context.configuration.black.set && + frameContext.sensor.exposure == exposure_ && + frameContext.sensor.gain == gain_) { context.configuration.black.changed = false; return; } @@ -61,7 +63,10 @@ void BlackLevel::process(IPAContext &context, if (seen >= pixelThreshold) { context.configuration.black.level = static_cast(i) / SwIspStats::kYHistogramSize; + context.configuration.black.set = true; context.configuration.black.changed = true; + exposure_ = frameContext.sensor.exposure; + gain_ = frameContext.sensor.gain; LOG(IPASoftBL, Debug) << "Auto-set black level: " << i << "/" << SwIspStats::kYHistogramSize diff --git a/src/ipa/simple/algorithms/blc.h b/src/ipa/simple/algorithms/blc.h index c8b8d92d..4ddd2c94 100644 --- a/src/ipa/simple/algorithms/blc.h +++ b/src/ipa/simple/algorithms/blc.h @@ -30,6 +30,10 @@ public: IPAFrameContext &frameContext, const SwIspStats *stats, ControlList &metadata) override; + +private: + uint32_t exposure_; + double gain_; }; } /* namespace ipa::soft::algorithms */