From patchwork Wed Jul 3 17:51:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20542 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2513ABEFBE for ; Wed, 3 Jul 2024 17:51:39 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 94E2863334; Wed, 3 Jul 2024 19:51:38 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="icFRcVis"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A8D7762E25 for ; Wed, 3 Jul 2024 19:51:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029091; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oIlrHc/1B4mCkerAdCOxDwu35utE13tK/dx0nfLF0OI=; b=icFRcVismR6OTio8ONmmg/As3VmrpV9SZt4C/iTJIChfzwZocyUJCausd9Fz/pdOC5Jo1+ gPqAmNruyGWF/92tJahqQ+6EM3Nlu1KavUI8zJiSzsUtiYhjomukeAKlmwXohe4lmQYcuy RcZ+WpOIYmCqbn8pz7xzQwzyNS/czG4= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-237-PKKvSOM3OmOWIklLxYvr0A-1; Wed, 03 Jul 2024 13:51:30 -0400 X-MC-Unique: PKKvSOM3OmOWIklLxYvr0A-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 2EBF91955BCC; Wed, 3 Jul 2024 17:51:29 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 8CF821954B0D; Wed, 3 Jul 2024 17:51:27 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 01/19] libcamera: software_isp: Remove superfluous includes Date: Wed, 3 Jul 2024 19:51:01 +0200 Message-ID: <20240703175119.1872585-2-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Remove unused libcamera internal headers bayer_format.h, framebuffer.h and mapped_frameBuffer.h. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- src/libcamera/software_isp/software_isp.cpp | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 20fb6f48..08a1afcc 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -16,10 +16,7 @@ #include #include -#include "libcamera/internal/bayer_format.h" -#include "libcamera/internal/framebuffer.h" #include "libcamera/internal/ipa_manager.h" -#include "libcamera/internal/mapped_framebuffer.h" #include "libcamera/internal/software_isp/debayer_params.h" #include "debayer_cpu.h" From patchwork Wed Jul 3 17:51:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20543 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 58246BEFBE for ; Wed, 3 Jul 2024 17:51:42 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 28D3263339; Wed, 3 Jul 2024 19:51:41 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="gmS9c4fo"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C0BF062C96 for ; Wed, 3 Jul 2024 19:51:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029093; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MxhWK9N1Ywoii8u6qPlsZyjJX1NOxClxhAn9uYiSVbk=; b=gmS9c4fobppLSp2DDnM/suMUYIjnZ2XyB9y7+4kGQ9AZnH9CYSnw0N27UaOhsZ+gV+hvrf hStNiYX03Chw4sleZrDQOXXAWUJmE37zZ/ZXlxF6vUsti0oN8JrmWpSLOwG+vWPIjqF9dB fVxtMWc2tEiEDjlboqMe4mvXc8l8jYQ= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-557-_HmrfdYaNnyPJzws133PFw-1; Wed, 03 Jul 2024 13:51:32 -0400 X-MC-Unique: _HmrfdYaNnyPJzws133PFw-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 5739219560B2; Wed, 3 Jul 2024 17:51:31 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 7EB791954B0D; Wed, 3 Jul 2024 17:51:29 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 02/19] libcamera: software_isp: Move BlackLevel to libcamera::ipa::soft Date: Wed, 3 Jul 2024 19:51:02 +0200 Message-ID: <20240703175119.1872585-3-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" To be in the same namespace as the other software ISP IPA stuff. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- src/ipa/simple/black_level.cpp | 5 +++++ src/ipa/simple/black_level.h | 4 ++++ 2 files changed, 9 insertions(+) diff --git a/src/ipa/simple/black_level.cpp b/src/ipa/simple/black_level.cpp index cc490eb5..37e0109c 100644 --- a/src/ipa/simple/black_level.cpp +++ b/src/ipa/simple/black_level.cpp @@ -15,6 +15,8 @@ namespace libcamera { LOG_DEFINE_CATEGORY(IPASoftBL) +namespace ipa::soft { + /** * \class BlackLevel * \brief Object providing black point level for software ISP @@ -85,4 +87,7 @@ void BlackLevel::update(SwIspStats::Histogram &yHistogram) } }; } + +} /* namespace ipa::soft */ + } /* namespace libcamera */ diff --git a/src/ipa/simple/black_level.h b/src/ipa/simple/black_level.h index 5e032f9f..a04230c9 100644 --- a/src/ipa/simple/black_level.h +++ b/src/ipa/simple/black_level.h @@ -14,6 +14,8 @@ namespace libcamera { +namespace ipa::soft { + class BlackLevel { public: @@ -26,4 +28,6 @@ private: bool blackLevelSet_; }; +} /* namespace ipa::soft */ + } /* namespace libcamera */ From patchwork Wed Jul 3 17:51:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20544 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B67B5BEFBE for ; Wed, 3 Jul 2024 17:51:45 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 68D0D6333F; Wed, 3 Jul 2024 19:51:43 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="fVGT5fem"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4FD5562E23 for ; Wed, 3 Jul 2024 19:51:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029096; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cBDv0d/VTKmYiWIPRUoqAdhAgGt/Q92W3o72oLQy79c=; b=fVGT5fem/CmLEgmoValaODWCcnl5VVHu+egsE3rk/NC8AVNWaxBAtF5g/HQhntU9beQgiX 0KdJRdoLMS8pvgFUyUAsoDdoZRTSJHA/Y/RK7GtKXvLzdPHeyB4YhvtqpIJKC/O62Jwm6p QQZF48YK2awipCd/vq8LDewUTDlsauM= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-519-3AkIvwYEN4WMNgIzSVPWMg-1; Wed, 03 Jul 2024 13:51:34 -0400 X-MC-Unique: 3AkIvwYEN4WMNgIzSVPWMg-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 45B291955BCF; Wed, 3 Jul 2024 17:51:33 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id A3C461954B0D; Wed, 3 Jul 2024 17:51:31 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 03/19] libcamera: simple: Fix a typo in a doc string Date: Wed, 3 Jul 2024 19:51:03 +0200 Message-ID: <20240703175119.1872585-4-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Rectify incorrect phrase in the sentence -- s/their share/they share/. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- src/libcamera/pipeline/simple/simple.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index eb36578e..7abe1d1d 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -163,7 +163,7 @@ LOG_DEFINE_CATEGORY(SimplePipeline) * handler has no a priori knowledge of. The pipeline handler thus implements a * heuristic to handle sharing of hardware resources in a generic fashion. * - * Two cameras are considered to be mutually exclusive if their share common + * Two cameras are considered to be mutually exclusive if they share common * pads along the pipeline from the camera sensor to the video node. An entity * can thus be used concurrently by multiple cameras, as long as pads are * distinct. From patchwork Wed Jul 3 17:51:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20545 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 3B386BEFBE for ; Wed, 3 Jul 2024 17:51:51 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E71B663361; Wed, 3 Jul 2024 19:51:50 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="cUXyR4Gs"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8E06A63331 for ; Wed, 3 Jul 2024 19:51:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029097; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Pd9GIB2cYy4KPFFO1D66dkzvdr0ywjwv5wWI59wfFYM=; b=cUXyR4Gs2h7tchrHKA9hVCeIYT2yQAbS1NoFh1agcI3dZNzHUB8997sIe9mBGLMVzWp1ob XOMWgJwyA7CL8koPnUZYKiaOqTGFrHD9eUwf7ptcl1YSbxm2P/loM3Voj9JJOSlTnZzFXg 6ER5SAYj5Sd4jNCn1w7FMLFk9dyMwCw= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-510-CERAIC07N_-xqAlH3Y0mjQ-1; Wed, 03 Jul 2024 13:51:35 -0400 X-MC-Unique: CERAIC07N_-xqAlH3Y0mjQ-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id E65D71955BC0; Wed, 3 Jul 2024 17:51:34 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id A471A1954B0D; Wed, 3 Jul 2024 17:51:33 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 04/19] libcamera: software_isp: Define skeletons for IPA refactoring Date: Wed, 3 Jul 2024 19:51:04 +0200 Message-ID: <20240703175119.1872585-5-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Software ISP image processing algorithms are currently defined in a simplified way, different from other libcamera pipelines. This is not good for several reasons: - It makes the software ISP code harder to understand due to its different structuring. - Adding more algorithms may make the code harder to understand generally. - Mass libcamera code changes may not be easily applicable to software ISP. - Algorithm sharing with other pipelines is not easily possible. This patch introduces basic software ISP IPA skeletons structured similarly to the other pipelines. The newly added files are currently not used or compiled and the general skeleton structures don't contain anything particular. It is just a preparation step for a larger refactoring and the code will be actually used and extended as needed in followup patches. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- src/ipa/simple/algorithms/algorithm.h | 22 +++++++++++ src/ipa/simple/ipa_context.cpp | 53 +++++++++++++++++++++++++++ src/ipa/simple/ipa_context.h | 34 +++++++++++++++++ src/ipa/simple/meson.build | 1 + src/ipa/simple/module.h | 28 ++++++++++++++ 5 files changed, 138 insertions(+) create mode 100644 src/ipa/simple/algorithms/algorithm.h create mode 100644 src/ipa/simple/ipa_context.cpp create mode 100644 src/ipa/simple/ipa_context.h create mode 100644 src/ipa/simple/module.h diff --git a/src/ipa/simple/algorithms/algorithm.h b/src/ipa/simple/algorithms/algorithm.h new file mode 100644 index 00000000..41f63170 --- /dev/null +++ b/src/ipa/simple/algorithms/algorithm.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Red Hat, Inc. + * + * Software ISP control algorithm interface + */ + +#pragma once + +#include + +#include "module.h" + +namespace libcamera { + +namespace ipa::soft { + +using Algorithm = libcamera::ipa::Algorithm; + +} /* namespace ipa::soft */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/ipa_context.cpp b/src/ipa/simple/ipa_context.cpp new file mode 100644 index 00000000..0898e591 --- /dev/null +++ b/src/ipa/simple/ipa_context.cpp @@ -0,0 +1,53 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Google Inc. + * Copyright (C) 2024 Red Hat Inc. + * + * Software ISP IPA Context + */ + +#include "ipa_context.h" + +/** + * \file ipa_context.h + * \brief Context and state information shared between the algorithms + */ + +namespace libcamera::ipa::soft { + +/** + * \struct IPASessionConfiguration + * \brief Session configuration for the IPA module + * + * The session configuration contains all IPA configuration parameters that + * remain constant during the capture session, from IPA module start to stop. + * It is typically set during the configure() operation of the IPA module, but + * may also be updated in the start() operation. + */ + +/** + * \struct IPAActiveState + * \brief The active state of the IPA algorithms + * + * The IPA is fed with the statistics generated from the latest frame captured + * by the hardware. The statistics are then processed by the IPA algorithms to + * compute ISP parameters required for the next frame capture. The current state + * of the algorithms is reflected through the IPAActiveState to store the values + * most recently computed by the IPA algorithms. + */ + +/** + * \struct IPAContext + * \brief Global IPA context data shared between all algorithms + * + * \var IPAContext::configuration + * \brief The IPA session configuration, immutable during the session + * + * \var IPAContext::frameContexts + * \brief Ring buffer of the IPAFrameContext(s) + * + * \var IPAContext::activeState + * \brief The current state of IPA algorithms + */ + +} /* namespace libcamera::ipa::soft */ diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h new file mode 100644 index 00000000..bc1235b6 --- /dev/null +++ b/src/ipa/simple/ipa_context.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Red Hat, Inc. + * + * Simple pipeline IPA Context + * + */ + +#pragma once + +#include + +namespace libcamera { + +namespace ipa::soft { + +struct IPASessionConfiguration { +}; + +struct IPAActiveState { +}; + +struct IPAFrameContext : public FrameContext { +}; + +struct IPAContext { + IPASessionConfiguration configuration; + IPAActiveState activeState; + FCQueue frameContexts; +}; + +} /* namespace ipa::soft */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build index 33d1c96a..363251fb 100644 --- a/src/ipa/simple/meson.build +++ b/src/ipa/simple/meson.build @@ -3,6 +3,7 @@ ipa_name = 'ipa_soft_simple' soft_simple_sources = files([ + 'ipa_context.cpp', 'soft_simple.cpp', 'black_level.cpp', ]) diff --git a/src/ipa/simple/module.h b/src/ipa/simple/module.h new file mode 100644 index 00000000..33a7d1db --- /dev/null +++ b/src/ipa/simple/module.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024 Red Hat, Inc. + * + * Software ISP IPA Module + */ + +#pragma once + +#include + +#include "libcamera/internal/software_isp/debayer_params.h" +#include "libcamera/internal/software_isp/swisp_stats.h" + +#include + +#include "ipa_context.h" + +namespace libcamera { + +namespace ipa::soft { + +using Module = ipa::Module; + +} /* namespace ipa::soft */ + +} /* namespace libcamera */ From patchwork Wed Jul 3 17:51:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20546 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 14F0FBEFBE for ; Wed, 3 Jul 2024 17:51:53 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id AD42163331; Wed, 3 Jul 2024 19:51:52 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="YYmqgKk9"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0FB9D632EA for ; Wed, 3 Jul 2024 19:51:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029099; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Pt48zVht3hFaa5dlYx3Pijez8CSay73FZWL1sWaKnuw=; b=YYmqgKk9Ty0vUldnh0f3rYu0cKjs89/q08OOkUX0Fp6KXqy/dYkb4k4m/C5Rol7luPVoKB w5Xn2jXwWDm7kxjzEzsgBRWdpnnbGYYyq0yuDKmczFbT0YT82t0Ecte0SFOoeyffSZXVIU 9uES3AYig9WvJFTmzAUAM/hgix1Hp90= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-584-0Gy8_rq3NSyI8Jmu_tTZJw-1; Wed, 03 Jul 2024 13:51:38 -0400 X-MC-Unique: 0Gy8_rq3NSyI8Jmu_tTZJw-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 069C219560B2; Wed, 3 Jul 2024 17:51:37 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 6FEF61954B0D; Wed, 3 Jul 2024 17:51:35 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 05/19] libcamera: software_isp: Let IPASoftSimple inherit Module Date: Wed, 3 Jul 2024 19:51:05 +0200 Message-ID: <20240703175119.1872585-6-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The Module class is a base class for all IPA modules. In addition, implement logPrefix() of the module for the softIPA. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- src/ipa/simple/soft_simple.cpp | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 3cc873a2..b41b24c6 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -29,6 +29,7 @@ #include "libipa/camera_sensor_helper.h" #include "black_level.h" +#include "module.h" namespace libcamera { LOG_DEFINE_CATEGORY(IPASoft) @@ -53,7 +54,7 @@ static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; */ static constexpr float kExposureSatisfactory = 0.2; -class IPASoftSimple : public ipa::soft::IPASoftInterface +class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() @@ -75,6 +76,9 @@ public: void processStats(const ControlList &sensorControls) override; +protected: + std::string logPrefix() const override; + private: void updateExposure(double exposureMSV); @@ -420,6 +424,11 @@ void IPASoftSimple::updateExposure(double exposureMSV) again_ = std::clamp(again_, againMin_, againMax_); } +std::string IPASoftSimple::logPrefix() const +{ + return "IPASoft"; +} + } /* namespace ipa::soft */ /* From patchwork Wed Jul 3 17:51:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20547 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 609B1C3240 for ; Wed, 3 Jul 2024 17:51:54 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5E25463361; Wed, 3 Jul 2024 19:51:53 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="W+CzzgkF"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4FD8062E22 for ; Wed, 3 Jul 2024 19:51:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029101; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=T3xgEn9AJJROtO74ZJAWDhFBjvjV/zHnxs+zv+Cm6t8=; b=W+CzzgkF1an3p/KWP5uEqHJGyTLBUy/pnD5xOmpOYVt3HKuhEZUJqBG4zP61ayZwrgLvZt vkQYgM+1o7ssF66rVdOh+jsQe3qqB3NKrCK5QoXA0qOo8Rckipsaq/wCrg9BqXsBzfdG6l UCQnB53JWXBWWagcCkUi2gtPek0qO9w= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-446-div3Gz8uNyagqLnHs4zE0Q-1; Wed, 03 Jul 2024 13:51:39 -0400 X-MC-Unique: div3Gz8uNyagqLnHs4zE0Q-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D182119560AE; Wed, 3 Jul 2024 17:51:38 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 4FA971954B0D; Wed, 3 Jul 2024 17:51:37 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 06/19] libcamera: software_isp: Make stats frame and buffer aware Date: Wed, 3 Jul 2024 19:51:06 +0200 Message-ID: <20240703175119.1872585-7-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds frame and bufferId arguments to stats related calls. Although the parameters are currently unused, because frame ids are not tracked and used and the stats buffer is passed around directly rather than being referred by its id, they bring the internal APIs closer to their counterparts in hardware pipelines. It serves as a preparation for followup patches that will introduce: - Frame number tracking in order to switch to DelayedControls (software ISP TODO #11 + #12). - A ring buffer for stats in order to improve passing the stats (software ISP TODO #2). Frame and buffer ids are unrelated for the given purposes but since they are passed together at the same places, the change is implemented as a single patch rather than two, basically the same, patches. Signed-off-by: Milan Zamazal --- .../libcamera/internal/software_isp/software_isp.h | 8 +++++--- include/libcamera/ipa/soft.mojom | 2 +- src/ipa/simple/soft_simple.cpp | 8 ++++++-- src/libcamera/pipeline/simple/simple.cpp | 11 +++++++---- src/libcamera/software_isp/debayer_cpu.cpp | 8 +++++++- src/libcamera/software_isp/software_isp.cpp | 11 +++++++---- src/libcamera/software_isp/swstats_cpu.cpp | 4 ++-- src/libcamera/software_isp/swstats_cpu.h | 4 ++-- 8 files changed, 37 insertions(+), 19 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index c5338c05..6a03b17f 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -65,7 +66,8 @@ public: int exportBuffers(unsigned int output, unsigned int count, std::vector> *buffers); - void processStats(const ControlList &sensorControls); + void processStats(const uint32_t frame, const uint32_t bufferId, + const ControlList &sensorControls); int start(); void stop(); @@ -77,13 +79,13 @@ public: Signal inputBufferReady; Signal outputBufferReady; - Signal<> ispStatsReady; + Signal ispStatsReady; Signal setSensorControls; private: void saveIspParams(); void setSensorCtrls(const ControlList &sensorControls); - void statsReady(); + void statsReady(uint32_t frame, uint32_t bufferId); void inputReady(FrameBuffer *input); void outputReady(FrameBuffer *output); diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index 3aa2066e..bd48ece9 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -19,7 +19,7 @@ interface IPASoftInterface { configure(libcamera.ControlInfoMap sensorCtrlInfoMap) => (int32 ret); - [async] processStats(libcamera.ControlList sensorControls); + [async] processStats(uint32 frame, uint32 bufferId, libcamera.ControlList sensorControls); }; interface IPASoftEventInterface { diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index b41b24c6..09e3c8f6 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -74,7 +74,8 @@ public: int start() override; void stop() override; - void processStats(const ControlList &sensorControls) override; + void processStats(const uint32_t frame, const uint32_t bufferId, + const ControlList &sensorControls) override; protected: std::string logPrefix() const override; @@ -248,7 +249,10 @@ void IPASoftSimple::stop() { } -void IPASoftSimple::processStats(const ControlList &sensorControls) +void IPASoftSimple::processStats( + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] const uint32_t bufferId, + const ControlList &sensorControls) { SwIspStats::Histogram histogram = stats_->yHistogram; if (ignoreUpdates_ > 0) diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 7abe1d1d..b1bf0d16 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -290,7 +291,7 @@ private: void conversionInputDone(FrameBuffer *buffer); void conversionOutputDone(FrameBuffer *buffer); - void ispStatsReady(); + void ispStatsReady(uint32_t frame, uint32_t bufferId); void setSensorControls(const ControlList &sensorControls); }; @@ -890,11 +891,13 @@ void SimpleCameraData::conversionOutputDone(FrameBuffer *buffer) pipe->completeRequest(request); } -void SimpleCameraData::ispStatsReady() +void SimpleCameraData::ispStatsReady(uint32_t frame, uint32_t bufferId) { /* \todo Use the DelayedControls class */ - swIsp_->processStats(sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, - V4L2_CID_EXPOSURE })); + swIsp_->processStats( + frame, bufferId, + sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, + V4L2_CID_EXPOSURE })); } void SimpleCameraData::setSensorControls(const ControlList &sensorControls) diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index f8d2677d..1575cedb 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -784,7 +784,13 @@ void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams } } - stats_->finishFrame(); + /* + * Frame and buffer ids are currently not used, so pass zeros as parameters. + * + * \todo Pass real values once frame is passed here and stats buffer passing + * is changed. + */ + stats_->finishFrame(0, 0); outputBufferReady.emit(output); inputBufferReady.emit(input); } diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 08a1afcc..3fc4a64b 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -156,15 +156,18 @@ SoftwareIsp::~SoftwareIsp() /** * \brief Process the statistics gathered + * \param[in] frame The frame number + * \param[in] bufferId ID of the statistics buffer * \param[in] sensorControls The sensor controls * * Requests the IPA to calculate new parameters for ISP and new control * values for the sensor. */ -void SoftwareIsp::processStats(const ControlList &sensorControls) +void SoftwareIsp::processStats(const uint32_t frame, const uint32_t bufferId, + const ControlList &sensorControls) { ASSERT(ipa_); - ipa_->processStats(sensorControls); + ipa_->processStats(frame, bufferId, sensorControls); } /** @@ -356,9 +359,9 @@ void SoftwareIsp::setSensorCtrls(const ControlList &sensorControls) setSensorControls.emit(sensorControls); } -void SoftwareIsp::statsReady() +void SoftwareIsp::statsReady(uint32_t frame, uint32_t bufferId) { - ispStatsReady.emit(); + ispStatsReady.emit(frame, bufferId); } void SoftwareIsp::inputReady(FrameBuffer *input) diff --git a/src/libcamera/software_isp/swstats_cpu.cpp b/src/libcamera/software_isp/swstats_cpu.cpp index 815c4d4f..a9b1f35a 100644 --- a/src/libcamera/software_isp/swstats_cpu.cpp +++ b/src/libcamera/software_isp/swstats_cpu.cpp @@ -314,10 +314,10 @@ void SwStatsCpu::startFrame(void) * * This may only be called after a successful setWindow() call. */ -void SwStatsCpu::finishFrame(void) +void SwStatsCpu::finishFrame(uint32_t frame, uint32_t bufferId) { *sharedStats_ = stats_; - statsReady.emit(); + statsReady.emit(frame, bufferId); } /** diff --git a/src/libcamera/software_isp/swstats_cpu.h b/src/libcamera/software_isp/swstats_cpu.h index 363e326f..26a2f462 100644 --- a/src/libcamera/software_isp/swstats_cpu.h +++ b/src/libcamera/software_isp/swstats_cpu.h @@ -41,7 +41,7 @@ public: int configure(const StreamConfiguration &inputCfg); void setWindow(const Rectangle &window); void startFrame(); - void finishFrame(); + void finishFrame(uint32_t frame, uint32_t bufferId); void processLine0(unsigned int y, const uint8_t *src[]) { @@ -61,7 +61,7 @@ public: (this->*stats2_)(src); } - Signal<> statsReady; + Signal statsReady; private: using statsProcessFn = void (SwStatsCpu::*)(const uint8_t *src[]); From patchwork Wed Jul 3 17:51:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20548 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E3219BEFBE for ; Wed, 3 Jul 2024 17:51:55 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 1512B63360; Wed, 3 Jul 2024 19:51:55 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="MiNo96nt"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 080AC62E27 for ; Wed, 3 Jul 2024 19:51:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029103; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DpD1SrsK+tr3SOAopeWApMTYw8NHevFi/7WDQdrN71o=; b=MiNo96ntGnIY56oSvvpfKc6DBVjGSTdw+cwp2BkBYfu12edky+Ggdwc0QyaIVkU2FJxXdj tZpDRc63/jwfBLTgnzOJ9WKsurhgv3E3afjMuWRQmaY3vvJIrs2vQJrA0+yhTrtyanUkoh lHigSAZuay9VMsQHyFFoxHfXWvLqOJA= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-258-aq0qzKQzNWquZA7_YkF7DQ-1; Wed, 03 Jul 2024 13:51:41 -0400 X-MC-Unique: aq0qzKQzNWquZA7_YkF7DQ-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id B097B19560B4; Wed, 3 Jul 2024 17:51:40 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 239801954B0D; Wed, 3 Jul 2024 17:51:38 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 07/19] libcamera: software_isp: Remove final dots in debayer.cpp docstrings Date: Wed, 3 Jul 2024 19:51:07 +0200 Message-ID: <20240703175119.1872585-8-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" As required by the policy and the style checker. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- src/libcamera/software_isp/debayer.cpp | 48 +++++++++++++------------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/src/libcamera/software_isp/debayer.cpp b/src/libcamera/software_isp/debayer.cpp index f4a299d5..4d101798 100644 --- a/src/libcamera/software_isp/debayer.cpp +++ b/src/libcamera/software_isp/debayer.cpp @@ -58,47 +58,47 @@ Debayer::~Debayer() /** * \fn int Debayer::configure(const StreamConfiguration &inputCfg, const std::vector> &outputCfgs) - * \brief Configure the debayer object according to the passed in parameters. - * \param[in] inputCfg The input configuration. - * \param[in] outputCfgs The output configurations. + * \brief Configure the debayer object according to the passed in parameters + * \param[in] inputCfg The input configuration + * \param[in] outputCfgs The output configurations * - * \return 0 on success, a negative errno on failure. + * \return 0 on success, a negative errno on failure */ /** * \fn Size Debayer::patternSize(PixelFormat inputFormat) - * \brief Get the width and height at which the bayer pattern repeats. - * \param[in] inputFormat The input format. + * \brief Get the width and height at which the bayer pattern repeats + * \param[in] inputFormat The input format * * Valid sizes are: 2x2, 4x2 or 4x4. * - * \return Pattern size or an empty size for unsupported inputFormats. + * \return Pattern size or an empty size for unsupported inputFormats */ /** * \fn std::vector Debayer::formats(PixelFormat inputFormat) - * \brief Get the supported output formats. - * \param[in] inputFormat The input format. + * \brief Get the supported output formats + * \param[in] inputFormat The input format * - * \return All supported output formats or an empty vector if there are none. + * \return All supported output formats or an empty vector if there are none */ /** * \fn std::tuple Debayer::strideAndFrameSize(const PixelFormat &outputFormat, const Size &size) - * \brief Get the stride and the frame size. - * \param[in] outputFormat The output format. - * \param[in] size The output size. + * \brief Get the stride and the frame size + * \param[in] outputFormat The output format + * \param[in] size The output size * * \return A tuple of the stride and the frame size, or a tuple with 0,0 if - * there is no valid output config. + * there is no valid output config */ /** * \fn void Debayer::process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) - * \brief Process the bayer data into the requested format. - * \param[in] input The input buffer. - * \param[in] output The output buffer. - * \param[in] params The parameters to be used in debayering. + * \brief Process the bayer data into the requested format + * \param[in] input The input buffer + * \param[in] output The output buffer + * \param[in] params The parameters to be used in debayering * * \note DebayerParams is passed by value deliberately so that a copy is passed * when this is run in another thread by invokeMethod(). @@ -106,21 +106,21 @@ Debayer::~Debayer() /** * \fn virtual SizeRange Debayer::sizes(PixelFormat inputFormat, const Size &inputSize) - * \brief Get the supported output sizes for the given input format and size. - * \param[in] inputFormat The input format. - * \param[in] inputSize The input size. + * \brief Get the supported output sizes for the given input format and size + * \param[in] inputFormat The input format + * \param[in] inputSize The input size * - * \return The valid size ranges or an empty range if there are none. + * \return The valid size ranges or an empty range if there are none */ /** * \var Signal Debayer::inputBufferReady - * \brief Signals when the input buffer is ready. + * \brief Signals when the input buffer is ready */ /** * \var Signal Debayer::outputBufferReady - * \brief Signals when the output buffer is ready. + * \brief Signals when the output buffer is ready */ } /* namespace libcamera */ From patchwork Wed Jul 3 17:51:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20549 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 30F4BC3240 for ; Wed, 3 Jul 2024 17:51:57 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E028162E27; Wed, 3 Jul 2024 19:51:55 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="bEhP49d6"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id AEAF662E23 for ; Wed, 3 Jul 2024 19:51:45 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029104; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HVnCTL08SgGgzUjtYGxqSZU2K0FTigrybQoXf5huc3o=; b=bEhP49d6byLOr64MAkrHNHPzsAj6uWvzXnh8BbRS0lEmQxyk4VzXPUSTiC6RL8ju6XrfaP 4tWOlQUYR6QcIL663SBj3q9a5/L68jItJNFXCjW4HfgWyP7G84Ttsj6nAKV1gEbR0xNhXr Vgj1gvCg/qYl5rIiSeU21EkLdjYzOT8= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-168-qGFSKgBHPwOTUJvnDE3K5g-1; Wed, 03 Jul 2024 13:51:43 -0400 X-MC-Unique: qGFSKgBHPwOTUJvnDE3K5g-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8237519560B4; Wed, 3 Jul 2024 17:51:42 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id F1A671954B0E; Wed, 3 Jul 2024 17:51:40 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 08/19] libcamera: software_isp: Track and pass frame ids Date: Wed, 3 Jul 2024 19:51:08 +0200 Message-ID: <20240703175119.1872585-9-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" A previous preparation patch implemented passing frame ids to stats processing but without actual meaningful frame id value passed there. This patch extends that by actually providing the frame id and passing it through to the stats processor. The frame id is taken from the request sequence number, the same as in hardware pipelines. Dear reviewers: I'm confused even after looking at commit 6084217cd3b52ba5677e3ca2de0e21008fdaa735. What's the relationship between requests, buffers and frames? It's not 1:1:1 or is it? Could you please provide some explanation that could be put here? Signed-off-by: Milan Zamazal --- .../libcamera/internal/software_isp/software_isp.h | 4 ++-- src/libcamera/pipeline/simple/simple.cpp | 8 +++++++- src/libcamera/software_isp/debayer.cpp | 3 ++- src/libcamera/software_isp/debayer.h | 2 +- src/libcamera/software_isp/debayer_cpu.cpp | 9 ++++----- src/libcamera/software_isp/debayer_cpu.h | 2 +- src/libcamera/software_isp/software_isp.cpp | 11 +++++++---- 7 files changed, 24 insertions(+), 15 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index 6a03b17f..7365b49a 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -72,10 +72,10 @@ public: int start(); void stop(); - int queueBuffers(FrameBuffer *input, + int queueBuffers(uint32_t frame, FrameBuffer *input, const std::map &outputs); - void process(FrameBuffer *input, FrameBuffer *output); + void process(uint32_t frame, FrameBuffer *input, FrameBuffer *output); Signal inputBufferReady; Signal outputBufferReady; diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index b1bf0d16..e96674ae 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -864,7 +864,13 @@ void SimpleCameraData::bufferReady(FrameBuffer *buffer) if (converter_) converter_->queueBuffers(buffer, conversionQueue_.front()); else - swIsp_->queueBuffers(buffer, conversionQueue_.front()); + /* + * request->sequence() cannot be retrieved from `buffer' inside + * queueBuffers because unique_ptr's make buffer->request() invalid + * already here. + */ + swIsp_->queueBuffers(request->sequence(), buffer, + conversionQueue_.front()); conversionQueue_.pop(); return; diff --git a/src/libcamera/software_isp/debayer.cpp b/src/libcamera/software_isp/debayer.cpp index 4d101798..e2295f35 100644 --- a/src/libcamera/software_isp/debayer.cpp +++ b/src/libcamera/software_isp/debayer.cpp @@ -94,8 +94,9 @@ Debayer::~Debayer() */ /** - * \fn void Debayer::process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) + * \fn void Debayer::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params) * \brief Process the bayer data into the requested format + * \param[in] frame The frame number * \param[in] input The input buffer * \param[in] output The output buffer * \param[in] params The parameters to be used in debayering diff --git a/src/libcamera/software_isp/debayer.h b/src/libcamera/software_isp/debayer.h index c151fe5d..d7ca060d 100644 --- a/src/libcamera/software_isp/debayer.h +++ b/src/libcamera/software_isp/debayer.h @@ -40,7 +40,7 @@ public: virtual std::tuple strideAndFrameSize(const PixelFormat &outputFormat, const Size &size) = 0; - virtual void process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) = 0; + virtual void process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params) = 0; virtual SizeRange sizes(PixelFormat inputFormat, const Size &inputSize) = 0; diff --git a/src/libcamera/software_isp/debayer_cpu.cpp b/src/libcamera/software_isp/debayer_cpu.cpp index 1575cedb..c75b8967 100644 --- a/src/libcamera/software_isp/debayer_cpu.cpp +++ b/src/libcamera/software_isp/debayer_cpu.cpp @@ -731,7 +731,7 @@ static inline int64_t timeDiff(timespec &after, timespec &before) (int64_t)after.tv_nsec - (int64_t)before.tv_nsec; } -void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams params) +void DebayerCpu::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params) { timespec frameStartTime; @@ -785,12 +785,11 @@ void DebayerCpu::process(FrameBuffer *input, FrameBuffer *output, DebayerParams } /* - * Frame and buffer ids are currently not used, so pass zeros as parameters. + * Buffer ids are currently not used, so pass zeros as its parameter. * - * \todo Pass real values once frame is passed here and stats buffer passing - * is changed. + * \todo Pass real bufferId once stats buffer passing is changed. */ - stats_->finishFrame(0, 0); + stats_->finishFrame(frame, 0); outputBufferReady.emit(output); inputBufferReady.emit(input); } diff --git a/src/libcamera/software_isp/debayer_cpu.h b/src/libcamera/software_isp/debayer_cpu.h index 1dac6435..6a9cb4c7 100644 --- a/src/libcamera/software_isp/debayer_cpu.h +++ b/src/libcamera/software_isp/debayer_cpu.h @@ -36,7 +36,7 @@ public: std::vector formats(PixelFormat input); std::tuple strideAndFrameSize(const PixelFormat &outputFormat, const Size &size); - void process(FrameBuffer *input, FrameBuffer *output, DebayerParams params); + void process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params); SizeRange sizes(PixelFormat inputFormat, const Size &inputSize); /** diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 3fc4a64b..0f4b23d3 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -14,6 +14,7 @@ #include #include +#include #include #include "libcamera/internal/ipa_manager.h" @@ -279,12 +280,13 @@ int SoftwareIsp::exportBuffers(unsigned int output, unsigned int count, /** * \brief Queue buffers to Software ISP + * \param[in] frame The frame number * \param[in] input The input framebuffer * \param[in] outputs The container holding the output stream indexes and * their respective frame buffer outputs * \return 0 on success, a negative errno on failure */ -int SoftwareIsp::queueBuffers(FrameBuffer *input, +int SoftwareIsp::queueBuffers(uint32_t frame, FrameBuffer *input, const std::map &outputs) { unsigned int mask = 0; @@ -308,7 +310,7 @@ int SoftwareIsp::queueBuffers(FrameBuffer *input, mask |= 1 << index; } - process(input, outputs.at(0)); + process(frame, input, outputs.at(0)); return 0; } @@ -340,13 +342,14 @@ void SoftwareIsp::stop() /** * \brief Passes the input framebuffer to the ISP worker to process + * \param[in] frame The frame number * \param[in] input The input framebuffer * \param[out] output The framebuffer to write the processed frame to */ -void SoftwareIsp::process(FrameBuffer *input, FrameBuffer *output) +void SoftwareIsp::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output) { debayer_->invokeMethod(&DebayerCpu::process, - ConnectionTypeQueued, input, output, debayerParams_); + ConnectionTypeQueued, frame, input, output, debayerParams_); } void SoftwareIsp::saveIspParams() From patchwork Wed Jul 3 17:51:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20550 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 41AC9BEFBE for ; Wed, 3 Jul 2024 17:51:58 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A5E2363331; Wed, 3 Jul 2024 19:51:56 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="PuucoWtb"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7164A6333F for ; Wed, 3 Jul 2024 19:51:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029106; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/6BCnKnHWS97PmlRW52YeLNopayab9Iq9rGegxGF62o=; b=PuucoWtbSv3BExLW/wXLVn9GFC8PiCnSHGjnGUKoVXyXReehz68+yC007EftNcSN5fCVmd 7ll4K8WZwnajHTt0XOf8/72ZZUv7MIREnY4iy55MusfM/+lLi0wDgMiMhpwu1U5DzYCp7K k1UvAQR2q6CoVmK6BNXKRe2SWIGYZkI= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-490-2LbwXxuYN4Su9imKr6ZXSA-1; Wed, 03 Jul 2024 13:51:45 -0400 X-MC-Unique: 2LbwXxuYN4Su9imKr6ZXSA-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1FA811955BC2; Wed, 3 Jul 2024 17:51:44 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id A7E831954B0E; Wed, 3 Jul 2024 17:51:42 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 09/19] libcamera: software_isp: Create algorithms Date: Wed, 3 Jul 2024 19:51:09 +0200 Message-ID: <20240703175119.1872585-10-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We are ready to introduce algorithms now. First, let's create algorithms. The algorithms are not called yet, calls to them will be added in followup patches. The maximum number of contexts is set to the same value as in hardware pipelines. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- src/ipa/simple/algorithms/meson.build | 4 ++++ src/ipa/simple/data/uncalibrated.yaml | 1 + src/ipa/simple/meson.build | 7 +++++-- src/ipa/simple/soft_simple.cpp | 14 ++++++++++++++ 4 files changed, 24 insertions(+), 2 deletions(-) create mode 100644 src/ipa/simple/algorithms/meson.build diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build new file mode 100644 index 00000000..31d26e43 --- /dev/null +++ b/src/ipa/simple/algorithms/meson.build @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: CC0-1.0 + +soft_simple_ipa_algorithms = files([ +]) diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index ff981a1a..2cdc39a8 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -2,4 +2,5 @@ %YAML 1.1 --- version: 1 +algorithms: ... diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build index 363251fb..7515a8d8 100644 --- a/src/ipa/simple/meson.build +++ b/src/ipa/simple/meson.build @@ -1,5 +1,8 @@ # SPDX-License-Identifier: CC0-1.0 +subdir('algorithms') +subdir('data') + ipa_name = 'ipa_soft_simple' soft_simple_sources = files([ @@ -8,6 +11,8 @@ soft_simple_sources = files([ 'black_level.cpp', ]) +soft_simple_sources += soft_simple_ipa_algorithms + mod = shared_module(ipa_name, [soft_simple_sources, libcamera_generated_ipa_headers], name_prefix : '', @@ -25,6 +30,4 @@ if ipa_sign_module build_by_default : true) endif -subdir('data') - ipa_names += ipa_name diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 09e3c8f6..dd300387 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -53,12 +53,15 @@ static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; * enough to prevent the exposure from wobbling around the optimal value. */ static constexpr float kExposureSatisfactory = 0.2; +/* Maximum number of frame contexts to be held */ +static constexpr uint32_t kMaxFrameContexts = 16; class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() : params_(nullptr), stats_(nullptr), blackLevel_(BlackLevel()), + context_({ {}, {}, { kMaxFrameContexts } }), ignoreUpdates_(0) { } @@ -92,6 +95,8 @@ private: static constexpr unsigned int kGammaLookupSize = 1024; std::array gammaTable_; int lastBlackLevel_ = -1; + /* Local parameter storage */ + struct IPAContext context_; int32_t exposureMin_, exposureMax_; int32_t exposure_; @@ -138,6 +143,15 @@ int IPASoftSimple::init(const IPASettings &settings, unsigned int version = (*data)["version"].get(0); LOG(IPASoft, Debug) << "Tuning file version " << version; + if (!data->contains("algorithms")) { + LOG(IPASoft, Error) << "Tuning file doesn't contain algorithms"; + return -EINVAL; + } + + int ret = createAlgorithms(context_, (*data)["algorithms"]); + if (ret) + return ret; + params_ = nullptr; stats_ = nullptr; From patchwork Wed Jul 3 17:51:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20551 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 466BFC3240 for ; Wed, 3 Jul 2024 17:51:59 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 43FBC63368; Wed, 3 Jul 2024 19:51:58 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="RBcbWY0H"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 397E662E26 for ; Wed, 3 Jul 2024 19:51:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029111; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bm7lFhcbwazjcjxChPZfLnE1kemQthOS5fvyqSVA+UU=; b=RBcbWY0HMTX1uFcXkm72adTiiG2NJjKLjELWfTZftbKUgF9Ry3r5JzxUIrbrerRwWEbSOW sPUNWD9blYK+/excF6QeS0o1ZvLUfqn2DmSSs69FgfkI4KrSBRK5cTvzB3MmmcsEmCrcHC 4TOt+uDBb/Ehvg0naKzuQIad1kG5NiQ= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-684-VyTMsswuNNGzipqDDWJ4Fw-1; Wed, 03 Jul 2024 13:51:47 -0400 X-MC-Unique: VyTMsswuNNGzipqDDWJ4Fw-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 2F21419560B0; Wed, 3 Jul 2024 17:51:46 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 8AA011954B0E; Wed, 3 Jul 2024 17:51:44 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 10/19] libcamera: software_isp: Call Algorithm::configure Date: Wed, 3 Jul 2024 19:51:10 +0200 Message-ID: <20240703175119.1872585-11-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds Algorithm::configure call for the defined algorithms. This is preparation only since there are currently no Algorithm based algorithms defined. A part of this change is passing IPAConfigInfo instead of ControlInfoMap to configure() calls as this is what Algorithm::configure expects. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- .../libcamera/internal/software_isp/software_isp.h | 2 +- include/libcamera/ipa/soft.mojom | 6 +++++- src/ipa/simple/module.h | 4 +++- src/ipa/simple/soft_simple.cpp | 12 +++++++++--- src/libcamera/pipeline/simple/simple.cpp | 11 +++++++---- src/libcamera/software_isp/software_isp.cpp | 7 ++++--- 6 files changed, 29 insertions(+), 13 deletions(-) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index 7365b49a..ff26b1d4 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -61,7 +61,7 @@ public: int configure(const StreamConfiguration &inputCfg, const std::vector> &outputCfgs, - const ControlInfoMap &sensorControls); + const ipa::soft::IPAConfigInfo &configInfo); int exportBuffers(unsigned int output, unsigned int count, std::vector> *buffers); diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index bd48ece9..5e124016 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -8,6 +8,10 @@ module ipa.soft; import "include/libcamera/ipa/core.mojom"; +struct IPAConfigInfo { + libcamera.ControlInfoMap sensorControls; +}; + interface IPASoftInterface { init(libcamera.IPASettings settings, libcamera.SharedFD fdStats, @@ -16,7 +20,7 @@ interface IPASoftInterface { => (int32 ret); start() => (int32 ret); stop(); - configure(libcamera.ControlInfoMap sensorCtrlInfoMap) + configure(IPAConfigInfo configInfo) => (int32 ret); [async] processStats(uint32 frame, uint32 bufferId, libcamera.ControlList sensorControls); diff --git a/src/ipa/simple/module.h b/src/ipa/simple/module.h index 33a7d1db..8d4d53fb 100644 --- a/src/ipa/simple/module.h +++ b/src/ipa/simple/module.h @@ -9,6 +9,8 @@ #include +#include + #include "libcamera/internal/software_isp/debayer_params.h" #include "libcamera/internal/software_isp/swisp_stats.h" @@ -20,7 +22,7 @@ namespace libcamera { namespace ipa::soft { -using Module = ipa::Module; } /* namespace ipa::soft */ diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index dd300387..49fff312 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -72,7 +72,7 @@ public: const SharedFD &fdStats, const SharedFD &fdParams, const ControlInfoMap &sensorInfoMap) override; - int configure(const ControlInfoMap &sensorInfoMap) override; + int configure(const IPAConfigInfo &configInfo) override; int start() override; void stop() override; @@ -206,9 +206,9 @@ int IPASoftSimple::init(const IPASettings &settings, return 0; } -int IPASoftSimple::configure(const ControlInfoMap &sensorInfoMap) +int IPASoftSimple::configure(const IPAConfigInfo &configInfo) { - sensorInfoMap_ = sensorInfoMap; + sensorInfoMap_ = configInfo.sensorControls; const ControlInfo &exposureInfo = sensorInfoMap_.find(V4L2_CID_EXPOSURE)->second; const ControlInfo &gainInfo = sensorInfoMap_.find(V4L2_CID_ANALOGUE_GAIN)->second; @@ -247,6 +247,12 @@ int IPASoftSimple::configure(const ControlInfoMap &sensorInfoMap) againMinStep_ = 1.0; } + for (auto const &algo : algorithms()) { + int ret = algo->configure(context_, configInfo); + if (ret) + return ret; + } + LOG(IPASoft, Info) << "Exposure " << exposureMin_ << "-" << exposureMax_ << ", gain " << againMin_ << "-" << againMax_ << " (" << againMinStep_ << ")"; diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index e96674ae..e0c9fe5c 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -1294,10 +1294,13 @@ int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c) inputCfg.stride = captureFormat.planes[0].bpl; inputCfg.bufferCount = kNumInternalBuffers; - return data->converter_ - ? data->converter_->configure(inputCfg, outputCfgs) - : data->swIsp_->configure(inputCfg, outputCfgs, - data->sensor_->controls()); + if (data->converter_) + return data->converter_->configure(inputCfg, outputCfgs); + else { + ipa::soft::IPAConfigInfo configInfo; + configInfo.sensorControls = data->sensor_->controls(); + return data->swIsp_->configure(inputCfg, outputCfgs, configInfo); + } } int SimplePipelineHandler::exportFrameBuffers(Camera *camera, Stream *stream, diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 0f4b23d3..2bbb86da 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -224,16 +224,17 @@ SoftwareIsp::strideAndFrameSize(const PixelFormat &outputFormat, const Size &siz * \brief Configure the SoftwareIsp object according to the passed in parameters * \param[in] inputCfg The input configuration * \param[in] outputCfgs The output configurations - * \param[in] sensorControls ControlInfoMap of the controls supported by the sensor + * \param[in] configInfo The IPA configuration data, received from the pipeline + * handler * \return 0 on success, a negative errno on failure */ int SoftwareIsp::configure(const StreamConfiguration &inputCfg, const std::vector> &outputCfgs, - const ControlInfoMap &sensorControls) + const ipa::soft::IPAConfigInfo &configInfo) { ASSERT(ipa_ && debayer_); - int ret = ipa_->configure(sensorControls); + int ret = ipa_->configure(configInfo); if (ret < 0) return ret; From patchwork Wed Jul 3 17:51:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20553 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 83860BEFBE for ; Wed, 3 Jul 2024 17:52:07 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 1384E6335E; Wed, 3 Jul 2024 19:52:07 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="ERPzFpFr"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1A56363362 for ; Wed, 3 Jul 2024 19:51:55 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029113; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2luqaJnYGTPMgjSgzM9RHdLsJ8Yl8itRFEar3wtfCj4=; b=ERPzFpFraptRZcMmjZaDgPoAkFFYRilSFhXSfcFJitgUyeMjKMFKOOnSi/kDdWIUMnhKlK 4JyUQy1EbwLlkpu36DEwItv7xIk0QeeNt5HnB3rAJpJPa50mPrUw6y8jJ00Md7aTMPZxz2 UM/G9HdmanZg7trD1B0vP99mIeDgmvE= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-263-RSPV5GbAPcK8rz_VcRjW2w-1; Wed, 03 Jul 2024 13:51:50 -0400 X-MC-Unique: RSPV5GbAPcK8rz_VcRjW2w-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 2D36419560B1; Wed, 3 Jul 2024 17:51:48 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 7289A1954B0D; Wed, 3 Jul 2024 17:51:46 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 11/19] libcamera: software_isp: Call Algorithm::queueRequest Date: Wed, 3 Jul 2024 19:51:11 +0200 Message-ID: <20240703175119.1872585-12-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds Algorithm::queueRequest call for the defined algorithms. This is preparation only since there are currently no Algorithm based algorithms defined. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- include/libcamera/internal/software_isp/software_isp.h | 1 + include/libcamera/ipa/soft.mojom | 1 + src/ipa/simple/soft_simple.cpp | 9 +++++++++ src/libcamera/pipeline/simple/simple.cpp | 2 ++ src/libcamera/software_isp/software_isp.cpp | 10 ++++++++++ 5 files changed, 23 insertions(+) diff --git a/include/libcamera/internal/software_isp/software_isp.h b/include/libcamera/internal/software_isp/software_isp.h index ff26b1d4..1ba9eb62 100644 --- a/include/libcamera/internal/software_isp/software_isp.h +++ b/include/libcamera/internal/software_isp/software_isp.h @@ -72,6 +72,7 @@ public: int start(); void stop(); + void queueRequest(const uint32_t frame, const ControlList &controls); int queueBuffers(uint32_t frame, FrameBuffer *input, const std::map &outputs); diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index 5e124016..426fee18 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -23,6 +23,7 @@ interface IPASoftInterface { configure(IPAConfigInfo configInfo) => (int32 ret); + queueRequest(uint32 frame, libcamera.ControlList sensorControls); [async] processStats(uint32 frame, uint32 bufferId, libcamera.ControlList sensorControls); }; diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 49fff312..bd0dea5f 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -77,6 +77,7 @@ public: int start() override; void stop() override; + void queueRequest(const uint32_t frame, const ControlList &controls) override; void processStats(const uint32_t frame, const uint32_t bufferId, const ControlList &sensorControls) override; @@ -269,6 +270,14 @@ void IPASoftSimple::stop() { } +void IPASoftSimple::queueRequest(const uint32_t frame, const ControlList &controls) +{ + IPAFrameContext &frameContext = context_.frameContexts.alloc(frame); + + for (auto const &algo : algorithms()) + algo->queueRequest(context_, frame, frameContext, controls); +} + void IPASoftSimple::processStats( [[maybe_unused]] const uint32_t frame, [[maybe_unused]] const uint32_t bufferId, diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index e0c9fe5c..6fb80209 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -1430,6 +1430,8 @@ int SimplePipelineHandler::queueRequestDevice(Camera *camera, Request *request) if (data->useConversion_) data->conversionQueue_.push(std::move(buffers)); + data->swIsp_->queueRequest(request->sequence(), request->controls()); + return 0; } diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 2bbb86da..4e6b3a2e 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -279,6 +279,16 @@ int SoftwareIsp::exportBuffers(unsigned int output, unsigned int count, return count; } +/** + * \brief Queue a request and process the control list from the application + * \param[in] frame The number of the frame which will be processed next + * \param[in] controls The controls for the \a frame + */ +void SoftwareIsp::queueRequest(const uint32_t frame, const ControlList &controls) +{ + ipa_->queueRequest(frame, controls); +} + /** * \brief Queue buffers to Software ISP * \param[in] frame The frame number From patchwork Wed Jul 3 17:51:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20552 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A55B8BEFBE for ; Wed, 3 Jul 2024 17:52:01 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 46CD76336A; Wed, 3 Jul 2024 19:52:01 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="annxouwW"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C14F26333F for ; Wed, 3 Jul 2024 19:51:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029111; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nCoLM2KF6HK30JK+jvt1Z3YICnoWIBPximKL0pAyEAo=; b=annxouwWN+dzccIn7KHieRFcg6VHIVfVjYzemCJ1B/A3chpdL2fn/0Ef4FcEw7D8buD6x7 PtVEZ1S1WRAgVdM9TJQza7TwxYX7LyEYeAZvBtb20eB/DNl6uLHC6CPCiEB7iZqoR1oNys anhg4Q6e63HVuKfdQ6MOTW7Z4BKo0T8= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-511-HIdUJ2v3PmytTPN7WUbWOQ-1; Wed, 03 Jul 2024 13:51:50 -0400 X-MC-Unique: HIdUJ2v3PmytTPN7WUbWOQ-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8D7A91955BC1; Wed, 3 Jul 2024 17:51:49 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 10F3F1954B0F; Wed, 3 Jul 2024 17:51:47 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 12/19] libcamera: software_isp: Call Algorithm::prepare Date: Wed, 3 Jul 2024 19:51:12 +0200 Message-ID: <20240703175119.1872585-13-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds Algorithm::prepare call for the defined algorithms. This is preparation only since there are currently no Algorithm based algorithms defined. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- include/libcamera/ipa/soft.mojom | 1 + src/ipa/simple/soft_simple.cpp | 8 ++++++++ src/libcamera/software_isp/software_isp.cpp | 1 + 3 files changed, 10 insertions(+) diff --git a/include/libcamera/ipa/soft.mojom b/include/libcamera/ipa/soft.mojom index 426fee18..1777012a 100644 --- a/include/libcamera/ipa/soft.mojom +++ b/include/libcamera/ipa/soft.mojom @@ -24,6 +24,7 @@ interface IPASoftInterface { => (int32 ret); queueRequest(uint32 frame, libcamera.ControlList sensorControls); + prepare(uint32 frame); [async] processStats(uint32 frame, uint32 bufferId, libcamera.ControlList sensorControls); }; diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index bd0dea5f..f4a6bdf4 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -78,6 +78,7 @@ public: void stop() override; void queueRequest(const uint32_t frame, const ControlList &controls) override; + void prepare(const uint32_t frame) override; void processStats(const uint32_t frame, const uint32_t bufferId, const ControlList &sensorControls) override; @@ -278,6 +279,13 @@ void IPASoftSimple::queueRequest(const uint32_t frame, const ControlList &contro algo->queueRequest(context_, frame, frameContext, controls); } +void IPASoftSimple::prepare(const uint32_t frame) +{ + IPAFrameContext &frameContext = context_.frameContexts.get(frame); + for (auto const &algo : algorithms()) + algo->prepare(context_, frame, frameContext, params_); +} + void IPASoftSimple::processStats( [[maybe_unused]] const uint32_t frame, [[maybe_unused]] const uint32_t bufferId, diff --git a/src/libcamera/software_isp/software_isp.cpp b/src/libcamera/software_isp/software_isp.cpp index 4e6b3a2e..38a284ef 100644 --- a/src/libcamera/software_isp/software_isp.cpp +++ b/src/libcamera/software_isp/software_isp.cpp @@ -359,6 +359,7 @@ void SoftwareIsp::stop() */ void SoftwareIsp::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output) { + ipa_->prepare(frame); debayer_->invokeMethod(&DebayerCpu::process, ConnectionTypeQueued, frame, input, output, debayerParams_); } From patchwork Wed Jul 3 17:51:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20555 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 023FCBEFBE for ; Wed, 3 Jul 2024 17:52:11 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 47FC963365; Wed, 3 Jul 2024 19:52:10 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="egaD99xy"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2487C6336B for ; Wed, 3 Jul 2024 19:51:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029115; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KmDUksM2FS5LDTE735TKANJ9WsmBJoUm/opuUSeLPhc=; b=egaD99xykaEGa7VFC6IdfRAMe0ST7KMp/2j/HRFKyaGF3FmnjiYhFRBh3p7Gpp3rd7ad5U pNaWlIfyWnxGlj7DFyARL+w3xj5SjOVHA/ifY59u7EGj4fddJD70lqPCbsZfFvlVdWnpka MfDiE070OTIUIPIZPtCyIKnGwNdX/60= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-439-NIZAYf5nMGi36S5_rGtjhQ-1; Wed, 03 Jul 2024 13:51:52 -0400 X-MC-Unique: NIZAYf5nMGi36S5_rGtjhQ-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id ABA0E1955BC2; Wed, 3 Jul 2024 17:51:51 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id F3ECD1954B0D; Wed, 3 Jul 2024 17:51:49 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 13/19] libcamera: software_isp: Call Algorithm::process Date: Wed, 3 Jul 2024 19:51:13 +0200 Message-ID: <20240703175119.1872585-14-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch adds Algorithm::process call for the defined algorithms. This is preparation only since there are currently no Algorithm based algorithms defined. As software ISP currently doesn't produce any metadata, a dummy and unused metadata instance is created to satisfy Algorithm::process API. This should be changed in future. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- src/ipa/simple/soft_simple.cpp | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index f4a6bdf4..a8499f29 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -287,10 +287,21 @@ void IPASoftSimple::prepare(const uint32_t frame) } void IPASoftSimple::processStats( - [[maybe_unused]] const uint32_t frame, + const uint32_t frame, [[maybe_unused]] const uint32_t bufferId, const ControlList &sensorControls) { + IPAFrameContext &frameContext = context_.frameContexts.get(frame); + /* + * Software ISP currently does not produce any metadata. Use an empty + * ControlList for now. + * + * \todo Implement proper metadata handling + */ + ControlList metadata(controls::controls); + for (auto const &algo : algorithms()) + algo->process(context_, frame, frameContext, stats_, metadata); + SwIspStats::Histogram histogram = stats_->yHistogram; if (ignoreUpdates_ > 0) blackLevel_.update(histogram); From patchwork Wed Jul 3 17:51:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20554 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id F16C6C3240 for ; Wed, 3 Jul 2024 17:52:08 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 69B4362C95; Wed, 3 Jul 2024 19:52:08 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="Q20c3J03"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 11F616336E for ; Wed, 3 Jul 2024 19:51:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029116; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PA1kbdhcXc+/iJ4Bx/lQCZSP1tPcBpQTgGMAcyT6xUc=; b=Q20c3J03bLmxKX38Oohwj96MhNTyqUBsLjQ4oCGbGTwS5JKYY4I/mpTH2o+Z66SLAXBqlS HZ2gjYBXMDh7qc7Pzc8dUUy1yfkr1ObiMa1XVR1iBx4CZJtbvTAy/cK3Hy8trY0MUbxFOA QuTLPwC9DeafUdPCJELOL1tLDIi22TE= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-350-vdq_kYzEPlOK-uL8wCfcsQ-1; Wed, 03 Jul 2024 13:51:54 -0400 X-MC-Unique: vdq_kYzEPlOK-uL8wCfcsQ-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D3E3719560AE; Wed, 3 Jul 2024 17:51:53 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 3D6C81954B0D; Wed, 3 Jul 2024 17:51:51 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 14/19] libcamera: software_isp: Move black level to an algorithm module Date: Wed, 3 Jul 2024 19:51:14 +0200 Message-ID: <20240703175119.1872585-15-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The black level determination, already present as a separate class, can be moved to the prepared Algorithm processing structure. It is the first of the current software ISP algorithms that is called in the stats processing sequence, which means it is also the first one that should be moved to the new structure. Stats processing starts with calling Algorithm-based processing so the call order of the algorithms is retained. Movement of this algorithm is relatively straightforward because all it does is processing stats. Black level is now recomputed on each stats processing. In a future patch, after DelayedControls are used, this will be changed to recompute the black level only after exposure/gain changes. Signed-off-by: Milan Zamazal Reviewed-by: Umang Jain --- src/ipa/simple/algorithms/blc.cpp | 71 ++++++++++++++++++++ src/ipa/simple/algorithms/blc.h | 37 +++++++++++ src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/black_level.cpp | 93 --------------------------- src/ipa/simple/black_level.h | 33 ---------- src/ipa/simple/data/uncalibrated.yaml | 1 + src/ipa/simple/ipa_context.cpp | 8 +++ src/ipa/simple/ipa_context.h | 5 ++ src/ipa/simple/meson.build | 1 - src/ipa/simple/soft_simple.cpp | 8 +-- 10 files changed, 125 insertions(+), 133 deletions(-) create mode 100644 src/ipa/simple/algorithms/blc.cpp create mode 100644 src/ipa/simple/algorithms/blc.h delete mode 100644 src/ipa/simple/black_level.cpp delete mode 100644 src/ipa/simple/black_level.h diff --git a/src/ipa/simple/algorithms/blc.cpp b/src/ipa/simple/algorithms/blc.cpp new file mode 100644 index 00000000..e98c5804 --- /dev/null +++ b/src/ipa/simple/algorithms/blc.cpp @@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * black level handling + */ + +#include "blc.h" + +#include + +#include + +namespace libcamera { + +namespace ipa::soft::algorithms { + +LOG_DEFINE_CATEGORY(IPASoftBL) + +BlackLevel::BlackLevel() +{ +} + +int BlackLevel::init(IPAContext &context, + [[maybe_unused]] const YamlObject &tuningData) +{ + context.activeState.black.level = 255; + return 0; +} + +void BlackLevel::process(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + const SwIspStats *stats, + [[maybe_unused]] ControlList &metadata) +{ + const SwIspStats::Histogram &histogram = stats->yHistogram; + + /* + * The constant is selected to be "good enough", not overly conservative or + * aggressive. There is no magic about the given value. + */ + constexpr float ignoredPercentage_ = 0.02; + const unsigned int total = + std::accumulate(begin(histogram), end(histogram), 0); + const unsigned int pixelThreshold = ignoredPercentage_ * total; + const unsigned int histogramRatio = 256 / SwIspStats::kYHistogramSize; + const unsigned int currentBlackIdx = + context.activeState.black.level / histogramRatio; + + for (unsigned int i = 0, seen = 0; + i < currentBlackIdx && i < SwIspStats::kYHistogramSize; + i++) { + seen += histogram[i]; + if (seen >= pixelThreshold) { + context.activeState.black.level = i * histogramRatio; + LOG(IPASoftBL, Debug) + << "Auto-set black level: " + << i << "/" << SwIspStats::kYHistogramSize + << " (" << 100 * (seen - histogram[i]) / total << "% below, " + << 100 * seen / total << "% at or below)"; + break; + } + }; +} + +REGISTER_IPA_ALGORITHM(BlackLevel, "BlackLevel") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/blc.h b/src/ipa/simple/algorithms/blc.h new file mode 100644 index 00000000..c8b8d92d --- /dev/null +++ b/src/ipa/simple/algorithms/blc.h @@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * black level handling + */ + +#pragma once + +#include + +#include "libcamera/internal/software_isp/swisp_stats.h" + +#include "algorithm.h" +#include "ipa_context.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class BlackLevel : public Algorithm +{ +public: + BlackLevel(); + ~BlackLevel() = default; + + int init(IPAContext &context, const YamlObject &tuningData) + override; + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const SwIspStats *stats, + ControlList &metadata) override; +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index 31d26e43..1f63c220 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -1,4 +1,5 @@ # SPDX-License-Identifier: CC0-1.0 soft_simple_ipa_algorithms = files([ + 'blc.cpp', ]) diff --git a/src/ipa/simple/black_level.cpp b/src/ipa/simple/black_level.cpp deleted file mode 100644 index 37e0109c..00000000 --- a/src/ipa/simple/black_level.cpp +++ /dev/null @@ -1,93 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1-or-later */ -/* - * Copyright (C) 2024, Red Hat Inc. - * - * black level handling - */ - -#include "black_level.h" - -#include - -#include - -namespace libcamera { - -LOG_DEFINE_CATEGORY(IPASoftBL) - -namespace ipa::soft { - -/** - * \class BlackLevel - * \brief Object providing black point level for software ISP - * - * Black level can be provided in hardware tuning files or, if no tuning file is - * available for the given hardware, guessed automatically, with less accuracy. - * As tuning files are not yet implemented for software ISP, BlackLevel - * currently provides only guessed black levels. - * - * This class serves for tracking black level as a property of the underlying - * hardware, not as means of enhancing a particular scene or image. - * - * The class is supposed to be instantiated for the given camera stream. - * The black level can be retrieved using BlackLevel::get() method. It is - * initially 0 and may change when updated using BlackLevel::update() method. - */ - -BlackLevel::BlackLevel() - : blackLevel_(255), blackLevelSet_(false) -{ -} - -/** - * \brief Return the current black level - * - * \return The black level, in the range from 0 (minimum) to 255 (maximum). - * If the black level couldn't be determined yet, return 0. - */ -uint8_t BlackLevel::get() const -{ - return blackLevelSet_ ? blackLevel_ : 0; -} - -/** - * \brief Update black level from the provided histogram - * \param[in] yHistogram The histogram to be used for updating black level - * - * The black level is property of the given hardware, not image. It is updated - * only if it has not been yet set or if it is lower than the lowest value seen - * so far. - */ -void BlackLevel::update(SwIspStats::Histogram &yHistogram) -{ - /* - * The constant is selected to be "good enough", not overly conservative or - * aggressive. There is no magic about the given value. - */ - constexpr float ignoredPercentage_ = 0.02; - const unsigned int total = - std::accumulate(begin(yHistogram), end(yHistogram), 0); - const unsigned int pixelThreshold = ignoredPercentage_ * total; - const unsigned int histogramRatio = 256 / SwIspStats::kYHistogramSize; - const unsigned int currentBlackIdx = blackLevel_ / histogramRatio; - - for (unsigned int i = 0, seen = 0; - i < currentBlackIdx && i < SwIspStats::kYHistogramSize; - i++) { - seen += yHistogram[i]; - if (seen >= pixelThreshold) { - blackLevel_ = i * histogramRatio; - blackLevelSet_ = true; - LOG(IPASoftBL, Debug) - << "Auto-set black level: " - << i << "/" << SwIspStats::kYHistogramSize - << " (" << 100 * (seen - yHistogram[i]) / total << "% below, " - << 100 * seen / total << "% at or below)"; - break; - } - }; -} - -} /* namespace ipa::soft */ - -} /* namespace libcamera */ diff --git a/src/ipa/simple/black_level.h b/src/ipa/simple/black_level.h deleted file mode 100644 index a04230c9..00000000 --- a/src/ipa/simple/black_level.h +++ /dev/null @@ -1,33 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1-or-later */ -/* - * Copyright (C) 2024, Red Hat Inc. - * - * black level handling - */ - -#pragma once - -#include -#include - -#include "libcamera/internal/software_isp/swisp_stats.h" - -namespace libcamera { - -namespace ipa::soft { - -class BlackLevel -{ -public: - BlackLevel(); - uint8_t get() const; - void update(SwIspStats::Histogram &yHistogram); - -private: - uint8_t blackLevel_; - bool blackLevelSet_; -}; - -} /* namespace ipa::soft */ - -} /* namespace libcamera */ diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index 2cdc39a8..e0d03d96 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -3,4 +3,5 @@ --- version: 1 algorithms: + - BlackLevel: ... diff --git a/src/ipa/simple/ipa_context.cpp b/src/ipa/simple/ipa_context.cpp index 0898e591..bb5daa0d 100644 --- a/src/ipa/simple/ipa_context.cpp +++ b/src/ipa/simple/ipa_context.cpp @@ -50,4 +50,12 @@ namespace libcamera::ipa::soft { * \brief The current state of IPA algorithms */ +/** + * \var IPAActiveState::black + * \brief Context for the Black Level algorithm + * + * \var IPAActiveState::black.level + * \brief Current determined black level + */ + } /* namespace libcamera::ipa::soft */ diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index bc1235b6..ed07dbb8 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -8,6 +8,8 @@ #pragma once +#include + #include namespace libcamera { @@ -18,6 +20,9 @@ struct IPASessionConfiguration { }; struct IPAActiveState { + struct { + uint8_t level; + } black; }; struct IPAFrameContext : public FrameContext { diff --git a/src/ipa/simple/meson.build b/src/ipa/simple/meson.build index 7515a8d8..92aa5744 100644 --- a/src/ipa/simple/meson.build +++ b/src/ipa/simple/meson.build @@ -8,7 +8,6 @@ ipa_name = 'ipa_soft_simple' soft_simple_sources = files([ 'ipa_context.cpp', 'soft_simple.cpp', - 'black_level.cpp', ]) soft_simple_sources += soft_simple_ipa_algorithms diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index a8499f29..2fb3a473 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -28,7 +28,6 @@ #include "libipa/camera_sensor_helper.h" -#include "black_level.h" #include "module.h" namespace libcamera { @@ -60,7 +59,7 @@ class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() - : params_(nullptr), stats_(nullptr), blackLevel_(BlackLevel()), + : params_(nullptr), stats_(nullptr), context_({ {}, {}, { kMaxFrameContexts } }), ignoreUpdates_(0) { @@ -92,7 +91,6 @@ private: SwIspStats *stats_; std::unique_ptr camHelper_; ControlInfoMap sensorInfoMap_; - BlackLevel blackLevel_; static constexpr unsigned int kGammaLookupSize = 1024; std::array gammaTable_; @@ -303,9 +301,7 @@ void IPASoftSimple::processStats( algo->process(context_, frame, frameContext, stats_, metadata); SwIspStats::Histogram histogram = stats_->yHistogram; - if (ignoreUpdates_ > 0) - blackLevel_.update(histogram); - const uint8_t blackLevel = blackLevel_.get(); + const uint8_t blackLevel = context_.activeState.black.level; /* * Black level must be subtracted to get the correct AWB ratios, they From patchwork Wed Jul 3 17:51:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20556 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 3BF10C3240 for ; Wed, 3 Jul 2024 17:52:12 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8EDE762C96; Wed, 3 Jul 2024 19:52:11 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="O7a1xZ+u"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4C79363365 for ; Wed, 3 Jul 2024 19:51:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029118; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9bNJzY9x7vb520WFL8Gi3vIkHzsTeGXJ+DP1aY8mBbI=; b=O7a1xZ+uDx+Ha5AKapaxKInIlGDcuP9u8KhzZQgoCStGvan4u9n2rLANKBN77oQDnpfRdo QoVpuZPYif2xWQ06MDTbbSbm+SSpp8aBlFJCBjZ+gUnwOgy3E++ySGiIa5Z7YE3I+W2UO8 GrjadfdH0ZlOEBwcs1/LQQdEElt4IK4= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-538-RSMOan0qM3GdWv4wpE_uSA-1; Wed, 03 Jul 2024 13:51:57 -0400 X-MC-Unique: RSMOan0qM3GdWv4wpE_uSA-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 147DE195609E; Wed, 3 Jul 2024 17:51:56 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 494E81954B0D; Wed, 3 Jul 2024 17:51:53 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 15/19] libcamera: software_isp: Move color handling to an algorithm module Date: Wed, 3 Jul 2024 19:51:15 +0200 Message-ID: <20240703175119.1872585-16-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" After black level handling has been moved to an algorithm module, white balance and the construction of color tables can be moved to an algorithm module too. This time, the moved code is split between stats processing and parameter construction methods. This is the only part of the software ISP algorithms that sets the parameters so emitting setIspParams can be moved to prepare() method. A more natural representation of the gains (and black level) would be floating point numbers. This is not done here in order to minimize changes in code movements. It will be addressed in a followup patch. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/colors.cpp | 121 ++++++++++++++++++++++++++ src/ipa/simple/algorithms/colors.h | 46 ++++++++++ src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/data/uncalibrated.yaml | 1 + src/ipa/simple/ipa_context.cpp | 30 +++++++ src/ipa/simple/ipa_context.h | 12 +++ src/ipa/simple/soft_simple.cpp | 66 +------------- 7 files changed, 215 insertions(+), 62 deletions(-) create mode 100644 src/ipa/simple/algorithms/colors.cpp create mode 100644 src/ipa/simple/algorithms/colors.h diff --git a/src/ipa/simple/algorithms/colors.cpp b/src/ipa/simple/algorithms/colors.cpp new file mode 100644 index 00000000..60fdca15 --- /dev/null +++ b/src/ipa/simple/algorithms/colors.cpp @@ -0,0 +1,121 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Color gains (AWB + gamma) + */ + +#include "colors.h" + +#include +#include +#include +#include + +#include + +#include "simple/ipa_context.h" + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoftColors) + +namespace ipa::soft::algorithms { + +static constexpr unsigned int kGammaLookupSize = 1024; + +Colors::Colors() +{ +} + +int Colors::init(IPAContext &context, + [[maybe_unused]] const YamlObject &tuningData) +{ + /* Gamma value is fixed */ + context.configuration.gamma = 0.5; + updateGammaTable(context); + + auto &gains = context.activeState.gains; + gains.red = gains.green = gains.blue = 256; + + return 0; +} + +void Colors::updateGammaTable(IPAContext &context) +{ + auto &gammaTable = context.activeState.gamma.gammaTable; + auto &blackLevel = context.activeState.gamma.blackLevel; + blackLevel = context.activeState.black.level; + const unsigned int blackIndex = + blackLevel * IPAActiveState::kGammaLookupSize / 256; + std::fill(gammaTable.begin(), gammaTable.begin() + blackIndex, 0); + const float divisor = kGammaLookupSize - blackIndex - 1.0; + for (unsigned int i = blackIndex; i < kGammaLookupSize; i++) + gammaTable[i] = UINT8_MAX * powf((i - blackIndex) / divisor, + context.configuration.gamma); +} + +void Colors::prepare(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + DebayerParams *params) +{ + /* Update the gamma table if needed */ + if (context.activeState.gamma.blackLevel != context.activeState.black.level) + updateGammaTable(context); + + auto &gains = context.activeState.gains; + auto &gammaTable = context.activeState.gamma.gammaTable; + for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { + constexpr unsigned int div = + static_cast(DebayerParams::kRGBLookupSize) * 256 / kGammaLookupSize; + /* Apply gamma after gain! */ + unsigned int idx; + idx = std::min({ i * gains.red / div, kGammaLookupSize - 1 }); + params->red[i] = gammaTable[idx]; + idx = std::min({ i * gains.green / div, kGammaLookupSize - 1 }); + params->green[i] = gammaTable[idx]; + idx = std::min({ i * gains.blue / div, kGammaLookupSize - 1 }); + params->blue[i] = gammaTable[idx]; + } +} + +void Colors::process(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + const SwIspStats *stats, + [[maybe_unused]] ControlList &metadata) +{ + const SwIspStats::Histogram &histogram = stats->yHistogram; + const uint8_t blackLevel = context.activeState.black.level; + + /* + * Black level must be subtracted to get the correct AWB ratios, they + * would be off if they were computed from the whole brightness range + * rather than from the sensor range. + */ + const uint64_t nPixels = std::accumulate( + histogram.begin(), histogram.end(), 0); + const uint64_t offset = blackLevel * nPixels; + const uint64_t sumR = stats->sumR_ - offset / 4; + const uint64_t sumG = stats->sumG_ - offset / 2; + const uint64_t sumB = stats->sumB_ - offset / 4; + + /* + * Calculate red and blue gains for AWB. + * Clamp max gain at 4.0, this also avoids 0 division. + * Gain: 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. + */ + auto &gains = context.activeState.gains; + gains.red = sumR <= sumG / 4 ? 1024 : 256 * sumG / sumR; + gains.blue = sumB <= sumG / 4 ? 1024 : 256 * sumG / sumB; + /* Green gain is fixed to 256 */ + + LOG(IPASoftColors, Debug) << "gain R/B " << gains.red << "/" << gains.blue; +} + +REGISTER_IPA_ALGORITHM(Colors, "Colors") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/colors.h b/src/ipa/simple/algorithms/colors.h new file mode 100644 index 00000000..2c2f7752 --- /dev/null +++ b/src/ipa/simple/algorithms/colors.h @@ -0,0 +1,46 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Color gains (AWB + gamma) + */ + +#pragma once + +#include + +#include "libcamera/internal/software_isp/debayer_params.h" +#include "libcamera/internal/software_isp/swisp_stats.h" + +#include "algorithm.h" +#include "ipa_context.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class Colors : public Algorithm +{ +public: + Colors(); + ~Colors() = default; + + int init(IPAContext &context, const YamlObject &tuningData) + override; + void prepare(IPAContext &context, + const uint32_t frame, + IPAFrameContext &frameContext, + DebayerParams *params) override; + void process(IPAContext &context, + const uint32_t frame, + IPAFrameContext &frameContext, + const SwIspStats *stats, + ControlList &metadata) override; + +private: + void updateGammaTable(IPAContext &context); +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index 1f63c220..324c7b4e 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -2,4 +2,5 @@ soft_simple_ipa_algorithms = files([ 'blc.cpp', + 'colors.cpp', ]) diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index e0d03d96..7e5149e5 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -4,4 +4,5 @@ version: 1 algorithms: - BlackLevel: + - Colors: ... diff --git a/src/ipa/simple/ipa_context.cpp b/src/ipa/simple/ipa_context.cpp index bb5daa0d..fc113527 100644 --- a/src/ipa/simple/ipa_context.cpp +++ b/src/ipa/simple/ipa_context.cpp @@ -50,6 +50,11 @@ namespace libcamera::ipa::soft { * \brief The current state of IPA algorithms */ +/** + * \var IPASessionConfiguration::gamma + * \brief Gamma value to be used in the raw image processing + */ + /** * \var IPAActiveState::black * \brief Context for the Black Level algorithm @@ -58,4 +63,29 @@ namespace libcamera::ipa::soft { * \brief Current determined black level */ +/** + * \var IPAActiveState::gains + * \brief Context for gains in the Colors algorithm + * + * \var IPAActiveState::gains.red + * \brief Gain of red color + * + * \var IPAActiveState::gains.green + * \brief Gain of green color + * + * \var IPAActiveState::gains.blue + * \brief Gain of blue color + */ + +/** + * \var IPAActiveState::gamma + * \brief Context for gamma in the Colors algorithm + * + * \var IPAActiveState::gamma.gammaTable + * \brief Computed gamma table + * + * \var IPAActiveState::gamma.blackLevel + * \brief Black level used for the gamma table computation + */ + } /* namespace libcamera::ipa::soft */ diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index ed07dbb8..979ddb8f 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -8,6 +8,7 @@ #pragma once +#include #include #include @@ -17,12 +18,23 @@ namespace libcamera { namespace ipa::soft { struct IPASessionConfiguration { + float gamma; }; struct IPAActiveState { struct { uint8_t level; } black; + struct { + unsigned int red; + unsigned int green; + unsigned int blue; + } gains; + static constexpr unsigned int kGammaLookupSize = 1024; + struct { + std::array gammaTable; + uint8_t blackLevel; + } gamma; }; struct IPAFrameContext : public FrameContext { diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 2fb3a473..2658e359 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -5,8 +5,6 @@ * Simple Software Image Processing Algorithm module */ -#include -#include #include #include @@ -92,9 +90,6 @@ private: std::unique_ptr camHelper_; ControlInfoMap sensorInfoMap_; - static constexpr unsigned int kGammaLookupSize = 1024; - std::array gammaTable_; - int lastBlackLevel_ = -1; /* Local parameter storage */ struct IPAContext context_; @@ -282,6 +277,7 @@ void IPASoftSimple::prepare(const uint32_t frame) IPAFrameContext &frameContext = context_.frameContexts.get(frame); for (auto const &algo : algorithms()) algo->prepare(context_, frame, frameContext, params_); + setIspParams.emit(); } void IPASoftSimple::processStats( @@ -300,62 +296,6 @@ void IPASoftSimple::processStats( for (auto const &algo : algorithms()) algo->process(context_, frame, frameContext, stats_, metadata); - SwIspStats::Histogram histogram = stats_->yHistogram; - const uint8_t blackLevel = context_.activeState.black.level; - - /* - * Black level must be subtracted to get the correct AWB ratios, they - * would be off if they were computed from the whole brightness range - * rather than from the sensor range. - */ - const uint64_t nPixels = std::accumulate( - histogram.begin(), histogram.end(), 0); - const uint64_t offset = blackLevel * nPixels; - const uint64_t sumR = stats_->sumR_ - offset / 4; - const uint64_t sumG = stats_->sumG_ - offset / 2; - const uint64_t sumB = stats_->sumB_ - offset / 4; - - /* - * Calculate red and blue gains for AWB. - * Clamp max gain at 4.0, this also avoids 0 division. - * Gain: 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. - */ - const unsigned int gainR = sumR <= sumG / 4 ? 1024 : 256 * sumG / sumR; - const unsigned int gainB = sumB <= sumG / 4 ? 1024 : 256 * sumG / sumB; - /* Green gain and gamma values are fixed */ - constexpr unsigned int gainG = 256; - - /* Update the gamma table if needed */ - if (blackLevel != lastBlackLevel_) { - constexpr float gamma = 0.5; - const unsigned int blackIndex = blackLevel * kGammaLookupSize / 256; - std::fill(gammaTable_.begin(), gammaTable_.begin() + blackIndex, 0); - const float divisor = kGammaLookupSize - blackIndex - 1.0; - for (unsigned int i = blackIndex; i < kGammaLookupSize; i++) - gammaTable_[i] = UINT8_MAX * - std::pow((i - blackIndex) / divisor, gamma); - - lastBlackLevel_ = blackLevel; - } - - for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { - constexpr unsigned int div = - DebayerParams::kRGBLookupSize * 256 / kGammaLookupSize; - unsigned int idx; - - /* Apply gamma after gain! */ - idx = std::min({ i * gainR / div, (kGammaLookupSize - 1) }); - params_->red[i] = gammaTable_[idx]; - - idx = std::min({ i * gainG / div, (kGammaLookupSize - 1) }); - params_->green[i] = gammaTable_[idx]; - - idx = std::min({ i * gainB / div, (kGammaLookupSize - 1) }); - params_->blue[i] = gammaTable_[idx]; - } - - setIspParams.emit(); - /* \todo Switch to the libipa/algorithm.h API someday. */ /* @@ -372,6 +312,7 @@ void IPASoftSimple::processStats( * Calculate Mean Sample Value (MSV) according to formula from: * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf */ + const uint8_t blackLevel = context_.activeState.black.level; const unsigned int blackLevelHistIdx = blackLevel / (256 / SwIspStats::kYHistogramSize); const unsigned int histogramSize = @@ -421,7 +362,8 @@ void IPASoftSimple::processStats( LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV << " exp " << exposure_ << " again " << again_ - << " gain R/B " << gainR << "/" << gainB + << " gain R/B " << context_.activeState.gains.red + << "/" << context_.activeState.gains.blue << " black level " << static_cast(blackLevel); } From patchwork Wed Jul 3 17:51:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20557 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 43EFCBEFBE for ; Wed, 3 Jul 2024 17:52:14 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D112463372; Wed, 3 Jul 2024 19:52:13 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="DfU7V9Ji"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id EDFC463370 for ; Wed, 3 Jul 2024 19:52:02 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029122; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+9dpGTI7Lb5OzMcvQBdIlrn1ts2cKKUtPJxz42cvmmM=; b=DfU7V9JivyIk+GpqRGdubgk2XggfuFgrXolKSwbXVfkdL9wlZlX377EE7q+zc042sCmxol CgnKH/yYeJyuuwEPY7trrFyszFi6nhNEj5eGJ6bzxt1qn3d89KV9WSmmOYBCax4OTFWRCS nERfnvwJU/z7vvsuj7fGtRZIWu62QB8= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-586-aDZFdukcN4ysfBbZDLLRxg-1; Wed, 03 Jul 2024 13:51:58 -0400 X-MC-Unique: aDZFdukcN4ysfBbZDLLRxg-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EFF011955F06; Wed, 3 Jul 2024 17:51:57 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 3BDC91954B0D; Wed, 3 Jul 2024 17:51:55 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 16/19] libcamera: software_isp: Use floating point for color parameters Date: Wed, 3 Jul 2024 19:51:16 +0200 Message-ID: <20240703175119.1872585-17-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" It's more natural to represent color gains and black level as floating point numbers rather than using a particular pixel-related representation. double is used rather than float because it's a more common floating point type in libcamera algorithms. Otherwise there is no obvious reason to select one over the other here. The constructed color tables still use integer representation for efficiency. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/blc.cpp | 8 ++++---- src/ipa/simple/algorithms/colors.cpp | 26 ++++++++++++++------------ src/ipa/simple/ipa_context.h | 11 +++++------ 3 files changed, 23 insertions(+), 22 deletions(-) diff --git a/src/ipa/simple/algorithms/blc.cpp b/src/ipa/simple/algorithms/blc.cpp index e98c5804..49f13682 100644 --- a/src/ipa/simple/algorithms/blc.cpp +++ b/src/ipa/simple/algorithms/blc.cpp @@ -24,7 +24,7 @@ BlackLevel::BlackLevel() int BlackLevel::init(IPAContext &context, [[maybe_unused]] const YamlObject &tuningData) { - context.activeState.black.level = 255; + context.activeState.black.level = 1.0; return 0; } @@ -44,16 +44,16 @@ void BlackLevel::process(IPAContext &context, const unsigned int total = std::accumulate(begin(histogram), end(histogram), 0); const unsigned int pixelThreshold = ignoredPercentage_ * total; - const unsigned int histogramRatio = 256 / SwIspStats::kYHistogramSize; const unsigned int currentBlackIdx = - context.activeState.black.level / histogramRatio; + context.activeState.black.level * SwIspStats::kYHistogramSize; for (unsigned int i = 0, seen = 0; i < currentBlackIdx && i < SwIspStats::kYHistogramSize; i++) { seen += histogram[i]; if (seen >= pixelThreshold) { - context.activeState.black.level = i * histogramRatio; + context.activeState.black.level = + static_cast(i) / SwIspStats::kYHistogramSize; LOG(IPASoftBL, Debug) << "Auto-set black level: " << i << "/" << SwIspStats::kYHistogramSize diff --git a/src/ipa/simple/algorithms/colors.cpp b/src/ipa/simple/algorithms/colors.cpp index 60fdca15..77492d9e 100644 --- a/src/ipa/simple/algorithms/colors.cpp +++ b/src/ipa/simple/algorithms/colors.cpp @@ -36,7 +36,7 @@ int Colors::init(IPAContext &context, updateGammaTable(context); auto &gains = context.activeState.gains; - gains.red = gains.green = gains.blue = 256; + gains.red = gains.green = gains.blue = 1.0; return 0; } @@ -47,7 +47,7 @@ void Colors::updateGammaTable(IPAContext &context) auto &blackLevel = context.activeState.gamma.blackLevel; blackLevel = context.activeState.black.level; const unsigned int blackIndex = - blackLevel * IPAActiveState::kGammaLookupSize / 256; + context.activeState.black.level * IPAActiveState::kGammaLookupSize; std::fill(gammaTable.begin(), gammaTable.begin() + blackIndex, 0); const float divisor = kGammaLookupSize - blackIndex - 1.0; for (unsigned int i = blackIndex; i < kGammaLookupSize; i++) @@ -67,15 +67,18 @@ void Colors::prepare(IPAContext &context, auto &gains = context.activeState.gains; auto &gammaTable = context.activeState.gamma.gammaTable; for (unsigned int i = 0; i < DebayerParams::kRGBLookupSize; i++) { - constexpr unsigned int div = - static_cast(DebayerParams::kRGBLookupSize) * 256 / kGammaLookupSize; + constexpr double div = + static_cast(DebayerParams::kRGBLookupSize) / kGammaLookupSize; /* Apply gamma after gain! */ unsigned int idx; - idx = std::min({ i * gains.red / div, kGammaLookupSize - 1 }); + idx = std::min({ static_cast(i * gains.red / div), + kGammaLookupSize - 1 }); params->red[i] = gammaTable[idx]; - idx = std::min({ i * gains.green / div, kGammaLookupSize - 1 }); + idx = std::min({ static_cast(i * gains.green / div), + kGammaLookupSize - 1 }); params->green[i] = gammaTable[idx]; - idx = std::min({ i * gains.blue / div, kGammaLookupSize - 1 }); + idx = std::min({ static_cast(i * gains.blue / div), + kGammaLookupSize - 1 }); params->blue[i] = gammaTable[idx]; } } @@ -87,7 +90,7 @@ void Colors::process(IPAContext &context, [[maybe_unused]] ControlList &metadata) { const SwIspStats::Histogram &histogram = stats->yHistogram; - const uint8_t blackLevel = context.activeState.black.level; + const double blackLevel = context.activeState.black.level; /* * Black level must be subtracted to get the correct AWB ratios, they @@ -104,12 +107,11 @@ void Colors::process(IPAContext &context, /* * Calculate red and blue gains for AWB. * Clamp max gain at 4.0, this also avoids 0 division. - * Gain: 128 = 0.5, 256 = 1.0, 512 = 2.0, etc. */ auto &gains = context.activeState.gains; - gains.red = sumR <= sumG / 4 ? 1024 : 256 * sumG / sumR; - gains.blue = sumB <= sumG / 4 ? 1024 : 256 * sumG / sumB; - /* Green gain is fixed to 256 */ + gains.red = sumR <= sumG / 4 ? 4.0 : static_cast(sumG) / sumR; + gains.blue = sumB <= sumG / 4 ? 4.0 : static_cast(sumG) / sumB; + /* Green gain is fixed to 1.0 */ LOG(IPASoftColors, Debug) << "gain R/B " << gains.red << "/" << gains.blue; } diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index 979ddb8f..552d6cde 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -9,7 +9,6 @@ #pragma once #include -#include #include @@ -23,17 +22,17 @@ struct IPASessionConfiguration { struct IPAActiveState { struct { - uint8_t level; + double level; } black; struct { - unsigned int red; - unsigned int green; - unsigned int blue; + double red; + double green; + double blue; } gains; static constexpr unsigned int kGammaLookupSize = 1024; struct { std::array gammaTable; - uint8_t blackLevel; + double blackLevel; } gamma; }; From patchwork Wed Jul 3 17:51:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20558 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 706CBBEFBE for ; Wed, 3 Jul 2024 17:52:16 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 841B66337C; Wed, 3 Jul 2024 19:52:15 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="iAjHKxO2"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 425DE63372 for ; Wed, 3 Jul 2024 19:52:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029122; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L7MlOh+qw73mQOEyoWAqoUxlB5WmS0x01cRTOrcozbI=; b=iAjHKxO2zSNMQBvA61OU9riukEG9nyiHvCzLpiVCDnWR6AlJ8oTL3YI+H6hb/5GGEtj3Fj O95qQAngq3XbxzOd7Jft8P8H9GyY4fWy1krfzGAPH1QdnG5z8JlLF8iDSq1MRcWUmWQbbq uxqfhLm9g9bwdskBLvheMNeyv3Qi3KU= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-232-RnaJU6CSPve27GWGjPxd_w-1; Wed, 03 Jul 2024 13:52:00 -0400 X-MC-Unique: RnaJU6CSPve27GWGjPxd_w-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A9D5A1955BC1; Wed, 3 Jul 2024 17:51:59 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 479E31954B0D; Wed, 3 Jul 2024 17:51:58 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 17/19] libcamera: software_isp: Use DelayedControls Date: Wed, 3 Jul 2024 19:51:17 +0200 Message-ID: <20240703175119.1872585-18-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Use the standard libcamera mechanism to report the "current" controls rather than delaying updates by counting from the last update. MY SPECULATION -- valid or not?: A problem is that with software ISP we cannot be sure about the sensor delay. The original implementation simply skips exposure updates for 2 frames, which should be enough in most cases. After this change, we assume the delay being exactly 2 frames, which may or may not be correct and may work with outdated values if the delay is shorter. This patch also prepares moving exposure+gain to an algorithm module where the original delay mechanism would be a (possibly unnecessary) complication. Resolves software ISP TODO #11 + #12. Signed-off-by: Milan Zamazal --- src/ipa/simple/soft_simple.cpp | 16 +------------ src/libcamera/pipeline/simple/simple.cpp | 17 ++++++++++++-- src/libcamera/software_isp/TODO | 29 ------------------------ 3 files changed, 16 insertions(+), 46 deletions(-) diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index 2658e359..cf2d628d 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -58,8 +58,7 @@ class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module public: IPASoftSimple() : params_(nullptr), stats_(nullptr), - context_({ {}, {}, { kMaxFrameContexts } }), - ignoreUpdates_(0) + context_({ {}, {}, { kMaxFrameContexts } }) { } @@ -97,7 +96,6 @@ private: int32_t exposure_; double againMin_, againMax_, againMinStep_; double again_; - unsigned int ignoreUpdates_; }; IPASoftSimple::~IPASoftSimple() @@ -298,16 +296,6 @@ void IPASoftSimple::processStats( /* \todo Switch to the libipa/algorithm.h API someday. */ - /* - * AE / AGC, use 2 frames delay to make sure that the exposure and - * the gain set have applied to the camera sensor. - * \todo This could be handled better with DelayedControls. - */ - if (ignoreUpdates_ > 0) { - --ignoreUpdates_; - return; - } - /* * Calculate Mean Sample Value (MSV) according to formula from: * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf @@ -356,8 +344,6 @@ void IPASoftSimple::processStats( ctrls.set(V4L2_CID_ANALOGUE_GAIN, static_cast(camHelper_ ? camHelper_->gainCode(again_) : again_)); - ignoreUpdates_ = 2; - setSensorControls.emit(ctrls); LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 6fb80209..68f650b4 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -32,6 +32,7 @@ #include "libcamera/internal/camera.h" #include "libcamera/internal/camera_sensor.h" #include "libcamera/internal/converter.h" +#include "libcamera/internal/delayed_controls.h" #include "libcamera/internal/device_enumerator.h" #include "libcamera/internal/media_device.h" #include "libcamera/internal/pipeline_handler.h" @@ -277,6 +278,8 @@ public: std::vector configs_; std::map> formats_; + std::unique_ptr delayedCtrls_; + std::vector> conversionBuffers_; std::queue> conversionQueue_; bool useConversion_; @@ -902,12 +905,12 @@ void SimpleCameraData::ispStatsReady(uint32_t frame, uint32_t bufferId) /* \todo Use the DelayedControls class */ swIsp_->processStats( frame, bufferId, - sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, - V4L2_CID_EXPOSURE })); + delayedCtrls_->get(frame)); } void SimpleCameraData::setSensorControls(const ControlList &sensorControls) { + delayedCtrls_->push(sensorControls); ControlList ctrls(sensorControls); sensor_->setControls(&ctrls); } @@ -1288,6 +1291,16 @@ int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c) if (outputCfgs.empty()) return 0; + std::unordered_map params = { + { V4L2_CID_ANALOGUE_GAIN, { 2, false } }, + { V4L2_CID_EXPOSURE, { 2, false } }, + }; + data->delayedCtrls_ = + std::make_unique(data->sensor_->device(), + params); + data->video_->frameStart.connect(data->delayedCtrls_.get(), + &DelayedControls::applyControls); + StreamConfiguration inputCfg; inputCfg.pixelFormat = pipeConfig->captureFormat; inputCfg.size = pipeConfig->captureSize; diff --git a/src/libcamera/software_isp/TODO b/src/libcamera/software_isp/TODO index 6bdc5905..6b1cb57a 100644 --- a/src/libcamera/software_isp/TODO +++ b/src/libcamera/software_isp/TODO @@ -228,35 +228,6 @@ At some point, yes. --- -11. Improve handling the sensor controls which take effect with a delay - -> void IPASoftSimple::processStats(const ControlList &sensorControls) -> { -> ... -> /* -> * AE / AGC, use 2 frames delay to make sure that the exposure and -> * the gain set have applied to the camera sensor. -> */ -> if (ignore_updates_ > 0) { -> --ignore_updates_; -> return; -> } - -This could be handled better with DelayedControls. - ---- - -12. Use DelayedControls class in ispStatsReady() - -> void SimpleCameraData::ispStatsReady() -> { -> swIsp_->processStats(sensor_->getControls({ V4L2_CID_ANALOGUE_GAIN, -> V4L2_CID_EXPOSURE })); - -You should use the DelayedControls class. - ---- - 13. Improve black level and colour gains application I think the black level should eventually be moved before debayering, and From patchwork Wed Jul 3 17:51:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20560 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 80778BEFBE for ; Wed, 3 Jul 2024 17:52:20 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 43FDB63375; Wed, 3 Jul 2024 19:52:20 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="i76Tq4lE"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6BA6563366 for ; Wed, 3 Jul 2024 19:52:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029127; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=blANfVDi5MIIajhHJ6kxFMGEc7GbTytWrOgiuYpKua0=; b=i76Tq4lEFz2YZPz9R94zqR1V0eXADVwlyXzUAbBPP6GressfiB4duW5uY23nheNbQQLfuN H3GEF0tjUuhLaSqfvgSlZ669onKen2aR+M6J6/BRlqVDPwQ6pNsDZeUO8U/IYK55NeJWIt lwdXZJnaLVeoDFuakk0CplyrVMyCcEk= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-529-ydneVMmCOaO-2WnjG-wCUw-1; Wed, 03 Jul 2024 13:52:02 -0400 X-MC-Unique: ydneVMmCOaO-2WnjG-wCUw-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D338E19560AD; Wed, 3 Jul 2024 17:52:01 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 16CC31954B0D; Wed, 3 Jul 2024 17:51:59 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 18/19] libcamera: software_isp: Move exposure+gain to an algorithm module Date: Wed, 3 Jul 2024 19:51:18 +0200 Message-ID: <20240703175119.1872585-19-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This is the last step to fully convert software ISP to Algorithm-based processing. The newly introduced frameContext.sensor properties are set, and the updated code moved, before calling Algorithm::process() to have the values up-to-date in stats processing. Resolves software ISP TODO #10. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/agc.cpp | 139 ++++++++++++++++++++++ src/ipa/simple/algorithms/agc.h | 36 ++++++ src/ipa/simple/algorithms/meson.build | 1 + src/ipa/simple/data/uncalibrated.yaml | 1 + src/ipa/simple/ipa_context.cpp | 11 ++ src/ipa/simple/ipa_context.h | 17 +++ src/ipa/simple/soft_simple.cpp | 159 +++++--------------------- src/libcamera/software_isp/TODO | 10 -- 8 files changed, 235 insertions(+), 139 deletions(-) create mode 100644 src/ipa/simple/algorithms/agc.cpp create mode 100644 src/ipa/simple/algorithms/agc.h diff --git a/src/ipa/simple/algorithms/agc.cpp b/src/ipa/simple/algorithms/agc.cpp new file mode 100644 index 00000000..a80022ff --- /dev/null +++ b/src/ipa/simple/algorithms/agc.cpp @@ -0,0 +1,139 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Exposure and gain + */ + +#include "agc.h" + +#include + +#include + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPASoftExposure) + +namespace ipa::soft::algorithms { + +/* + * The number of bins to use for the optimal exposure calculations. + */ +static constexpr unsigned int kExposureBinsCount = 5; + +/* + * The exposure is optimal when the mean sample value of the histogram is + * in the middle of the range. + */ +static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; + +/* + * The below value implements the hysteresis for the exposure adjustment. + * It is small enough to have the exposure close to the optimal, and is big + * enough to prevent the exposure from wobbling around the optimal value. + */ +static constexpr float kExposureSatisfactory = 0.2; + +Agc::Agc() +{ +} + +void Agc::updateExposure(IPAContext &context, double exposureMSV) +{ + /* + * kExpDenominator of 10 gives ~10% increment/decrement; + * kExpDenominator of 5 - about ~20% + */ + static constexpr uint8_t kExpDenominator = 10; + static constexpr uint8_t kExpNumeratorUp = kExpDenominator + 1; + static constexpr uint8_t kExpNumeratorDown = kExpDenominator - 1; + + double next; + int32_t &exposure = context.activeState.agc.exposure; + double &again = context.activeState.agc.again; + + if (exposureMSV < kExposureOptimal - kExposureSatisfactory) { + next = exposure * kExpNumeratorUp / kExpDenominator; + if (next - exposure < 1) + exposure += 1; + else + exposure = next; + if (exposure >= context.configuration.agc.exposureMax) { + next = again * kExpNumeratorUp / kExpDenominator; + if (next - again < context.configuration.agc.againMinStep) + again += context.configuration.agc.againMinStep; + else + again = next; + } + } + + if (exposureMSV > kExposureOptimal + kExposureSatisfactory) { + if (exposure == context.configuration.agc.exposureMax && + again > context.configuration.agc.againMin) { + next = again * kExpNumeratorDown / kExpDenominator; + if (again - next < context.configuration.agc.againMinStep) + again -= context.configuration.agc.againMinStep; + else + again = next; + } else { + next = exposure * kExpNumeratorDown / kExpDenominator; + if (exposure - next < 1) + exposure -= 1; + else + exposure = next; + } + } + + exposure = std::clamp(exposure, context.configuration.agc.exposureMin, + context.configuration.agc.exposureMax); + again = std::clamp(again, context.configuration.agc.againMin, + context.configuration.agc.againMax); + + LOG(IPASoftExposure, Debug) + << "exposureMSV " << exposureMSV + << " exp " << exposure << " again " << again; +} + +void Agc::process(IPAContext &context, + [[maybe_unused]] const uint32_t frame, + [[maybe_unused]] IPAFrameContext &frameContext, + const SwIspStats *stats, + [[maybe_unused]] ControlList &metadata) +{ + /* + * Calculate Mean Sample Value (MSV) according to formula from: + * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf + */ + const auto &histogram = stats->yHistogram; + const unsigned int blackLevelHistIdx = + context.configuration.black.level * SwIspStats::kYHistogramSize; + const unsigned int histogramSize = + SwIspStats::kYHistogramSize - blackLevelHistIdx; + const unsigned int yHistValsPerBin = histogramSize / kExposureBinsCount; + const unsigned int yHistValsPerBinMod = + histogramSize / (histogramSize % kExposureBinsCount + 1); + int exposureBins[kExposureBinsCount] = {}; + unsigned int denom = 0; + unsigned int num = 0; + + for (unsigned int i = 0; i < histogramSize; i++) { + unsigned int idx = (i - (i / yHistValsPerBinMod)) / yHistValsPerBin; + exposureBins[idx] += histogram[blackLevelHistIdx + i]; + } + + for (unsigned int i = 0; i < kExposureBinsCount; i++) { + LOG(IPASoftExposure, Debug) << i << ": " << exposureBins[i]; + denom += exposureBins[i]; + num += exposureBins[i] * (i + 1); + } + + float exposureMSV = (denom == 0 ? 0 : static_cast(num) / denom); + updateExposure(context, exposureMSV); +} + +REGISTER_IPA_ALGORITHM(Agc, "Agc") + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/agc.h b/src/ipa/simple/algorithms/agc.h new file mode 100644 index 00000000..a4aa656c --- /dev/null +++ b/src/ipa/simple/algorithms/agc.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Red Hat Inc. + * + * Exposure and gain + */ + +#pragma once + +#include "libcamera/internal/software_isp/swisp_stats.h" + +#include "algorithm.h" +#include "ipa_context.h" + +namespace libcamera { + +namespace ipa::soft::algorithms { + +class Agc : public Algorithm +{ +public: + Agc(); + ~Agc() = default; + + void process(IPAContext &context, const uint32_t frame, + IPAFrameContext &frameContext, + const SwIspStats *stats, + ControlList &metadata) override; + +private: + void updateExposure(IPAContext &context, double exposureMSV); +}; + +} /* namespace ipa::soft::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/simple/algorithms/meson.build b/src/ipa/simple/algorithms/meson.build index 324c7b4e..0b178e17 100644 --- a/src/ipa/simple/algorithms/meson.build +++ b/src/ipa/simple/algorithms/meson.build @@ -1,6 +1,7 @@ # SPDX-License-Identifier: CC0-1.0 soft_simple_ipa_algorithms = files([ + 'agc.cpp', 'blc.cpp', 'colors.cpp', ]) diff --git a/src/ipa/simple/data/uncalibrated.yaml b/src/ipa/simple/data/uncalibrated.yaml index 7e5149e5..7489c81f 100644 --- a/src/ipa/simple/data/uncalibrated.yaml +++ b/src/ipa/simple/data/uncalibrated.yaml @@ -5,4 +5,5 @@ version: 1 algorithms: - BlackLevel: - Colors: + - Agc: ... diff --git a/src/ipa/simple/ipa_context.cpp b/src/ipa/simple/ipa_context.cpp index fc113527..dd55d52d 100644 --- a/src/ipa/simple/ipa_context.cpp +++ b/src/ipa/simple/ipa_context.cpp @@ -77,6 +77,17 @@ namespace libcamera::ipa::soft { * \brief Gain of blue color */ +/** + * \var IPAActiveState::agc + * \brief Context for the AGC algorithm + * + * \var IPAActiveState::agc.exposure + * \brief Current exposure value + * + * \var IPAActiveState::agc.again + * \brief Current analog gain value + */ + /** * \var IPAActiveState::gamma * \brief Context for gamma in the Colors algorithm diff --git a/src/ipa/simple/ipa_context.h b/src/ipa/simple/ipa_context.h index 552d6cde..40a2efc2 100644 --- a/src/ipa/simple/ipa_context.h +++ b/src/ipa/simple/ipa_context.h @@ -10,6 +10,8 @@ #include +#include + #include namespace libcamera { @@ -18,6 +20,13 @@ namespace ipa::soft { struct IPASessionConfiguration { float gamma; + struct { + int32_t exposureMin, exposureMax; + double againMin, againMax, againMinStep; + } agc; + struct { + double level; + } black; }; struct IPAActiveState { @@ -29,6 +38,10 @@ struct IPAActiveState { double green; double blue; } gains; + struct { + int32_t exposure; + double again; + } agc; static constexpr unsigned int kGammaLookupSize = 1024; struct { std::array gammaTable; @@ -37,6 +50,10 @@ struct IPAActiveState { }; struct IPAFrameContext : public FrameContext { + struct { + uint32_t exposure; + double gain; + } sensor; }; struct IPAContext { diff --git a/src/ipa/simple/soft_simple.cpp b/src/ipa/simple/soft_simple.cpp index cf2d628d..547d0fcf 100644 --- a/src/ipa/simple/soft_simple.cpp +++ b/src/ipa/simple/soft_simple.cpp @@ -33,23 +33,6 @@ LOG_DEFINE_CATEGORY(IPASoft) namespace ipa::soft { -/* - * The number of bins to use for the optimal exposure calculations. - */ -static constexpr unsigned int kExposureBinsCount = 5; - -/* - * The exposure is optimal when the mean sample value of the histogram is - * in the middle of the range. - */ -static constexpr float kExposureOptimal = kExposureBinsCount / 2.0; - -/* - * The below value implements the hysteresis for the exposure adjustment. - * It is small enough to have the exposure close to the optimal, and is big - * enough to prevent the exposure from wobbling around the optimal value. - */ -static constexpr float kExposureSatisfactory = 0.2; /* Maximum number of frame contexts to be held */ static constexpr uint32_t kMaxFrameContexts = 16; @@ -57,8 +40,7 @@ class IPASoftSimple : public ipa::soft::IPASoftInterface, public Module { public: IPASoftSimple() - : params_(nullptr), stats_(nullptr), - context_({ {}, {}, { kMaxFrameContexts } }) + : context_({ {}, {}, { kMaxFrameContexts } }) { } @@ -91,11 +73,6 @@ private: /* Local parameter storage */ struct IPAContext context_; - - int32_t exposureMin_, exposureMax_; - int32_t exposure_; - double againMin_, againMax_, againMinStep_; - double again_; }; IPASoftSimple::~IPASoftSimple() @@ -206,20 +183,23 @@ int IPASoftSimple::configure(const IPAConfigInfo &configInfo) const ControlInfo &exposureInfo = sensorInfoMap_.find(V4L2_CID_EXPOSURE)->second; const ControlInfo &gainInfo = sensorInfoMap_.find(V4L2_CID_ANALOGUE_GAIN)->second; - exposureMin_ = exposureInfo.min().get(); - exposureMax_ = exposureInfo.max().get(); - if (!exposureMin_) { + context_.configuration.agc.exposureMin = exposureInfo.min().get(); + context_.configuration.agc.exposureMax = exposureInfo.max().get(); + if (!context_.configuration.agc.exposureMin) { LOG(IPASoft, Warning) << "Minimum exposure is zero, that can't be linear"; - exposureMin_ = 1; + context_.configuration.agc.exposureMin = 1; } int32_t againMin = gainInfo.min().get(); int32_t againMax = gainInfo.max().get(); if (camHelper_) { - againMin_ = camHelper_->gain(againMin); - againMax_ = camHelper_->gain(againMax); - againMinStep_ = (againMax_ - againMin_) / 100.0; + context_.configuration.agc.againMin = camHelper_->gain(againMin); + context_.configuration.agc.againMax = camHelper_->gain(againMax); + context_.configuration.agc.againMinStep = + (context_.configuration.agc.againMax - + context_.configuration.agc.againMin) / + 100.0; } else { /* * The camera sensor gain (g) is usually not equal to the value written @@ -231,13 +211,14 @@ int IPASoftSimple::configure(const IPAConfigInfo &configInfo) * the AGC algorithm (abrupt near one edge, and very small near the * other) we limit the range of the gain values used. */ - againMax_ = againMax; + context_.configuration.agc.againMax = againMax; if (!againMin) { LOG(IPASoft, Warning) << "Minimum gain is zero, that can't be linear"; - againMin_ = std::min(100, againMin / 2 + againMax / 2); + context_.configuration.agc.againMin = + std::min(100, againMin / 2 + againMax / 2); } - againMinStep_ = 1.0; + context_.configuration.agc.againMinStep = 1.0; } for (auto const &algo : algorithms()) { @@ -246,9 +227,12 @@ int IPASoftSimple::configure(const IPAConfigInfo &configInfo) return ret; } - LOG(IPASoft, Info) << "Exposure " << exposureMin_ << "-" << exposureMax_ - << ", gain " << againMin_ << "-" << againMax_ - << " (" << againMinStep_ << ")"; + LOG(IPASoft, Info) + << "Exposure " << context_.configuration.agc.exposureMin << "-" + << context_.configuration.agc.exposureMax + << ", gain " << context_.configuration.agc.againMin << "-" + << context_.configuration.agc.againMax + << " (" << context_.configuration.agc.againMinStep << ")"; return 0; } @@ -284,6 +268,12 @@ void IPASoftSimple::processStats( const ControlList &sensorControls) { IPAFrameContext &frameContext = context_.frameContexts.get(frame); + + frameContext.sensor.exposure = + sensorControls.get(V4L2_CID_EXPOSURE).get(); + int32_t again = sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get(); + frameContext.sensor.gain = camHelper_ ? camHelper_->gain(again) : again; + /* * Software ISP currently does not produce any metadata. Use an empty * ControlList for now. @@ -294,37 +284,6 @@ void IPASoftSimple::processStats( for (auto const &algo : algorithms()) algo->process(context_, frame, frameContext, stats_, metadata); - /* \todo Switch to the libipa/algorithm.h API someday. */ - - /* - * Calculate Mean Sample Value (MSV) according to formula from: - * https://www.araa.asn.au/acra/acra2007/papers/paper84final.pdf - */ - const uint8_t blackLevel = context_.activeState.black.level; - const unsigned int blackLevelHistIdx = - blackLevel / (256 / SwIspStats::kYHistogramSize); - const unsigned int histogramSize = - SwIspStats::kYHistogramSize - blackLevelHistIdx; - const unsigned int yHistValsPerBin = histogramSize / kExposureBinsCount; - const unsigned int yHistValsPerBinMod = - histogramSize / (histogramSize % kExposureBinsCount + 1); - int exposureBins[kExposureBinsCount] = {}; - unsigned int denom = 0; - unsigned int num = 0; - - for (unsigned int i = 0; i < histogramSize; i++) { - unsigned int idx = (i - (i / yHistValsPerBinMod)) / yHistValsPerBin; - exposureBins[idx] += stats_->yHistogram[blackLevelHistIdx + i]; - } - - for (unsigned int i = 0; i < kExposureBinsCount; i++) { - LOG(IPASoft, Debug) << i << ": " << exposureBins[i]; - denom += exposureBins[i]; - num += exposureBins[i] * (i + 1); - } - - float exposureMSV = static_cast(num) / denom; - /* Sanity check */ if (!sensorControls.contains(V4L2_CID_EXPOSURE) || !sensorControls.contains(V4L2_CID_ANALOGUE_GAIN)) { @@ -332,72 +291,14 @@ void IPASoftSimple::processStats( return; } - exposure_ = sensorControls.get(V4L2_CID_EXPOSURE).get(); - int32_t again = sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get(); - again_ = camHelper_ ? camHelper_->gain(again) : again; - - updateExposure(exposureMSV); - ControlList ctrls(sensorInfoMap_); - ctrls.set(V4L2_CID_EXPOSURE, exposure_); + auto &againNew = context_.activeState.agc.again; + ctrls.set(V4L2_CID_EXPOSURE, context_.activeState.agc.exposure); ctrls.set(V4L2_CID_ANALOGUE_GAIN, - static_cast(camHelper_ ? camHelper_->gainCode(again_) : again_)); + static_cast(camHelper_ ? camHelper_->gainCode(againNew) : againNew)); setSensorControls.emit(ctrls); - - LOG(IPASoft, Debug) << "exposureMSV " << exposureMSV - << " exp " << exposure_ << " again " << again_ - << " gain R/B " << context_.activeState.gains.red - << "/" << context_.activeState.gains.blue - << " black level " << static_cast(blackLevel); -} - -void IPASoftSimple::updateExposure(double exposureMSV) -{ - /* - * kExpDenominator of 10 gives ~10% increment/decrement; - * kExpDenominator of 5 - about ~20% - */ - static constexpr uint8_t kExpDenominator = 10; - static constexpr uint8_t kExpNumeratorUp = kExpDenominator + 1; - static constexpr uint8_t kExpNumeratorDown = kExpDenominator - 1; - - double next; - - if (exposureMSV < kExposureOptimal - kExposureSatisfactory) { - next = exposure_ * kExpNumeratorUp / kExpDenominator; - if (next - exposure_ < 1) - exposure_ += 1; - else - exposure_ = next; - if (exposure_ >= exposureMax_) { - next = again_ * kExpNumeratorUp / kExpDenominator; - if (next - again_ < againMinStep_) - again_ += againMinStep_; - else - again_ = next; - } - } - - if (exposureMSV > kExposureOptimal + kExposureSatisfactory) { - if (exposure_ == exposureMax_ && again_ > againMin_) { - next = again_ * kExpNumeratorDown / kExpDenominator; - if (again_ - next < againMinStep_) - again_ -= againMinStep_; - else - again_ = next; - } else { - next = exposure_ * kExpNumeratorDown / kExpDenominator; - if (exposure_ - next < 1) - exposure_ -= 1; - else - exposure_ = next; - } - } - - exposure_ = std::clamp(exposure_, exposureMin_, exposureMax_); - again_ = std::clamp(again_, againMin_, againMax_); } std::string IPASoftSimple::logPrefix() const diff --git a/src/libcamera/software_isp/TODO b/src/libcamera/software_isp/TODO index 6b1cb57a..42828895 100644 --- a/src/libcamera/software_isp/TODO +++ b/src/libcamera/software_isp/TODO @@ -218,16 +218,6 @@ Yes, because, well... all the other IPAs were doing that... --- -10. Switch to libipa/algorithm.h API in processStats - ->> void IPASoftSimple::processStats(const ControlList &sensorControls) ->> -> Do you envision switching to the libipa/algorithm.h API at some point ? - -At some point, yes. - ---- - 13. Improve black level and colour gains application I think the black level should eventually be moved before debayering, and From patchwork Wed Jul 3 17:51:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Milan Zamazal X-Patchwork-Id: 20559 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 95CB9BD87C for ; Wed, 3 Jul 2024 17:52:19 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3949F63379; Wed, 3 Jul 2024 19:52:19 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="E1OwqcD1"; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id CFEF062E23 for ; Wed, 3 Jul 2024 19:52:06 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720029125; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ut5xKDLT/mcVIaUJnw+pCWlMoQbRbIi651zu07x587g=; b=E1OwqcD1SF7gUjHv6pvrvyx4ssYqb6pQIrQaIJz/Ox+subVj1YHiwVxkb3H+Gt71K5Y6Dq WAH5RSGXLohSINgrlNkqvHxa/Np4ch9fqw1A1+qwyElpUrvYg87hiQqm8I+POyQ7f6Tz19 xOHci8icftwe7CYt/qOFPGL9OCwp38c= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-176-UsERWiqDPsCRpVZeyakuRw-1; Wed, 03 Jul 2024 13:52:04 -0400 X-MC-Unique: UsERWiqDPsCRpVZeyakuRw-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A8B111955F66; Wed, 3 Jul 2024 17:52:03 +0000 (UTC) Received: from nuthatch.redhat.com (unknown [10.45.224.13]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 392B01954B0D; Wed, 3 Jul 2024 17:52:02 +0000 (UTC) From: Milan Zamazal To: libcamera-devel@lists.libcamera.org Cc: Milan Zamazal , Umang Jain Subject: [PATCH v2 19/19] libcamera: software_isp: Update black level only on exposure changes Date: Wed, 3 Jul 2024 19:51:19 +0200 Message-ID: <20240703175119.1872585-20-mzamazal@redhat.com> In-Reply-To: <20240703175119.1872585-1-mzamazal@redhat.com> References: <20240703175119.1872585-1-mzamazal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The black level is likely to get updated, if ever, only after exposure or gain changes. Don't compute its possible updates if exposure and gain are unchanged. It's probably not worth trying to implement something more sophisticated. Better to spend the effort on supporting tuning files. Signed-off-by: Milan Zamazal --- src/ipa/simple/algorithms/blc.cpp | 9 ++++++++- src/ipa/simple/algorithms/blc.h | 4 ++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/src/ipa/simple/algorithms/blc.cpp b/src/ipa/simple/algorithms/blc.cpp index 49f13682..8f7828d9 100644 --- a/src/ipa/simple/algorithms/blc.cpp +++ b/src/ipa/simple/algorithms/blc.cpp @@ -30,10 +30,15 @@ int BlackLevel::init(IPAContext &context, void BlackLevel::process(IPAContext &context, [[maybe_unused]] const uint32_t frame, - [[maybe_unused]] IPAFrameContext &frameContext, + IPAFrameContext &frameContext, const SwIspStats *stats, [[maybe_unused]] ControlList &metadata) { + if (frameContext.sensor.exposure == exposure_ && + frameContext.sensor.gain == gain_) { + return; + } + const SwIspStats::Histogram &histogram = stats->yHistogram; /* @@ -54,6 +59,8 @@ void BlackLevel::process(IPAContext &context, if (seen >= pixelThreshold) { context.activeState.black.level = static_cast(i) / SwIspStats::kYHistogramSize; + exposure_ = frameContext.sensor.exposure; + gain_ = frameContext.sensor.gain; LOG(IPASoftBL, Debug) << "Auto-set black level: " << i << "/" << SwIspStats::kYHistogramSize diff --git a/src/ipa/simple/algorithms/blc.h b/src/ipa/simple/algorithms/blc.h index c8b8d92d..4ddd2c94 100644 --- a/src/ipa/simple/algorithms/blc.h +++ b/src/ipa/simple/algorithms/blc.h @@ -30,6 +30,10 @@ public: IPAFrameContext &frameContext, const SwIspStats *stats, ControlList &metadata) override; + +private: + uint32_t exposure_; + double gain_; }; } /* namespace ipa::soft::algorithms */