From patchwork Wed Nov 24 13:40:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14750 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id AA99DBF415 for ; Wed, 24 Nov 2021 13:40:27 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 48B946038B; Wed, 24 Nov 2021 14:40:27 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="Cm7gr6fB"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 38C5160231 for ; Wed, 24 Nov 2021 14:40:23 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:968b:bd0c:97fc:7c17]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id E7F76D78; Wed, 24 Nov 2021 14:40:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637761223; bh=6qkF2Tg7yCai9Xl8kh9FitL+Os2hwaUjtO8oJRvJdwQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Cm7gr6fB/khfnmJARKGuzdDizVxJg4v11wm/371Lf5MN7K1XAhttSa/BNQCSDQTO7 6k9L93IuXgvgKaIPbh01vI5fS/8ycbpYnR3+YNLc3fzrxjLpZCEt15hFELnSnpJuKT mK+BRFuCcwwCzeT1SHORc43/7lI6s4+t+j10H2SY= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Wed, 24 Nov 2021 14:40:09 +0100 Message-Id: <20211124134019.110765-2-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> References: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 01/11] libipa: Correct IMX219 in CameraSensorHelper X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The equation is badly reported in the CameraSensorHelper, as m1 and c0 are inverted. Correct it to have a proper gain calculation. Signed-off-by: Jean-Michel Hautbois Acked-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/ipa/libipa/camera_sensor_helper.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ipa/libipa/camera_sensor_helper.cpp b/src/ipa/libipa/camera_sensor_helper.cpp index 0b0eb503..5d052612 100644 --- a/src/ipa/libipa/camera_sensor_helper.cpp +++ b/src/ipa/libipa/camera_sensor_helper.cpp @@ -290,7 +290,7 @@ class CameraSensorHelperImx219 : public CameraSensorHelper public: CameraSensorHelperImx219() { - analogueGainConstants_ = { AnalogueGainLinear, 0, -1, 256, 256 }; + analogueGainConstants_ = { AnalogueGainLinear, 0, 256, -1, 256 }; } }; REGISTER_CAMERA_SENSOR_HELPER("imx219", CameraSensorHelperImx219) From patchwork Wed Nov 24 13:40:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14751 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 21F16BF415 for ; Wed, 24 Nov 2021 13:40:29 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 49EF4603F9; Wed, 24 Nov 2021 14:40:28 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ouAyD6bb"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 63C0660233 for ; Wed, 24 Nov 2021 14:40:23 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:968b:bd0c:97fc:7c17]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 22D4A90E; Wed, 24 Nov 2021 14:40:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637761223; bh=YehKf1iayUezpPnKMAd/oOrQ48+itY7/+GNTbzUaglE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ouAyD6bbg+DOGmfTEJpvMNTBLWdXDUlBIbJdABd/4il4pNKdAmowiACHl+mUf+8g4 KpsiU7qZps3fWEoPrhZmj2UvZkPjLKIj8/wwcst7C8+3B+fNYvgvhP237cSpOB6H8f 6CNpwf3dsk3j1sNn4uslXAwuAtcMm0Wb/Wrq57jM= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Wed, 24 Nov 2021 14:40:10 +0100 Message-Id: <20211124134019.110765-3-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> References: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 02/11] ipa: rkisp1: Pass IPASettings at init call X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When the IPA is initialized, it will need to know the sensor model used in order to properly call CameraSensorHelper for the analogue gain. Modify the init definition in the pipeline handler and in the IPA as well as the mojo interface to pass the IPASettings. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Laurent Pinchart Reviewed-by: Kieran Bingham --- include/libcamera/ipa/rkisp1.mojom | 4 +++- src/ipa/rkisp1/rkisp1.cpp | 5 +++-- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 2 +- 3 files changed, 7 insertions(+), 4 deletions(-) diff --git a/include/libcamera/ipa/rkisp1.mojom b/include/libcamera/ipa/rkisp1.mojom index cae757ea..a6991d4f 100644 --- a/include/libcamera/ipa/rkisp1.mojom +++ b/include/libcamera/ipa/rkisp1.mojom @@ -29,7 +29,9 @@ struct RkISP1Action { }; interface IPARkISP1Interface { - init(uint32 hwRevision) => (int32 ret); + init(libcamera.IPASettings settings, + uint32 hwRevision) + => (int32 ret); start() => (int32 ret); stop(); diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index bf2c13b6..7ecbf8ae 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -34,7 +34,7 @@ namespace ipa::rkisp1 { class IPARkISP1 : public IPARkISP1Interface { public: - int init(unsigned int hwRevision) override; + int init(const IPASettings &settings, unsigned int hwRevision) override; int start() override; void stop() override {} @@ -75,7 +75,8 @@ private: unsigned int hwHistogramWeightGridsSize_; }; -int IPARkISP1::init(unsigned int hwRevision) +int IPARkISP1::init([[maybe_unused]] const IPASettings &settings, + unsigned int hwRevision) { /* \todo Add support for other revisions */ switch (hwRevision) { diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index 98008862..6cdc4c03 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -319,7 +319,7 @@ int RkISP1CameraData::loadIPA(unsigned int hwRevision) ipa_->queueFrameAction.connect(this, &RkISP1CameraData::queueFrameAction); - int ret = ipa_->init(hwRevision); + int ret = ipa_->init(IPASettings{ "", sensor_->model() }, hwRevision); if (ret < 0) { LOG(RkISP1, Error) << "IPA initialization failure"; return ret; From patchwork Wed Nov 24 13:40:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14752 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B4B5EBF415 for ; Wed, 24 Nov 2021 13:40:30 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 559ED6042B; Wed, 24 Nov 2021 14:40:29 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="Vgfr0yng"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9A67D60128 for ; Wed, 24 Nov 2021 14:40:23 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:968b:bd0c:97fc:7c17]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 51124D78; Wed, 24 Nov 2021 14:40:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637761223; bh=6pybVIYFXVd1SgQSEoyDZDyd/ijoQp2Fy/E09A9+tuw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Vgfr0yngUrkMlVFQ+wS/FjI0nSSBv0paGowaTVEXqhN6Pxq7J6478ic0+ZR8MjMOf CPExJTBHu2om838e2rlNhXGkEkYKi5aJ+y2+/nRlMOHRkksFE5oFiIzzT3HP1X6cLt MnG2Dst4uvDsJvmboWpRlH6jCEQGyOal9blfqV2s= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Wed, 24 Nov 2021 14:40:11 +0100 Message-Id: <20211124134019.110765-4-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> References: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 03/11] ipa: Do not modify the sensor limits X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The driver is responsible for setting the proper limits for its controls. The IMX219 has an analogue gain of 1.0 when the gain code is set to 0, therefore we can not clamp to a minimum gain code of 1. Rework this for both IPU3 and RkISP1, for both Exposure and Gain controls. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Laurent Pinchart Reviewed-by: Kieran Bingham --- src/ipa/ipu3/ipu3.cpp | 4 ++-- src/ipa/rkisp1/rkisp1.cpp | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index a8d54a5d..b0c75541 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -444,11 +444,11 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo, return -EINVAL; } - minExposure_ = std::max(itExp->second.min().get(), 1); + minExposure_ = itExp->second.min().get(); maxExposure_ = itExp->second.max().get(); exposure_ = minExposure_; - minGain_ = std::max(itGain->second.min().get(), 1); + minGain_ = itGain->second.min().get(); maxGain_ = itGain->second.max().get(); gain_ = minGain_; diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 7ecbf8ae..910ad952 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -139,11 +139,11 @@ int IPARkISP1::configure([[maybe_unused]] const IPACameraSensorInfo &info, autoExposure_ = true; - minExposure_ = std::max(itExp->second.min().get(), 1); + minExposure_ = itExp->second.min().get(); maxExposure_ = itExp->second.max().get(); exposure_ = minExposure_; - minGain_ = std::max(itGain->second.min().get(), 1); + minGain_ = itGain->second.min().get(); maxGain_ = itGain->second.max().get(); gain_ = minGain_; From patchwork Wed Nov 24 13:40:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14753 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B02F4BF415 for ; Wed, 24 Nov 2021 13:40:32 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 6746060376; Wed, 24 Nov 2021 14:40:32 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="R9ClGYxm"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C33E460376 for ; Wed, 24 Nov 2021 14:40:23 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:968b:bd0c:97fc:7c17]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 805AE1B61; Wed, 24 Nov 2021 14:40:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637761223; bh=7EKJLxN6FyygVP3ODjD71RglMl4Afhijn0ZyokJr6KI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=R9ClGYxmR2srleIfSVoXUTEF7pqBsbH96u5ZUCedSUzsvJZ7FPEsW23laD1E3hwfT wpTaFfZXWdeziAt8LIlnC44gB7T7izIvnVqkSdd6IyzIzuhE6meE7l9sNHTgNDnZjE 5Rcy+dhp8Y6ldOwUvliVPqH7xcd0cBbU6tj7DCX0= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Wed, 24 Nov 2021 14:40:12 +0100 Message-Id: <20211124134019.110765-5-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> References: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 04/11] ipa: rkisp1: Instantiate CameraSensorHelper X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When the AGC will run, it will use analogue gains as double values. We will need those values to be converted to apply the control. Introduce CameraSensorHelper and call it at init(). Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/ipa/rkisp1/rkisp1.cpp | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 910ad952..73a7f582 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -25,6 +25,8 @@ #include +#include "libipa/camera_sensor_helper.h" + namespace libcamera { LOG_DEFINE_CATEGORY(IPARkISP1) @@ -73,10 +75,12 @@ private: unsigned int hwHistBinNMax_; unsigned int hwGammaOutMaxSamples_; unsigned int hwHistogramWeightGridsSize_; + + /* Interface to the Camera Helper */ + std::unique_ptr camHelper_; }; -int IPARkISP1::init([[maybe_unused]] const IPASettings &settings, - unsigned int hwRevision) +int IPARkISP1::init(const IPASettings &settings, unsigned int hwRevision) { /* \todo Add support for other revisions */ switch (hwRevision) { @@ -100,6 +104,15 @@ int IPARkISP1::init([[maybe_unused]] const IPASettings &settings, } LOG(IPARkISP1, Debug) << "Hardware revision is " << hwRevision; + + camHelper_ = CameraSensorHelperFactory::create(settings.sensorModel); + if (!camHelper_) { + LOG(IPARkISP1, Error) + << "Failed to create camera sensor helper for " + << settings.sensorModel; + return -ENODEV; + } + return 0; } From patchwork Wed Nov 24 13:40:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14754 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4A997C324F for ; Wed, 24 Nov 2021 13:40:33 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E3F6E60394; Wed, 24 Nov 2021 14:40:32 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="DE4xExIe"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0455F6038A for ; Wed, 24 Nov 2021 14:40:24 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:968b:bd0c:97fc:7c17]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id AF28F90E; Wed, 24 Nov 2021 14:40:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637761223; bh=0kqsbAEhRO7dIoewyB9Wlddg6WRUI67UYuCSprpPsz8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DE4xExIe27q3zjfv5rGQeWZu2hPlkn1tkTFve1SlvxeVWwvHiG+gFXKA7rf3c83eE vKnDZeblhoN4Ckgr50xsq/+kORNU4AlEaVgGWeUlldQIJ+7knhEtLFjkLuXrimSkIQ ktYYhu3dwcnsL7nvs1jOmWvT+FuxJ5kmhkNqe9Lw= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Wed, 24 Nov 2021 14:40:13 +0100 Message-Id: <20211124134019.110765-6-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> References: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 05/11] ipa: rkisp1: Introduce IPAContext X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Before using any algorithm, we want the IPAContext to be ready for those. Introduce the IPAContext following the existing design from IPA::IPU3. Each algorithm will then introduce the needed fields. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- v4: use #pragma once --- src/ipa/rkisp1/ipa_context.cpp | 58 ++++++++++++++++++++++++++++++++++ src/ipa/rkisp1/ipa_context.h | 27 ++++++++++++++++ src/ipa/rkisp1/meson.build | 7 +++- src/ipa/rkisp1/rkisp1.cpp | 7 ++++ 4 files changed, 98 insertions(+), 1 deletion(-) create mode 100644 src/ipa/rkisp1/ipa_context.cpp create mode 100644 src/ipa/rkisp1/ipa_context.h diff --git a/src/ipa/rkisp1/ipa_context.cpp b/src/ipa/rkisp1/ipa_context.cpp new file mode 100644 index 00000000..819b2c73 --- /dev/null +++ b/src/ipa/rkisp1/ipa_context.cpp @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Ideas On Board + * + * ipa_context.cpp - RkISP1 IPA Context + */ + +#include "ipa_context.h" + +/** + * \file ipa_context.h + * \brief Context and state information shared between the algorithms + */ + +namespace libcamera::ipa::rkisp1 { + +/** + * \struct IPASessionConfiguration + * \brief Session configuration for the IPA module + * + * The session configuration contains all IPA configuration parameters that + * remain constant during the capture session, from IPA module start to stop. + * It is typically set during the configure() operation of the IPA module, but + * may also be updated in the start() operation. + */ + +/** + * \struct IPAFrameContext + * \brief Per-frame context for algorithms + * + * The frame context stores data specific to a single frame processed by the + * IPA. Each frame processed by the IPA has a context associated with it, + * accessible through the IPAContext structure. + * + * \todo Detail how to access contexts for a particular frame + * + * Each of the fields in the frame context belongs to either a specific + * algorithm, or to the top-level IPA module. A field may be read by any + * algorithm, but should only be written by its owner. + */ + +/** + * \struct IPAContext + * \brief Global IPA context data shared between all algorithms + * + * \var IPAContext::configuration + * \brief The IPA session configuration, immutable during the session + * + * \var IPAContext::frameContext + * \brief The frame context for the frame being processed + * + * \todo While the frame context is supposed to be per-frame, this + * single frame context stores data related to both the current frame + * and the previous frames, with fields being updated as the algorithms + * are run. This needs to be turned into real per-frame data storage. + */ + +} /* namespace libcamera::ipa::rkisp1 */ diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h new file mode 100644 index 00000000..a02b7667 --- /dev/null +++ b/src/ipa/rkisp1/ipa_context.h @@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Ideas On Board + * + * ipa_context.h - RkISP1 IPA Context + * + */ +#pragma once + +namespace libcamera { + +namespace ipa::rkisp1 { + +struct IPASessionConfiguration { +}; + +struct IPAFrameContext { +}; + +struct IPAContext { + IPASessionConfiguration configuration; + IPAFrameContext frameContext; +}; + +} /* namespace ipa::rkisp1 */ + +} /* namespace libcamera*/ diff --git a/src/ipa/rkisp1/meson.build b/src/ipa/rkisp1/meson.build index f76b37f5..3683c922 100644 --- a/src/ipa/rkisp1/meson.build +++ b/src/ipa/rkisp1/meson.build @@ -2,8 +2,13 @@ ipa_name = 'ipa_rkisp1' +rkisp1_ipa_sources = files([ + 'ipa_context.cpp', + 'rkisp1.cpp', +]) + mod = shared_module(ipa_name, - ['rkisp1.cpp', libcamera_generated_ipa_headers], + [rkisp1_ipa_sources, libcamera_generated_ipa_headers], name_prefix : '', include_directories : [ipa_includes, libipa_includes], dependencies : libcamera_private, diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 73a7f582..34c3f9a2 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -25,6 +25,7 @@ #include +#include "ipa_context.h" #include "libipa/camera_sensor_helper.h" namespace libcamera { @@ -78,6 +79,9 @@ private: /* Interface to the Camera Helper */ std::unique_ptr camHelper_; + + /* Local parameter storage */ + struct IPAContext context_; }; int IPARkISP1::init(const IPASettings &settings, unsigned int hwRevision) @@ -164,6 +168,9 @@ int IPARkISP1::configure([[maybe_unused]] const IPACameraSensorInfo &info, << "Exposure: " << minExposure_ << "-" << maxExposure_ << " Gain: " << minGain_ << "-" << maxGain_; + /* Clean context at configuration */ + context_ = {}; + return 0; } From patchwork Wed Nov 24 13:40:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14755 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 31B89BF415 for ; Wed, 24 Nov 2021 13:40:34 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BF72E603BE; Wed, 24 Nov 2021 14:40:33 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="kAiklZha"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2BE1F6039C for ; Wed, 24 Nov 2021 14:40:24 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:968b:bd0c:97fc:7c17]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id DD9B0D78; Wed, 24 Nov 2021 14:40:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637761224; bh=nUNJZu16m8uatmAMoR5/WTY+cV6rJSZG9O0vYntEQSI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kAiklZhaq13/S3aRqhzZ6Z2/7BlB3SliYshJQsqE9vc0JVeFaEwhEV+qE3rPLMEeW DeBxVbADmcsXojB9cKBWU9wuYL+Y93b59Yk3DFsJ6nv7iF5o8XumoOGH4XXMvoAQzA rT74PK3DkyljPWmZ+npwkaSkS5BX/E3coKYRmDNI= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Wed, 24 Nov 2021 14:40:14 +0100 Message-Id: <20211124134019.110765-7-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> References: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 06/11] ipa: libipa: Introduce Algorithm class template X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The algorithms are using the same function names with specialized parameters. Instead of duplicating code, introduce a libipa Algorithm class which implements a base class with template parameters in libipa, and use it in each IPA. As we now won't need an algorithm class for each IPA, move the documentation to libipa, and make it agnostic of the IPA used. While at it, fix the IPU3::Algorithm::Awb documentation. Signed-off-by: Laurent Pinchart Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham --- v4: use #pragma once --- src/ipa/ipu3/algorithms/algorithm.h | 12 ++---- src/ipa/ipu3/algorithms/awb.cpp | 9 +++++ src/ipa/ipu3/algorithms/meson.build | 1 - .../{ipu3/algorithms => libipa}/algorithm.cpp | 38 ++++++++----------- src/ipa/libipa/algorithm.h | 38 +++++++++++++++++++ src/ipa/libipa/meson.build | 1 + 6 files changed, 67 insertions(+), 32 deletions(-) rename src/ipa/{ipu3/algorithms => libipa}/algorithm.cpp (75%) create mode 100644 src/ipa/libipa/algorithm.h diff --git a/src/ipa/ipu3/algorithms/algorithm.h b/src/ipa/ipu3/algorithms/algorithm.h index 16310ab1..d2eecc78 100644 --- a/src/ipa/ipu3/algorithms/algorithm.h +++ b/src/ipa/ipu3/algorithms/algorithm.h @@ -9,21 +9,15 @@ #include +#include + #include "ipa_context.h" namespace libcamera { namespace ipa::ipu3 { -class Algorithm -{ -public: - virtual ~Algorithm() {} - - virtual int configure(IPAContext &context, const IPAConfigInfo &configInfo); - virtual void prepare(IPAContext &context, ipu3_uapi_params *params); - virtual void process(IPAContext &context, const ipu3_uapi_stats_3a *stats); -}; +using Algorithm = libcamera::ipa::Algorithm; } /* namespace ipa::ipu3 */ diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index c7bcb20e..1dc27fc9 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -193,6 +193,9 @@ Awb::Awb() Awb::~Awb() = default; +/** + * \copydoc libcamera::ipa::Algorithm::configure + */ int Awb::configure(IPAContext &context, [[maybe_unused]] const IPAConfigInfo &configInfo) { @@ -373,6 +376,9 @@ void Awb::calculateWBGains(const ipu3_uapi_stats_3a *stats) } } +/** + * \copydoc libcamera::ipa::Algorithm::process + */ void Awb::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) { calculateWBGains(stats); @@ -394,6 +400,9 @@ constexpr uint16_t Awb::threshold(float value) return value * 8191; } +/** + * \copydoc libcamera::ipa::Algorithm::prepare + */ void Awb::prepare(IPAContext &context, ipu3_uapi_params *params) { /* diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build index 3ec42f72..4db6ae1d 100644 --- a/src/ipa/ipu3/algorithms/meson.build +++ b/src/ipa/ipu3/algorithms/meson.build @@ -2,7 +2,6 @@ ipu3_ipa_algorithms = files([ 'agc.cpp', - 'algorithm.cpp', 'awb.cpp', 'blc.cpp', 'tone_mapping.cpp', diff --git a/src/ipa/ipu3/algorithms/algorithm.cpp b/src/ipa/libipa/algorithm.cpp similarity index 75% rename from src/ipa/ipu3/algorithms/algorithm.cpp rename to src/ipa/libipa/algorithm.cpp index 3e7e3018..398d5372 100644 --- a/src/ipa/ipu3/algorithms/algorithm.cpp +++ b/src/ipa/libipa/algorithm.cpp @@ -2,7 +2,7 @@ /* * Copyright (C) 2021, Ideas On Board * - * algorithm.cpp - IPU3 control algorithm interface + * algorithm.cpp - IPA control algorithm interface */ #include "algorithm.h" @@ -14,11 +14,15 @@ namespace libcamera { -namespace ipa::ipu3 { +namespace ipa { /** * \class Algorithm - * \brief The base class for all IPU3 algorithms + * \brief The base class for all IPA algorithms + * \tparam Context The type of shared IPA context + * \tparam Config The type of the IPA configuration data + * \tparam Params The type of the ISP specific parameters + * \tparam Stats The type of the IPA statistics and ISP results * * The Algorithm class defines a standard interface for IPA algorithms. By * abstracting algorithms, it makes possible the implementation of generic code @@ -26,6 +30,7 @@ namespace ipa::ipu3 { */ /** + * \fn Algorithm::configure() * \brief Configure the Algorithm given an IPAConfigInfo * \param[in] context The shared IPA context * \param[in] configInfo The IPA configuration data, received from the pipeline @@ -39,37 +44,30 @@ namespace ipa::ipu3 { * * \return 0 if successful, an error code otherwise */ -int Algorithm::configure([[maybe_unused]] IPAContext &context, - [[maybe_unused]] const IPAConfigInfo &configInfo) -{ - return 0; -} /** + * \fn Algorithm::prepare() * \brief Fill the \a params buffer with ISP processing parameters for a frame * \param[in] context The shared IPA context - * \param[out] params The IPU3 specific parameters. + * \param[out] params The ISP specific parameters. * * This function is called for every frame when the camera is running before it - * is processed by the ImgU to prepare the ImgU processing parameters for that + * is processed by the ISP to prepare the ISP processing parameters for that * frame. * * Algorithms shall fill in the parameter structure fields appropriately to - * configure the ImgU processing blocks that they are responsible for. This + * configure the ISP processing blocks that they are responsible for. This * includes setting fields and flags that enable those processing blocks. */ -void Algorithm::prepare([[maybe_unused]] IPAContext &context, - [[maybe_unused]] ipu3_uapi_params *params) -{ -} /** + * \fn Algorithm::process() * \brief Process ISP statistics, and run algorithm operations * \param[in] context The shared IPA context - * \param[in] stats The IPU3 statistics and ISP results + * \param[in] stats The IPA statistics and ISP results * * This function is called while camera is running for every frame processed by - * the ImgU, to process statistics generated from that frame by the ImgU. + * the ISP, to process statistics generated from that frame by the ISP. * Algorithms shall use this data to run calculations and update their state * accordingly. * @@ -91,11 +89,7 @@ void Algorithm::prepare([[maybe_unused]] IPAContext &context, * Care shall be taken to ensure the ordering of access to the information * such that the algorithms use up to date state as required. */ -void Algorithm::process([[maybe_unused]] IPAContext &context, - [[maybe_unused]] const ipu3_uapi_stats_3a *stats) -{ -} -} /* namespace ipa::ipu3 */ +} /* namespace ipa */ } /* namespace libcamera */ diff --git a/src/ipa/libipa/algorithm.h b/src/ipa/libipa/algorithm.h new file mode 100644 index 00000000..766aee5d --- /dev/null +++ b/src/ipa/libipa/algorithm.h @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Ideas On Board + * + * algorithm.h - ISP control algorithm interface + */ +#pragma once + +namespace libcamera { + +namespace ipa { + +template +class Algorithm +{ +public: + virtual ~Algorithm() {} + + virtual int configure([[maybe_unused]] Context &context, + [[maybe_unused]] const Config &configInfo) + { + return 0; + } + + virtual void prepare([[maybe_unused]] Context &context, + [[maybe_unused]] Params *params) + { + } + + virtual void process([[maybe_unused]] Context &context, + [[maybe_unused]] const Stats *stats) + { + } +}; + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index 4d073a03..161cc5a1 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -1,6 +1,7 @@ # SPDX-License-Identifier: CC0-1.0 libipa_headers = files([ + 'algorithm.h', 'camera_sensor_helper.h', 'histogram.h' ]) From patchwork Wed Nov 24 13:40:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14756 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id EC865C3250 for ; Wed, 24 Nov 2021 13:40:34 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 87503603EE; Wed, 24 Nov 2021 14:40:34 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="Ux569/Xz"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5ECD660231 for ; Wed, 24 Nov 2021 14:40:24 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:968b:bd0c:97fc:7c17]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 159CF1B61; Wed, 24 Nov 2021 14:40:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637761224; bh=DoH6PDHGpmx1HKaude0FEcYU8GHsomwLvfLJ93Nx13w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ux569/XzRCo5cb6pNBld1159ROw31h/t+GFX4gLcrbslzL8okeWwKKyaQJstXPWVf aphHrUn70kcCdWrzdhMiyaVheKJfa7JxSVd3k9nOKbIdhskHIFevou8msYWmtLX8Ad /2I63PVtrFJ9VL8Abi1KaxD77nbzXMWVU/avcIV8= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Wed, 24 Nov 2021 14:40:15 +0100 Message-Id: <20211124134019.110765-8-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> References: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 07/11] ipa: rkisp1: Use the Algorithm class X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that libipa offers a templated class for Algorithm, use it in RkISP1. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Laurent Pinchart Reviewed-by: Kieran Bingham --- v4: use #pragma once --- src/ipa/rkisp1/algorithms/algorithm.h | 25 +++++++++++++++++++++++++ src/ipa/rkisp1/algorithms/meson.build | 4 ++++ src/ipa/rkisp1/meson.build | 4 ++++ src/ipa/rkisp1/rkisp1.cpp | 7 ++++++- 4 files changed, 39 insertions(+), 1 deletion(-) create mode 100644 src/ipa/rkisp1/algorithms/algorithm.h create mode 100644 src/ipa/rkisp1/algorithms/meson.build diff --git a/src/ipa/rkisp1/algorithms/algorithm.h b/src/ipa/rkisp1/algorithms/algorithm.h new file mode 100644 index 00000000..59333e30 --- /dev/null +++ b/src/ipa/rkisp1/algorithms/algorithm.h @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Ideas On Board + * + * algorithm.h - RkISP1 control algorithm interface + */ +#pragma once + +#include + +#include + +#include + +#include "ipa_context.h" + +namespace libcamera { + +namespace ipa::rkisp1 { + +using Algorithm = libcamera::ipa::Algorithm; + +} /* namespace ipa::rkisp1 */ + +} /* namespace libcamera */ diff --git a/src/ipa/rkisp1/algorithms/meson.build b/src/ipa/rkisp1/algorithms/meson.build new file mode 100644 index 00000000..1c6c59cf --- /dev/null +++ b/src/ipa/rkisp1/algorithms/meson.build @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: CC0-1.0 + +rkisp1_ipa_algorithms = files([ +]) diff --git a/src/ipa/rkisp1/meson.build b/src/ipa/rkisp1/meson.build index 3683c922..8c822fbb 100644 --- a/src/ipa/rkisp1/meson.build +++ b/src/ipa/rkisp1/meson.build @@ -1,5 +1,7 @@ # SPDX-License-Identifier: CC0-1.0 +subdir('algorithms') + ipa_name = 'ipa_rkisp1' rkisp1_ipa_sources = files([ @@ -7,6 +9,8 @@ rkisp1_ipa_sources = files([ 'rkisp1.cpp', ]) +rkisp1_ipa_sources += rkisp1_ipa_algorithms + mod = shared_module(ipa_name, [rkisp1_ipa_sources, libcamera_generated_ipa_headers], name_prefix : '', diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 34c3f9a2..225f1847 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -25,9 +25,11 @@ #include -#include "ipa_context.h" +#include "algorithms/algorithm.h" #include "libipa/camera_sensor_helper.h" +#include "ipa_context.h" + namespace libcamera { LOG_DEFINE_CATEGORY(IPARkISP1) @@ -82,6 +84,9 @@ private: /* Local parameter storage */ struct IPAContext context_; + + /* Maintain the algorithms used by the IPA */ + std::list> algorithms_; }; int IPARkISP1::init(const IPASettings &settings, unsigned int hwRevision) From patchwork Wed Nov 24 13:40:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14759 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 35E16C3251 for ; Wed, 24 Nov 2021 13:40:37 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D6E6C6042A; Wed, 24 Nov 2021 14:40:36 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="RDufbVat"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9DA54603C2 for ; Wed, 24 Nov 2021 14:40:24 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:968b:bd0c:97fc:7c17]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 440A490E; Wed, 24 Nov 2021 14:40:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637761224; bh=RXF5gd3nL/zL5elGMr1rrP0VB1zm1MMGKOQYwt6bjxI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RDufbVatT9nXXVyy9SZcUxEhCvHjq1r0nFhO9ivyVsvvob3J5ru4hN20ur+AkzWDy Cs5VR16VZ4Tqi7eNOezcBONafjND0OgfpkZiZVopqTINhg5mv4sbpb4lvTlAz0qr80 rBO3Euvq43zp8YcEgw4v/8ZANAQJ3FvS5PDftbAk= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Wed, 24 Nov 2021 14:40:16 +0100 Message-Id: <20211124134019.110765-9-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> References: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 08/11] ipa: rkisp1: Report and use sensor controls X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The pipeline handler populates a new sensorControls ControlList, to have the effective exposure and gain values for the current frame. This is done when a statistics buffer is received. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- include/libcamera/ipa/rkisp1.mojom | 2 ++ src/ipa/rkisp1/rkisp1.cpp | 2 +- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 3 ++- 3 files changed, 5 insertions(+), 2 deletions(-) diff --git a/include/libcamera/ipa/rkisp1.mojom b/include/libcamera/ipa/rkisp1.mojom index a6991d4f..c3a6d8e1 100644 --- a/include/libcamera/ipa/rkisp1.mojom +++ b/include/libcamera/ipa/rkisp1.mojom @@ -21,11 +21,13 @@ struct RkISP1Event { uint32 frame; uint32 bufferId; libcamera.ControlList controls; + libcamera.ControlList sensorControls; }; struct RkISP1Action { RkISP1Operations op; libcamera.ControlList controls; + libcamera.ControlList sensorControls; }; interface IPARkISP1Interface { diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 225f1847..2b073258 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -314,7 +314,7 @@ void IPARkISP1::setControls(unsigned int frame) ControlList ctrls(ctrls_); ctrls.set(V4L2_CID_EXPOSURE, static_cast(exposure_)); ctrls.set(V4L2_CID_ANALOGUE_GAIN, static_cast(gain_)); - op.controls = ctrls; + op.sensorControls = ctrls; queueFrameAction.emit(frame, op); } diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index 6cdc4c03..36ef6a02 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -333,7 +333,7 @@ void RkISP1CameraData::queueFrameAction(unsigned int frame, { switch (action.op) { case ipa::rkisp1::ActionV4L2Set: { - const ControlList &controls = action.controls; + const ControlList &controls = action.sensorControls; delayedCtrls_->push(controls); break; } @@ -1124,6 +1124,7 @@ void PipelineHandlerRkISP1::statReady(FrameBuffer *buffer) ev.op = ipa::rkisp1::EventSignalStatBuffer; ev.frame = info->frame; ev.bufferId = info->statBuffer->cookie(); + ev.sensorControls = data->delayedCtrls_->get(buffer->metadata().sequence); data->ipa_->processEvent(ev); } From patchwork Wed Nov 24 13:40:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14757 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 96F8DC324F for ; Wed, 24 Nov 2021 13:40:35 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4993D60376; Wed, 24 Nov 2021 14:40:35 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="gRAa1csR"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D201C603C9 for ; Wed, 24 Nov 2021 14:40:24 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:968b:bd0c:97fc:7c17]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 747EBD78; Wed, 24 Nov 2021 14:40:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637761224; bh=5FSzPQNYzFfSyh/qQNs3zUgsEWwHmrY0ZPfviu87fvY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gRAa1csRvCkffEm+i7SFs4461dsRQKY9OaWyKU1U1cr+2sftlWoTe5ANo7X/TuLd2 izEoJmS/NvSJiq17Blhb+Pa+5mHUpBV1pAOKGwfH4biIgKthfeIXGO2cAzKH2FEn6d GijfMfFMVvbMqULllJG7oM8VX16MVePt0E6WeHA0= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Wed, 24 Nov 2021 14:40:17 +0100 Message-Id: <20211124134019.110765-10-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> References: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 09/11] ipa: rkisp1: agc: Introduce HW revision in IPAContext X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The ISP can use 25 or 81 cells depending on its revision. Remove the cached value in IPARkISP1 and use IPASessionConfiguration to store it and pass it to AGC later. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Laurent Pinchart Reviewed-by: Kieran Bingham --- v4: - Only store HW revision, and use it in next patch --- src/ipa/rkisp1/ipa_context.cpp | 8 ++++++++ src/ipa/rkisp1/ipa_context.h | 5 +++++ src/ipa/rkisp1/rkisp1.cpp | 7 +++++++ 3 files changed, 20 insertions(+) diff --git a/src/ipa/rkisp1/ipa_context.cpp b/src/ipa/rkisp1/ipa_context.cpp index 819b2c73..6b53dfdf 100644 --- a/src/ipa/rkisp1/ipa_context.cpp +++ b/src/ipa/rkisp1/ipa_context.cpp @@ -55,4 +55,12 @@ namespace libcamera::ipa::rkisp1 { * are run. This needs to be turned into real per-frame data storage. */ +/** + * \var IPASessionConfiguration::hw + * \brief RkISP1-specific hardware information + * + * \var IPASessionConfiguration::hw.revision + * \brief Hardware revision of the ISP + */ + } /* namespace libcamera::ipa::rkisp1 */ diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index a02b7667..f9ac6833 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -7,11 +7,16 @@ */ #pragma once +#include + namespace libcamera { namespace ipa::rkisp1 { struct IPASessionConfiguration { + struct { + rkisp1_cif_isp_version revision; + } hw; }; struct IPAFrameContext { diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 2b073258..59676a70 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -74,6 +74,7 @@ private: uint32_t maxGain_; /* revision-specific data */ + rkisp1_cif_isp_version hwRevision_; unsigned int hwAeMeanMax_; unsigned int hwHistBinNMax_; unsigned int hwGammaOutMaxSamples_; @@ -114,6 +115,9 @@ int IPARkISP1::init(const IPASettings &settings, unsigned int hwRevision) LOG(IPARkISP1, Debug) << "Hardware revision is " << hwRevision; + /* Cache the value to set it in configure. */ + hwRevision_ = static_cast(hwRevision); + camHelper_ = CameraSensorHelperFactory::create(settings.sensorModel); if (!camHelper_) { LOG(IPARkISP1, Error) @@ -176,6 +180,9 @@ int IPARkISP1::configure([[maybe_unused]] const IPACameraSensorInfo &info, /* Clean context at configuration */ context_ = {}; + /* Set the hardware revision for the algorithms. */ + context_.configuration.hw.revision = hwRevision_; + return 0; } From patchwork Wed Nov 24 13:40:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14758 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 83D43BF415 for ; Wed, 24 Nov 2021 13:40:36 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 18E41603F9; Wed, 24 Nov 2021 14:40:36 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="NxXir3+n"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0D72C60233 for ; Wed, 24 Nov 2021 14:40:25 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:968b:bd0c:97fc:7c17]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id A405E1B61; Wed, 24 Nov 2021 14:40:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637761224; bh=sbspvGj11McE/Lr9FRGnL3wogJGYBeWxoCcXzrpm/Jg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NxXir3+nlfC8RS/rzhi17CE0nqZth+KJp78oV9X8WvKMufj7rEiAmVx0Fne+Tnl2m LQEneXlsZiLETuaA6DoIfQGF0quVczQsUX4l/5xSiZaeIn2v9uucXB9NVWH51BXSKF D/k507L9TS/qxvQ3WvcXUf96b4NOEzsYNpFTSQRo= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Wed, 24 Nov 2021 14:40:18 +0100 Message-Id: <20211124134019.110765-11-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> References: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 10/11] ipa: rkisp1: Introduce AGC X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that we have IPAContext and Algorithm, we can implement a simple AGC based on the IPU3 one. It is very similar, except that there is no histogram used for an inter quantile mean. The RkISP1 is returning a 5x5 array (for V10) of luminance means. Estimating the relative luminance is thus a simple mean of all the blocks already calculated by the ISP. Signed-off-by: Jean-Michel Hautbois --- v4: - use #pragma once - Return filtered value from the function - Store line duration in IPASessionConfiguration - Use the hw revision to configure the number of AE cells --- src/ipa/rkisp1/algorithms/agc.cpp | 279 ++++++++++++++++++++++++++ src/ipa/rkisp1/algorithms/agc.h | 45 +++++ src/ipa/rkisp1/algorithms/meson.build | 1 + src/ipa/rkisp1/ipa_context.cpp | 45 +++++ src/ipa/rkisp1/ipa_context.h | 19 ++ src/ipa/rkisp1/rkisp1.cpp | 73 ++++--- 6 files changed, 424 insertions(+), 38 deletions(-) create mode 100644 src/ipa/rkisp1/algorithms/agc.cpp create mode 100644 src/ipa/rkisp1/algorithms/agc.h diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp new file mode 100644 index 00000000..84ff8c7a --- /dev/null +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -0,0 +1,279 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Ideas On Board + * + * agc.cpp - AGC/AEC mean-based control algorithm + */ + +#include "agc.h" + +#include +#include +#include + +#include + +#include + +/** + * \file agc.h + */ + +namespace libcamera { + +using namespace std::literals::chrono_literals; + +namespace ipa::rkisp1::algorithms { + +/** + * \class Agc + * \brief A mean-based auto-exposure algorithm + */ + +LOG_DEFINE_CATEGORY(RkISP1Agc) + +/* Limits for analogue gain values */ +static constexpr double kMinAnalogueGain = 1.0; +static constexpr double kMaxAnalogueGain = 8.0; + +/* \todo Honour the FrameDurationLimits control instead of hardcoding a limit */ +static constexpr utils::Duration kMaxShutterSpeed = 60ms; + +/* Number of frames to wait before calculating stats on minimum exposure */ +static constexpr uint32_t kNumStartupFrames = 10; + +/* + * Relative luminance target. + * + * It's a number that's chosen so that, when the camera points at a grey + * target, the resulting image brightness is considered right. + */ +static constexpr double kRelativeLuminanceTarget = 0.4; + +Agc::Agc() + : frameCount_(0), currentExposure_(0s) +{ +} + +/** + * \brief Configure the AGC given a configInfo + * \param[in] context The shared IPA context + * \param[in] configInfo The IPA configuration data + * + * \return 0 + */ +int Agc::configure(IPAContext &context, + [[maybe_unused]] const IPACameraSensorInfo &configInfo) +{ + /* Configure the default exposure and gain. */ + context.frameContext.agc.gain = std::max(context.configuration.agc.minAnalogueGain, kMinAnalogueGain); + context.frameContext.agc.exposure = 10ms / context.configuration.agc.lineDuration; + + /* + * According to the RkISP1 documentation: + * - versions <= V11 have RKISP1_CIF_ISP_AE_MEAN_MAX_V10 entries, + * - versions >= V12 have RKISP1_CIF_ISP_AE_MEAN_MAX_V12 entries. + */ + if (context.configuration.hw.revision < RKISP1_V12) + numCells_ = RKISP1_CIF_ISP_AE_MEAN_MAX_V10; + else + numCells_ = RKISP1_CIF_ISP_AE_MEAN_MAX_V12; + + return 0; +} + +/** + * \brief Apply a filter on the exposure value to limit the speed of changes + */ +utils::Duration Agc::filterExposure() +{ + utils::Duration filteredExposure = currentExposure_; + double speed = 0.2; + + /* Adapt instantly if we are in startup phase */ + if (frameCount_ < kNumStartupFrames) + speed = 1.0; + + if (filteredExposure == 0s) { + filteredExposure = currentExposure_; + } else { + /* + * If we are close to the desired result, go faster to avoid making + * multiple micro-adjustments. + * \todo Make this customisable? + */ + if (filteredExposure < 1.2 * currentExposure_ && + filteredExposure > 0.8 * currentExposure_) + speed = sqrt(speed); + + filteredExposure = speed * currentExposure_ + + filteredExposure * (1.0 - speed); + } + + LOG(RkISP1Agc, Debug) << "After filtering, total_exposure " << filteredExposure; + + return filteredExposure; +} + +/** + * \brief Estimate the new exposure and gain values + * \param[inout] frameContext The shared IPA frame Context + * \param[in] yGain The gain calculated on the current brightness level + */ +void Agc::computeExposure(IPAContext &context, double yGain) +{ + IPASessionConfiguration &configuration = context.configuration; + IPAFrameContext &frameContext = context.frameContext; + + /* Get the effective exposure and gain applied on the sensor. */ + uint32_t exposure = frameContext.sensor.exposure; + double analogueGain = frameContext.sensor.gain; + + utils::Duration minShutterSpeed = configuration.agc.minShutterSpeed; + utils::Duration maxShutterSpeed = std::min(configuration.agc.maxShutterSpeed, + kMaxShutterSpeed); + + double minAnalogueGain = std::max(configuration.agc.minAnalogueGain, + kMinAnalogueGain); + double maxAnalogueGain = std::min(configuration.agc.maxAnalogueGain, + kMaxAnalogueGain); + + /* Consider within 1% of the target as correctly exposed */ + if (std::abs(yGain - 1.0) < 0.01) + LOG(RkISP1Agc, Debug) << "We are well exposed (iqMean = " + << yGain << ")"; + + /* extracted from Rpi::Agc::computeTargetExposure */ + + /* Calculate the shutter time in seconds */ + utils::Duration currentShutter = exposure * configuration.agc.lineDuration; + + /* + * Update the exposure value for the next computation using the values + * of exposure and gain really used by the sensor. + */ + utils::Duration effectiveExposureValue = currentShutter * analogueGain; + + LOG(RkISP1Agc, Debug) << "Actual total exposure " << currentShutter * analogueGain + << " Shutter speed " << currentShutter + << " Gain " << analogueGain + << " Needed ev gain " << yGain; + + /* + * Calculate the current exposure value for the scene as the latest + * exposure value applied multiplied by the new estimated gain. + */ + currentExposure_ = effectiveExposureValue * yGain; + + /* Clamp the exposure value to the min and max authorized */ + utils::Duration maxTotalExposure = maxShutterSpeed * maxAnalogueGain; + currentExposure_ = std::min(currentExposure_, maxTotalExposure); + LOG(RkISP1Agc, Debug) << "Target total exposure " << currentExposure_ + << ", maximum is " << maxTotalExposure; + + /* + * Divide the exposure value as new exposure and gain values + * \todo: estimate if we need to desaturate + */ + utils::Duration exposureValue = filterExposure(); + utils::Duration shutterTime; + + /* + * Push the shutter time up to the maximum first, and only then + * increase the gain. + */ + shutterTime = std::clamp(exposureValue / minAnalogueGain, + minShutterSpeed, maxShutterSpeed); + double stepGain = std::clamp(exposureValue / shutterTime, + minAnalogueGain, maxAnalogueGain); + LOG(RkISP1Agc, Debug) << "Divided up shutter and gain are " + << shutterTime << " and " + << stepGain; + + /* Update the estimated exposure and gain. */ + frameContext.agc.exposure = shutterTime / configuration.agc.lineDuration; + frameContext.agc.gain = stepGain; +} + +/** + * \brief Estimate the relative luminance of the frame with a given gain + * \param[in] ae The RkISP1 statistics and ISP results + * \param[in] gain The gain calculated on the current brightness level + * \return The relative luminance + * + * This function estimates the average relative luminance of the frame that + * would be output by the sensor if an additional \a gain was applied. + * + * The estimation is based on the AE statistics for the current frame. Y + * averages for all cells are first multiplied by the gain, and then saturated + * to approximate the sensor behaviour at high brightness values. The + * approximation is quite rough, as it doesn't take into account non-linearities + * when approaching saturation. + * + * The values are normalized to the [0.0, 1.0] range, where 1.0 corresponds to a + * theoretical perfect reflector of 100% reference white. + * + * More detailed information can be found in: + * https://en.wikipedia.org/wiki/Relative_luminance + */ +double Agc::estimateLuminance(const rkisp1_cif_isp_ae_stat *ae, + double gain) +{ + double ySum = 0.0; + unsigned int num = 0; + + /* Sum the averages, saturated to 255. */ + for (unsigned int aeCell = 0; aeCell < numCells_; aeCell++) { + ySum += std::min(ae->exp_mean[aeCell] * gain, 255.0); + num++; + } + + /* \todo Weight with the AWB gains */ + + return ySum / num / 255; +} + +/** + * \brief Process RkISP1 statistics, and run AGC operations + * \param[in] context The shared IPA context + * \param[in] stats The RKIsp1 statistics and ISP results + * + * Identify the current image brightness, and use that to estimate the optimal + * new exposure and gain for the scene. + */ +void Agc::process(IPAContext &context, const rkisp1_stat_buffer *stats) +{ + const rkisp1_cif_isp_stat *params = &stats->params; + ASSERT(stats->meas_type & RKISP1_CIF_ISP_STAT_AUTOEXP); + + const rkisp1_cif_isp_ae_stat *ae = ¶ms->ae; + + /* + * Estimate the gain needed to achieve a relative luminance target. To + * account for non-linearity caused by saturation, the value needs to be + * estimated in an iterative process, as multiplying by a gain will not + * increase the relative luminance by the same factor if some image + * regions are saturated. + */ + double yGain = 1.0; + double yTarget = kRelativeLuminanceTarget; + + for (unsigned int i = 0; i < 8; i++) { + double yValue = estimateLuminance(ae, yGain); + double extra_gain = std::min(10.0, yTarget / (yValue + .001)); + + yGain *= extra_gain; + LOG(RkISP1Agc, Debug) << "Y value: " << yValue + << ", Y target: " << yTarget + << ", gives gain " << yGain; + if (extra_gain < 1.01) + break; + } + + computeExposure(context, yGain); + frameCount_++; +} + +} /* namespace ipa::rkisp1::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h new file mode 100644 index 00000000..a59ae9c8 --- /dev/null +++ b/src/ipa/rkisp1/algorithms/agc.h @@ -0,0 +1,45 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Ideas On Board + * + * agc.h - RkISP1 AGC/AEC mean-based control algorithm + */ +#pragma once + +#include + +#include + +#include + +#include "algorithm.h" + +namespace libcamera { + +struct IPACameraSensorInfo; + +namespace ipa::rkisp1::algorithms { + +class Agc : public Algorithm +{ +public: + Agc(); + ~Agc() = default; + + int configure(IPAContext &context, const IPACameraSensorInfo &configInfo) override; + void process(IPAContext &context, const rkisp1_stat_buffer *stats) override; + +private: + void computeExposure(IPAContext &Context, double yGain); + utils::Duration filterExposure(); + double estimateLuminance(const rkisp1_cif_isp_ae_stat *ae, double gain); + + uint64_t frameCount_; + + uint32_t numCells_; + + utils::Duration currentExposure_; +}; + +} /* namespace ipa::rkisp1::algorithms */ +} /* namespace libcamera */ diff --git a/src/ipa/rkisp1/algorithms/meson.build b/src/ipa/rkisp1/algorithms/meson.build index 1c6c59cf..a19c1a4f 100644 --- a/src/ipa/rkisp1/algorithms/meson.build +++ b/src/ipa/rkisp1/algorithms/meson.build @@ -1,4 +1,5 @@ # SPDX-License-Identifier: CC0-1.0 rkisp1_ipa_algorithms = files([ + 'agc.cpp', ]) diff --git a/src/ipa/rkisp1/ipa_context.cpp b/src/ipa/rkisp1/ipa_context.cpp index 6b53dfdf..dff00362 100644 --- a/src/ipa/rkisp1/ipa_context.cpp +++ b/src/ipa/rkisp1/ipa_context.cpp @@ -56,6 +56,24 @@ namespace libcamera::ipa::rkisp1 { */ /** + * \var IPASessionConfiguration::agc + * \brief AGC parameters configuration of the IPA + * + * \var IPASessionConfiguration::agc.minShutterSpeed + * \brief Minimum shutter speed supported with the configured sensor + * + * \var IPASessionConfiguration::agc.maxShutterSpeed + * \brief Maximum shutter speed supported with the configured sensor + * + * \var IPASessionConfiguration::agc.minAnalogueGain + * \brief Minimum analogue gain supported with the configured sensor + * + * \var IPASessionConfiguration::agc.maxAnalogueGain + * \brief Maximum analogue gain supported with the configured sensor + * + * \var IPASessionConfiguration::agc.lineDuration + * \brief Line duration in microseconds + * * \var IPASessionConfiguration::hw * \brief RkISP1-specific hardware information * @@ -63,4 +81,31 @@ namespace libcamera::ipa::rkisp1 { * \brief Hardware revision of the ISP */ +/** + * \var IPAFrameContext::agc + * \brief Context for the Automatic Gain Control algorithm + * + * The exposure and gain determined are expected to be applied to the sensor + * at the earliest opportunity. + * + * \var IPAFrameContext::agc.exposure + * \brief Exposure time expressed as a number of lines + * + * \var IPAFrameContext::agc.gain + * \brief Analogue gain multiplier + * + * The gain should be adapted to the sensor specific gain code before applying. + */ + +/** + * \var IPAFrameContext::sensor + * \brief Effective sensor values + * + * \var IPAFrameContext::sensor.exposure + * \brief Exposure time expressed as a number of lines + * + * \var IPAFrameContext::sensor.gain + * \brief Analogue gain multiplier + */ + } /* namespace libcamera::ipa::rkisp1 */ diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index f9ac6833..653d7e65 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -9,17 +9,36 @@ #include +#include + namespace libcamera { namespace ipa::rkisp1 { struct IPASessionConfiguration { + struct { + utils::Duration minShutterSpeed; + utils::Duration maxShutterSpeed; + double minAnalogueGain; + double maxAnalogueGain; + utils::Duration lineDuration; + } agc; + struct { rkisp1_cif_isp_version revision; } hw; }; struct IPAFrameContext { + struct { + uint32_t exposure; + double gain; + } agc; + + struct { + uint32_t exposure; + double gain; + } sensor; }; struct IPAContext { diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 59676a70..5c0a0330 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -25,6 +25,7 @@ #include +#include "algorithms/agc.h" #include "algorithms/algorithm.h" #include "libipa/camera_sensor_helper.h" @@ -34,6 +35,8 @@ namespace libcamera { LOG_DEFINE_CATEGORY(IPARkISP1) +using namespace std::literals::chrono_literals; + namespace ipa::rkisp1 { class IPARkISP1 : public IPARkISP1Interface @@ -75,7 +78,6 @@ private: /* revision-specific data */ rkisp1_cif_isp_version hwRevision_; - unsigned int hwAeMeanMax_; unsigned int hwHistBinNMax_; unsigned int hwGammaOutMaxSamples_; unsigned int hwHistogramWeightGridsSize_; @@ -95,13 +97,11 @@ int IPARkISP1::init(const IPASettings &settings, unsigned int hwRevision) /* \todo Add support for other revisions */ switch (hwRevision) { case RKISP1_V10: - hwAeMeanMax_ = RKISP1_CIF_ISP_AE_MEAN_MAX_V10; hwHistBinNMax_ = RKISP1_CIF_ISP_HIST_BIN_N_MAX_V10; hwGammaOutMaxSamples_ = RKISP1_CIF_ISP_GAMMA_OUT_MAX_SAMPLES_V10; hwHistogramWeightGridsSize_ = RKISP1_CIF_ISP_HISTOGRAM_WEIGHT_GRIDS_SIZE_V10; break; case RKISP1_V12: - hwAeMeanMax_ = RKISP1_CIF_ISP_AE_MEAN_MAX_V12; hwHistBinNMax_ = RKISP1_CIF_ISP_HIST_BIN_N_MAX_V12; hwGammaOutMaxSamples_ = RKISP1_CIF_ISP_GAMMA_OUT_MAX_SAMPLES_V12; hwHistogramWeightGridsSize_ = RKISP1_CIF_ISP_HISTOGRAM_WEIGHT_GRIDS_SIZE_V12; @@ -126,6 +126,9 @@ int IPARkISP1::init(const IPASettings &settings, unsigned int hwRevision) return -ENODEV; } + /* Construct our Algorithms */ + algorithms_.push_back(std::make_unique()); + return 0; } @@ -183,6 +186,26 @@ int IPARkISP1::configure([[maybe_unused]] const IPACameraSensorInfo &info, /* Set the hardware revision for the algorithms. */ context_.configuration.hw.revision = hwRevision_; + context_.configuration.agc.lineDuration = info.lineLength * 1.0s / info.pixelRate; + + /* + * When the AGC computes the new exposure values for a frame, it needs + * to know the limits for shutter speed and analogue gain. + * As it depends on the sensor, update it with the controls. + * + * \todo take VBLANK into account for maximum shutter speed + */ + context_.configuration.agc.minShutterSpeed = minExposure_ * context_.configuration.agc.lineDuration; + context_.configuration.agc.maxShutterSpeed = maxExposure_ * context_.configuration.agc.lineDuration; + context_.configuration.agc.minAnalogueGain = camHelper_->gain(minGain_); + context_.configuration.agc.maxAnalogueGain = camHelper_->gain(maxGain_); + + for (auto const &algo : algorithms_) { + int ret = algo->configure(context_, info); + if (ret) + return ret; + } + return 0; } @@ -227,6 +250,9 @@ void IPARkISP1::processEvent(const RkISP1Event &event) reinterpret_cast( mappedBuffers_.at(bufferId).planes()[0].data()); + context_.frameContext.sensor.exposure = event.sensorControls.get(V4L2_CID_EXPOSURE).get(); + context_.frameContext.sensor.gain = camHelper_->gain(event.sensorControls.get(V4L2_CID_ANALOGUE_GAIN).get()); + updateStatistics(frame, stats); break; } @@ -271,44 +297,12 @@ void IPARkISP1::queueRequest(unsigned int frame, rkisp1_params_cfg *params, void IPARkISP1::updateStatistics(unsigned int frame, const rkisp1_stat_buffer *stats) { - const rkisp1_cif_isp_stat *params = &stats->params; unsigned int aeState = 0; - if (stats->meas_type & RKISP1_CIF_ISP_STAT_AUTOEXP) { - const rkisp1_cif_isp_ae_stat *ae = ¶ms->ae; - - const unsigned int target = 60; - - unsigned int value = 0; - unsigned int num = 0; - for (unsigned int i = 0; i < hwAeMeanMax_; i++) { - if (ae->exp_mean[i] <= 15) - continue; - - value += ae->exp_mean[i]; - num++; - } - value /= num; + for (auto const &algo : algorithms_) + algo->process(context_, stats); - double factor = (double)target / value; - - if (frame % 3 == 0) { - double exposure; - - exposure = factor * exposure_ * gain_ / minGain_; - exposure_ = std::clamp((uint64_t)exposure, - minExposure_, - maxExposure_); - - exposure = exposure / exposure_ * minGain_; - gain_ = std::clamp((uint64_t)exposure, - minGain_, maxGain_); - - setControls(frame + 1); - } - - aeState = fabs(factor - 1.0f) < 0.05f ? 2 : 1; - } + setControls(frame); metadataReady(frame, aeState); } @@ -318,6 +312,9 @@ void IPARkISP1::setControls(unsigned int frame) RkISP1Action op; op.op = ActionV4L2Set; + exposure_ = context_.frameContext.agc.exposure; + gain_ = camHelper_->gainCode(context_.frameContext.agc.gain); + ControlList ctrls(ctrls_); ctrls.set(V4L2_CID_EXPOSURE, static_cast(exposure_)); ctrls.set(V4L2_CID_ANALOGUE_GAIN, static_cast(gain_)); From patchwork Wed Nov 24 13:40:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14760 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 187BEC3250 for ; Wed, 24 Nov 2021 13:40:38 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9737060394; Wed, 24 Nov 2021 14:40:37 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="LTlMvoLL"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 499F4603BE for ; Wed, 24 Nov 2021 14:40:25 +0100 (CET) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:968b:bd0c:97fc:7c17]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id D0E2190E; Wed, 24 Nov 2021 14:40:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637761224; bh=5Crdsyz3SjtVysgJUqaTefvgYfb5Gx+AOkK3zgUmOn4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LTlMvoLLQ4eIWOeGYrHLq+wOeXSmPnPP+/iEBh3eA+LuzAmHqlBbP5vBw8u8VxN7J 6ojYLLLX+II6ic0Zjp1ns0BdRYtktvw73Ig4LITB5aNXVSC4BXH0g9fqnsFi0o/8WN xJOJnNYiwlBvK94K44ydJuY5w3xVCYImsWmsXQoQ= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Wed, 24 Nov 2021 14:40:19 +0100 Message-Id: <20211124134019.110765-12-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> References: <20211124134019.110765-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v4 11/11] ipa: rkisp1: agc: Introduce prepare call X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When a new parameter buffer needs to be queued, we need to specify which algorithm is activated or not in the ISP. Add a simple prepare function in AGC for that, which may later evolve to take the exposure locking into account. For that function to be called, we also need to add the loop on the algorithms in IPARkISP1::queueRequest. We no longer disable the AE algorithm based on the controls::AeEnable, which will be handled in a different manner later. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Laurent Pinchart Reviewed-by: Kieran Bingham --- src/ipa/rkisp1/algorithms/agc.cpp | 7 +++++++ src/ipa/rkisp1/algorithms/agc.h | 1 + src/ipa/rkisp1/rkisp1.cpp | 12 +++--------- 3 files changed, 11 insertions(+), 9 deletions(-) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 84ff8c7a..4827b9ac 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -274,6 +274,13 @@ void Agc::process(IPAContext &context, const rkisp1_stat_buffer *stats) frameCount_++; } +void Agc::prepare([[maybe_unused]] IPAContext &context, + rkisp1_params_cfg *params) +{ + params->module_ens |= RKISP1_CIF_ISP_MODULE_AEC; + params->module_en_update |= RKISP1_CIF_ISP_MODULE_AEC; +} + } /* namespace ipa::rkisp1::algorithms */ } /* namespace libcamera */ diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h index a59ae9c8..279d21b6 100644 --- a/src/ipa/rkisp1/algorithms/agc.h +++ b/src/ipa/rkisp1/algorithms/agc.h @@ -27,6 +27,7 @@ public: ~Agc() = default; int configure(IPAContext &context, const IPACameraSensorInfo &configInfo) override; + void prepare(IPAContext &context, rkisp1_params_cfg *params) override; void process(IPAContext &context, const rkisp1_stat_buffer *stats) override; private: diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 5c0a0330..539f10af 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -274,19 +274,13 @@ void IPARkISP1::processEvent(const RkISP1Event &event) } void IPARkISP1::queueRequest(unsigned int frame, rkisp1_params_cfg *params, - const ControlList &controls) + [[maybe_unused]] const ControlList &controls) { /* Prepare parameters buffer. */ memset(params, 0, sizeof(*params)); - /* Auto Exposure on/off. */ - if (controls.contains(controls::AeEnable)) { - autoExposure_ = controls.get(controls::AeEnable); - if (autoExposure_) - params->module_ens = RKISP1_CIF_ISP_MODULE_AEC; - - params->module_en_update = RKISP1_CIF_ISP_MODULE_AEC; - } + for (auto const &algo : algorithms_) + algo->prepare(context_, params); RkISP1Action op; op.op = ActionParamFilled;