From patchwork Mon Aug 23 12:49:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13436 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 44E90BD87D for ; Mon, 23 Aug 2021 12:49:46 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A268668915; Mon, 23 Aug 2021 14:49:44 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="PXzPaZGq"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E364068892 for ; Mon, 23 Aug 2021 14:49:42 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:b920:776:a08c:1d1f]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 84AC24A3; Mon, 23 Aug 2021 14:49:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1629722982; bh=Oy2vfvLsZuh7Gc2BWBlypvUyxoIvvZXJDpS+VojAvbw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PXzPaZGqQFOtazM2AczClZ1OcdvgoiKNuLBaH8zeY12nICABoKYZPIIZT4khf1eSm nx+fPrl/DxDTacw2jlBFS6JYRqOw76bLjMFYYZcQwZf2g48Ck3XPG7Cm2vsHK9Oxyi MeSVeIzVDub9wavPgKixquQicoPET46F5hX0jlsc= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 23 Aug 2021 14:49:31 +0200 Message-Id: <20210823124937.253539-2-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210823124937.253539-1-jeanmichel.hautbois@ideasonboard.com> References: <20210823124937.253539-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 1/7] ipa: ipu3: Move the AWB stats structures X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The structure Ipu3AwbCell describes the AWB stats layout on the kernel side. We will need it to be used by the AGC algorithm to be introduced later, so let's make it visible from ipa::ipu3::algorithms and not only for the AWB class. This structure should probably go into the intel-ipu3.h file, whichs means a kernel patch, let's keep it in mind for the moment. The other structures RGB, IspStatsRegion and AwbStatus will also be used elsewhere so move them at the same time. Signed-off-by: Jean-Michel Hautbois --- src/ipa/ipu3/algorithms/awb.h | 77 ++++++++++++++++++----------------- 1 file changed, 39 insertions(+), 38 deletions(-) diff --git a/src/ipa/ipu3/algorithms/awb.h b/src/ipa/ipu3/algorithms/awb.h index a16dd68d..332652d0 100644 --- a/src/ipa/ipu3/algorithms/awb.h +++ b/src/ipa/ipu3/algorithms/awb.h @@ -23,6 +23,45 @@ namespace ipa::ipu3::algorithms { static constexpr uint32_t kAwbStatsSizeX = 16; static constexpr uint32_t kAwbStatsSizeY = 12; +/* \todo Move the cell layout into intel-ipu3.h kernel header */ +struct Ipu3AwbCell { + unsigned char greenRedAvg; + unsigned char redAvg; + unsigned char blueAvg; + unsigned char greenBlueAvg; + unsigned char satRatio; + unsigned char padding[3]; +}; + +/* \todo Make these structs available to all the ISPs ? */ +struct RGB { + RGB(double _R = 0, double _G = 0, double _B = 0) + : R(_R), G(_G), B(_B) + { + } + double R, G, B; + RGB &operator+=(RGB const &other) + { + R += other.R, G += other.G, B += other.B; + return *this; + } +}; + +struct IspStatsRegion { + unsigned int counted; + unsigned int uncounted; + unsigned long long rSum; + unsigned long long gSum; + unsigned long long bSum; +}; + +struct AwbStatus { + double temperatureK; + double redGain; + double greenGain; + double blueGain; +}; + class Awb : public Algorithm { public: @@ -32,44 +71,6 @@ public: void prepare(IPAContext &context, ipu3_uapi_params *params) override; void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override; - struct Ipu3AwbCell { - unsigned char greenRedAvg; - unsigned char redAvg; - unsigned char blueAvg; - unsigned char greenBlueAvg; - unsigned char satRatio; - unsigned char padding[3]; - } __attribute__((packed)); - - /* \todo Make these three structs available to all the ISPs ? */ - struct RGB { - RGB(double _R = 0, double _G = 0, double _B = 0) - : R(_R), G(_G), B(_B) - { - } - double R, G, B; - RGB &operator+=(RGB const &other) - { - R += other.R, G += other.G, B += other.B; - return *this; - } - }; - - struct IspStatsRegion { - unsigned int counted; - unsigned int uncounted; - unsigned long long rSum; - unsigned long long gSum; - unsigned long long bSum; - }; - - struct AwbStatus { - double temperatureK; - double redGain; - double greenGain; - double blueGain; - }; - private: void calculateWBGains(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid); From patchwork Mon Aug 23 12:49:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13437 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 43D64BD87D for ; Mon, 23 Aug 2021 12:49:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3FB646891B; Mon, 23 Aug 2021 14:49:45 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="SFcXRnCn"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 19E27688A2 for ; Mon, 23 Aug 2021 14:49:43 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:b920:776:a08c:1d1f]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id B3015DEF; Mon, 23 Aug 2021 14:49:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1629722982; bh=VWhok++3B0jlwziB2rCBoWl/dG/OOAP3G17I8TEEInM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SFcXRnCnwE9T8inqwD6zuEoSYnY21zkJA2zKER4k3TR7gVTmkF896sbkPo1z8VD1N l0GHzPqqGy/lWzp7pfVaeJZm7YYMyuECLB9mkxgGLij809xsXRbdlcB5k4BAYPJqu7 bblyXaJl/KUZ10p0Kh2WhoCGak5QfCwUAJ6gZyR4= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 23 Aug 2021 14:49:32 +0200 Message-Id: <20210823124937.253539-3-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210823124937.253539-1-jeanmichel.hautbois@ideasonboard.com> References: <20210823124937.253539-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 2/7] ipa: ipu3: awb: Correct the relevant zones proportion X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The algorithm uses the statistics of a cell only if there is not too much saturated pixels in it. The grey world algorithm works fine when there are a limited number of outliers. Consider a valid zone to be at least 80% of unsaturated cells in it. This value could very well be configurable, and make the algorithm more or less tolerant. Signed-off-by: Jean-Michel Hautbois --- src/ipa/ipu3/algorithms/awb.cpp | 15 ++++++++++++--- src/ipa/ipu3/algorithms/awb.h | 1 + 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index e05647c9..9497a21b 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -17,8 +17,6 @@ namespace ipa::ipu3::algorithms { LOG_DEFINE_CATEGORY(IPU3Awb) -static constexpr uint32_t kMinZonesCounted = 16; -static constexpr uint32_t kMinGreenLevelInZone = 32; /** * \struct IspStatsRegion @@ -114,6 +112,9 @@ static const struct ipu3_uapi_ccm_mat_config imguCssCcmDefault = { 0, 0, 8191, 0 }; +/* Minimum level of green in a given zone */ +static constexpr uint32_t kMinGreenLevelInZone = 16; + Awb::Awb() : Algorithm() { @@ -123,6 +124,7 @@ Awb::Awb() asyncResults_.temperatureK = 4500; zones_.reserve(kAwbStatsSizeX * kAwbStatsSizeY); + minZonesCounted_ = 0; } Awb::~Awb() = default; @@ -163,7 +165,7 @@ void Awb::generateZones(std::vector &zones) for (unsigned int i = 0; i < kAwbStatsSizeX * kAwbStatsSizeY; i++) { RGB zone; double counted = awbStats_[i].counted; - if (counted >= kMinZonesCounted) { + if (counted >= minZonesCounted_) { zone.G = awbStats_[i].gSum / counted; if (zone.G >= kMinGreenLevelInZone) { zone.R = awbStats_[i].rSum / counted; @@ -181,6 +183,13 @@ void Awb::generateAwbStats(const ipu3_uapi_stats_3a *stats, uint32_t regionWidth = round(grid.width / static_cast(kAwbStatsSizeX)); uint32_t regionHeight = round(grid.height / static_cast(kAwbStatsSizeY)); + /* + * It is the minimum proportion of pixels counted within AWB region + * for it to be relevant. + * \todo This proportion could be configured. + */ + minZonesCounted_ = (regionWidth * regionHeight) * 80 / 100; + /* * Generate a (kAwbStatsSizeX x kAwbStatsSizeY) array from the IPU3 grid which is * (grid.width x grid.height). diff --git a/src/ipa/ipu3/algorithms/awb.h b/src/ipa/ipu3/algorithms/awb.h index 332652d0..95238b6a 100644 --- a/src/ipa/ipu3/algorithms/awb.h +++ b/src/ipa/ipu3/algorithms/awb.h @@ -84,6 +84,7 @@ private: std::vector zones_; IspStatsRegion awbStats_[kAwbStatsSizeX * kAwbStatsSizeY]; AwbStatus asyncResults_; + uint32_t minZonesCounted_; }; } /* namespace ipa::ipu3::algorithms */ From patchwork Mon Aug 23 12:49:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13438 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D204EC3241 for ; Mon, 23 Aug 2021 12:49:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B06FF688E5; Mon, 23 Aug 2021 14:49:45 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="JiTSZXiO"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4003E688A5 for ; Mon, 23 Aug 2021 14:49:43 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:b920:776:a08c:1d1f]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id E76032A5; Mon, 23 Aug 2021 14:49:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1629722983; bh=gqjBD44EQGMxs4ueVeR9c84sLsZma5VM+oPubRRlHq4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JiTSZXiOSAiPBAYez+or4yntsdx0NYvnHQSgKJwb9NmjBVCEAXFr4337Nw/j3vu3o A3CbRURLSdHNmC4e87TD735ADviE9p6b9J/fcxCi9beI81QIq4LNUR1MA1W0gxeM4S vs81dUqZ8wchN1pJWuwjSFYUs4rKgiBq/PsNj8iM= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 23 Aug 2021 14:49:33 +0200 Message-Id: <20210823124937.253539-4-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210823124937.253539-1-jeanmichel.hautbois@ideasonboard.com> References: <20210823124937.253539-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 3/7] ipa: ipu3: awb: Correct the gain multipliers X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The gains have a precision u3.13, range [0, 8) which means that a gain multiplier value of 1.0 is represented as a multiplication by 8192 in the ImgU. Correct the gains as this was misunderstood in the first place. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Laurent Pinchart --- src/ipa/ipu3/algorithms/awb.cpp | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index 9497a21b..60a5fc52 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -329,13 +329,12 @@ void Awb::prepare(IPAContext &context, ipu3_uapi_params *params) * params->acc_param.bnr.opt_center.y_reset; /* * Green gains should not be touched and considered 1. - * Default is 16, so do not change it at all. - * 4096 is the value for a gain of 1.0 + * 8192 is the multiplier for a gain of 1.0 */ - params->acc_param.bnr.wb_gains.gr = 16 * context.frameContext.awb.gains.green; - params->acc_param.bnr.wb_gains.r = 4096 * context.frameContext.awb.gains.red; - params->acc_param.bnr.wb_gains.b = 4096 * context.frameContext.awb.gains.blue; - params->acc_param.bnr.wb_gains.gb = 16 * context.frameContext.awb.gains.green; + params->acc_param.bnr.wb_gains.gr = 8192 * context.frameContext.awb.gains.green; + params->acc_param.bnr.wb_gains.r = 8192 * context.frameContext.awb.gains.red; + params->acc_param.bnr.wb_gains.b = 8192 * context.frameContext.awb.gains.blue; + params->acc_param.bnr.wb_gains.gb = 8192 * context.frameContext.awb.gains.green; LOG(IPU3Awb, Debug) << "Color temperature estimated: " << asyncResults_.temperatureK; From patchwork Mon Aug 23 12:49:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13439 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 3F090C3242 for ; Mon, 23 Aug 2021 12:49:48 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4693B68909; Mon, 23 Aug 2021 14:49:46 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="pQdJjj3V"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 650456025B for ; Mon, 23 Aug 2021 14:49:43 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:b920:776:a08c:1d1f]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 223844A3; Mon, 23 Aug 2021 14:49:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1629722983; bh=pV7DKxt1Yd6rLgTQTnfWNm9pzjtDNmtZP0Bw0MOYhE8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pQdJjj3Vj6hp6Qwe6Ww2qfsyUHvUDhsFh+ffz5FUvZ7CZyQ9oPVHiV+tbWMH/dgMJ 1aNgZRGzXPXnuL8BBbqdcEsGFSMj/s+aEyzgeI8eQ19RpxtlkCFU1BadNIJ/SjXoIV 137jRvjCTfSnK6cNJ6rFVdJfLLug4eCTByZ01fRE= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 23 Aug 2021 14:49:34 +0200 Message-Id: <20210823124937.253539-5-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210823124937.253539-1-jeanmichel.hautbois@ideasonboard.com> References: <20210823124937.253539-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 4/7] ipa: ipu3: awb: Add some documentation on the algorithm used X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The AWB algorithm is based on the Grey world algorithm and uses the statistics generated by the ImgU for that. Explain how it uses those, and reference the original algorithm at the same time. Signed-off-by: Jean-Michel Hautbois --- src/ipa/ipu3/algorithms/awb.cpp | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index 60a5fc52..8e4230b5 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -17,6 +17,31 @@ namespace ipa::ipu3::algorithms { LOG_DEFINE_CATEGORY(IPU3Awb) +/** + * The Grey World algorithm assumes that the scene, in average, is neutral grey. + * Reference: Lam, Edmund & Fung, George. (2008). Automatic White Balancing in + * Digital Photography. 10.1201/9781420054538.ch10. + * + * The IPU3 generates statistics from the Bayer Down Scaler output into a grid + * defined in the ipu3_uapi_awb_config_s structure. + * + * For example, when the BDS outputs a frame of 2592x1944, the grid may be + * configured to 81x30 cells each with a size of 32x64 pixels. + * We then have an average of 2048 R, G and B pixels per cell. + * + * The AWB algorithm uses a fixed grid size of kAwbStatsSizeX x kAwbStatsSizeY. + * Each of this new grid cell will be called a zone. + * + * Before calculating the gains, we will convert the statistics from the BDS + * grid to an internal grid configuration in generateAwbStats. + * As part of converting the statistics to an internal grid, the saturation + * flag from the originating grid cell is used to decide if the zone contains + * saturated pixels or not, making the zone relevant or not. + * A saturated zone will be excluded from the calculation. + * + * The Grey World algorithm will then estimate the red and blue gains to apply, and + * store the results in the metadata. + */ /** * \struct IspStatsRegion From patchwork Mon Aug 23 12:49:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13440 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 29425BD87D for ; Mon, 23 Aug 2021 12:49:51 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E6889688AE; Mon, 23 Aug 2021 14:49:50 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="u+7v1b0Z"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 99CDE68892 for ; Mon, 23 Aug 2021 14:49:43 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:b920:776:a08c:1d1f]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 509AFDEF; Mon, 23 Aug 2021 14:49:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1629722983; bh=1vBod9TGeRYs2oPIlzmj9W8zwjl+281J1OAaooAQwWQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=u+7v1b0ZxlESCCE5SzPQhwfr0WQefLrKiy7rGd79my7WgGGe5YbAb4sjWXLrAWos6 hRl6z86oclffRlxdxGWTzgJAl5gEpEnwSf+OUykDaq69d5J3tC1NUAUWHiBVUlzmyL 7yRIsIEuqaipyiyqExLo1MA2U1cu2/AaCj0QAakA= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 23 Aug 2021 14:49:35 +0200 Message-Id: <20210823124937.253539-6-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210823124937.253539-1-jeanmichel.hautbois@ideasonboard.com> References: <20210823124937.253539-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 5/7] ipa: ipu3: rename AGC algorithm X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The initial AGC algorithm is a simple one, which calculates the exposure and gain values by trying to have a mean histogram value to 128. Rename it as "AgcMean" to make it easy to distinguish. Now that we are modular, we can have multiple algorithms for one functionnality (here, AGC). Naming those algorithms makes it easier to chose between them. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Laurent Pinchart --- .../ipu3/algorithms/{agc.cpp => agc_mean.cpp} | 31 +++++++++---------- src/ipa/ipu3/algorithms/{agc.h => agc_mean.h} | 8 ++--- src/ipa/ipu3/algorithms/meson.build | 2 +- src/ipa/ipu3/ipu3.cpp | 4 +-- 4 files changed, 22 insertions(+), 23 deletions(-) rename src/ipa/ipu3/algorithms/{agc.cpp => agc_mean.cpp} (87%) rename src/ipa/ipu3/algorithms/{agc.h => agc_mean.h} (90%) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc_mean.cpp similarity index 87% rename from src/ipa/ipu3/algorithms/agc.cpp rename to src/ipa/ipu3/algorithms/agc_mean.cpp index 5ff50f4a..193f6e9a 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc_mean.cpp @@ -2,10 +2,10 @@ /* * Copyright (C) 2021, Ideas On Board * - * ipu3_agc.cpp - AGC/AEC control algorithm + * agc_mean.cpp - AGC/AEC control algorithm */ -#include "agc.h" +#include "agc_mean.h" #include #include @@ -23,7 +23,7 @@ using namespace std::literals::chrono_literals; namespace ipa::ipu3::algorithms { -LOG_DEFINE_CATEGORY(IPU3Agc) +LOG_DEFINE_CATEGORY(IPU3AgcMean) /* Number of frames to wait before calculating stats on minimum exposure */ static constexpr uint32_t kInitialFrameMinAECount = 4; @@ -50,16 +50,15 @@ static constexpr double kEvGainTarget = 0.5; /* A cell is 8 bytes and contains averages for RGB values and saturation ratio */ static constexpr uint8_t kCellSize = 8; -Agc::Agc() +AgcMean::AgcMean() : frameCount_(0), lastFrame_(0), iqMean_(0.0), lineDuration_(0s), maxExposureTime_(0s), prevExposure_(0s), prevExposureNoDg_(0s), currentExposure_(0s), currentExposureNoDg_(0s) { } -int Agc::configure([[maybe_unused]] IPAContext &context, - const IPAConfigInfo &configInfo) -{ +int AgcMean::configure([[maybe_unused]] IPAContext &context, + const IPAConfigInfo &configInfo){ lineDuration_ = configInfo.sensorInfo.lineLength * 1.0s / configInfo.sensorInfo.pixelRate; maxExposureTime_ = kMaxExposure * lineDuration_; @@ -67,7 +66,7 @@ int Agc::configure([[maybe_unused]] IPAContext &context, return 0; } -void Agc::processBrightness(const ipu3_uapi_stats_3a *stats, +void AgcMean::processBrightness(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid) { const struct ipu3_uapi_grid_config statsAeGrid = stats->stats_4a_config.awb_config.grid; @@ -109,7 +108,7 @@ void Agc::processBrightness(const ipu3_uapi_stats_3a *stats, iqMean_ = Histogram(Span(hist)).interQuantileMean(0.98, 1.0); } -void Agc::filterExposure() +void AgcMean::filterExposure() { double speed = 0.2; if (prevExposure_ == 0s) { @@ -140,10 +139,10 @@ void Agc::filterExposure() if (prevExposureNoDg_ < prevExposure_ * fastReduceThreshold) prevExposureNoDg_ = prevExposure_ * fastReduceThreshold; - LOG(IPU3Agc, Debug) << "After filtering, total_exposure " << prevExposure_; + LOG(IPU3AgcMean, Debug) << "After filtering, total_exposure " << prevExposure_; } -void Agc::lockExposureGain(uint32_t &exposure, double &gain) +void AgcMean::lockExposureGain(uint32_t &exposure, double &gain) { /* Algorithm initialization should wait for first valid frames */ /* \todo - have a number of frames given by DelayedControls ? @@ -153,20 +152,20 @@ void Agc::lockExposureGain(uint32_t &exposure, double &gain) /* Are we correctly exposed ? */ if (std::abs(iqMean_ - kEvGainTarget * knumHistogramBins) <= 1) { - LOG(IPU3Agc, Debug) << "!!! Good exposure with iqMean = " << iqMean_; + LOG(IPU3AgcMean, Debug) << "!!! Good exposure with iqMean = " << iqMean_; } else { double newGain = kEvGainTarget * knumHistogramBins / iqMean_; /* extracted from Rpi::Agc::computeTargetExposure */ libcamera::utils::Duration currentShutter = exposure * lineDuration_; currentExposureNoDg_ = currentShutter * gain; - LOG(IPU3Agc, Debug) << "Actual total exposure " << currentExposureNoDg_ + LOG(IPU3AgcMean, Debug) << "Actual total exposure " << currentExposureNoDg_ << " Shutter speed " << currentShutter << " Gain " << gain; currentExposure_ = currentExposureNoDg_ * newGain; libcamera::utils::Duration maxTotalExposure = maxExposureTime_ * kMaxGain; currentExposure_ = std::min(currentExposure_, maxTotalExposure); - LOG(IPU3Agc, Debug) << "Target total exposure " << currentExposure_; + LOG(IPU3AgcMean, Debug) << "Target total exposure " << currentExposure_; /* \todo: estimate if we need to desaturate */ filterExposure(); @@ -181,12 +180,12 @@ void Agc::lockExposureGain(uint32_t &exposure, double &gain) newExposure = currentExposure_ / gain; exposure = std::clamp(static_cast(exposure * currentExposure_ / newExposure), kMinExposure, kMaxExposure); } - LOG(IPU3Agc, Debug) << "Adjust exposure " << exposure * lineDuration_ << " and gain " << gain; + LOG(IPU3AgcMean, Debug) << "Adjust exposure " << exposure * lineDuration_ << " and gain " << gain; } lastFrame_ = frameCount_; } -void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) +void AgcMean::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) { uint32_t &exposure = context.frameContext.agc.exposure; double &gain = context.frameContext.agc.gain; diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc_mean.h similarity index 90% rename from src/ipa/ipu3/algorithms/agc.h rename to src/ipa/ipu3/algorithms/agc_mean.h index e36797d3..97114121 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc_mean.h @@ -2,7 +2,7 @@ /* * Copyright (C) 2021, Ideas On Board * - * agc.h - IPU3 AGC/AEC control algorithm + * agc_mean.h - IPU3 AGC/AEC control algorithm */ #ifndef __LIBCAMERA_IPU3_ALGORITHMS_AGC_H__ #define __LIBCAMERA_IPU3_ALGORITHMS_AGC_H__ @@ -23,11 +23,11 @@ namespace ipa::ipu3::algorithms { using utils::Duration; -class Agc : public Algorithm +class AgcMean : public Algorithm { public: - Agc(); - ~Agc() = default; + AgcMean(); + ~AgcMean() = default; int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override; diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build index deae225b..807b53ea 100644 --- a/src/ipa/ipu3/algorithms/meson.build +++ b/src/ipa/ipu3/algorithms/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: CC0-1.0 ipu3_ipa_algorithms = files([ - 'agc.cpp', + 'agc_mean.cpp', 'algorithm.cpp', 'awb.cpp', 'tone_mapping.cpp', diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 0ed0a6f1..b73c4f7b 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -29,7 +29,7 @@ #include "libcamera/internal/mapped_framebuffer.h" -#include "algorithms/agc.h" +#include "algorithms/agc_mean.h" #include "algorithms/algorithm.h" #include "algorithms/awb.h" #include "algorithms/tone_mapping.h" @@ -266,7 +266,7 @@ int IPAIPU3::init(const IPASettings &settings, *ipaControls = ControlInfoMap(std::move(controls), controls::controls); /* Construct our Algorithms */ - algorithms_.push_back(std::make_unique()); + algorithms_.push_back(std::make_unique()); algorithms_.push_back(std::make_unique()); algorithms_.push_back(std::make_unique()); From patchwork Mon Aug 23 12:49:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13441 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9EA9BC3241 for ; Mon, 23 Aug 2021 12:49:51 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5CB7468909; Mon, 23 Aug 2021 14:49:51 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="El0e0CTF"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C1EC0688A2 for ; Mon, 23 Aug 2021 14:49:43 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:b920:776:a08c:1d1f]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 7EC1B2A5; Mon, 23 Aug 2021 14:49:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1629722983; bh=MDGzhbsKcWFUHzJNmt/iJyqMDBA2mMekCD/YATXQXEw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=El0e0CTFZFb9HhjV7SYNrbLZH5n6lu29seh9oQVnXBuLedLR61qIYPfoebBUoKZma RdIDyLEIdmeunQ3OgtWeSQjQgfRIInIyIOnw8WrbFnNzDA+J8juB0hFrIuS49T3AgA BEkRaaOX8fngakpALCrs923FLw4J53tK5Aj9uRro= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 23 Aug 2021 14:49:36 +0200 Message-Id: <20210823124937.253539-7-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210823124937.253539-1-jeanmichel.hautbois@ideasonboard.com> References: <20210823124937.253539-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 6/7] ipa: ipu3: Introduce a new AGC algorithm X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The algorithm used until then is a simple one, let's introduce a new one, based on the one used by the Raspberry Pi code. We can keep both compiled, and chose to instanciate only one, which demonstrates the modularity and ease to add functionnalities to the IPA. This algorithm uses the IPAFrameContext to get the latest AWB gains applied and use them to estimate the next shutter time and gain values to set. Signed-off-by: Jean-Michel Hautbois --- src/ipa/ipu3/algorithms/agc_metering.cpp | 336 +++++++++++++++++++++++ src/ipa/ipu3/algorithms/agc_metering.h | 81 ++++++ src/ipa/ipu3/algorithms/awb.cpp | 4 +- src/ipa/ipu3/algorithms/meson.build | 1 + src/ipa/ipu3/ipa_context.h | 8 + src/ipa/ipu3/ipu3.cpp | 12 +- 6 files changed, 438 insertions(+), 4 deletions(-) create mode 100644 src/ipa/ipu3/algorithms/agc_metering.cpp create mode 100644 src/ipa/ipu3/algorithms/agc_metering.h diff --git a/src/ipa/ipu3/algorithms/agc_metering.cpp b/src/ipa/ipu3/algorithms/agc_metering.cpp new file mode 100644 index 00000000..3479c269 --- /dev/null +++ b/src/ipa/ipu3/algorithms/agc_metering.cpp @@ -0,0 +1,336 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Based on the implementation from the Raspberry Pi IPA, + * Copyright (C) 2019-2021, Raspberry Pi (Trading) Ltd. + * Copyright (C) 2021, Ideas On Board + * + * agc_metering.cpp - AGC/AEC control algorithm + */ + +#include "agc_metering.h" +#include "awb.h" + +#include +#include +#include +#include + +#include + +#include +#include + +#include "libipa/histogram.h" + +namespace libcamera { + +using namespace std::literals::chrono_literals; + +namespace ipa::ipu3::algorithms { + +LOG_DEFINE_CATEGORY(IPU3AgcMetering) + +/* Histogram constants */ +static constexpr uint32_t knumHistogramBins = 256; + +/* seems to be a 10-bit pipeline */ +static constexpr uint8_t kPipelineBits = 10; + +/* width of the AGC stats grid */ +static constexpr uint32_t kAgcStatsSizeX = 7; +/* height of the AGC stats grid */ +static constexpr uint32_t kAgcStatsSizeY = 5; +/* size of the AGC stats grid */ +static constexpr uint32_t kAgcStatsSize = kAgcStatsSizeX * kAgcStatsSizeY; + +/** + * The AGC algorithm uses region-based metering. + * The image is divided up into regions as: + * + * +--+--------------+--+ + * |11| 9 |12| + * +--+--+--------+--+--+ + * | | | 3 | | | + * | | +--+--+--+ | | + * |7 |5 |1 |0 |2 |6 |8 | + * | | +--+--+--+ | | + * | | | 4 | | | + * +--+--+--------+--+--+ + * |13| 10 |14| + * +--+--------------+--+ + * An average luminance value for the image is calculated according to: + * \f$Y = \frac{\sum_{i=0}^{i=kNumAgcWeightedZones}{kCenteredWeights_{i}Y_{i}}} + * {\sum_{i=0}^{i=kNumAgcWeightedZones}{w_{i}}}\f$ + */ + +/* Weight applied on each region */ +static constexpr double kCenteredWeights[kNumAgcWeightedZones] = { 3, 3, 3, 2, 2, 2, 2, 1, 1, 1, 1, 0, 0, 0, 0 }; +/* Region number repartition in the image */ +static constexpr uint32_t kAgcStatsRegions[kAgcStatsSize] = { + 11, 9, 9, 9, 9, 9, 12, + 7, 5, 3, 3, 3, 6, 8, + 7, 5, 1, 0, 2, 6, 8, + 7, 5, 4, 4, 4, 6, 8, + 13, 10, 10, 10, 10, 10, 14 +}; + +AgcMetering::AgcMetering() + : iqMean_(0.0), prevExposure_(0s), prevExposureNoDg_(0s), + currentExposure_(0s), currentExposureNoDg_(0s), currentShutter_(1.0s), + currentAnalogueGain_(1.0) +{ +} + +int AgcMetering::configure(IPAContext &context, const IPAConfigInfo &configInfo) +{ + context.configuration.agc.lineDuration = configInfo.sensorInfo.lineLength + * (1.0s / configInfo.sensorInfo.pixelRate); + + /* \todo: those values need to be extracted from a configuration file */ + shutterConstraints_.push_back(100us); + shutterConstraints_.push_back(10ms); + shutterConstraints_.push_back(33ms); + gainConstraints_.push_back(1.0); + gainConstraints_.push_back(4.0); + gainConstraints_.push_back(16.0); + + fixedShutter_ = 0s; + fixedAnalogueGain_ = 0.0; + + return 0; +} + +/* Translate the IPU3 statistics into the default statistics region array */ +void AgcMetering::generateStats(const ipu3_uapi_stats_3a *stats, + const ipu3_uapi_grid_config &grid) +{ + uint32_t regionWidth = round(grid.width / static_cast(kAgcStatsSizeX)); + uint32_t regionHeight = round(grid.height / static_cast(kAgcStatsSizeY)); + uint32_t hist[knumHistogramBins] = { 0 }; + + LOG(IPU3AgcMetering, Debug) << "[" << (int)grid.width + << "x" << (int)grid.height << "] regions" + << " scaled to [" << regionWidth + << "x" << regionHeight << "] AGC stats"; + + /* + * Generate a (kAgcStatsSizeX x kAgcStatsSizeY) array from the IPU3 grid + * which is (grid.width x grid.height). + */ + for (unsigned int j = 0; j < kAgcStatsSizeY * regionHeight; j++) { + for (unsigned int i = 0; i < kAgcStatsSizeX * regionWidth; i++) { + uint32_t cellPosition = j * grid.width + i; + uint32_t cellX = (cellPosition / regionWidth) + % kAgcStatsSizeX; + uint32_t cellY = ((cellPosition / grid.width) / regionHeight) + % kAgcStatsSizeY; + + uint32_t agcRegionPosition = kAgcStatsRegions[cellY * kAgcStatsSizeX + cellX]; + weights_[agcRegionPosition] = kCenteredWeights[agcRegionPosition]; + cellPosition *= sizeof(Ipu3AwbCell); + + /* Cast the initial IPU3 structure to simplify the reading */ + Ipu3AwbCell *currentCell = reinterpret_cast(const_cast(&stats->awb_raw_buffer.meta_data[cellPosition])); + if (currentCell->satRatio == 0) { + /* The cell is not saturated, use the current cell */ + agcStats_[agcRegionPosition].counted++; + uint32_t greenValue = currentCell->greenRedAvg + currentCell->greenBlueAvg; + hist[greenValue / 2]++; + agcStats_[agcRegionPosition].gSum += greenValue / 2; + agcStats_[agcRegionPosition].rSum += currentCell->redAvg; + agcStats_[agcRegionPosition].bSum += currentCell->blueAvg; + } + } + } + + /* Estimate the quantile mean of the top 2% of the histogram */ + iqMean_ = Histogram(Span(hist)).interQuantileMean(0.98, 1.0); +} + +void AgcMetering::clearStats() +{ + for (unsigned int i = 0; i < kNumAgcWeightedZones; i++) { + agcStats_[i].bSum = 0; + agcStats_[i].rSum = 0; + agcStats_[i].gSum = 0; + agcStats_[i].counted = 0; + agcStats_[i].uncounted = 0; + } +} + +void AgcMetering::filterExposure() +{ + double speed = 0.08; + if (prevExposure_ == 0s) { + /* DG stands for digital gain.*/ + prevExposure_ = currentExposure_; + prevExposureNoDg_ = currentExposureNoDg_; + } else { + /* + * If we are close to the desired result, go faster to avoid + * making multiple micro-adjustments. + * \ todo: Make this customisable? + */ + if (prevExposure_ < 1.2 * currentExposure_ && + prevExposure_ > 0.8 * currentExposure_) + speed = sqrt(speed); + + prevExposure_ = speed * currentExposure_ + + prevExposure_ * (1.0 - speed); + prevExposureNoDg_ = speed * currentExposureNoDg_ + + prevExposureNoDg_ * (1.0 - speed); + } + /* + * We can't let the no_dg exposure deviate too far below the + * total exposure, as there might not be enough digital gain available + * in the ISP to hide it (which will cause nasty oscillation). + */ + double fastReduceThreshold = 0.3; + if (prevExposureNoDg_ < + prevExposure_ * fastReduceThreshold) + prevExposureNoDg_ = prevExposure_ * fastReduceThreshold; + LOG(IPU3AgcMetering, Debug) << "After filtering, total_exposure " << prevExposure_; +} + +double AgcMetering::computeInitialY(double gain) +{ + /* + * Note how the calculation below means that equal weights_ give you + * "average" metering (i.e. all pixels equally important). + */ + double redSum = 0, greenSum = 0, blueSum = 0, pixelSum = 0; + for (unsigned int i = 0; i < kNumAgcWeightedZones; i++) { + double counted = agcStats_[i].counted; + double rSum = std::min(agcStats_[i].rSum * gain, ((1 << kPipelineBits) - 1) * counted); + double gSum = std::min(agcStats_[i].gSum * gain, ((1 << kPipelineBits) - 1) * counted); + double bSum = std::min(agcStats_[i].bSum * gain, ((1 << kPipelineBits) - 1) * counted); + redSum += rSum * weights_[i]; + greenSum += gSum * weights_[i]; + blueSum += bSum * weights_[i]; + pixelSum += counted * weights_[i]; + } + if (pixelSum == 0.0) { + LOG(IPU3AgcMetering, Warning) << "computeInitialY: pixel_sum is zero"; + return 0; + } + double Y_sum = redSum * awbStatus_.redGain * .299 + + greenSum * awbStatus_.greenGain * .587 + + blueSum * awbStatus_.blueGain * .114; + + return Y_sum / pixelSum / (1 << kPipelineBits); +} + +void AgcMetering::computeTargetExposure(double gain) +{ + currentExposure_ = currentExposureNoDg_ * gain; + /* \todo: have a list of shutter speeds */ + Duration maxShutterSpeed = shutterConstraints_.back(); + Duration maxTotalExposure = maxShutterSpeed * gainConstraints_.back(); + + currentExposure_ = std::min(currentExposure_, maxTotalExposure); + LOG(IPU3AgcMetering, Debug) << "Target total_exposure " << currentExposure_; +} + +void AgcMetering::divideUpExposure() +{ + Duration exposureValue = prevExposure_; + Duration shutterTime; + double analogueGain; + shutterTime = shutterConstraints_[0]; + shutterTime = std::min(shutterTime, shutterConstraints_.back()); + analogueGain = gainConstraints_[0]; + + if (shutterTime * analogueGain < exposureValue) { + for (unsigned int stage = 1; + stage < gainConstraints_.size(); stage++) { + if (fixedShutter_ == 0s) { + Duration stageShutter = + std::min(shutterConstraints_[stage], shutterConstraints_.back()); + if (stageShutter * analogueGain >= + exposureValue) { + shutterTime = + exposureValue / analogueGain; + break; + } + shutterTime = stageShutter; + } + if (fixedAnalogueGain_ == 0.0) { + if (gainConstraints_[stage] * shutterTime >= exposureValue) { + analogueGain = exposureValue / shutterTime; + break; + } + analogueGain = gainConstraints_[stage]; + } + } + } + LOG(IPU3AgcMetering, Debug) << "Divided up shutter and gain are " + << shutterTime << " and " << analogueGain; + + /* \todo: flickering avoidance ? */ + filteredShutter_ = shutterTime; + filteredAnalogueGain_ = analogueGain; +} + +void AgcMetering::computeGain(double ¤tGain) +{ + currentGain = 1.0; + /* \todo: the target Y needs to be grabbed from a configuration */ + double targetY = 0.162; + for (int i = 0; i < 8; i++) { + double initialY = computeInitialY(currentGain); + double extra_gain = std::min(10.0, targetY / (initialY + .001)); + + currentGain *= extra_gain; + LOG(IPU3AgcMetering, Debug) << "Initial Y " << initialY + << " target " << targetY + << " gives gain " << currentGain; + if (extra_gain < 1.01) + break; + } + + /* + * Require the top 2% of pixels to lie at or below 0.8 in the pixel + * range (for a range from 0 to 255, it is 205). This lowers the + * exposure to stop pixels saturating. + */ + double newGain = (0.8 * knumHistogramBins) / iqMean_; + LOG(IPU3AgcMetering, Debug) << "gain: " << currentGain + << " new gain: " << newGain; + + /* Are we at the upper bound ? */ + if (newGain < currentGain) + currentGain = newGain; +} + +void AgcMetering::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) +{ + ASSERT(stats->stats_3a_status.awb_en); + clearStats(); + generateStats(stats, context.configuration.grid.bdsGrid); + + currentShutter_ = context.frameContext.agc.exposure + * context.configuration.agc.lineDuration; + currentAnalogueGain_ = context.frameContext.agc.gain; + + /* Estimate the current exposure value */ + currentExposureNoDg_ = currentShutter_ * currentAnalogueGain_; + + /* Get the current awb gains from IPAFrameContext */ + awbStatus_.redGain = context.frameContext.awb.gains.red; + awbStatus_.greenGain = context.frameContext.awb.gains.green; + awbStatus_.blueGain = context.frameContext.awb.gains.blue; + + double currentGain = 1.0; + computeGain(currentGain); + computeTargetExposure(currentGain); + filterExposure(); + divideUpExposure(); + + context.frameContext.agc.exposure = filteredShutter_ + / context.configuration.agc.lineDuration; + context.frameContext.agc.gain = filteredAnalogueGain_; +} + +} /* namespace ipa::ipu3::algorithms */ + +} /* namespace libcamera */ diff --git a/src/ipa/ipu3/algorithms/agc_metering.h b/src/ipa/ipu3/algorithms/agc_metering.h new file mode 100644 index 00000000..c10846e7 --- /dev/null +++ b/src/ipa/ipu3/algorithms/agc_metering.h @@ -0,0 +1,81 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Based on the implementation from the Raspberry Pi IPA, + * Copyright (C) 2019-2021, Raspberry Pi (Trading) Ltd. + * Copyright (C) 2021, Ideas On Board + * + * agc_metering.h - IPU3 AGC/AEC control algorithm + */ +#ifndef __LIBCAMERA_IPU3_AGC_H__ +#define __LIBCAMERA_IPU3_AGC_H__ + +#include + +#include + +#include + +#include "algorithm.h" +#include "awb.h" + +namespace libcamera { + +struct IPACameraSensorInfo; + +namespace ipa::ipu3::algorithms { + +using utils::Duration; + +/* Number of weighted zones for metering */ +static constexpr uint32_t kNumAgcWeightedZones = 15; + +class AgcMetering : public Algorithm +{ +public: + AgcMetering(); + ~AgcMetering() = default; + + int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; + void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override; + +private: + void processBrightness(const ipu3_uapi_stats_3a *stats); + void filterExposure(); + void lockExposureGain(uint32_t &exposure, double &gain); + void generateStats(const ipu3_uapi_stats_3a *stats, + const ipu3_uapi_grid_config &grid); + void clearStats(); + void generateZones(std::vector &zones); + double computeInitialY(double gain); + void computeTargetExposure(double currentGain); + void divideUpExposure(); + void computeGain(double ¤tGain); + + double weights_[kNumAgcWeightedZones]; + struct IspStatsRegion agcStats_[kNumAgcWeightedZones]; + + double iqMean_; + + Duration prevExposure_; + Duration prevExposureNoDg_; + Duration currentExposure_; + Duration currentExposureNoDg_; + + Duration currentShutter_; + std::vector shutterConstraints_; + Duration fixedShutter_; + Duration filteredShutter_; + + double currentAnalogueGain_; + std::vector gainConstraints_; + double fixedAnalogueGain_; + double filteredAnalogueGain_; + + struct AwbStatus awbStatus_; +}; + +} /* namespace ipa::ipu3::algorithms */ + +} /* namespace libcamera */ + +#endif /* __LIBCAMERA_IPU3_AGC_H__ */ diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index 8e4230b5..294871b1 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -138,7 +138,7 @@ static const struct ipu3_uapi_ccm_mat_config imguCssCcmDefault = { }; /* Minimum level of green in a given zone */ -static constexpr uint32_t kMinGreenLevelInZone = 16; +static constexpr uint32_t kMinGreenLevelInZone = 32; Awb::Awb() : Algorithm() @@ -213,7 +213,7 @@ void Awb::generateAwbStats(const ipu3_uapi_stats_3a *stats, * for it to be relevant. * \todo This proportion could be configured. */ - minZonesCounted_ = (regionWidth * regionHeight) * 80 / 100; + minZonesCounted_ = (regionWidth * regionHeight) * 50 / 100; /* * Generate a (kAwbStatsSizeX x kAwbStatsSizeY) array from the IPU3 grid which is diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build index 807b53ea..f31b2070 100644 --- a/src/ipa/ipu3/algorithms/meson.build +++ b/src/ipa/ipu3/algorithms/meson.build @@ -2,6 +2,7 @@ ipu3_ipa_algorithms = files([ 'agc_mean.cpp', + 'agc_metering.cpp', 'algorithm.cpp', 'awb.cpp', 'tone_mapping.cpp', diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index 9d9444dc..0a987da4 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -10,13 +10,21 @@ #include +#include + #include namespace libcamera { namespace ipa::ipu3 { +using utils::Duration; + struct IPASessionConfiguration { + struct { + Duration lineDuration; + } agc; + struct { ipu3_uapi_grid_config bdsGrid; Size bdsOutputSize; diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index b73c4f7b..1425b344 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -29,7 +29,7 @@ #include "libcamera/internal/mapped_framebuffer.h" -#include "algorithms/agc_mean.h" +#include "algorithms/agc_metering.h" #include "algorithms/algorithm.h" #include "algorithms/awb.h" #include "algorithms/tone_mapping.h" @@ -81,6 +81,14 @@ * are run. This needs to be turned into real per-frame data storage. */ +/** + * \struct IPASessionConfiguration::agc + * \brief AGC configuration of the IPA + * + * \var IPASessionConfiguration::agc::lineDuration + * \brief Duration of one line dependant on the sensor configuration + */ + /** * \struct IPASessionConfiguration::grid * \brief Grid configuration of the IPA @@ -266,7 +274,7 @@ int IPAIPU3::init(const IPASettings &settings, *ipaControls = ControlInfoMap(std::move(controls), controls::controls); /* Construct our Algorithms */ - algorithms_.push_back(std::make_unique()); + algorithms_.push_back(std::make_unique()); algorithms_.push_back(std::make_unique()); algorithms_.push_back(std::make_unique()); From patchwork Mon Aug 23 12:49:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13442 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 312DFC3242 for ; Mon, 23 Aug 2021 12:49:52 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E5F6E68915; Mon, 23 Aug 2021 14:49:51 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="d71oWskJ"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0369F6025B for ; Mon, 23 Aug 2021 14:49:44 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:b920:776:a08c:1d1f]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id AD07B4A3; Mon, 23 Aug 2021 14:49:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1629722983; bh=bUvMTU7i9QQFkbbpFMujK4Q/ei2+xVpjZ1j7qSsqoh4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=d71oWskJyVeg2jatLWdqzB+vnfQfrGHPB3hSJWZiI7HNk4wI7WIHj1OBjHYQG/55Y cWgOItkLqXytvs2QP09JqLwXTKtMlAdfzrg276XqDUeWLRl3XLSLsDqWLaQsZcywnE Eqn9pAyH29VR+FJQ8QxRZlTjQEefMAkQ6geeIs90= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 23 Aug 2021 14:49:37 +0200 Message-Id: <20210823124937.253539-8-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210823124937.253539-1-jeanmichel.hautbois@ideasonboard.com> References: <20210823124937.253539-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 7/7] ipa: ipu3: Introduce Black Level Correction X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The pixels output by the camera normally include a black level, because sensors do not always report a signal level of '0' for black. Pixels at or below this level should be considered black. This can be taken into account by reading the lowest value of a special region on sensors which is not exposed to the lens. This provides a scaling factor to be able to adjust the expected black levels in the resultant images. For a camera outputting 10-bit pixel values (in the range 0 to 1023) a typical black level might be 64, but the correct value would be found in the sensor manaufacturer’s datasheet. Signed-off-by: Jean-Michel Hautbois --- src/ipa/ipu3/algorithms/awb.cpp | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index 294871b1..ad3784d3 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -366,9 +366,17 @@ void Awb::prepare(IPAContext &context, ipu3_uapi_params *params) /* The CCM matrix may change when color temperature will be used */ params->acc_param.ccm = imguCssCcmDefault; + /* The Optical Black Level correction values */ + params->obgrid_param.gr = 64; + params->obgrid_param.r = 63; + params->obgrid_param.b = 63; + params->obgrid_param.gb = 64; + params->use.acc_awb = 1; params->use.acc_bnr = 1; params->use.acc_ccm = 1; + params->use.obgrid = 1; + params->use.obgrid_param = 1; } } /* namespace ipa::ipu3::algorithms */