From patchwork Mon Mar 29 19:18:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 11776 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 98B4FC32F0 for ; Mon, 29 Mar 2021 19:18:37 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 547E96878B; Mon, 29 Mar 2021 21:18:37 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="CTubhu3B"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D246168782 for ; Mon, 29 Mar 2021 21:18:34 +0200 (CEST) Received: from localhost.localdomain (unknown [IPv6:2a01:e0a:169:7140:9714:da21:50fd:f63f]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 705A0503; Mon, 29 Mar 2021 21:18:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1617045514; bh=HNiVdnso9zkQx0R1GQccz0JWQu1u19jLKCjZR2JRbdI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CTubhu3BXg8qH8EykKRXPYuv5Tk4T9z1qSmpGCo4A1e+umxd7vLbxVYvGpGVZ7HQi PCAl+VukDHVp24xHh+HcemB1m4zjbpTIQU3hKMFWZj6PnTMKJ8TLtHUM44BK+nrRkd hMCouf3n2Pp12ILNK26NMCasE9do/cz06VTP+KCs= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 29 Mar 2021 21:18:22 +0200 Message-Id: <20210329191826.77817-2-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210329191826.77817-1-jeanmichel.hautbois@ideasonboard.com> References: <20210329191826.77817-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 1/5] ipa: Add a common interface for algorithm objects X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In order to instanciate and use algorithms (AWB, AGC, etc.) there is a need for a common class to define mandatory methods. Instead of reinventing the wheel, reuse what Raspberry Pi has done and adapt to the minimum requirements expected. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Laurent Pinchart --- src/ipa/libipa/algorithm.cpp | 39 ++++++++++++++++++++++++++++++++++++ src/ipa/libipa/algorithm.h | 24 ++++++++++++++++++++++ src/ipa/libipa/meson.build | 4 +++- 3 files changed, 66 insertions(+), 1 deletion(-) create mode 100644 src/ipa/libipa/algorithm.cpp create mode 100644 src/ipa/libipa/algorithm.h diff --git a/src/ipa/libipa/algorithm.cpp b/src/ipa/libipa/algorithm.cpp new file mode 100644 index 00000000..930f9353 --- /dev/null +++ b/src/ipa/libipa/algorithm.cpp @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Ideas On Board + * + * algorithm.cpp - ISP control algorithms + */ + +#include "algorithm.h" + +/** + * \file algorithm.h + * \brief Algorithm common interface + */ + +namespace libcamera { + +/** + * \brief The IPA namespace + * + * The IPA namespace groups all types specific to IPA modules. It serves as the + * top-level namespace for the IPA library libipa, and also contains + * module-specific namespaces for IPA modules. + */ +namespace ipa { + +/** + * \class Algorithm + * \brief The base class for all IPA algorithms + * + * The Algorithm class defines a standard interface for IPA algorithms. By + * abstracting algorithms, it makes possible the implementation of generic code + * to manage algorithms regardless of their specific type. + */ + +Algorithm::~Algorithm() = default; + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/algorithm.h b/src/ipa/libipa/algorithm.h new file mode 100644 index 00000000..89cee4c4 --- /dev/null +++ b/src/ipa/libipa/algorithm.h @@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Ideas On Board + * + * algorithm.h - ISP control algorithm interface + */ +#ifndef __LIBCAMERA_IPA_LIBIPA_ALGORITHM_H__ +#define __LIBCAMERA_IPA_LIBIPA_ALGORITHM_H__ + +namespace libcamera { + +namespace ipa { + +class Algorithm +{ +public: + virtual ~Algorithm(); +}; + +} /* namespace ipa */ + +} /* namespace libcamera */ + +#endif /* __LIBCAMERA_IPA_LIBIPA_ALGORITHM_H__ */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index b29ef0f4..585d02a3 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -1,13 +1,15 @@ # SPDX-License-Identifier: CC0-1.0 libipa_headers = files([ + 'algorithm.h', ]) libipa_sources = files([ + 'algorithm.cpp', ]) libipa_includes = include_directories('..') -libipa = static_library('ipa', libipa_sources, +libipa = static_library('ipa', [libipa_sources, libipa_headers], include_directories : ipa_includes, dependencies : libcamera_dep) From patchwork Mon Mar 29 19:18:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 11777 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 19945C32F0 for ; Mon, 29 Mar 2021 19:18:41 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id CC52168793; Mon, 29 Mar 2021 21:18:40 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="n6JDQ0q7"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9EE0968783 for ; Mon, 29 Mar 2021 21:18:35 +0200 (CEST) Received: from localhost.localdomain (unknown [IPv6:2a01:e0a:169:7140:9714:da21:50fd:f63f]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 4AF22503; Mon, 29 Mar 2021 21:18:35 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1617045515; bh=4ugc+u64tJwtiF9ySV7Cyo9jk10r8fkAzax7dpg0aIE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=n6JDQ0q7rvXdNQBE0xmzZ5YTmBOQe+R4DptCJRyN5e4Iyh01fLH29/E2PF7GsAxw8 8oFJ9JH4yibltB2+17bvGCWbEjIB1wboXQ88m9tifw1zydmmpCF6pPGKPCGQQuam+f bLDxTTIZqQ64JVKivqFGEt0wFGgme00JlAk4dEnQ= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 29 Mar 2021 21:18:23 +0200 Message-Id: <20210329191826.77817-3-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210329191826.77817-1-jeanmichel.hautbois@ideasonboard.com> References: <20210329191826.77817-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 2/5] ipa: ipu3: Add an histogram class X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This class will be used at least by AGC algorithm when quantiles are needed for example. Signed-off-by: Jean-Michel Hautbois --- src/ipa/libipa/histogram.cpp | 102 +++++++++++++++++++++++++++++++++++ src/ipa/libipa/histogram.h | 62 +++++++++++++++++++++ src/ipa/libipa/meson.build | 2 + 3 files changed, 166 insertions(+) create mode 100644 src/ipa/libipa/histogram.cpp create mode 100644 src/ipa/libipa/histogram.h diff --git a/src/ipa/libipa/histogram.cpp b/src/ipa/libipa/histogram.cpp new file mode 100644 index 00000000..bea52687 --- /dev/null +++ b/src/ipa/libipa/histogram.cpp @@ -0,0 +1,102 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2019, Raspberry Pi (Trading) Limited + * + * histogram.cpp - histogram calculations + */ +#include + +#include "histogram.h" + +/** + * \file histogram.h + * \brief Class to represent Histograms and manipulate them + */ + +namespace libcamera { + +namespace ipa { + +/** + * \class Histogram + * \brief The base class for creating histograms + * + * The Histogram class defines a standard interface for IPA algorithms. By + * abstracting histograms, it makes possible the implementation of generic code + * to manage algorithms regardless of their specific type. + */ + +/** + * \brief Cumulative frequency up to a (fractional) point in a bin. + * \param[in] bin the number of bins + * \return The number of bins cumulated + */ +uint64_t Histogram::cumulativeFreq(double bin) const +{ + if (bin <= 0) + return 0; + else if (bin >= bins()) + return total(); + int b = (int)bin; + return cumulative_[b] + + (bin - b) * (cumulative_[b + 1] - cumulative_[b]); +} + +/** + * \brief Return the (fractional) bin of the point through the histogram + * \param[in] q the desired point (0 <= q <= 1) + * \param[in] first low limit (optionnal if -1) + * \param[in] last high limit (optionnal if -1) + * \return The fractionnal bin of the point + */ +double Histogram::quantile(double q, int first, int last) const +{ + if (first == -1) + first = 0; + if (last == -1) + last = cumulative_.size() - 2; + assert(first <= last); + uint64_t items = q * total(); + /* Binary search to find the right bin */ + while (first < last) { + int middle = (first + last) / 2; + /* Is it between first and middle ? */ + if (cumulative_[middle + 1] > items) + last = middle; + else + first = middle + 1; + } + assert(items >= cumulative_[first] && items <= cumulative_[last + 1]); + double frac = cumulative_[first + 1] == cumulative_[first] ? 0 + : (double)(items - cumulative_[first]) / + (cumulative_[first + 1] - cumulative_[first]); + return first + frac; +} + +/** + * \brief Calculate the mean between two quantiles + * \param[in] lowQuantile low Quantile + * \param[in] highQuantile high Quantile + * \return The average histogram bin value between the two quantiles + */ +double Histogram::interQuantileMean(double lowQuantile, double highQuantile) const +{ + assert(highQuantile > lowQuantile); + double lowPoint = quantile(lowQuantile); + double highPoint = quantile(highQuantile, (int)lowPoint); + double sumBinFreq = 0, cumulFreq = 0; + for (double p_next = floor(lowPoint) + 1.0; p_next <= ceil(highPoint); + lowPoint = p_next, p_next += 1.0) { + int bin = floor(lowPoint); + double freq = (cumulative_[bin + 1] - cumulative_[bin]) * + (std::min(p_next, highPoint) - lowPoint); + sumBinFreq += bin * freq; + cumulFreq += freq; + } + /* add 0.5 to give an average for bin mid-points */ + return sumBinFreq / cumulFreq + 0.5; +} + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/histogram.h b/src/ipa/libipa/histogram.h new file mode 100644 index 00000000..a610f675 --- /dev/null +++ b/src/ipa/libipa/histogram.h @@ -0,0 +1,62 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2019, Raspberry Pi (Trading) Limited + * + * histogram.h - histogram calculation interface + */ +#ifndef __LIBCAMERA_IPA_LIBIPA_HISTOGRAM_H__ +#define __LIBCAMERA_IPA_LIBIPA_HISTOGRAM_H__ + +#include +#include +#include + +// A simple histogram class, for use in particular to find "quantiles" and +// averages between "quantiles". + +namespace libcamera { + +namespace ipa { + +class Histogram +{ +public: + template + /** + * \brief Create a cumulative histogram with a bin number of intervals + * \param[in] histogram a reference to the histogram + * \param[in] num the number of bins + */ + Histogram(T *histogram, int num) + { + assert(num); + cumulative_.reserve(num + 1); + cumulative_.push_back(0); + for (int i = 0; i < num; i++) + cumulative_.push_back(cumulative_.back() + + histogram[i]); + } + + /** + * \brief getter for number of bins + * \return number of bins + */ + uint32_t bins() const { return cumulative_.size() - 1; } + /** + * \brief getter for number of values + * \return number of values + */ + uint64_t total() const { return cumulative_[cumulative_.size() - 1]; } + uint64_t cumulativeFreq(double bin) const; + double quantile(double q, int first = -1, int last = -1) const; + double interQuantileMean(double lowQuantile, double hiQuantile) const; + +private: + std::vector cumulative_; +}; + +} /* namespace ipa */ + +} /* namespace libcamera */ + +#endif /* __LIBCAMERA_IPA_LIBIPA_HISTOGRAM_H__ */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index 585d02a3..53edeef9 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -2,10 +2,12 @@ libipa_headers = files([ 'algorithm.h', + 'histogram.h' ]) libipa_sources = files([ 'algorithm.cpp', + 'histogram.cpp' ]) libipa_includes = include_directories('..') From patchwork Mon Mar 29 19:18:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 11778 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A1099C32F0 for ; Mon, 29 Mar 2021 19:18:41 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 55F3C6878D; Mon, 29 Mar 2021 21:18:41 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="NJ0jCwGc"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 64B1A6877D for ; Mon, 29 Mar 2021 21:18:36 +0200 (CEST) Received: from localhost.localdomain (unknown [IPv6:2a01:e0a:169:7140:9714:da21:50fd:f63f]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id F261AA49; Mon, 29 Mar 2021 21:18:35 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1617045516; bh=qc9pD4A5hjZO2xSNHx4vTYsDKsqv2di52PIgpw102DM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NJ0jCwGcgfPiqmZ7gguILvyedtCwym3EoNRoqVaFQpSIx/chL8DeeobMGRcpKTSHx 6v21xe9Wghhhts2PgyI8FNdLoGpApU+4086sM9T/sSInAMwT1QZOSYKx5Ax3azeRRb x/E3foAYzSrraFy1ie9DuxgYr3xmdM1E7qMgx4wI= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 29 Mar 2021 21:18:24 +0200 Message-Id: <20210329191826.77817-4-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210329191826.77817-1-jeanmichel.hautbois@ideasonboard.com> References: <20210329191826.77817-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 3/5] ipa: ipu3: Use a local parameter object and prepare for 3A algorithms X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The IPA will locally modify the parameters before they are passed down to the imgU. Use a local parameter object to give a reference to those algorithms. AGC algorithm calculates a brightness level at min and max exposure/gains. Starting at the minimum value for both of them shortens the time to converge (as the first frames will already have the correct level). Signed-off-by: Jean-Michel Hautbois Reviewed-by: Laurent Pinchart --- src/ipa/ipu3/ipu3.cpp | 11 +++++++++-- src/ipa/ipu3/meson.build | 6 +++++- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 34a907f2..07dbc24a 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -61,6 +61,9 @@ private: uint32_t gain_; uint32_t minGain_; uint32_t maxGain_; + + /* Local parameter storage */ + ipu3_uapi_params params_; }; int IPAIPU3::start() @@ -92,11 +95,15 @@ void IPAIPU3::configure(const std::map &entityControls minExposure_ = std::max(itExp->second.min().get(), 1); maxExposure_ = itExp->second.max().get(); - exposure_ = maxExposure_; + exposure_ = minExposure_; minGain_ = std::max(itGain->second.min().get(), 1); maxGain_ = itGain->second.max().get(); - gain_ = maxGain_; + gain_ = minGain_; + + params_ = {}; + + setControls(0); } void IPAIPU3::mapBuffers(const std::vector &buffers) diff --git a/src/ipa/ipu3/meson.build b/src/ipa/ipu3/meson.build index a241f617..52d98c8e 100644 --- a/src/ipa/ipu3/meson.build +++ b/src/ipa/ipu3/meson.build @@ -2,8 +2,12 @@ ipa_name = 'ipa_ipu3' +ipu3_ipa_sources = files([ + 'ipu3.cpp', +]) + mod = shared_module(ipa_name, - ['ipu3.cpp', libcamera_generated_ipa_headers], + [ipu3_ipa_sources, libcamera_generated_ipa_headers], name_prefix : '', include_directories : [ipa_includes, libipa_includes], dependencies : libcamera_dep, From patchwork Mon Mar 29 19:18:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 11779 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7D3E3C32F0 for ; Mon, 29 Mar 2021 19:18:42 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 090C068784; Mon, 29 Mar 2021 21:18:42 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="O9dxFOIb"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0D4F86877D for ; Mon, 29 Mar 2021 21:18:37 +0200 (CEST) Received: from localhost.localdomain (unknown [IPv6:2a01:e0a:169:7140:9714:da21:50fd:f63f]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id A4D89503; Mon, 29 Mar 2021 21:18:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1617045516; bh=q6cUeQ3WHRddgl7yIRCgfZQ8AsnH3df/yVyhQmZf4/U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=O9dxFOIbHcSjpnWNmKc9gPET7kb4rTstLGocoY4pa9DsdxPXh81gkthBXTchi347t AAiu+73I29ihEhM/jUk/LuF9w54IqpKo9lrS4eLQBRLtbXC6nBLvu4qhfG85pZrK1z FZW5HKkCeXENwjkhgfxNcfiBEHV6FUxoePYNCxr0= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 29 Mar 2021 21:18:25 +0200 Message-Id: <20210329191826.77817-5-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210329191826.77817-1-jeanmichel.hautbois@ideasonboard.com> References: <20210329191826.77817-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 4/5] ipa: ipu3: Add support for IPU3 AWB algorithm X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Inherit from the Algorithm class to implement basic AWB functions. Once AWB is done, a color temperature is estimated and a default CCM matrice is used (yet to be tuned). Implement a basic "grey-world" AWB algorithm just for demonstration purpose. BDS output size is passed by the pipeline handler to the IPA. The best grid is then calculated to maximize the number of pixels taken into account in each cells. As commented in the source code, it can be improved, as ithas (at least) one limitation: if a cell is big (say 128 pixels wide) and indicated as saturated, it won't be taken into account at all. Maybe is it possible to have a smaller one, at the cost of a few pixels to lose. In which case we can center the grid using the x_start and y_start parameters. Signed-off-by: Jean-Michel Hautbois --- src/ipa/ipu3/ipu3.cpp | 74 +++++++++++- src/ipa/ipu3/ipu3_awb.cpp | 233 ++++++++++++++++++++++++++++++++++++++ src/ipa/ipu3/ipu3_awb.h | 44 +++++++ src/ipa/ipu3/meson.build | 1 + 4 files changed, 346 insertions(+), 6 deletions(-) create mode 100644 src/ipa/ipu3/ipu3_awb.cpp create mode 100644 src/ipa/ipu3/ipu3_awb.h diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 07dbc24a..cc741d69 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -21,6 +21,11 @@ #include "libcamera/internal/buffer.h" #include "libcamera/internal/log.h" +#include "ipu3_awb.h" + +static const uint32_t kMaxCellWidthPerSet = 160; +static const uint32_t kMaxCellHeightPerSet = 80; + namespace libcamera { LOG_DEFINE_CATEGORY(IPAIPU3) @@ -62,8 +67,12 @@ private: uint32_t minGain_; uint32_t maxGain_; + /* Interface to the AWB algorithm */ + std::unique_ptr awbAlgo_; /* Local parameter storage */ - ipu3_uapi_params params_; + struct ipu3_uapi_params params_; + + struct ipu3_uapi_grid_config bdsGrid_; }; int IPAIPU3::start() @@ -73,6 +82,51 @@ int IPAIPU3::start() return 0; } +/* This method calculates a grid for the AWB algorithm in the IPU3 firmware. + * Its input it the BDS output size calculated in the imgU. + * It is limited for now to the simplest method: find the lesser error + * with the width/height and respective log2 width/height of the cells. + * \todo smaller cells are better so adapt x_start to lose a bit but have + * a better average resolution. If a cell is saturated, it might make a big difference. */ +void IPAIPU3::calculateBdsGrid(const Size &bdsOutputSize) +{ + std::vector log2WidthVector = { 3, 4, 5, 6, 7 }; + std::vector log2HeightVector = { 3, 4, 5, 6, 7 }; + uint32_t minError = std::numeric_limits::max(); + uint32_t bestWidth = 0; + uint32_t bestHeight = 0; + uint32_t bestLog2Width = 0; + uint32_t bestLog2Height = 0; + bdsGrid_ = {}; + + for (uint32_t i = 0; i < log2WidthVector.size(); ++i) { + uint32_t width = std::min(kMaxCellWidthPerSet, bdsOutputSize.width >> log2WidthVector[i]); + width = width << log2WidthVector[i]; + for (uint32_t j = 0; j < log2HeightVector.size(); ++j) { + int32_t height = std::min(kMaxCellHeightPerSet, bdsOutputSize.height >> log2HeightVector[j]); + height = height << log2HeightVector[j]; + uint32_t error = std::abs((int)(width - bdsOutputSize.width)) + std::abs((int)(height - bdsOutputSize.height)); + + if (error > minError) + continue; + + minError = error; + bestWidth = width; + bestHeight = height; + bestLog2Width = log2WidthVector[i]; + bestLog2Height = log2HeightVector[j]; + } + } + + bdsGrid_.width = bestWidth >> bestLog2Width; + bdsGrid_.block_width_log2 = bestLog2Width; + bdsGrid_.height = bestHeight >> bestLog2Height; + bdsGrid_.block_height_log2 = bestLog2Height; + LOG(IPAIPU3, Debug) << "Best grid found is: (" + << (int)bdsGrid_.width << " << " << (int)bdsGrid_.block_width_log2 << ") x (" + << (int)bdsGrid_.height << "<<" << (int)bdsGrid_.block_height_log2 << ")"; +} + void IPAIPU3::configure(const std::map &entityControls, [[maybe_unused]] const Size &bdsOutputSize) { @@ -103,6 +157,9 @@ void IPAIPU3::configure(const std::map &entityControls params_ = {}; + awbAlgo_ = std::make_unique(); + awbAlgo_->initialise(params_, bdsOutputSize, bdsGrid_); + setControls(0); } @@ -175,10 +232,9 @@ void IPAIPU3::processControls([[maybe_unused]] unsigned int frame, void IPAIPU3::fillParams(unsigned int frame, ipu3_uapi_params *params) { - /* Prepare parameters buffer. */ - memset(params, 0, sizeof(*params)); + awbAlgo_->updateWbParameters(params_, 1.0); - /* \todo Fill in parameters buffer. */ + *params = params_; ipa::ipu3::IPU3Action op; op.op = ipa::ipu3::ActionParamFilled; @@ -191,8 +247,14 @@ void IPAIPU3::parseStatistics(unsigned int frame, { ControlList ctrls(controls::controls); - /* \todo React to statistics and update internal state machine. */ - /* \todo Add meta-data information to ctrls. */ + if (!stats->stats_3a_status.awb_en) { + LOG(IPAIPU3, Error) << "AWB stats are not enabled"; + } else { + agcAlgo_->process(stats, exposure_, gain_); + awbAlgo_->calculateWBGains(stats); + if (agcAlgo_->updateControls()) + setControls(frame); + } ipa::ipu3::IPU3Action op; op.op = ipa::ipu3::ActionMetadataReady; diff --git a/src/ipa/ipu3/ipu3_awb.cpp b/src/ipa/ipu3/ipu3_awb.cpp new file mode 100644 index 00000000..c63eb08f --- /dev/null +++ b/src/ipa/ipu3/ipu3_awb.cpp @@ -0,0 +1,233 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Ideas On Board + * + * ipu3_awb.cpp - AWB control algorithm + */ +#include "ipu3_awb.h" + +#include +#include +#include + +#include "libcamera/internal/log.h" + +namespace libcamera { + +namespace ipa { + +LOG_DEFINE_CATEGORY(IPU3Awb) + +static const struct ipu3_uapi_bnr_static_config imguCssBnrDefaults = { + .wb_gains = { 16, 16, 16, 16 }, + .wb_gains_thr = { 255, 255, 255, 255 }, + .thr_coeffs = { 1700, 0, 31, 31, 0, 16 }, + .thr_ctrl_shd = { 26, 26, 26, 26 }, + .opt_center{ -648, 0, -366, 0 }, + .lut = { + { 17, 23, 28, 32, 36, 39, 42, 45, + 48, 51, 53, 55, 58, 60, 62, 64, + 66, 68, 70, 72, 73, 75, 77, 78, + 80, 82, 83, 85, 86, 88, 89, 90 } }, + .bp_ctrl = { 20, 0, 1, 40, 0, 6, 0, 6, 0 }, + .dn_detect_ctrl{ 9, 3, 4, 0, 8, 0, 1, 1, 1, 1, 0 }, + .column_size = 1296, + .opt_center_sqr = { 419904, 133956 }, +}; + +/* settings for Auto White Balance */ +static const struct ipu3_uapi_awb_config_s imguCssAwbDefaults = { + .rgbs_thr_gr = 8191, + .rgbs_thr_r = 8191, + .rgbs_thr_gb = 8191, + .rgbs_thr_b = 8191 | IPU3_UAPI_AWB_RGBS_THR_B_EN | IPU3_UAPI_AWB_RGBS_THR_B_INCL_SAT, + .grid = { + .width = 160, + .height = 36, + .block_width_log2 = 3, + .block_height_log2 = 4, + .height_per_slice = 1, /* Overridden by kernel. */ + .x_start = 0, + .y_start = 0, + .x_end = 0, + .y_end = 0, + }, +}; + +static const struct ipu3_uapi_ccm_mat_config imguCssCcmDefault = { + 8191, 0, 0, 0, + 0, 8191, 0, 0, + 0, 0, 8191, 0 +}; + +IPU3Awb::IPU3Awb() + : Algorithm() +{ +} + +IPU3Awb::~IPU3Awb() +{ +} + +void IPU3Awb::initialise(ipu3_uapi_params ¶ms, const Size &bdsOutputSize, struct ipu3_uapi_grid_config &bdsGrid) +{ + params.use.acc_awb = 1; + params.acc_param.awb.config = imguCssAwbDefaults; + + awbGrid_ = bdsGrid; + params.acc_param.awb.config.grid = awbGrid_; + + params.use.obgrid = 0; + params.obgrid_param.gr = 20; + params.obgrid_param.r = 28; + params.obgrid_param.b = 28; + params.obgrid_param.gb = 20; + + params.use.acc_bnr = 1; + params.acc_param.bnr = imguCssBnrDefaults; + params.acc_param.bnr.opt_center.x_reset = -1 * (bdsOutputSize.width / 2); + params.acc_param.bnr.opt_center.y_reset = -1 * (bdsOutputSize.height / 2); + params.acc_param.bnr.column_size = bdsOutputSize.width; + params.acc_param.bnr.opt_center_sqr.x_sqr_reset = params.acc_param.bnr.opt_center.x_reset * params.acc_param.bnr.opt_center.x_reset; + params.acc_param.bnr.opt_center_sqr.y_sqr_reset = params.acc_param.bnr.opt_center.y_reset * params.acc_param.bnr.opt_center.y_reset; + + params.use.acc_ccm = 1; + params.acc_param.ccm = imguCssCcmDefault; + + params.use.acc_gamma = 1; + params.acc_param.gamma.gc_ctrl.enable = 1; + + params.use.acc_green_disparity = 0; + params.acc_param.green_disparity.gd_black = 2440; + params.acc_param.green_disparity.gd_red = 4; + params.acc_param.green_disparity.gd_blue = 4; + params.acc_param.green_disparity.gd_green = 4; + params.acc_param.green_disparity.gd_shading = 24; + params.acc_param.green_disparity.gd_support = 2; + params.acc_param.green_disparity.gd_clip = 1; + params.acc_param.green_disparity.gd_central_weight = 5; + + params.use.acc_cds = 1; + params.acc_param.cds.csc_en = 1; + params.acc_param.cds.uv_bin_output = 0; + params.acc_param.cds.ds_c00 = 0; + params.acc_param.cds.ds_c01 = 1; + params.acc_param.cds.ds_c02 = 1; + params.acc_param.cds.ds_c03 = 0; + params.acc_param.cds.ds_c10 = 0; + params.acc_param.cds.ds_c11 = 1; + params.acc_param.cds.ds_c12 = 1; + params.acc_param.cds.ds_c13 = 0; + params.acc_param.cds.ds_nf = 2; + + wbGains_[0] = 16; + wbGains_[1] = 4096; + wbGains_[2] = 4096; + wbGains_[3] = 16; + + frame_count_ = 0; +} + +uint32_t IPU3Awb::estimateCCT(uint8_t red, uint8_t green, uint8_t blue) +{ + double X = (-0.14282) * (red) + (1.54924) * (green) + (-0.95641) * (blue); + double Y = (-0.32466) * (red) + (1.57837) * (green) + (-0.73191) * (blue); + double Z = (-0.68202) * (red) + (0.77073) * (green) + (0.56332) * (blue); + + double x = X / (X + Y + Z); + double y = Y / (X + Y + Z); + + double n = (x - 0.3320) / (0.1858 - y); + return static_cast(449 * n * n * n + 3525 * n * n + 6823.3 * n + 5520.33); +} + +double meanValue(std::vector colorValues) +{ + uint32_t count = 0; + uint32_t hist[256] = { 0 }; + for (uint32_t const &val : colorValues) { + hist[val]++; + count++; + } + + double mean = 0.0; + for (uint32_t i = 0; i < 256; i++) { + mean += hist[i] * i; + } + return mean /= count; +} + +void IPU3Awb::calculateWBGains(const ipu3_uapi_stats_3a *stats) +{ + std::vector redValues, greenValues, blueValues; + const struct ipu3_uapi_grid_config statsAwbGrid = stats->stats_4a_config.awb_config.grid; + Rectangle awbRegion = { statsAwbGrid.x_start, + statsAwbGrid.y_start, + static_cast(statsAwbGrid.x_end - statsAwbGrid.x_start) + 1, + static_cast(statsAwbGrid.y_end - statsAwbGrid.y_start) + 1 }; + + Point topleft = awbRegion.topLeft(); + uint32_t startY = (topleft.y >> awbGrid_.block_height_log2) * awbGrid_.width << awbGrid_.block_width_log2; + uint32_t startX = (topleft.x >> awbGrid_.block_width_log2) << awbGrid_.block_width_log2; + uint32_t endX = (startX + (awbRegion.size().width >> awbGrid_.block_width_log2)) << awbGrid_.block_width_log2; + uint32_t count = 0; + uint32_t i, j; + + awbCounted_ = 0; + for (j = (topleft.y >> awbGrid_.block_height_log2); + j < (topleft.y >> awbGrid_.block_height_log2) + (awbRegion.size().height >> awbGrid_.block_height_log2); + j++) { + for (i = startX + startY; i < endX + startY; i += 8) { + if (stats->awb_raw_buffer.meta_data[i + 4 + j * awbGrid_.width] == 0) { + greenValues.push_back(stats->awb_raw_buffer.meta_data[i + j * awbGrid_.width]); + redValues.push_back(stats->awb_raw_buffer.meta_data[i + 1 + j * awbGrid_.width]); + blueValues.push_back(stats->awb_raw_buffer.meta_data[i + 2 + j * awbGrid_.width]); + greenValues.push_back(stats->awb_raw_buffer.meta_data[i + 3 + j * awbGrid_.width]); + awbCounted_++; + } + count++; + } + } + + double rMean = meanValue(redValues); + double bMean = meanValue(blueValues); + double gMean = meanValue(greenValues); + + double rGain = gMean / rMean; + double bGain = gMean / bMean; + + wbGains_[0] = 16; + wbGains_[1] = 4096 * rGain; + wbGains_[2] = 4096 * bGain; + wbGains_[3] = 16; + + frame_count_++; + + cct_ = estimateCCT(rMean, gMean, bMean); +} + +void IPU3Awb::updateWbParameters(ipu3_uapi_params ¶ms, double agcGamma) +{ + if ((wbGains_[0] == 0) || (wbGains_[1] == 0) || (wbGains_[2] == 0) || (wbGains_[3] == 0)) { + LOG(IPU3Awb, Error) << "Gains can't be 0, check the stats"; + } else { + params.acc_param.bnr.wb_gains.gr = wbGains_[0]; + params.acc_param.bnr.wb_gains.r = wbGains_[1]; + params.acc_param.bnr.wb_gains.b = wbGains_[2]; + params.acc_param.bnr.wb_gains.gb = wbGains_[3]; + + LOG(IPU3Awb, Debug) << "Color temperature estimated: " << cct_ + << " and gamma calculated: " << agcGamma; + params.acc_param.ccm = imguCssCcmDefault; + + for (uint32_t i = 0; i < 256; i++) { + double j = i / 255.0; + double gamma = std::pow(j, 1.0 / agcGamma); + params.acc_param.gamma.gc_lut.lut[i] = gamma * 8191; + } + } +} + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/ipu3/ipu3_awb.h b/src/ipa/ipu3/ipu3_awb.h new file mode 100644 index 00000000..a14401a0 --- /dev/null +++ b/src/ipa/ipu3/ipu3_awb.h @@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Ideas On Board + * + * ipu3_awb.h - IPU3 AWB control algorithm + */ +#ifndef __LIBCAMERA_IPU3_AWB_H__ +#define __LIBCAMERA_IPU3_AWB_H__ + +#include + +#include + +#include "libipa/algorithm.h" + +namespace libcamera { + +namespace ipa { + +class IPU3Awb : public Algorithm +{ +public: + IPU3Awb(); + ~IPU3Awb(); + + void initialise(ipu3_uapi_params ¶ms, const Size &bdsOutputSize, struct ipu3_uapi_grid_config &bdsGrid); + void calculateWBGains(const ipu3_uapi_stats_3a *stats); + void updateWbParameters(ipu3_uapi_params ¶ms, double agcGamma); + +private: + uint32_t estimateCCT(uint8_t red, uint8_t green, uint8_t blue); + + /* WB calculated gains */ + uint16_t wbGains_[4]; + uint32_t cct_; + uint32_t awbCounted_; + struct ipu3_uapi_grid_config awbGrid_; + uint32_t frame_count_; +}; + +} /* namespace ipa */ + +} /* namespace libcamera*/ +#endif /* __LIBCAMERA_IPU3_AWB_H__ */ diff --git a/src/ipa/ipu3/meson.build b/src/ipa/ipu3/meson.build index 52d98c8e..07a864c8 100644 --- a/src/ipa/ipu3/meson.build +++ b/src/ipa/ipu3/meson.build @@ -4,6 +4,7 @@ ipa_name = 'ipa_ipu3' ipu3_ipa_sources = files([ 'ipu3.cpp', + 'ipu3_awb.cpp', ]) mod = shared_module(ipa_name, From patchwork Mon Mar 29 19:18:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 11780 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 245F6C32F0 for ; Mon, 29 Mar 2021 19:18:43 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C93266878F; Mon, 29 Mar 2021 21:18:42 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="hFpu4ebp"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5221068788 for ; Mon, 29 Mar 2021 21:18:38 +0200 (CEST) Received: from localhost.localdomain (unknown [IPv6:2a01:e0a:169:7140:9714:da21:50fd:f63f]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 5D385A49; Mon, 29 Mar 2021 21:18:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1617045517; bh=0rl1kk4O8LqjDsPc6RST9Cis5vPWbA/rCwpp+p9JlZM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hFpu4ebpszEcx80qHZuN9oiJopr0ARB2D1MH2+4ht2mhTg5mNmpqvWXFSm5Aye/Gi Wz76noUQA2chnxvbZzugvT+/8bMXf2hLIFxK6JamGZaoFRs/zXPSV9RE0vlOlDc4QA uBef0XzbcCoQ2JjS6+Srkc0nLJOotsWPDu+NuT3w= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 29 Mar 2021 21:18:26 +0200 Message-Id: <20210329191826.77817-6-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210329191826.77817-1-jeanmichel.hautbois@ideasonboard.com> References: <20210329191826.77817-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 5/5] ipa: ipu3: Add support for IPU3 AEC/AGC algorithm X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Inherit from the Algorithm class to implement basic auto-exposure and auto-gain functions. Extract computeTargetExposure() and computeGain() and adapt those to the IPU3 structure. Filetering was added too as it avoids big steps when exposure changes a lot. Signed-off-by: Jean-Michel Hautbois --- src/ipa/ipu3/ipu3.cpp | 12 +- src/ipa/ipu3/ipu3_agc.cpp | 229 ++++++++++++++++++++++++++++++++++++++ src/ipa/ipu3/ipu3_agc.h | 67 +++++++++++ src/ipa/ipu3/meson.build | 1 + 4 files changed, 307 insertions(+), 2 deletions(-) create mode 100644 src/ipa/ipu3/ipu3_agc.cpp create mode 100644 src/ipa/ipu3/ipu3_agc.h diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index cc741d69..59ed09e6 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -21,6 +21,7 @@ #include "libcamera/internal/buffer.h" #include "libcamera/internal/log.h" +#include "ipu3_agc.h" #include "ipu3_awb.h" static const uint32_t kMaxCellWidthPerSet = 160; @@ -54,6 +55,7 @@ private: const ipu3_uapi_stats_3a *stats); void setControls(unsigned int frame); + void calculateBdsGrid(const Size &bdsOutputSize); std::map buffers_; @@ -69,6 +71,8 @@ private: /* Interface to the AWB algorithm */ std::unique_ptr awbAlgo_; + /* Interface to the AEC/AGC algorithm */ + std::unique_ptr agcAlgo_; /* Local parameter storage */ struct ipu3_uapi_params params_; @@ -157,10 +161,13 @@ void IPAIPU3::configure(const std::map &entityControls params_ = {}; + calculateBdsGrid(bdsOutputSize); + awbAlgo_ = std::make_unique(); awbAlgo_->initialise(params_, bdsOutputSize, bdsGrid_); - setControls(0); + agcAlgo_ = std::make_unique(); + agcAlgo_->initialise(bdsGrid_); } void IPAIPU3::mapBuffers(const std::vector &buffers) @@ -232,7 +239,8 @@ void IPAIPU3::processControls([[maybe_unused]] unsigned int frame, void IPAIPU3::fillParams(unsigned int frame, ipu3_uapi_params *params) { - awbAlgo_->updateWbParameters(params_, 1.0); + if (agcAlgo_->updateControls()) + awbAlgo_->updateWbParameters(params_, agcAlgo_->gamma()); *params = params_; diff --git a/src/ipa/ipu3/ipu3_agc.cpp b/src/ipa/ipu3/ipu3_agc.cpp new file mode 100644 index 00000000..14baad9b --- /dev/null +++ b/src/ipa/ipu3/ipu3_agc.cpp @@ -0,0 +1,229 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Ideas On Board + * + * ipu3_agc.cpp - AGC/AEC control algorithm + */ + +#include "ipu3_agc.h" + +#include +#include +#include + +#include "libcamera/internal/log.h" + +#include "libipa/histogram.h" + +namespace libcamera { + +namespace ipa { + +LOG_DEFINE_CATEGORY(IPU3Agc) + +/* Number of frames to wait before calculating stats on minimum exposure */ +static const uint32_t kInitialFrameMinAECount = 4; +/* Number of frames to wait between new gain/exposure estimations */ +static const uint32_t kFrameSkipCount = 6; + +/* Maximum ISO value for analogue gain */ +static const uint32_t kMinISO = 100; +static const uint32_t kMaxISO = 1500; +/* Maximum analogue gain value + * \todo grab it from a camera helper */ +static const uint32_t kMinGain = kMinISO / 100; +static const uint32_t kMaxGain = kMaxISO / 100; +/* \todo use calculated value based on sensor */ +static const uint32_t kMinExposure = 1; +static const uint32_t kMaxExposure = 1976; +/* \todo those should be get from pipeline handler ! */ +/* line duration in microseconds */ +static const double kLineDuration = 16.8; +static const double kMaxExposureTime = kMaxExposure * kLineDuration; +/* Histogram constants */ +static const uint32_t knumHistogramBins = 256; +static const double kEvGainTarget = 0.5; + +IPU3Agc::IPU3Agc() + : frameCount_(0), lastFrame_(0), + converged_(false), updateControls_(false) +{ + iqMean_ = 0.0; + gamma_ = 1.0; + histLow_ = 0; + histHigh_ = 255; + prevTotalExposure_ = 0.0; + prevTotalExposureNoDg_ = 0.0; + currentTotalExposure_ = 0.0; + currentTotalExposureNoDg_ = 0.0; +} + +IPU3Agc::~IPU3Agc() +{ +} + +void IPU3Agc::initialise(struct ipu3_uapi_grid_config &bdsGrid) +{ + aeGrid_ = bdsGrid; +} +void IPU3Agc::processBrightness(const ipu3_uapi_stats_3a *stats) +{ + const struct ipu3_uapi_grid_config statsAeGrid = stats->stats_4a_config.awb_config.grid; + Rectangle aeRegion = { statsAeGrid.x_start, + statsAeGrid.y_start, + static_cast(statsAeGrid.x_end - statsAeGrid.x_start) + 1, + static_cast(statsAeGrid.y_end - statsAeGrid.y_start) + 1 }; + Point topleft = aeRegion.topLeft(); + uint32_t startY = (topleft.y >> aeGrid_.block_height_log2) * aeGrid_.width << aeGrid_.block_width_log2; + uint32_t startX = (topleft.x >> aeGrid_.block_width_log2) << aeGrid_.block_width_log2; + uint32_t endX = (startX + (aeRegion.size().width >> aeGrid_.block_width_log2)) << aeGrid_.block_width_log2; + uint32_t i, j; + uint32_t count = 0; + + cellsBrightness_.clear(); + + for (j = (topleft.y >> aeGrid_.block_height_log2); + j < (topleft.y >> aeGrid_.block_height_log2) + (aeRegion.size().height >> aeGrid_.block_height_log2); + j++) { + for (i = startX + startY; i < endX + startY; i += 8) { + /* grid width (and maybe height) is not reliable. + * We observed a bit shift which makes the value 160 to be 32 in the stats grid. + * Use the one passed at init time. */ + if (stats->awb_raw_buffer.meta_data[i + 4 + j * aeGrid_.width] == 0) { + uint8_t Gr = stats->awb_raw_buffer.meta_data[i + j * aeGrid_.width]; + uint8_t R = stats->awb_raw_buffer.meta_data[i + 1 + j * aeGrid_.width]; + uint8_t B = stats->awb_raw_buffer.meta_data[i + 2 + j * aeGrid_.width]; + uint8_t Gb = stats->awb_raw_buffer.meta_data[i + 3 + j * aeGrid_.width]; + + cellsBrightness_.push_back(static_cast(0.2125 * R + 0.7154 * (Gr + Gb) / 2 + 0.0722 * B)); + count++; + } + } + } + std::vector::iterator maxIntensity = std::max_element(cellsBrightness_.begin(), cellsBrightness_.end()); + LOG(IPU3Agc, Debug) << "Most frequent intensity is " << *maxIntensity << " at " << std::distance(cellsBrightness_.begin(), maxIntensity); + + /* \todo create a class to generate histograms ! */ + uint32_t hist[knumHistogramBins] = { 0 }; + for (uint32_t const &val : cellsBrightness_) + hist[val]++; + + double mean = 0.0; + for (i = 0; i < knumHistogramBins; i++) { + mean += hist[i] * i; + } + mean /= count; + + double variance = 0.0; + for (i = 0; i < knumHistogramBins; i++) { + variance += ((i - mean) * (i - mean)) * hist[i]; + } + variance /= count; + variance = std::sqrt(variance); + + LOG(IPU3Agc, Debug) << "mean value is: " << mean << " and variance is " << variance; + /* Limit the gamma effect for now */ + gamma_ = 1.1; + + const auto [minBrightness, maxBrightness] = std::minmax_element(cellsBrightness_.begin(), cellsBrightness_.end()); + histLow_ = *minBrightness; + histHigh_ = *maxBrightness; + + Histogram histogram(hist, knumHistogramBins); + iqMean_ = histogram.interQuantileMean(0.98, 1.0); +} + +void IPU3Agc::filterExposure(bool desaturate) +{ + double speed = 0.2; + if (prevTotalExposure_ == 0.0) { + prevTotalExposure_ = currentTotalExposure_; + prevTotalExposureNoDg_ = currentTotalExposureNoDg_; + } else { + /* If close to the result go faster, to save making so many + * micro-adjustments on the way. + * \ todo: Make this customisable? */ + if (prevTotalExposure_ < 1.2 * currentTotalExposure_ && + prevTotalExposure_ > 0.8 * currentTotalExposure_) + speed = sqrt(speed); + prevTotalExposure_ = speed * currentTotalExposure_ + + prevTotalExposure_ * (1.0 - speed); + /* When desaturing, take a big jump down in exposure_no_dg, + * which we'll hide with digital gain. */ + if (desaturate) + prevTotalExposureNoDg_ = + currentTotalExposureNoDg_; + else + prevTotalExposureNoDg_ = + speed * currentTotalExposureNoDg_ + + prevTotalExposureNoDg_ * (1.0 - speed); + } + /* We can't let the no_dg exposure deviate too far below the + * total exposure, as there might not be enough digital gain available + * in the ISP to hide it (which will cause nasty oscillation). */ + double fastReduceThreshold = 0.4; + if (prevTotalExposureNoDg_ < + prevTotalExposure_ * fastReduceThreshold) + prevTotalExposureNoDg_ = prevTotalExposure_ * fastReduceThreshold; + LOG(IPU3Agc, Debug) << "After filtering, total_exposure " << prevTotalExposure_; +} + +void IPU3Agc::lockExposureGain(uint32_t &exposure, uint32_t &gain) +{ + updateControls_ = false; + + /* Algorithm initialization wait for first valid frames */ + /* \todo - have a number of frames given by DelayedControls ? + * - implement a function for IIR */ + if ((frameCount_ == kInitialFrameMinAECount) || (frameCount_ - lastFrame_ >= kFrameSkipCount)) { + /* Are we correctly exposed ? */ + double newGain = kEvGainTarget * knumHistogramBins / iqMean_; + + if (std::abs(iqMean_ - kEvGainTarget * knumHistogramBins) <= 1) { + LOG(IPU3Agc, Debug) << "!!! Good exposure with iqMean = " << iqMean_; + converged_ = true; + } else { + /* extracted from Rpi::Agc::computeTargetExposure */ + double currentShutter = exposure * kLineDuration; + currentTotalExposureNoDg_ = currentShutter * gain; + LOG(IPU3Agc, Debug) << "Actual total exposure " << currentTotalExposureNoDg_ + << " Shutter speed " << currentShutter + << " Gain " << gain; + currentTotalExposure_ = currentTotalExposureNoDg_ * newGain; + double maxTotalExposure = kMaxExposureTime * kMaxGain; + currentTotalExposure_ = std::min(currentTotalExposure_, maxTotalExposure); + LOG(IPU3Agc, Debug) << "Target total exposure " << currentTotalExposure_; + + /* \todo: estimate if we need to desaturate */ + filterExposure(false); + + double newExposure = 0.0; + if (currentShutter < kMaxExposureTime) { + exposure = std::clamp(static_cast(exposure * currentTotalExposure_ / currentTotalExposureNoDg_), kMinExposure, kMaxExposure); + newExposure = currentTotalExposure_ / exposure; + gain = std::clamp(static_cast(gain * currentTotalExposure_ / newExposure), kMinGain, kMaxGain); + updateControls_ = true; + } else if (currentShutter >= kMaxExposureTime) { + gain = std::clamp(static_cast(gain * currentTotalExposure_ / currentTotalExposureNoDg_), kMinGain, kMaxGain); + newExposure = currentTotalExposure_ / gain; + exposure = std::clamp(static_cast(exposure * currentTotalExposure_ / newExposure), kMinExposure, kMaxExposure); + updateControls_ = true; + } + LOG(IPU3Agc, Debug) << "Adjust exposure " << exposure * kLineDuration << " and gain " << gain; + } + lastFrame_ = frameCount_; + } else { + updateControls_ = false; + } +} + +void IPU3Agc::process(const ipu3_uapi_stats_3a *stats, uint32_t &exposure, uint32_t &gain) +{ + processBrightness(stats); + lockExposureGain(exposure, gain); + frameCount_++; +} + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/ipu3/ipu3_agc.h b/src/ipa/ipu3/ipu3_agc.h new file mode 100644 index 00000000..d4657a81 --- /dev/null +++ b/src/ipa/ipu3/ipu3_agc.h @@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Ideas On Board + * + * ipu3_agc.h - IPU3 AGC/AEC control algorithm + */ +#ifndef __LIBCAMERA_IPU3_AGC_H__ +#define __LIBCAMERA_IPU3_AGC_H__ + +#include +#include + +#include + +#include + +#include "libipa/algorithm.h" + +namespace libcamera { + +namespace ipa { + +class IPU3Agc : public Algorithm +{ +public: + IPU3Agc(); + ~IPU3Agc(); + + void initialise(struct ipu3_uapi_grid_config &bdsGrid); + void process(const ipu3_uapi_stats_3a *stats, uint32_t &exposure, uint32_t &gain); + bool converged() { return converged_; } + bool updateControls() { return updateControls_; } + /* \todo Use a metadata exchange between IPAs */ + double gamma() { return gamma_; } + +private: + void processBrightness(const ipu3_uapi_stats_3a *stats); + void filterExposure(bool desaturate); + void lockExposureGain(uint32_t &exposure, uint32_t &gain); + + struct ipu3_uapi_grid_config aeGrid_; + + uint64_t frameCount_; + uint64_t lastFrame_; + + /* Vector of calculated brightness for each cell */ + std::vector cellsBrightness_; + + bool converged_; + bool updateControls_; + + double iqMean_; + double gamma_; + uint32_t histLow_; + uint32_t histHigh_; + + double prevTotalExposure_; + double prevTotalExposureNoDg_; + double currentTotalExposure_; + double currentTotalExposureNoDg_; +}; + +} /* namespace ipa */ + +} /* namespace libcamera */ + +#endif /* __LIBCAMERA_IPU3_AGC_H__ */ diff --git a/src/ipa/ipu3/meson.build b/src/ipa/ipu3/meson.build index 07a864c8..43ad0e0d 100644 --- a/src/ipa/ipu3/meson.build +++ b/src/ipa/ipu3/meson.build @@ -4,6 +4,7 @@ ipa_name = 'ipa_ipu3' ipu3_ipa_sources = files([ 'ipu3.cpp', + 'ipu3_agc.cpp', 'ipu3_awb.cpp', ])