From patchwork Mon Sep 13 14:58:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13829 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 6491CBDB1D for ; Mon, 13 Sep 2021 14:58:19 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C132F6919A; Mon, 13 Sep 2021 16:58:16 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="lhgMhTos"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 48A5C6916F for ; Mon, 13 Sep 2021 16:58:14 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:edc5:688b:2ede:8b4b]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id DB20B510; Mon, 13 Sep 2021 16:58:13 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631545094; bh=2W/X5O5p+CdyJRpLIEzDukAF1PpMvb4JKDB2ylT0s2c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lhgMhTosiT0YCNEPgJnnYWsVAmXEPPCuJPNFhaeJN9Gvsxq9jyGnmB0fM9uKWW7af meMlzo/O01kAUdAWXBm9Q1K4Xiz0wDgqM4oWYkpwedNRU4sRPCs7iUSNjXaTGaf3vy aQVHkYaX1f1gT+gWA2IeXqcJ9fAz2AF4bV2ggof8= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 13 Sep 2021 16:58:00 +0200 Message-Id: <20210913145810.66515-2-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> References: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 01/11] ipa: ipu3: Document IPAIPU3 class interface X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The IPU3 IPA is maturing to a modular and extensible system capable of handling specific algorithms for the accelleration clusters on the ImgU. Provide a top-level class documentation to provide an overview of the IPA, detailing what events are used and what algorithms are currently supported, as well as the limitations currently imposed. Signed-off-by: Jean-Michel Hautbois Signed-off-by: Kieran Bingham --- src/ipa/ipu3/ipu3.cpp | 64 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index c925cf64..90053069 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -145,6 +145,70 @@ LOG_DEFINE_CATEGORY(IPAIPU3) namespace ipa::ipu3 { +/** + * \brief The IPU3 IPA implementation + * + * The IPU3 Pipeline defines an IPU3 specific interface for communication + * between the PipelineHandler, and the IPA module. + * + * We extend the IPAIPU3Interface to implement our algorithms and handle events + * from the IPU3 PipelineHandler to satisfy requests from the application. + * + * At initialisation time, a CameraSensorHelper is instantiated to support + * camera specific calculations, while the default controls are computed, and + * the algorithms are constructed and placed in an ordered list. + * + * The IPU3 ImgU operates with a grid layout to divide the overall frame into + * rectangular cells of pixels. When the IPA is configured, we determine the + * best grid for the statistics based on the pipeline handler Bayer Down Scaler + * output size. + * + * Two main events are then handled to facilitate the operation of the IPU3 ImgU + * by populating its parameter buffer, and adapting the settings of the sensor + * attached to the IPU3 CIO2 through sensor specific V4L2 controls. + * + * When the event \a EventFillParams occurs we populate the ImgU parameter + * buffer with settings to configure the device in preparation for handling the + * frame queued in the Request. + * + * When the frame has completed processing, the ImgU will generate a statistics + * buffer which is given to the IPA as part of the \a EventStatReady event. At + * this event we run the algorithms to parse the statistics and cache any + * results for the next \a EventFillParams event. + * + * The individual algorithms are split into modular components that are called + * iteratively to allow them to process statistics from the ImgU in a defined + * order. + * + * The current implementation supports three core algorithms: + * - Automatic white balance (AWB) + * - Automatic gain and exposure control (AGC) + * - Tonemapping (Gamma) + * + * AWB is implemented using a Greyworld algorithm, and calculates the red and + * blue gains to apply to generate a neutral grey frame overall. + * + * AGC is handled by calculating a histogram of the green channel to estimate an + * analogue gain and shutter time which will provide a well exposed frame. An + * IIR filter is used to smooth the changes to the sensor to reduce perceivable + * steps. + * + * The Tonemapping algorithm provides a gamma correction table to improve the + * contrast of the scene. + * + * The IPU3 ImgU has further accellerator clusters to support image quality + * improvements through bayer and temporal noise reductions, however those are + * not supported in the current implementation, and will use default settings as + * provided by the kernel driver. + * + * Demosaicing is operating on the default values and could be further optimised + * to provide improved sharpening coefficients, checker artifact removal, and + * false color correction. + * + * Additional image enhancements can be made by providing lens and sensor + * specific tuning to adapt for Black Level compensation (BLC), Lens shading + * correction (SHD) and Color correction (CCM) + */ class IPAIPU3 : public IPAIPU3Interface { public: From patchwork Mon Sep 13 14:58:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13830 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 63B5EBDB1D for ; Mon, 13 Sep 2021 14:58:20 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 66C186919E; Mon, 13 Sep 2021 16:58:17 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="rFtIbZxp"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 648AC69181 for ; Mon, 13 Sep 2021 16:58:14 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:edc5:688b:2ede:8b4b]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 19D4BB2B; Mon, 13 Sep 2021 16:58:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631545094; bh=ev9EYZqNEM+QpWmZ7GJvB0KehupDBJH9gQNrxZjN0dI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rFtIbZxpH19QT1/m71SG0xqrayMLWEdiE+ayu/t7B+SVWMdvomKx5tfxvZByJngdD a5l/23wkvttgeITOi1OoeQsdDBEAXlyIMyHj6YWAn5ChJptIl9aNXohPaFenjZxPlV eovRgBU3ksBSgFyM9TgTsapgNyt+Mb5jHKNbXNxI= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 13 Sep 2021 16:58:01 +0200 Message-Id: <20210913145810.66515-3-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> References: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 02/11] ipa: ipu3: Improve the documentation of BDS grid X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The BDS grid has its own limitations for the log2 width and height for each cell of pixels. Use constexpr for those and comment it to help understanding their usage. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Laurent Pinchart --- src/ipa/ipu3/ipu3.cpp | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 90053069..c71f2a6c 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -345,6 +345,8 @@ int IPAIPU3::start() } /** + * \brief Calculate a grid for the AWB statistics + * * This function calculates a grid for the AWB algorithm in the IPU3 firmware. * Its input is the BDS output size calculated in the ImgU. * It is limited for now to the simplest method: find the lesser error @@ -364,16 +366,24 @@ void IPAIPU3::calculateBdsGrid(const Size &bdsOutputSize) /* Set the BDS output size in the IPAConfiguration structure */ context_.configuration.grid.bdsOutputSize = bdsOutputSize; - for (uint32_t widthShift = 3; widthShift <= 7; ++widthShift) { + /* + * The log2 of the width and height of each cell is limited by the ImgU + * in the interval [3, 7] according to the kernel header. + */ + constexpr uint32_t kCellMin = 3; + constexpr uint32_t kCellMax = 7; + + for (uint32_t widthShift = kCellMin; widthShift <= kCellMax; ++widthShift) { uint32_t width = std::min(kMaxCellWidthPerSet, bdsOutputSize.width >> widthShift); width = width << widthShift; - for (uint32_t heightShift = 3; heightShift <= 7; ++heightShift) { + + for (uint32_t heightShift = kCellMin; heightShift <= kCellMax; ++heightShift) { int32_t height = std::min(kMaxCellHeightPerSet, bdsOutputSize.height >> heightShift); height = height << heightShift; uint32_t error = std::abs(static_cast(width - bdsOutputSize.width)) - + std::abs(static_cast(height - bdsOutputSize.height)); + + std::abs(static_cast(height - bdsOutputSize.height)); if (error > minError) continue; From patchwork Mon Sep 13 14:58:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13831 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 28B40BDB1D for ; Mon, 13 Sep 2021 14:58:21 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5EAA86918F; Mon, 13 Sep 2021 16:58:18 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="nDH20rNK"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9868769184 for ; Mon, 13 Sep 2021 16:58:14 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:edc5:688b:2ede:8b4b]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 48600499; Mon, 13 Sep 2021 16:58:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631545094; bh=rDJh2rIFSQDO+LzmtivU1lp6Tgi/ysejHSR1f39pa04=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nDH20rNKfvSSGQ5xGLqJloJWyP5FR0u+9tEKisyYOI6ix+fb5b/gqSu2yg3yB7kj8 1J0kcytOX/Iqt51pjaXLAxTSqHJDjDZipEdNuuiikvf3hWXtP8WdSPlmwl8+kmqC2Q T1gmOHkN/119zM+YvRkdXHIfuqhyeU3+TcxgUJvY= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 13 Sep 2021 16:58:02 +0200 Message-Id: <20210913145810.66515-4-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> References: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 03/11] ipa: ipu3: Document IPAIPU3::configure X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Further extend the documentation for the IPAIPU3::configure operation. Signed-off-by: Jean-Michel Hautbois Signed-off-by: Kieran Bingham --- src/ipa/ipu3/ipu3.cpp | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index c71f2a6c..c0004ea6 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -409,6 +409,18 @@ void IPAIPU3::calculateBdsGrid(const Size &bdsOutputSize) << (int)bdsGrid.height << " << " << (int)bdsGrid.block_height_log2 << ")"; } +/** + * \brief Configure the IPU3 IPA. + * \param[in] configInfo The IPA configuration data, received from the pipeline + * handler + * + * Calculate the best grid for the statistics based on the Pipeline Handler BDS + * output, and parse the minimum and maximum exposure and analogue gain control + * values. + * + * All algorithm modules are called to allow them to prepare the + * \a IPASessionConfiguration structure for the \a IPAContext. + */ int IPAIPU3::configure(const IPAConfigInfo &configInfo) { if (configInfo.entityControls.empty()) { From patchwork Mon Sep 13 14:58:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13832 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id BA8FFBDB1D for ; Mon, 13 Sep 2021 14:58:21 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4119A6919F; Mon, 13 Sep 2021 16:58:19 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="rmPeNcXo"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C68706024C for ; Mon, 13 Sep 2021 16:58:14 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:edc5:688b:2ede:8b4b]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 7C259510; Mon, 13 Sep 2021 16:58:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631545094; bh=VxYWF6HJHNZTL376kV1ThaRo0SvqX+/qsp0LUYzhs7g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rmPeNcXo1ZY0WRMt/JQOwIPQKjbxNO4VWoKyjiqdeqDaAXysQ7u24q2MA0uFgOQ6L FrCwEMgIeTLAAKTIi1HQbj1ufmJRH5yz4vNC3sc8j26DZEUh/NiUqRPFUYLUq3JLqs tAfieUmWsfdpw1Jd4zY4eesff/nur1XukiGaApik= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 13 Sep 2021 16:58:03 +0200 Message-Id: <20210913145810.66515-5-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> References: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 04/11] ipa: ipu3: Document the IPAIPU3 class X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Clarify the roles and interactions between the pipeline handler events and the algorithm calls by documenting all the remaining functions of the class. Signed-off-by: Jean-Michel Hautbois Signed-off-by: Kieran Bingham --- src/ipa/ipu3/ipu3.cpp | 65 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 62 insertions(+), 3 deletions(-) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index c0004ea6..df64b0f1 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -262,7 +262,7 @@ private: }; /** - * Initialize the IPA module and its controls. + * \brief Initialize the IPA module and its controls. * * This function receives the camera sensor information from the pipeline * handler, computes the limits of the controls it handles and returns @@ -337,8 +337,15 @@ int IPAIPU3::init(const IPASettings &settings, return 0; } +/** + * \brief Perform any processing required before the first frame. + */ int IPAIPU3::start() { + /* + * Set the sensors V4L2 controls before the first frame to ensure that + * we have an expected and known configuration from the start. + */ setControls(0); return 0; @@ -474,6 +481,10 @@ int IPAIPU3::configure(const IPAConfigInfo &configInfo) return 0; } +/** + * \brief Map the parameters and stats buffers allocated in the Pipeline Handler + * \param[out] buffers A vector with param and stat buffers + */ void IPAIPU3::mapBuffers(const std::vector &buffers) { for (const IPABuffer &buffer : buffers) { @@ -483,6 +494,10 @@ void IPAIPU3::mapBuffers(const std::vector &buffers) } } +/** + * \brief Unmap the parameters and stats buffers + * \param[in] ids A list of buffer ids + */ void IPAIPU3::unmapBuffers(const std::vector &ids) { for (unsigned int id : ids) { @@ -494,6 +509,10 @@ void IPAIPU3::unmapBuffers(const std::vector &ids) } } +/** + * \brief Process events generated by the Pipeline Handler + * \param[in] event An event sent from Pipeline Handler + */ void IPAIPU3::processEvent(const IPU3Event &event) { switch (event.op) { @@ -535,12 +554,26 @@ void IPAIPU3::processEvent(const IPU3Event &event) } } +/** + * \brief Process a control list for a request from the application + * + * Parse the request to handle any IPA managed controls that were set from the + * application such as manual sensor settings. + */ void IPAIPU3::processControls([[maybe_unused]] unsigned int frame, [[maybe_unused]] const ControlList &controls) { /* \todo Start processing for 'frame' based on 'controls'. */ } +/** + * \brief Fill the ImgU parameter buffer for the next frame + * \param[in] frame The current frame number + * \param[inout] params the parameter buffer to update + * + * Algorithms are expected to fill the IPU3 parameter buffer for the next + * frame given their most recent processing of the ImgU statistics. + */ void IPAIPU3::fillParams(unsigned int frame, ipu3_uapi_params *params) { for (auto const &algo : algorithms_) @@ -552,6 +585,16 @@ void IPAIPU3::fillParams(unsigned int frame, ipu3_uapi_params *params) queueFrameAction.emit(frame, op); } +/** + * \brief Process the statistics generated by the ImgU + * \param[in] frame The current frame number + * \param[in] frameTimestamp The current frame timestamp + * \param[in] stats The IPU3 statistics and ISP results + * + * Parse the most recently processed image statistics from the ImgU. The + * statistics are passed to each algorithm module to run their calculations and + * update their state accordingly. + */ void IPAIPU3::parseStatistics(unsigned int frame, [[maybe_unused]] int64_t frameTimestamp, [[maybe_unused]] const ipu3_uapi_stats_3a *stats) @@ -579,6 +622,13 @@ void IPAIPU3::parseStatistics(unsigned int frame, queueFrameAction.emit(frame, op); } +/** + * \brief Handle V4L2 controls for a given \a frame number + * \param[in] frame The frame on which the V4L2 controls should be set + * + * Send the desired sensor control values to the Pipeline handler to request + * that they are applied on the Camera sensor. + */ void IPAIPU3::setControls(unsigned int frame) { IPU3Action op; @@ -597,10 +647,15 @@ void IPAIPU3::setControls(unsigned int frame) } /* namespace ipa::ipu3 */ -/* +/** * External IPA module interface + * + * The IPAModuleInfo is required to match an IPA module construction against the + * intented pipeline handler with the module. The API and Pipeline handler + * versions must match the corresponding IPA interface and pipeline handler. + * + * \sa struct IPAModuleInfo */ - extern "C" { const struct IPAModuleInfo ipaModuleInfo = { IPA_MODULE_API_VERSION, @@ -609,6 +664,10 @@ const struct IPAModuleInfo ipaModuleInfo = { "ipu3", }; +/** + * When matched against with a pipeline handler, the IPAManager will construct + * an IPA instance for each associated Camera. + */ IPAInterface *ipaCreate() { return new ipa::ipu3::IPAIPU3(); From patchwork Mon Sep 13 14:58:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13833 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5A7D1BDB1D for ; Mon, 13 Sep 2021 14:58:22 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9994669195; Mon, 13 Sep 2021 16:58:20 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="LQzfnweu"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0B9D06916F for ; Mon, 13 Sep 2021 16:58:15 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:edc5:688b:2ede:8b4b]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id AEBCD499; Mon, 13 Sep 2021 16:58:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631545094; bh=RQKHVG1Ey3m3oKFmSOjlVgR6IHmAZBd/CKqvo7iXBno=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LQzfnweuwWilfclwwXnM1m4RymzQjfJWbnptpesaLx/OuMbgBY+ydGva40dFPF2Jv ehoGux7SQHP1xVdahchFYieITcrphbFBnoY+/Tz0JHerLwBRqQS8EBgqoqdRIXWRs1 k4aHGxIqu51bFx6urGo9SSiKPBZuE6Ka9qNzMqHY= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 13 Sep 2021 16:58:04 +0200 Message-Id: <20210913145810.66515-6-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> References: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 05/11] ipa: ipu3: Explicitly use the statistics parameter X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The stats pointer is marked as [[maybe_unused]]. This is a leftover from a previous commit which was here to keep the compatibility while transitioning to the new iterative algorithms. Remove this attribute to make it explicit that stats are really used to feed the algorithms. Signed-off-by: Jean-Michel Hautbois Signed-off-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/ipa/ipu3/ipu3.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index df64b0f1..5b487dcd 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -597,7 +597,7 @@ void IPAIPU3::fillParams(unsigned int frame, ipu3_uapi_params *params) */ void IPAIPU3::parseStatistics(unsigned int frame, [[maybe_unused]] int64_t frameTimestamp, - [[maybe_unused]] const ipu3_uapi_stats_3a *stats) + const ipu3_uapi_stats_3a *stats) { ControlList ctrls(controls::controls); From patchwork Mon Sep 13 14:58:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13834 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D0D49BDB1D for ; Mon, 13 Sep 2021 14:58:22 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 75D8D6919B; Mon, 13 Sep 2021 16:58:21 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="WdQYhaCj"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3D6D269181 for ; Mon, 13 Sep 2021 16:58:15 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:edc5:688b:2ede:8b4b]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id E2554510; Mon, 13 Sep 2021 16:58:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631545095; bh=u/R0B1lt/cG9gs3pLw+ld3FyrrYHLqI6ag/XioyEL/8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WdQYhaCjBItiSRWBv51f0LQpJ2ZBYARXiPZRDr5CnAwDhVW/qUJPd5xDw50psICUx JaZKbua+tUKEpKQk9Udr+fh+b/GKnSiM3LMol/Zro8eQIQ3lsiotzk2VbBQVUBKP16 ovGzyMCbEoaHdYwrafF7JN4HVwvsCJSimcxezSUU= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 13 Sep 2021 16:58:05 +0200 Message-Id: <20210913145810.66515-7-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> References: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 06/11] ipa: ipu3: awb: Add AWB class documentation X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The AWB algorithm is based on the Grey world algorithm and uses the statistics generated by the ImgU for that. Explain how it uses those, and reference the original algorithm at the same time. Signed-off-by: Jean-Michel Hautbois --- src/ipa/ipu3/algorithms/awb.cpp | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index 36e80306..887ac4b5 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -11,6 +11,10 @@ #include +/** + * \file awb.h + */ + namespace libcamera { namespace ipa::ipu3::algorithms { @@ -20,6 +24,35 @@ LOG_DEFINE_CATEGORY(IPU3Awb) static constexpr uint32_t kMinZonesCounted = 16; static constexpr uint32_t kMinGreenLevelInZone = 32; +/** + * \class Awb + * \brief A Grey world white balance correction algorithm + * + * The Grey World algorithm assumes that the scene, in average, is neutral grey. + * Reference: Lam, Edmund & Fung, George. (2008). Automatic White Balancing in + * Digital Photography. 10.1201/9781420054538.ch10. + * + * The IPU3 generates statistics from the Bayer Down Scaler output into a grid + * defined in the ipu3_uapi_awb_config_s structure. + * + * For example, when the BDS outputs a frame of 2592x1944, the grid may be + * configured to 81x30 cells each with a size of 32x64 pixels. + * We then have an average of 2048 R, G and B pixels per cell. + * + * The AWB algorithm uses a fixed grid size of kAwbStatsSizeX x kAwbStatsSizeY. + * Each of this new grid cell will be called a zone. + * + * Before calculating the gains, we will convert the statistics from the BDS + * grid to an internal grid configuration in generateAwbStats. + * As part of converting the statistics to an internal grid, the saturation + * flag from the originating grid cell is used to decide if the zone contains + * saturated pixels or not, making the zone relevant or not. + * A saturated zone will be excluded from the calculation. + * + * The Grey World algorithm will then estimate the red and blue gains to apply, and + * store the results in the metadata. + */ + /** * \struct Accumulator * \brief RGB statistics for a given zone From patchwork Mon Sep 13 14:58:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13835 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 99560BDB1D for ; Mon, 13 Sep 2021 14:58:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 67F2A6919D; Mon, 13 Sep 2021 16:58:24 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="DB48oslx"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 61A8269184 for ; Mon, 13 Sep 2021 16:58:15 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:edc5:688b:2ede:8b4b]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 1BB34B2B; Mon, 13 Sep 2021 16:58:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631545095; bh=D1/r6bOWaxa0Aed+GRFPuZVR/7tTNArW0YQN15hDMTU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DB48oslxSEZ2nhgi16IehB32OzkHAFUIXRn3eG87uqpSdF/Ro60WBTPvItQfd93G7 g17p/IpPkJCyEDxnhVgS5smAzr0I/NjKw6NlqTwTVCvnXlP7OZ7AzEgYrmflAilRLR wRnJWIDhvBrYroLa75OdZo1mkG1qfzkVr1w1pwjI= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 13 Sep 2021 16:58:06 +0200 Message-Id: <20210913145810.66515-8-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> References: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 07/11] ipu3: ipa: agc: Determine cell size from ipu3_uapi_awb_set_item X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The kCellSize was hardcoded from the size determined by examining the statistics from the ImgU. The cell definition is now described as a struct. Utilise the struct size to express the cell size. Signed-off-by: Jean-Michel Hautbois --- src/ipa/ipu3/algorithms/agc.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 8740dcdf..e085599a 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -48,7 +48,7 @@ static constexpr uint32_t knumHistogramBins = 256; static constexpr double kEvGainTarget = 0.5; /* A cell is 8 bytes and contains averages for RGB values and saturation ratio */ -static constexpr uint8_t kCellSize = 8; +static constexpr uint8_t kCellSize = sizeof(ipu3_uapi_awb_set_item); Agc::Agc() : frameCount_(0), lastFrame_(0), iqMean_(0.0), lineDuration_(0s), From patchwork Mon Sep 13 14:58:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13836 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id F1F9EBEFBE for ; Mon, 13 Sep 2021 14:58:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BD0B069194; Mon, 13 Sep 2021 16:58:24 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="YIYc2/ZU"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 90E896024C for ; Mon, 13 Sep 2021 16:58:15 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:edc5:688b:2ede:8b4b]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 4B85C499; Mon, 13 Sep 2021 16:58:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631545095; bh=qPlf4t0ZOVRRshTPLFOUUIu8JCdGeeaipQGmrQy6nn4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YIYc2/ZUhZBcCrrXF0ulhPvoQidDogzJ1MxsMnhtsTZ421htwvl5Y3dF24rHiaScI cP6vNa6ZNKD6RpDu/a4mCeumoxgclF6fdtE+JSDahJPVIF4EK4qt2gbN3TawONOPwQ wqN8MkP3bdAmqtRgfkuWceDroCo9HbfOmxWWgd28= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 13 Sep 2021 16:58:07 +0200 Message-Id: <20210913145810.66515-9-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> References: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 08/11] ipa: ipu3: agc: Remove unused variable count X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When processing brightness, a local variable is incremented at each loop. There is no need for it, so remove it. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Laurent Pinchart --- src/ipa/ipu3/algorithms/agc.cpp | 2 -- 1 file changed, 2 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index e085599a..2ef998b7 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -84,7 +84,6 @@ void Agc::processBrightness(const ipu3_uapi_stats_3a *stats, uint32_t startY = topleftY * grid.width << grid.block_width_log2; uint32_t endX = (startX + (aeRegion.size().width >> grid.block_width_log2)) << grid.block_width_log2; uint32_t i, j; - uint32_t count = 0; uint32_t hist[knumHistogramBins] = { 0 }; for (j = topleftY; @@ -101,7 +100,6 @@ void Agc::processBrightness(const ipu3_uapi_stats_3a *stats, uint8_t Gr = currentCell->Gr_avg; uint8_t Gb = currentCell->Gb_avg; hist[(Gr + Gb) / 2]++; - count++; } } } From patchwork Mon Sep 13 14:58:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13837 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 66CBABDB1D for ; Mon, 13 Sep 2021 14:58:25 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2026A69199; Mon, 13 Sep 2021 16:58:25 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ZVZQYLwB"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C561B6918A for ; Mon, 13 Sep 2021 16:58:15 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:edc5:688b:2ede:8b4b]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 7B332510; Mon, 13 Sep 2021 16:58:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631545095; bh=N4WiT6HCg2xzi0XLnQoniIH/xebceEZlDejew+etlxo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZVZQYLwB9z+7LPOeh1OP2WuSX2y4Obg+Jxysp2KJZ6DxBFVNS3JTKatwzyDUgTt/X 2BRVwqa/DSJwuIaSGPuXDmX07IDbsdeJ6wmizg5lnvGkvKH3faFt4zrQ233gmyQgyy /lF/6hNbMZmrW+pMgNV0+z86WWM5KcADgvWAnwpM= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 13 Sep 2021 16:58:08 +0200 Message-Id: <20210913145810.66515-10-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> References: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 09/11] ipa: ipu3: agc: Document AGC mean-based algorithm X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The AGC class was not documented while developing. Extend that to reference the origins of the implementation, and improve the descriptions on how the algorithm operates internally. Signed-off-by: Jean-Michel Hautbois Signed-off-by: Kieran Bingham --- src/ipa/ipu3/algorithms/agc.cpp | 116 +++++++++++++++++++++++++++----- src/ipa/ipu3/algorithms/agc.h | 2 +- 2 files changed, 101 insertions(+), 17 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 2ef998b7..9a55bb72 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -2,7 +2,7 @@ /* * Copyright (C) 2021, Ideas On Board * - * ipu3_agc.cpp - AGC/AEC control algorithm + * ipu3_agc.cpp - AGC/AEC mean-based control algorithm */ #include "agc.h" @@ -17,27 +17,49 @@ #include "libipa/histogram.h" +/** + * \file agc.h + */ + namespace libcamera { using namespace std::literals::chrono_literals; namespace ipa::ipu3::algorithms { +/** + * \class Agc + * \brief A mean-based auto-exposure algorithm + * + * This algorithm calculates a shutter time and a gain so that the average value + * of the green channel of the brightest 2% of pixels approaches 0.5. The AWB + * gains are not used here, and all cells in the grid have the same weight, like + * an average-metering case. In this metering mode, the camera uses light + * information from the entire scene and creates an average for the final + * exposure setting, giving no weighting to any particular portion of the + * metered area. + * + * Reference: Battiato, Messina & Castorina. (2008). Exposure + * Correction for Imaging Devices: An Overview. 10.1201/9781420054538.ch12. + */ + LOG_DEFINE_CATEGORY(IPU3Agc) /* Number of frames to wait before calculating stats on minimum exposure */ static constexpr uint32_t kInitialFrameMinAECount = 4; -/* Number of frames to wait between new gain/exposure estimations */ +/* Number of frames to wait between new gain/shutter time estimations */ static constexpr uint32_t kFrameSkipCount = 6; -/* Maximum ISO value for analogue gain */ -static constexpr uint32_t kMinISO = 100; -static constexpr uint32_t kMaxISO = 1500; - -/* Maximum analogue gain value - * \todo grab it from a camera helper */ -static constexpr uint32_t kMinGain = kMinISO / 100; -static constexpr uint32_t kMaxGain = kMaxISO / 100; +/* + * Minimum analogue gain value + * \todo grab it from a camera helper + */ +static constexpr uint32_t kMinGain = 1; +/* + * Maximum analogue gain value + * \todo grab it from a camera helper + */ +static constexpr uint32_t kMaxGain = 15; /* \todo use calculated value based on sensor */ static constexpr uint32_t kMinExposure = 1; @@ -45,6 +67,7 @@ static constexpr uint32_t kMaxExposure = 1976; /* Histogram constants */ static constexpr uint32_t knumHistogramBins = 256; +/* Target value to reach for the top 2% of the histogram */ static constexpr double kEvGainTarget = 0.5; /* A cell is 8 bytes and contains averages for RGB values and saturation ratio */ @@ -57,9 +80,17 @@ Agc::Agc() { } +/** + * \brief Configure the AGC given a configInfo + * \param[in] context The shared IPA context + * \param[in] configInfo The IPA configuration data + * + * \return 0 + */ int Agc::configure([[maybe_unused]] IPAContext &context, - const IPAConfigInfo &configInfo) + const IPAConfigInfo &configInfo) { + /* \todo use the configInfo fields and IPAContext to store the limits */ lineDuration_ = configInfo.sensorInfo.lineLength * 1.0s / configInfo.sensorInfo.pixelRate; maxExposureTime_ = kMaxExposure * lineDuration_; @@ -67,9 +98,22 @@ int Agc::configure([[maybe_unused]] IPAContext &context, return 0; } +/** + * \brief Estimate the mean quantile of the top 2% of the histogram + * \param[in] stats The statistics computed by the ImgU + * \param[in] grid The grid used to store the statistics in the IPU3 + */ void Agc::processBrightness(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid) { + /* + * Get the applied grid from the statistics buffer. When the kernel + * receives a grid from the parameters buffer, it will check and align + * all the values. For instance, it will automatically fill the x_end + * value based on x_start, grid width and log2 width. + * \todo Use the grid calculated in configure as there is a bug in IPU3 + * causing the width (maybe height) to be bit-shifted. + */ const struct ipu3_uapi_grid_config statsAeGrid = stats->stats_4a_config.awb_config.grid; Rectangle aeRegion = { statsAeGrid.x_start, statsAeGrid.y_start, @@ -85,6 +129,7 @@ void Agc::processBrightness(const ipu3_uapi_stats_3a *stats, uint32_t endX = (startX + (aeRegion.size().width >> grid.block_width_log2)) << grid.block_width_log2; uint32_t i, j; + /* Initialise the histogram array */ uint32_t hist[knumHistogramBins] = { 0 }; for (j = topleftY; j < topleftY + (aeRegion.size().height >> grid.block_height_log2); @@ -92,13 +137,18 @@ void Agc::processBrightness(const ipu3_uapi_stats_3a *stats, for (i = startX + startY; i < endX + startY; i += kCellSize) { /* * The grid width (and maybe height) is not reliable. - * We observed a bit shift which makes the value 160 to be 32 in the stats grid. - * Use the one passed at init time. + * We observed a bit shift which makes the value 160 to + * be 32 in the stats grid. Use the one from configure. */ const ipu3_uapi_awb_set_item *currentCell = &stats->awb_raw_buffer.meta_data[i + j * grid.width]; if (currentCell->sat_ratio == 0) { uint8_t Gr = currentCell->Gr_avg; uint8_t Gb = currentCell->Gb_avg; + /* + * Store the average green value to estimate the + * brightness. Even the over exposed pixels are + * taken into account. + */ hist[(Gr + Gb) / 2]++; } } @@ -108,18 +158,24 @@ void Agc::processBrightness(const ipu3_uapi_stats_3a *stats, iqMean_ = Histogram(Span(hist)).interQuantileMean(0.98, 1.0); } +/** + * \brief Apply a filter on the exposure value to limit the speed of changes + */ void Agc::filterExposure() { double speed = 0.2; if (prevExposure_ == 0s) { - /* DG stands for digital gain.*/ + /* + * DG stands for digital gain, which is always 1.0 for now as it + * is not implemented right now. + */ prevExposure_ = currentExposure_; prevExposureNoDg_ = currentExposureNoDg_; } else { /* * If we are close to the desired result, go faster to avoid making * multiple micro-adjustments. - * \ todo: Make this customisable? + * \todo: Make this customisable? */ if (prevExposure_ < 1.2 * currentExposure_ && prevExposure_ > 0.8 * currentExposure_) @@ -130,10 +186,12 @@ void Agc::filterExposure() prevExposureNoDg_ = speed * currentExposureNoDg_ + prevExposureNoDg_ * (1.0 - speed); } + /* * We can't let the no_dg exposure deviate too far below the * total exposure, as there might not be enough digital gain available * in the ISP to hide it (which will cause nasty oscillation). + * \todo implement digital gain setting */ double fastReduceThreshold = 0.4; if (prevExposureNoDg_ < @@ -142,6 +200,11 @@ void Agc::filterExposure() LOG(IPU3Agc, Debug) << "After filtering, total_exposure " << prevExposure_; } +/** + * \brief Estimate the new exposure and gain values + * \param[inout] exposure The exposure value reference as a number of lines + * \param[inout] gain The gain reference to be updated + */ void Agc::lockExposureGain(uint32_t &exposure, double &gain) { /* Algorithm initialization should wait for first valid frames */ @@ -154,22 +217,33 @@ void Agc::lockExposureGain(uint32_t &exposure, double &gain) if (std::abs(iqMean_ - kEvGainTarget * knumHistogramBins) <= 1) { LOG(IPU3Agc, Debug) << "!!! Good exposure with iqMean = " << iqMean_; } else { + /* Estimate the gain needed to have the proportion wanted */ double newGain = kEvGainTarget * knumHistogramBins / iqMean_; /* extracted from Rpi::Agc::computeTargetExposure */ + /* Calculate the shutter time in seconds */ libcamera::utils::Duration currentShutter = exposure * lineDuration_; + + /* + * Estimate the current exposure value for the scene as shutter + * time multiplicated by the analogue gain. + */ currentExposureNoDg_ = currentShutter * gain; LOG(IPU3Agc, Debug) << "Actual total exposure " << currentExposureNoDg_ << " Shutter speed " << currentShutter << " Gain " << gain; + + /* Apply the gain calculated to the current exposure value */ currentExposure_ = currentExposureNoDg_ * newGain; + + /* Clamp the exposure value to the min and max authorized */ libcamera::utils::Duration maxTotalExposure = maxExposureTime_ * kMaxGain; currentExposure_ = std::min(currentExposure_, maxTotalExposure); LOG(IPU3Agc, Debug) << "Target total exposure " << currentExposure_; - /* \todo: estimate if we need to desaturate */ filterExposure(); + /* Divide the exposure value as new exposure and gain values */ libcamera::utils::Duration newExposure = 0.0s; if (currentShutter < maxExposureTime_) { exposure = std::clamp(static_cast(exposure * currentExposure_ / currentExposureNoDg_), kMinExposure, kMaxExposure); @@ -185,10 +259,20 @@ void Agc::lockExposureGain(uint32_t &exposure, double &gain) lastFrame_ = frameCount_; } +/** + * \brief Process IPU3 statistics, and run AGC operations + * \param[in] context The shared IPA context + * \param[in] stats The IPU3 statistics and ISP results + * + * Identify the current image brightness, and use that to estimate the optimal + * new exposure and gain for the scene. + */ void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) { + /* Get the latest exposure and gain applied */ uint32_t &exposure = context.frameContext.agc.exposure; double &gain = context.frameContext.agc.gain; + processBrightness(stats, context.configuration.grid.bdsGrid); lockExposureGain(exposure, gain); frameCount_++; diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index e36797d3..9449ba48 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -2,7 +2,7 @@ /* * Copyright (C) 2021, Ideas On Board * - * agc.h - IPU3 AGC/AEC control algorithm + * agc.h - IPU3 AGC/AEC mean-based control algorithm */ #ifndef __LIBCAMERA_IPU3_ALGORITHMS_AGC_H__ #define __LIBCAMERA_IPU3_ALGORITHMS_AGC_H__ From patchwork Mon Sep 13 14:58:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13838 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B5C48BEFBE for ; Mon, 13 Sep 2021 14:58:25 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 7304D6919C; Mon, 13 Sep 2021 16:58:25 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="tUPXaoD6"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 09E416916F for ; Mon, 13 Sep 2021 16:58:16 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:edc5:688b:2ede:8b4b]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id AE531499; Mon, 13 Sep 2021 16:58:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631545095; bh=u7Pe7c5yntjEsJ6hw0iOP5bOGktGU3TeCdshjm76Oh0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tUPXaoD6CbZjC4aajLYnuDnx60smLCuutVd0oxvrKn2MxFBj31k7JAXI5atcfV4Li 3vnYq3iE0A2ol3yUdgCLqHE9Y1Ih9ayd/6HX99zkaPszdp0m/UtVO83Gri9VHK+aTU ki3TdbShGnMFDb1ugSuMMsX9QgQj/pMDKrvVULs8= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 13 Sep 2021 16:58:09 +0200 Message-Id: <20210913145810.66515-11-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> References: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 10/11] ipa: ipu3: tonemapping: Generate the LUT only on gamma change X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The Tonemapping algorithm calculates the gamma curve for every frame, regardless of whether the gamma value has changed or not. This issue is exasperated as we currently hardcode the gamma to a single value. Optimise the implementation to only recalculate the look up table when the gamma setting is changed, and store the gamma setting of the LUT curve as part of the IPA context. Signed-off-by: Jean-Michel Hautbois Signed-off-by: Kieran Bingham --- src/ipa/ipu3/algorithms/tone_mapping.cpp | 5 +++++ src/ipa/ipu3/ipa_context.h | 1 + src/ipa/ipu3/ipu3.cpp | 3 +++ 3 files changed, 9 insertions(+) diff --git a/src/ipa/ipu3/algorithms/tone_mapping.cpp b/src/ipa/ipu3/algorithms/tone_mapping.cpp index 3af96261..40337f9d 100644 --- a/src/ipa/ipu3/algorithms/tone_mapping.cpp +++ b/src/ipa/ipu3/algorithms/tone_mapping.cpp @@ -43,6 +43,9 @@ void ToneMapping::process([[maybe_unused]] IPAContext &context, */ gamma_ = 1.1; + if (context.frameContext.toneMapping.gamma == gamma_) + return; + struct ipu3_uapi_gamma_corr_lut &lut = context.frameContext.toneMapping.gammaCorrection; @@ -53,6 +56,8 @@ void ToneMapping::process([[maybe_unused]] IPAContext &context, /* The output value is expressed on 13 bits. */ lut.lut[i] = gamma * 8191; } + + context.frameContext.toneMapping.gamma = gamma_; } } /* namespace ipa::ipu3::algorithms */ diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h index 9d9444dc..bffa7634 100644 --- a/src/ipa/ipu3/ipa_context.h +++ b/src/ipa/ipu3/ipa_context.h @@ -38,6 +38,7 @@ struct IPAFrameContext { } awb; struct { + double gamma; struct ipu3_uapi_gamma_corr_lut gammaCorrection; } toneMapping; }; diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 5b487dcd..f76627f6 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -129,6 +129,9 @@ * \struct IPAFrameContext::toneMapping * \brief Context for ToneMapping and Gamma control * + * \var IPAFrameContext::toneMapping::gamma + * \brief Gamma value for the LUT + * * \var IPAFrameContext::toneMapping::gammaCorrection * \brief Per-pixel tone mapping implemented as a LUT * From patchwork Mon Sep 13 14:58:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 13839 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 26E78BDB1D for ; Mon, 13 Sep 2021 14:58:26 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id CE1F3691AE; Mon, 13 Sep 2021 16:58:25 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="XTY9uERZ"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 35E8969194 for ; Mon, 13 Sep 2021 16:58:16 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:edc5:688b:2ede:8b4b]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id E17C2510; Mon, 13 Sep 2021 16:58:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631545096; bh=LFh8BNmwrDEmjBPspaawQKvWN2mk1Rd89f6qorKcmug=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XTY9uERZVQqZBhXMjegzM0Do2W0XsSDsbOlTaydEeIDCEhf5KYBP+9jPry/2gP/oX HvMJWYg+c01KNmMkIvh0Ma9L1YIVPR77zUJ0+vGJUo9Zqf6TvGLlWwIWf0nooghIam M99gqBV0I/g476a3yd7MhTnew8rh/11PlrjFiv50= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Mon, 13 Sep 2021 16:58:10 +0200 Message-Id: <20210913145810.66515-12-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> References: <20210913145810.66515-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 11/11] ipa: ipu3: tonemapping: Add the documentation for ToneMapping X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The Tonemapping algorithm is currently undocumented. Provide an introduction and overview to the implementation as the class definition and document how the algorithm operates in the process and prepare methods. Signed-off-by: Jean-Michel Hautbois Signed-off-by: Kieran Bingham --- src/ipa/ipu3/algorithms/tone_mapping.cpp | 35 +++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/src/ipa/ipu3/algorithms/tone_mapping.cpp b/src/ipa/ipu3/algorithms/tone_mapping.cpp index 40337f9d..15e9ab3f 100644 --- a/src/ipa/ipu3/algorithms/tone_mapping.cpp +++ b/src/ipa/ipu3/algorithms/tone_mapping.cpp @@ -10,15 +10,40 @@ #include #include +/** + * \file tone_mapping.h + */ + namespace libcamera { namespace ipa::ipu3::algorithms { +/** + * \class ToneMapping + * \brief A class to handle tone mapping based on gamma + * + * This algorithm improves the image dynamic using a look-up table which is + * generated based on a gamma parameter. + * + * Gamma values less than one have the effect of compressing the image histogram + * while values over 1 will expand it. + * + * Expanding the histogram has the effect of providing better overall contrast. + */ + ToneMapping::ToneMapping() : gamma_(1.0) { } +/** + * \brief Fill in the parameter structure, and enable gamma control + * \param context The shared IPA context + * \param params The IPU3 parameters + * + * Populate the IPU3 parameter structure with our gamma correction table, and + * enable the gamma control module in the accelerator cluster. + */ void ToneMapping::prepare([[maybe_unused]] IPAContext &context, ipu3_uapi_params *params) { @@ -33,7 +58,15 @@ void ToneMapping::prepare([[maybe_unused]] IPAContext &context, params->acc_param.gamma.gc_ctrl.enable = 1; } -void ToneMapping::process([[maybe_unused]] IPAContext &context, +/** + * \brief Calculate the Gamma curve + * \param context The shared IPA context + * \param stats The IPU3 statistics and ISP results + * + * The gamma correction look up table is generated as an inverse power curve + * from our gamma setting. + */ +void ToneMapping::process(IPAContext &context, [[maybe_unused]] const ipu3_uapi_stats_3a *stats) { /*