From patchwork Tue Feb 23 16:40:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 11367 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E353BBD1F1 for ; Tue, 23 Feb 2021 16:40:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C589D68A3A; Tue, 23 Feb 2021 17:40:46 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="vJhAKLD9"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C1BC368A2F for ; Tue, 23 Feb 2021 17:40:43 +0100 (CET) Received: from localhost.localdomain (unknown [IPv6:2a01:e0a:169:7140:d7c9:eb3:c460:c24a]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 63A3E90C; Tue, 23 Feb 2021 17:40:43 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1614098443; bh=jZrk5k22e9JNhGw2NKZ78BZohYFlK5c8a5TKKJMGkpA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vJhAKLD9mStQ/OxQiOr/L2MifDpFBhB4baSMULlwSWsk8ZliaFJizEwDHFyIgL63l TD0c0aFX7zb1Dh1tEJ1b7LsIwRzVRRPimft9xkutCJUYXtM27ZMfIHA2Mu/1OLp7hO B3XOuDvCIJEMR079yPuRYaFPJIJ5993jcxRwPKB4= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Tue, 23 Feb 2021 17:40:39 +0100 Message-Id: <20210223164041.49932-2-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210223164041.49932-1-jeanmichel.hautbois@ideasonboard.com> References: <20210223164041.49932-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH v2 1/3] WIP: ipa: Add a common interface for algorithm objects X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In order to instanciate and use algorithms (AWB, AGC, etc.) there is a need for a common class to define mandatory methods. Instead of reinventing the wheel, reuse what Raspberry Pi has done and adapt to the minimum requirements expected. Signed-off-by: Jean-Michel Hautbois Reviewed-by: Laurent Pinchart --- src/ipa/libipa/algorithm.cpp | 25 +++++++++++++++++++++++++ src/ipa/libipa/algorithm.h | 29 +++++++++++++++++++++++++++++ src/ipa/libipa/meson.build | 4 +++- 3 files changed, 57 insertions(+), 1 deletion(-) create mode 100644 src/ipa/libipa/algorithm.cpp create mode 100644 src/ipa/libipa/algorithm.h diff --git a/src/ipa/libipa/algorithm.cpp b/src/ipa/libipa/algorithm.cpp new file mode 100644 index 00000000..24cd0ae2 --- /dev/null +++ b/src/ipa/libipa/algorithm.cpp @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2019, Raspberry Pi (Trading) Limited + * + * algorithm.cpp - ISP control algorithms + */ + +#include "algorithm.h" + +/** + * \file algorithm.h + * \brief Algorithm common interface + */ + +namespace libcamera { + +namespace ipa { + +void Algorithm::initialise() {} + +void Algorithm::process() {} + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/algorithm.h b/src/ipa/libipa/algorithm.h new file mode 100644 index 00000000..56cd8172 --- /dev/null +++ b/src/ipa/libipa/algorithm.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2019, Raspberry Pi (Trading) Limited + * + * algorithm.h - ISP control algorithm interface + */ +#ifndef __LIBCAMERA_IPA_LIBIPA_ALGORITHM_H__ +#define __LIBCAMERA_IPA_LIBIPA_ALGORITHM_H__ + +namespace libcamera { + +namespace ipa { + +class Algorithm +{ +public: + Algorithm() + { + } + virtual ~Algorithm() = default; + virtual void initialise(); + virtual void process(); +}; + +} /* namespace ipa */ + +} /* namespace libcamera */ + +#endif /* __LIBCAMERA_IPA_LIBIPA_ALGORITHM_H__ */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index b29ef0f4..585d02a3 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -1,13 +1,15 @@ # SPDX-License-Identifier: CC0-1.0 libipa_headers = files([ + 'algorithm.h', ]) libipa_sources = files([ + 'algorithm.cpp', ]) libipa_includes = include_directories('..') -libipa = static_library('ipa', libipa_sources, +libipa = static_library('ipa', [libipa_sources, libipa_headers], include_directories : ipa_includes, dependencies : libcamera_dep) From patchwork Tue Feb 23 16:40:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 11368 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id CB340BD1F1 for ; Tue, 23 Feb 2021 16:40:48 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9007468A3F; Tue, 23 Feb 2021 17:40:47 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="Q5L1NVT9"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E38B268A30 for ; Tue, 23 Feb 2021 17:40:43 +0100 (CET) Received: from localhost.localdomain (unknown [IPv6:2a01:e0a:169:7140:d7c9:eb3:c460:c24a]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 95158968; Tue, 23 Feb 2021 17:40:43 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1614098443; bh=nzw4QLjEJdYoJP5wfoaBoNkRsiLj5ozaSOFSmE0EwLs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Q5L1NVT9j1IWIOONl6Z+LtXLyXLVzPqWQv4UTyO2JuUHq8c4Ibfoo6E3lOBiXTe56 suXQbvgdFJQYQxV7T+k9tpGPK1T1rvjVtaQ2as+uKsi1Lc3GGykdSLg+tNQLhX8R/T dHjGZTUwDlBN4ysLa6CPmL+I9C6OKsif7gUbWt/8= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Tue, 23 Feb 2021 17:40:40 +0100 Message-Id: <20210223164041.49932-3-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210223164041.49932-1-jeanmichel.hautbois@ideasonboard.com> References: <20210223164041.49932-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH v2 2/3] WIP: ipa: ipu3: Add support for IPU3 AWB algorithm X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Inherit from the Algorithm class to implement basic AWB functions. Once AWB is done, a color temperature is estimated and default CCM matrices are used (yet to be tuned). Implement a basic "grey-world" AWB algorithm just for demonstration purpose. Signed-off-by: Jean-Michel Hautbois --- src/ipa/ipu3/ipu3.cpp | 26 +++-- src/ipa/ipu3/ipu3_awb.cpp | 199 ++++++++++++++++++++++++++++++++++++++ src/ipa/ipu3/ipu3_awb.h | 47 +++++++++ src/ipa/ipu3/meson.build | 7 +- 4 files changed, 270 insertions(+), 9 deletions(-) create mode 100644 src/ipa/ipu3/ipu3_awb.cpp create mode 100644 src/ipa/ipu3/ipu3_awb.h diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index b63e58be..6fae5160 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -21,6 +21,8 @@ #include "libcamera/internal/buffer.h" #include "libcamera/internal/log.h" +#include "ipu3_awb.h" + namespace libcamera { LOG_DEFINE_CATEGORY(IPAIPU3) @@ -60,6 +62,11 @@ private: uint32_t gain_; uint32_t minGain_; uint32_t maxGain_; + + /* Interface to the AWB algorithm */ + ipa::IPU3Awb *awbAlgo_; + /* Local parameter storage */ + ipu3_uapi_params params_; }; void IPAIPU3::configure(const std::map &entityControls) @@ -83,11 +90,16 @@ void IPAIPU3::configure(const std::map &entityControls minExposure_ = std::max(itExp->second.min().get(), 1); maxExposure_ = itExp->second.max().get(); - exposure_ = maxExposure_; + exposure_ = minExposure_; minGain_ = std::max(itGain->second.min().get(), 1); maxGain_ = itGain->second.max().get(); - gain_ = maxGain_; + gain_ = minGain_; + + params_ = {}; + + awbAlgo_ = new ipa::IPU3Awb(); + awbAlgo_->initialise(params_); setControls(0); } @@ -161,10 +173,8 @@ void IPAIPU3::processControls([[maybe_unused]] unsigned int frame, void IPAIPU3::fillParams(unsigned int frame, ipu3_uapi_params *params) { - /* Prepare parameters buffer. */ - memset(params, 0, sizeof(*params)); - - /* \todo Fill in parameters buffer. */ + awbAlgo_->updateWbParameters(params_); + *params = params_; ipa::ipu3::IPU3Action op; op.op = ipa::ipu3::ActionParamFilled; @@ -177,8 +187,8 @@ void IPAIPU3::parseStatistics(unsigned int frame, { ControlList ctrls(controls::controls); - /* \todo React to statistics and update internal state machine. */ - /* \todo Add meta-data information to ctrls. */ + awbAlgo_->calculateWBGains(Rectangle(250, 160, 800, 400), stats); + setControls(frame); ipa::ipu3::IPU3Action op; op.op = ipa::ipu3::ActionMetadataReady; diff --git a/src/ipa/ipu3/ipu3_awb.cpp b/src/ipa/ipu3/ipu3_awb.cpp new file mode 100644 index 00000000..3ff239c0 --- /dev/null +++ b/src/ipa/ipu3/ipu3_awb.cpp @@ -0,0 +1,199 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2019, Raspberry Pi (Trading) Limited + * + * ipu3_awb.cpp - AWB control algorithm + */ +#include +#include + +#include "libcamera/internal/log.h" + +#include "ipu3_awb.h" + +namespace libcamera { + +namespace ipa { + +LOG_DEFINE_CATEGORY(IPU3Awb) + +static const struct ipu3_uapi_bnr_static_config imgu_css_bnr_defaults = { + { 16, 16, 16, 16 }, /* wb_gains */ + { 255, 255, 255, 255 }, /* wb_gains_thr */ + { 0, 0, 8, 6, 0, 14 }, /* thr_coeffs */ + { 0, 0, 0, 0 }, /* thr_ctrl_shd */ + { -648, 0, -366, 0 }, /* opt_center */ + { /* lut */ + { 17, 23, 28, 32, 36, 39, 42, 45, + 48, 51, 53, 55, 58, 60, 62, 64, + 66, 68, 70, 72, 73, 75, 77, 78, + 80, 82, 83, 85, 86, 88, 89, 90 } }, + { 4, 0, 1, 8, 0, 8, 0, 8, 0 }, /* bp_ctrl */ + { 8, 4, 4, 0, 8, 0, 1, 1, 1, 1, 0 }, /* dn_detect_ctrl */ + 1296, + { 419904, 133956 }, +}; + +/* settings for Auto White Balance */ +static const struct ipu3_uapi_awb_config_s imgu_css_awb_defaults = { + 8191, + 8191, + 8191, + 8191 | /* rgbs_thr_gr/r/gb/b */ + IPU3_UAPI_AWB_RGBS_THR_B_EN | IPU3_UAPI_AWB_RGBS_THR_B_INCL_SAT, + .grid = { + .width = 160, + .height = 45, + .block_width_log2 = 3, + .block_height_log2 = 4, + .x_start = 0, + .y_start = 0, + }, +}; + +static const struct ipu3_uapi_ccm_mat_config imgu_css_ccm_6000k = { + 7239, -750, -37, 0, + -215, 8196, -200, 0, + -70, -589, 6810, 0 +}; + +static const struct ipu3_uapi_ccm_mat_config imgu_css_ccm_3800k = { + 7379, -526, -296, 0, + -411, 7397, -415, 0, + -224, -564, 7244, 0 +}; + +IPU3Awb::IPU3Awb() + : Algorithm() +{ +} + +IPU3Awb::~IPU3Awb() +{ +} + +void IPU3Awb::initialise() +{ +} + +void IPU3Awb::initialise(ipu3_uapi_params ¶ms) +{ + params.use.acc_awb = 1; + /*\todo fill the grid calculated based on BDS configuration */ + params.acc_param.awb.config = imgu_css_awb_defaults; + + params.use.acc_bnr = 1; + params.acc_param.bnr = imgu_css_bnr_defaults; + + params.use.acc_ccm = 1; + params.acc_param.ccm = imgu_css_ccm_3800k; + + params.use.acc_gamma = 1; + params.acc_param.gamma.gc_ctrl.enable = 1; + + uint32_t a = (32 * 245) / (245 - 9); + + for (uint32_t i = 0; i < 10; i++) + params.acc_param.gamma.gc_lut.lut[i] = 0; + for (uint32_t i = 10; i < 245; i++) + params.acc_param.gamma.gc_lut.lut[i] = a * i + (0 - a * 9); + for (uint32_t i = 245; i < 255; i++) + params.acc_param.gamma.gc_lut.lut[i] = 32 * 245; + + wbGains_[0] = 8192 * 0.8; + wbGains_[1] = 8192; + wbGains_[2] = 8192; + wbGains_[3] = 8192 * 0.8; + + frame_count_ = 0; +} + +uint32_t IPU3Awb::estimateCCT(uint8_t red, uint8_t green, uint8_t blue) +{ + double X = (-0.14282) * (red) + (1.54924) * (green) + (-0.95641) * (blue); + double Y = (-0.32466) * (red) + (1.57837) * (green) + (-0.73191) * (blue); + double Z = (-0.68202) * (red) + (0.77073) * (green) + (0.56332) * (blue); + + double x = X / (X + Y + Z); + double y = Y / (X + Y + Z); + + double n = (x - 0.3320) / (0.1858 - y); + return static_cast(449 * n * n * n + 3525 * n * n + 6823.3 * n + 5520.33); +} + +void IPU3Awb::calculateWBGains([[maybe_unused]] Rectangle roi, + const ipu3_uapi_stats_3a *stats) +{ + std::vector redValues, greenRedValues, greenBlueValues, blueValues; + Point topleft = roi.topLeft(); + uint32_t startY = (topleft.y / 16) * 160 * 8; + uint32_t startX = (topleft.x / 8) * 8; + uint32_t endX = (startX + (roi.size().width / 8)) * 8; + + for (uint32_t j = (topleft.y / 16); j < (topleft.y / 16) + (roi.size().height / 16); j++) { + for (uint32_t i = startX + startY; i < endX + startY; i += 8) { + greenRedValues.push_back(stats->awb_raw_buffer.meta_data[i]); + redValues.push_back(stats->awb_raw_buffer.meta_data[i + 1]); + blueValues.push_back(stats->awb_raw_buffer.meta_data[i + 2]); + greenBlueValues.push_back(stats->awb_raw_buffer.meta_data[i + 3]); + } + } + + std::sort(redValues.begin(), redValues.end()); + std::sort(greenRedValues.begin(), greenRedValues.end()); + std::sort(blueValues.begin(), blueValues.end()); + std::sort(greenBlueValues.begin(), greenBlueValues.end()); + + double Grmed = greenRedValues[greenRedValues.size() / 2]; + double Rmed = redValues[redValues.size() / 2]; + double Bmed = blueValues[blueValues.size() / 2]; + double Gbmed = greenBlueValues[greenBlueValues.size() / 2]; + + double Rgain = Grmed / Rmed; + double Bgain = Gbmed / Bmed; + LOG(IPU3Awb, Debug) << "max R, Gr, B, Gb: " + << redValues.back() << "," + << greenRedValues.back() << "," + << blueValues.back() << "," + << greenBlueValues.back(); + tint_ = ((Rmed / Grmed) + (Bmed / Gbmed)) / 2; + + /* \todo Those are corrections when light is really low + * it should be taken into account by AGC somehow */ + if ((Rgain >= 2) && (Bgain < 2)) { + wbGains_[0] = 4096 * tint_; + wbGains_[1] = 8192 * Rgain; + wbGains_[2] = 4096 * Bgain; + wbGains_[3] = 4096 * tint_; + } else if ((Bgain >= 2) && (Rgain < 2)) { + wbGains_[0] = 4096 * tint_; + wbGains_[1] = 4096 * Rgain; + wbGains_[2] = 8192 * Bgain; + wbGains_[3] = 4096 * tint_; + } else { + wbGains_[0] = 8192 * tint_; + wbGains_[1] = 8192 * Rgain; + wbGains_[2] = 8192 * Bgain; + wbGains_[3] = 8192 * tint_; + } + + frame_count_++; + + cct_ = estimateCCT(Rmed, (Grmed + Gbmed) / 2, Bmed); +} + +void IPU3Awb::updateWbParameters(ipu3_uapi_params ¶ms) +{ + params.acc_param.bnr.wb_gains.gr = wbGains_[0]; + params.acc_param.bnr.wb_gains.r = wbGains_[1]; + params.acc_param.bnr.wb_gains.b = wbGains_[2]; + params.acc_param.bnr.wb_gains.gb = wbGains_[3]; + if (cct_ < 5500) + params.acc_param.ccm = imgu_css_ccm_3800k; + else + params.acc_param.ccm = imgu_css_ccm_6000k; +} + +} /* namespace ipa */ + +} /* namespace libcamera */ \ No newline at end of file diff --git a/src/ipa/ipu3/ipu3_awb.h b/src/ipa/ipu3/ipu3_awb.h new file mode 100644 index 00000000..ff6906f2 --- /dev/null +++ b/src/ipa/ipu3/ipu3_awb.h @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2019, Raspberry Pi (Trading) Limited + * + * ipu3_awb.h - IPU3 AWB control algorithm + */ +#ifndef __LIBCAMERA_IPU3_AWB_H__ +#define __LIBCAMERA_IPU3_AWB_H__ + +#include + +#include + +#include "libipa/algorithm.h" + +namespace libcamera { + +namespace ipa { + +class IPU3Awb : public Algorithm +{ +public: + IPU3Awb(); + ~IPU3Awb(); + + void initialise() override; + + void initialise(ipu3_uapi_params ¶ms); + void calculateWBGains(Rectangle roi, + const ipu3_uapi_stats_3a *stats); + void updateWbParameters(ipu3_uapi_params ¶ms); + +private: + uint32_t estimateCCT(uint8_t red, uint8_t green, uint8_t blue); + + /* WB calculated gains */ + uint16_t wbGains_[4]; + double tint_; + uint32_t cct_; + + uint32_t frame_count_; +}; + +} /* namespace ipa */ + +} /* namespace libcamera*/ +#endif /* __LIBCAMERA_IPU3_AWB_H__ */ diff --git a/src/ipa/ipu3/meson.build b/src/ipa/ipu3/meson.build index a241f617..07a864c8 100644 --- a/src/ipa/ipu3/meson.build +++ b/src/ipa/ipu3/meson.build @@ -2,8 +2,13 @@ ipa_name = 'ipa_ipu3' +ipu3_ipa_sources = files([ + 'ipu3.cpp', + 'ipu3_awb.cpp', +]) + mod = shared_module(ipa_name, - ['ipu3.cpp', libcamera_generated_ipa_headers], + [ipu3_ipa_sources, libcamera_generated_ipa_headers], name_prefix : '', include_directories : [ipa_includes, libipa_includes], dependencies : libcamera_dep, From patchwork Tue Feb 23 16:40:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 11369 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 65734BD1F1 for ; Tue, 23 Feb 2021 16:40:49 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 715F768A35; Tue, 23 Feb 2021 17:40:48 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ORzSSGu/"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2490E68A31 for ; Tue, 23 Feb 2021 17:40:44 +0100 (CET) Received: from localhost.localdomain (unknown [IPv6:2a01:e0a:169:7140:d7c9:eb3:c460:c24a]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C6F8B9E2; Tue, 23 Feb 2021 17:40:43 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1614098443; bh=2KXdTw7glBCyYisb7cqKMHtmm379qQ3rGt3x/iJ+ORY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ORzSSGu//UST0P04CwlyHc2ti/ORM0zxzIqSIAcwxqzyZKZX9j1CXFvg9iQcYrSaR mE0x3JdqsZ+diPMYjVwyhJ/lsyqWwPDIDR0Orxvb975j4fXlrPd37nbfl0IVW1eoxZ 4BYGotz4b9aJWE6QAbt2OLtIMYWuyYxAiMxT2BSw= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Tue, 23 Feb 2021 17:40:41 +0100 Message-Id: <20210223164041.49932-4-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210223164041.49932-1-jeanmichel.hautbois@ideasonboard.com> References: <20210223164041.49932-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH v2 3/3] WIP: ipa: ipu3: Add support for IPU3 AEC/AGC algorithm X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Inherit from the Algorithm class to implement basic auto-exposure and auto-gain functions. While exposure is not locked, AWB is not calculated and corrected. Implement a basic "skewness-based" for demonstration purpose. Signed-off-by: Jean-Michel Hautbois --- src/ipa/ipu3/ipu3.cpp | 10 ++- src/ipa/ipu3/ipu3_agc.cpp | 171 ++++++++++++++++++++++++++++++++++++++ src/ipa/ipu3/ipu3_agc.h | 62 ++++++++++++++ src/ipa/ipu3/meson.build | 1 + 4 files changed, 243 insertions(+), 1 deletion(-) create mode 100644 src/ipa/ipu3/ipu3_agc.cpp create mode 100644 src/ipa/ipu3/ipu3_agc.h diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 6fae5160..cabd0c71 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -21,6 +21,7 @@ #include "libcamera/internal/buffer.h" #include "libcamera/internal/log.h" +#include "ipu3_agc.h" #include "ipu3_awb.h" namespace libcamera { @@ -65,6 +66,8 @@ private: /* Interface to the AWB algorithm */ ipa::IPU3Awb *awbAlgo_; + /* Interface to the AEC/AGC algorithm */ + ipa::IPU3Agc *agcAlgo_; /* Local parameter storage */ ipu3_uapi_params params_; }; @@ -101,6 +104,8 @@ void IPAIPU3::configure(const std::map &entityControls awbAlgo_ = new ipa::IPU3Awb(); awbAlgo_->initialise(params_); + agcAlgo_ = new ipa::IPU3Agc(); + setControls(0); } @@ -187,7 +192,10 @@ void IPAIPU3::parseStatistics(unsigned int frame, { ControlList ctrls(controls::controls); - awbAlgo_->calculateWBGains(Rectangle(250, 160, 800, 400), stats); + agcAlgo_->process(stats, exposure_, gain_); + if (agcAlgo_->converged()) + awbAlgo_->calculateWBGains(Rectangle(250, 160, 800, 400), stats); + setControls(frame); ipa::ipu3::IPU3Action op; diff --git a/src/ipa/ipu3/ipu3_agc.cpp b/src/ipa/ipu3/ipu3_agc.cpp new file mode 100644 index 00000000..2636e2ea --- /dev/null +++ b/src/ipa/ipu3/ipu3_agc.cpp @@ -0,0 +1,171 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2019, Raspberry Pi (Trading) Limited + * + * ipu3_agc.cpp - AGC/AEC control algorithm + */ + +#include "ipu3_agc.h" + +#include + +#include "libcamera/internal/log.h" + +namespace libcamera { + +namespace ipa { + +LOG_DEFINE_CATEGORY(IPU3Agc) + +/* Number of frames to wait before calculating stats on minimum exposure */ +static const uint32_t kInitialFrameMinAECount = 4; +/* Number of frames to wait before calculating stats on maximum exposure */ +static const uint32_t kInitialFrameMaxAECount = 8; +/* Number of frames to wait before calculating stats and estimate gain/exposure */ +static const uint32_t kInitialFrameSkipCount = 12; + +/* Number of frames to wait between new gain/exposure estimations */ +static const uint32_t kFrameSkipCount = 4; + +IPU3Agc::IPU3Agc() + : frameCount_(0), converged_(false) +{ +} + +IPU3Agc::~IPU3Agc() +{ +} + +void IPU3Agc::initialise() +{ +} + +void IPU3Agc::process() +{ +} + +/* + * \todo This function is taken from numerical recipes and calculates all + * moments. It needs to be rewritten properly and maybe in a "math" class ? + */ +void IPU3Agc::moments(std::unordered_map &data, int n) +{ + int j; + double ep = 0.0, s, p; + double ave, adev, sdev; + double var, skew, curt; + + s = 0.0; + for (j = 1; j <= n; j++) + s += data[j]; + + ave = s / n; + adev = var = skew = curt = 0.0; + + for (j = 1; j <= n; j++) { + adev += s = data[j] - (ave); + ep += s; + var += (p = s * s); + skew += (p *= s); + curt += (p *= s); + } + + adev /= n; + var = (var - ep * ep / n) / (n - 1); + sdev = std::sqrt(var); + + if (var) { + skew /= n * var * sdev; + curt = curt / (n * var * var) - 3.0; + } + skew_ = skew; +} + +void IPU3Agc::processBrightness(const ipu3_uapi_stats_3a *stats) +{ + cellsBrightness_.clear(); + + /*\todo Replace constant values with real BDS configuration */ + for (uint32_t j = 0; j < 45; j++) { + for (uint32_t i = 0; i < 160 * 45 * 8; i += 8) { + uint8_t Gr = stats->awb_raw_buffer.meta_data[i]; + uint8_t R = stats->awb_raw_buffer.meta_data[i + 1]; + uint8_t B = stats->awb_raw_buffer.meta_data[i + 2]; + uint8_t Gb = stats->awb_raw_buffer.meta_data[i + 3]; + cellsBrightness_.push_back(static_cast(0.299 * R + 0.587 * (Gr + Gb) / 2 + 0.114 * B)); + } + } + std::sort(cellsBrightness_.begin(), cellsBrightness_.end()); + + /* \todo create a class to generate histograms ! */ + std::unordered_map hist; + for (uint32_t const &val : cellsBrightness_) + hist[val]++; + moments(hist, 256); +} + +/* \todo make this function a math one ? */ +uint32_t IPU3Agc::rootApproximation() +{ + return (currentExposure_ * prevSkew_ + prevExposure_ * currentSkew_) / (prevSkew_ + currentSkew_); +} + +void IPU3Agc::lockExposure(uint32_t &exposure, uint32_t &gain) +{ + /* Algorithm initialization wait for first valid frames */ + /* \todo - have a number of frames given by DelayedControls ? + * - implement a function for IIR */ + if (frameCount_ == kInitialFrameMinAECount) { + prevExposure_ = exposure; + + prevSkew_ = skew_; + /* \todo use configured values */ + exposure = 800; + gain = 8; + currentExposure_ = exposure; + } else if (frameCount_ == kInitialFrameMaxAECount) { + currentSkew_ = skew_; + exposure = rootApproximation(); + nextExposure_ = exposure; + lastFrame_ = frameCount_; + } else if ((frameCount_ >= kInitialFrameSkipCount) && (frameCount_ - lastFrame_ >= kFrameSkipCount)) { + currentSkew_ = skew_; + /* \todo properly calculate a gain */ + if (frameCount_ == kInitialFrameSkipCount) + gain = ((8 * prevSkew_) + (1 * currentSkew_)) / (prevSkew_ + currentSkew_); + + if (currentSkew_ - prevSkew_ > 1) { + /* under exposed */ + prevExposure_ = nextExposure_; + exposure = rootApproximation(); + nextExposure_ = exposure; + } else if (currentSkew_ - prevSkew_ < -1) { + /* over exposed */ + currentExposure_ = nextExposure_; + exposure = rootApproximation(); + nextExposure_ = exposure; + } else { + /* we have converged */ + converged_ = true; + } + lastFrame_ = frameCount_; + prevSkew_ = currentSkew_; + } +} + +void IPU3Agc::process(const ipu3_uapi_stats_3a *stats, uint32_t &exposure, uint32_t &gain) +{ + processBrightness(stats); + if (!converged_) + lockExposure(exposure, gain); + else { + /* Are we still well exposed ? */ + if ((skew_ < 2) || (skew_ > 4)) + converged_ = false; + } + frameCount_++; +} + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/ipu3/ipu3_agc.h b/src/ipa/ipu3/ipu3_agc.h new file mode 100644 index 00000000..b14a2a2f --- /dev/null +++ b/src/ipa/ipu3/ipu3_agc.h @@ -0,0 +1,62 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2019, Raspberry Pi (Trading) Limited + * + * ipu3_agc.h - IPU3 AGC/AEC control algorithm + */ +#ifndef __LIBCAMERA_IPU3_AGC_H__ +#define __LIBCAMERA_IPU3_AGC_H__ + +#include +#include + +#include + +#include + +#include "libipa/algorithm.h" + +namespace libcamera { + +namespace ipa { + +class IPU3Agc : public Algorithm +{ +public: + IPU3Agc(); + ~IPU3Agc(); + + void initialise() override; + void process() override; + + void process(const ipu3_uapi_stats_3a *stats, uint32_t &exposure, uint32_t &gain); + bool converged() { return converged_; } + +private: + void moments(std::unordered_map &data, int n); + void processBrightness(const ipu3_uapi_stats_3a *stats); + uint32_t rootApproximation(); + void lockExposure(uint32_t &exposure, uint32_t &gain); + + uint64_t frameCount_; + uint64_t lastFrame_; + + /* Vector of calculated brightness for each cell */ + std::vector cellsBrightness_; + + /* Values for filtering */ + uint32_t prevExposure_; + uint32_t currentExposure_; + uint32_t nextExposure_; + + double skew_; + double prevSkew_; + double currentSkew_; + bool converged_; +}; + +} /* namespace ipa */ + +} /* namespace libcamera */ + +#endif /* __LIBCAMERA_IPU3_AGC_H__ */ diff --git a/src/ipa/ipu3/meson.build b/src/ipa/ipu3/meson.build index 07a864c8..43ad0e0d 100644 --- a/src/ipa/ipu3/meson.build +++ b/src/ipa/ipu3/meson.build @@ -4,6 +4,7 @@ ipa_name = 'ipa_ipu3' ipu3_ipa_sources = files([ 'ipu3.cpp', + 'ipu3_agc.cpp', 'ipu3_awb.cpp', ])