From patchwork Thu Sep 30 09:55:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Michel Hautbois X-Patchwork-Id: 14002 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9E2C9C3243 for ; Thu, 30 Sep 2021 09:55:29 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8828E691C8; Thu, 30 Sep 2021 11:55:26 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="YUJblwbb"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8212A691A3 for ; Thu, 30 Sep 2021 11:55:18 +0200 (CEST) Received: from tatooine.ideasonboard.com (unknown [IPv6:2a01:e0a:169:7140:bab4:22c5:662d:e478]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 3CF162A8; Thu, 30 Sep 2021 11:55:18 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1632995718; bh=pzB+SmLd/iPDdUY9sZq09VPRGurVnZ0DZvxabPCWeEc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YUJblwbb0SEOeCtqnOPg+jZGD8mvVdKV3By88VFuvMr998Quu0VUXsmJY3DgVoRIG ejso4LvopnHQ9VxXAP8Ejg4/WfJ8UzqQK8YIl9NpQQ9spOQypkLH+VGwIVCL0Wqimx V5/PqGqvXPb0c1RZ6BcIP6SLIUL90ukmZ64PDcDw= From: Jean-Michel Hautbois To: libcamera-devel@lists.libcamera.org Date: Thu, 30 Sep 2021 11:55:01 +0200 Message-Id: <20210930095513.76213-7-jeanmichel.hautbois@ideasonboard.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210930095513.76213-1-jeanmichel.hautbois@ideasonboard.com> References: <20210930095513.76213-1-jeanmichel.hautbois@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 06/18] ipa: ipu3: agc: Document AGC mean-based algorithm X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The AGC class was not documented while developing. Extend that to reference the origins of the implementation, and improve the descriptions on how the algorithm operates internally. Signed-off-by: Jean-Michel Hautbois Signed-off-by: Kieran Bingham --- src/ipa/ipu3/algorithms/agc.cpp | 102 ++++++++++++++++++++++++++++---- src/ipa/ipu3/algorithms/agc.h | 2 +- 2 files changed, 90 insertions(+), 14 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 44008632..2ddbf438 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -2,7 +2,7 @@ /* * Copyright (C) 2021, Ideas On Board * - * ipu3_agc.cpp - AGC/AEC control algorithm + * ipu3_agc.cpp - AGC/AEC mean-based control algorithm */ #include "agc.h" @@ -17,27 +17,49 @@ #include "libipa/histogram.h" +/** + * \file agc.h + */ + namespace libcamera { using namespace std::literals::chrono_literals; namespace ipa::ipu3::algorithms { +/** + * \class Agc + * \brief A mean-based auto-exposure algorithm + * + * This algorithm calculates a shutter time and a gain so that the average value + * of the green channel of the brightest 2% of pixels approaches 0.5. The AWB + * gains are not used here, and all cells in the grid have the same weight, like + * an average-metering case. In this metering mode, the camera uses light + * information from the entire scene and creates an average for the final + * exposure setting, giving no weighting to any particular portion of the + * metered area. + * + * Reference: Battiato, Messina & Castorina. (2008). Exposure + * Correction for Imaging Devices: An Overview. 10.1201/9781420054538.ch12. + */ + LOG_DEFINE_CATEGORY(IPU3Agc) /* Number of frames to wait before calculating stats on minimum exposure */ static constexpr uint32_t kInitialFrameMinAECount = 4; -/* Number of frames to wait between new gain/exposure estimations */ +/* Number of frames to wait between new gain/shutter time estimations */ static constexpr uint32_t kFrameSkipCount = 6; -/* Maximum ISO value for analogue gain */ -static constexpr uint32_t kMinISO = 100; -static constexpr uint32_t kMaxISO = 1500; - -/* Maximum analogue gain value - * \todo grab it from a camera helper */ -static constexpr uint32_t kMinGain = kMinISO / 100; -static constexpr uint32_t kMaxGain = kMaxISO / 100; +/* + * Minimum analogue gain value + * \todo grab it from a camera helper + */ +static constexpr uint32_t kMinGain = 1; +/* + * Maximum analogue gain value + * \todo grab it from a camera helper + */ +static constexpr uint32_t kMaxGain = 15; /* \todo use calculated value based on sensor */ static constexpr uint32_t kMinExposure = 1; @@ -45,6 +67,7 @@ static constexpr uint32_t kMaxExposure = 1976; /* Histogram constants */ static constexpr uint32_t knumHistogramBins = 256; +/* Target value to reach for the top 2% of the histogram */ static constexpr double kEvGainTarget = 0.5; Agc::Agc() @@ -54,10 +77,18 @@ Agc::Agc() { } +/** + * \brief Configure the AGC given a configInfo + * \param[in] context The shared IPA context + * \param[in] configInfo The IPA configuration data + * + * \return 0 + */ int Agc::configure(IPAContext &context, const IPAConfigInfo &configInfo) { stride_ = context.configuration.grid.stride; + /* \todo use the configInfo fields and IPAContext to store the limits */ lineDuration_ = configInfo.sensorInfo.lineLength * 1.0s / configInfo.sensorInfo.pixelRate; maxExposureTime_ = kMaxExposure * lineDuration_; @@ -65,9 +96,15 @@ int Agc::configure(IPAContext &context, const IPAConfigInfo &configInfo) return 0; } +/** + * \brief Estimate the mean quantile of the top 2% of the histogram + * \param[in] stats The statistics computed by the ImgU + * \param[in] grid The grid used to store the statistics in the IPU3 + */ void Agc::processBrightness(const ipu3_uapi_stats_3a *stats, const ipu3_uapi_grid_config &grid) { + /* Initialise the histogram array */ uint32_t hist[knumHistogramBins] = { 0 }; for (unsigned int cellY = 0; cellY < grid.height; cellY++) { @@ -83,6 +120,11 @@ void Agc::processBrightness(const ipu3_uapi_stats_3a *stats, if (cell->sat_ratio == 0) { uint8_t gr = cell->Gr_avg; uint8_t gb = cell->Gb_avg; + /* + * Store the average green value to estimate the + * brightness. Even the over exposed pixels are + * taken into account. + */ hist[(gr + gb) / 2]++; } } @@ -92,18 +134,24 @@ void Agc::processBrightness(const ipu3_uapi_stats_3a *stats, iqMean_ = Histogram(Span(hist)).interQuantileMean(0.98, 1.0); } +/** + * \brief Apply a filter on the exposure value to limit the speed of changes + */ void Agc::filterExposure() { double speed = 0.2; if (prevExposure_ == 0s) { - /* DG stands for digital gain.*/ + /* + * DG stands for digital gain, which is always 1.0 for now as it + * is not implemented right now. + */ prevExposure_ = currentExposure_; prevExposureNoDg_ = currentExposureNoDg_; } else { /* * If we are close to the desired result, go faster to avoid making * multiple micro-adjustments. - * \ todo: Make this customisable? + * \todo: Make this customisable? */ if (prevExposure_ < 1.2 * currentExposure_ && prevExposure_ > 0.8 * currentExposure_) @@ -114,10 +162,12 @@ void Agc::filterExposure() prevExposureNoDg_ = speed * currentExposureNoDg_ + prevExposureNoDg_ * (1.0 - speed); } + /* * We can't let the no_dg exposure deviate too far below the * total exposure, as there might not be enough digital gain available * in the ISP to hide it (which will cause nasty oscillation). + * \todo implement digital gain setting */ double fastReduceThreshold = 0.4; if (prevExposureNoDg_ < @@ -126,6 +176,11 @@ void Agc::filterExposure() LOG(IPU3Agc, Debug) << "After filtering, total_exposure " << prevExposure_; } +/** + * \brief Estimate the new exposure and gain values + * \param[inout] exposure The exposure value reference as a number of lines + * \param[inout] gain The gain reference to be updated + */ void Agc::lockExposureGain(uint32_t &exposure, double &gain) { /* Algorithm initialization should wait for first valid frames */ @@ -138,22 +193,33 @@ void Agc::lockExposureGain(uint32_t &exposure, double &gain) if (std::abs(iqMean_ - kEvGainTarget * knumHistogramBins) <= 1) { LOG(IPU3Agc, Debug) << "!!! Good exposure with iqMean = " << iqMean_; } else { + /* Estimate the gain needed to have the proportion wanted */ double newGain = kEvGainTarget * knumHistogramBins / iqMean_; /* extracted from Rpi::Agc::computeTargetExposure */ + /* Calculate the shutter time in seconds */ libcamera::utils::Duration currentShutter = exposure * lineDuration_; + + /* + * Estimate the current exposure value for the scene as shutter + * time multiplicated by the analogue gain. + */ currentExposureNoDg_ = currentShutter * gain; LOG(IPU3Agc, Debug) << "Actual total exposure " << currentExposureNoDg_ << " Shutter speed " << currentShutter << " Gain " << gain; + + /* Apply the gain calculated to the current exposure value */ currentExposure_ = currentExposureNoDg_ * newGain; + + /* Clamp the exposure value to the min and max authorized */ libcamera::utils::Duration maxTotalExposure = maxExposureTime_ * kMaxGain; currentExposure_ = std::min(currentExposure_, maxTotalExposure); LOG(IPU3Agc, Debug) << "Target total exposure " << currentExposure_; - /* \todo: estimate if we need to desaturate */ filterExposure(); + /* Divide the exposure value as new exposure and gain values */ libcamera::utils::Duration newExposure = 0.0s; if (currentShutter < maxExposureTime_) { exposure = std::clamp(static_cast(exposure * currentExposure_ / currentExposureNoDg_), kMinExposure, kMaxExposure); @@ -169,10 +235,20 @@ void Agc::lockExposureGain(uint32_t &exposure, double &gain) lastFrame_ = frameCount_; } +/** + * \brief Process IPU3 statistics, and run AGC operations + * \param[in] context The shared IPA context + * \param[in] stats The IPU3 statistics and ISP results + * + * Identify the current image brightness, and use that to estimate the optimal + * new exposure and gain for the scene. + */ void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats) { + /* Get the latest exposure and gain applied */ uint32_t &exposure = context.frameContext.agc.exposure; double &gain = context.frameContext.agc.gain; + processBrightness(stats, context.configuration.grid.bdsGrid); lockExposureGain(exposure, gain); frameCount_++; diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 64b71c65..c81ed6e7 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -2,7 +2,7 @@ /* * Copyright (C) 2021, Ideas On Board * - * agc.h - IPU3 AGC/AEC control algorithm + * agc.h - IPU3 AGC/AEC mean-based control algorithm */ #ifndef __LIBCAMERA_IPU3_ALGORITHMS_AGC_H__ #define __LIBCAMERA_IPU3_ALGORITHMS_AGC_H__