[libcamera-devel,v3,4/5] ipa: rkisp1: agc: Add a histogram-based gain

15380 diff mbox series

As for the IPU3, we can estimate the histogram of the luminance. The
RkISP1 can estimate multiple ones, the R, G and B ones, the Y only one
and a combination of RGB. The one we are interested by in AGC is the Y
histogram.

Use the hardware revision to determine the number of bins of the
produced histogram, and use it to populate a vector passed down to the
libipa::Histogram class.

Signed-off-by: Jean-Michel Hautbois <jeanmichel.hautbois@ideasonboard.com>
Tested-by: Peter Griffin <peter.griffin@linaro.org>
---
v3:
    - Don't copy histogram values, pass it to the Histogram class
      directly
    - Change the date of the copyright
---
 src/ipa/rkisp1/algorithms/agc.cpp | 94 ++++++++++++++++++++++++++-----
 src/ipa/rkisp1/algorithms/agc.h   |  6 +-
 src/ipa/rkisp1/ipa_context.cpp    |  3 +
 src/ipa/rkisp1/ipa_context.h      |  1 +
 4 files changed, 88 insertions(+), 16 deletions(-)

Hi Jean-Michel,

Thank you for the patch.

On Thu, Feb 24, 2022 at 12:33:46PM +0100, Jean-Michel Hautbois wrote:
> As for the IPU3, we can estimate the histogram of the luminance. The
> RkISP1 can estimate multiple ones, the R, G and B ones, the Y only one
> and a combination of RGB. The one we are interested by in AGC is the Y
> histogram.
> 
> Use the hardware revision to determine the number of bins of the
> produced histogram, and use it to populate a vector passed down to the
> libipa::Histogram class.

You can drop the second part of the sentence, you don't copy the values
anymore.

> Signed-off-by: Jean-Michel Hautbois <jeanmichel.hautbois@ideasonboard.com>
> Tested-by: Peter Griffin <peter.griffin@linaro.org>
> ---
> v3:
>     - Don't copy histogram values, pass it to the Histogram class
>       directly
>     - Change the date of the copyright
> ---
>  src/ipa/rkisp1/algorithms/agc.cpp | 94 ++++++++++++++++++++++++++-----
>  src/ipa/rkisp1/algorithms/agc.h   |  6 +-
>  src/ipa/rkisp1/ipa_context.cpp    |  3 +
>  src/ipa/rkisp1/ipa_context.h      |  1 +
>  4 files changed, 88 insertions(+), 16 deletions(-)
> 
> diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp
> index dd97afc0..2e95ee05 100644
> --- a/src/ipa/rkisp1/algorithms/agc.cpp
> +++ b/src/ipa/rkisp1/algorithms/agc.cpp
> @@ -1,6 +1,6 @@
>  /* SPDX-License-Identifier: LGPL-2.1-or-later */
>  /*
> - * Copyright (C) 2021, Ideas On Board
> + * Copyright (C) 2021-2022, Ideas On Board
>   *
>   * agc.cpp - AGC/AEC mean-based control algorithm
>   */
> @@ -16,6 +16,8 @@
>  
>  #include <libcamera/ipa/core_ipa_interface.h>
>  
> +#include "libipa/histogram.h"
> +
>  /**
>   * \file agc.h
>   */
> @@ -43,18 +45,22 @@ static constexpr utils::Duration kMaxShutterSpeed = 60ms;
>  /* Number of frames to wait before calculating stats on minimum exposure */
>  static constexpr uint32_t kNumStartupFrames = 10;
>  
> +/* Target value to reach for the top 2% of the histogram */
> +static constexpr double kEvGainTarget = 0.5;
> +
>  /*
>   * Relative luminance target.
>   *
>   * It's a number that's chosen so that, when the camera points at a grey
>   * target, the resulting image brightness is considered right.
>   *
> - * \todo Why is the value different between IPU3 and RkISP1 ?
> + * \todo The value is not the same for IPU3 and RkIsp1 because we don't have
> + * sensor degamma yet.

The sensor output is supposed to be nearly linear already. I'd be very
surprised if this alone explained the difference between the two values.
I'd keep the original comment.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

>   */
>  static constexpr double kRelativeLuminanceTarget = 0.4;
>  
>  Agc::Agc()
> -	: frameCount_(0), numCells_(0), filteredExposure_(0s)
> +	: frameCount_(0), numCells_(0), numHistBins_(0), filteredExposure_(0s)
>  {
>  }
>  
> @@ -65,8 +71,7 @@ Agc::Agc()
>   *
>   * \return 0
>   */
> -int Agc::configure(IPAContext &context,
> -		   [[maybe_unused]] const IPACameraSensorInfo &configInfo)
> +int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo)
>  {
>  	/* Configure the default exposure and gain. */
>  	context.frameContext.agc.gain = std::max(context.configuration.agc.minAnalogueGain, kMinAnalogueGain);
> @@ -77,10 +82,22 @@ int Agc::configure(IPAContext &context,
>  	 * - versions < V12 have RKISP1_CIF_ISP_AE_MEAN_MAX_V10 entries,
>  	 * - versions >= V12 have RKISP1_CIF_ISP_AE_MEAN_MAX_V12 entries.
>  	 */
> -	if (context.configuration.hw.revision < RKISP1_V12)
> +	if (context.configuration.hw.revision < RKISP1_V12) {
>  		numCells_ = RKISP1_CIF_ISP_AE_MEAN_MAX_V10;
> -	else
> +		numHistBins_ = RKISP1_CIF_ISP_HIST_BIN_N_MAX_V10;
> +	} else {
>  		numCells_ = RKISP1_CIF_ISP_AE_MEAN_MAX_V12;
> +		numHistBins_ = RKISP1_CIF_ISP_HIST_BIN_N_MAX_V12;
> +	}
> +
> +	/*
> +	 * Define the measurement window for AGC as a centered rectangle
> +	 * covering 3/4 of the image width and height.
> +	 */
> +	context.configuration.agc.measureWindow.h_offs = configInfo.outputSize.width / 8;
> +	context.configuration.agc.measureWindow.v_offs = configInfo.outputSize.height / 8;
> +	context.configuration.agc.measureWindow.h_size = 3 * configInfo.outputSize.width / 4;
> +	context.configuration.agc.measureWindow.v_size = 3 * configInfo.outputSize.height / 4;
>  
>  	/* \todo Use actual frame index by populating it in the frameContext. */
>  	frameCount_ = 0;
> @@ -126,8 +143,9 @@ utils::Duration Agc::filterExposure(utils::Duration exposureValue)
>   * \brief Estimate the new exposure and gain values
>   * \param[inout] frameContext The shared IPA frame Context
>   * \param[in] yGain The gain calculated on the current brightness level
> + * \param[in] iqMeanGain The gain calculated based on the relative luminance target
>   */
> -void Agc::computeExposure(IPAContext &context, double yGain)
> +void Agc::computeExposure(IPAContext &context, double yGain, double iqMeanGain)
>  {
>  	IPASessionConfiguration &configuration = context.configuration;
>  	IPAFrameContext &frameContext = context.frameContext;
> @@ -136,6 +154,9 @@ void Agc::computeExposure(IPAContext &context, double yGain)
>  	uint32_t exposure = frameContext.sensor.exposure;
>  	double analogueGain = frameContext.sensor.gain;
>  
> +	/* Use the highest of the two gain estimates. */
> +	double evGain = std::max(yGain, iqMeanGain);
> +
>  	utils::Duration minShutterSpeed = configuration.agc.minShutterSpeed;
>  	utils::Duration maxShutterSpeed = std::min(configuration.agc.maxShutterSpeed,
>  						   kMaxShutterSpeed);
> @@ -146,7 +167,7 @@ void Agc::computeExposure(IPAContext &context, double yGain)
>  					  kMaxAnalogueGain);
>  
>  	/* Consider within 1% of the target as correctly exposed. */
> -	if (utils::abs_diff(yGain, 1.0) < 0.01)
> +	if (utils::abs_diff(evGain, 1.0) < 0.01)
>  		return;
>  
>  	/* extracted from Rpi::Agc::computeTargetExposure. */
> @@ -163,13 +184,13 @@ void Agc::computeExposure(IPAContext &context, double yGain)
>  	LOG(RkISP1Agc, Debug) << "Actual total exposure " << currentShutter * analogueGain
>  			      << " Shutter speed " << currentShutter
>  			      << " Gain " << analogueGain
> -			      << " Needed ev gain " << yGain;
> +			      << " Needed ev gain " << evGain;
>  
>  	/*
>  	 * Calculate the current exposure value for the scene as the latest
>  	 * exposure value applied multiplied by the new estimated gain.
>  	 */
> -	utils::Duration exposureValue = effectiveExposureValue * yGain;
> +	utils::Duration exposureValue = effectiveExposureValue * evGain;
>  
>  	/* Clamp the exposure value to the min and max authorized. */
>  	utils::Duration maxTotalExposure = maxShutterSpeed * maxAnalogueGain;
> @@ -240,6 +261,18 @@ double Agc::estimateLuminance(const rkisp1_cif_isp_ae_stat *ae,
>  	return ySum / numCells_ / 255;
>  }
>  
> +/**
> + * \brief Estimate the mean value of the top 2% of the histogram
> + * \param[in] hist The histogram statistics computed by the ImgU
> + * \return The mean value of the top 2% of the histogram
> + */
> +double Agc::measureBrightness(const rkisp1_cif_isp_hist_stat *hist) const
> +{
> +	Histogram histogram{ Span<const uint32_t>(hist->hist_bins, numHistBins_) };
> +	/* Estimate the quantile mean of the top 2% of the histogram. */
> +	return histogram.interQuantileMean(0.98, 1.0);
> +}
> +
>  /**
>   * \brief Process RkISP1 statistics, and run AGC operations
>   * \param[in] context The shared IPA context
> @@ -254,6 +287,10 @@ void Agc::process(IPAContext &context, const rkisp1_stat_buffer *stats)
>  	ASSERT(stats->meas_type & RKISP1_CIF_ISP_STAT_AUTOEXP);
>  
>  	const rkisp1_cif_isp_ae_stat *ae = &params->ae;
> +	const rkisp1_cif_isp_hist_stat *hist = &params->hist;
> +
> +	double iqMean = measureBrightness(hist);
> +	double iqMeanGain = kEvGainTarget * numHistBins_ / iqMean;
>  
>  	/*
>  	 * Estimate the gain needed to achieve a relative luminance target. To
> @@ -277,15 +314,44 @@ void Agc::process(IPAContext &context, const rkisp1_stat_buffer *stats)
>  			break;
>  	}
>  
> -	computeExposure(context, yGain);
> +	computeExposure(context, yGain, iqMeanGain);
>  	frameCount_++;
>  }
>  
> -void Agc::prepare([[maybe_unused]] IPAContext &context,
> -		  rkisp1_params_cfg *params)
> +/**
> + * \copydoc libcamera::ipa::Algorithm::prepare
> + */
> +void Agc::prepare(IPAContext &context, rkisp1_params_cfg *params)
>  {
> +	if (context.frameContext.frameCount > 0)
> +		return;
> +
> +	/* Configure the measurement window. */
> +	params->meas.aec_config.meas_window = context.configuration.agc.measureWindow;
> +	/* Use a continuous method for measure. */
> +	params->meas.aec_config.autostop = RKISP1_CIF_ISP_EXP_CTRL_AUTOSTOP_0;
> +	/* Estimate Y as (R + G + B) x (85/256). */
> +	params->meas.aec_config.mode = RKISP1_CIF_ISP_EXP_MEASURING_MODE_1;
> +
> +	params->module_cfg_update |= RKISP1_CIF_ISP_MODULE_AEC;
>  	params->module_ens |= RKISP1_CIF_ISP_MODULE_AEC;
>  	params->module_en_update |= RKISP1_CIF_ISP_MODULE_AEC;
> +
> +	/* Configure histogram. */
> +	params->meas.hst_config.meas_window = context.configuration.agc.measureWindow;
> +	/* Produce the luminance histogram. */
> +	params->meas.hst_config.mode = RKISP1_CIF_ISP_HISTOGRAM_MODE_Y_HISTOGRAM;
> +	/* Set an average weighted histogram. */
> +	for (unsigned int histBin = 0; histBin < numHistBins_; histBin++)
> +		params->meas.hst_config.hist_weight[histBin] = 1;
> +	/* Step size can't be less than 3. */
> +	params->meas.hst_config.histogram_predivider = 4;
> +
> +	/* Update the configuration for histogram. */
> +	params->module_cfg_update |= RKISP1_CIF_ISP_MODULE_HST;
> +	/* Enable the histogram measure unit. */
> +	params->module_ens |= RKISP1_CIF_ISP_MODULE_HST;
> +	params->module_en_update |= RKISP1_CIF_ISP_MODULE_HST;
>  }
>  
>  } /* namespace ipa::rkisp1::algorithms */
> diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h
> index 942c9d7a..ce1adf27 100644
> --- a/src/ipa/rkisp1/algorithms/agc.h
> +++ b/src/ipa/rkisp1/algorithms/agc.h
> @@ -1,6 +1,6 @@
>  /* SPDX-License-Identifier: LGPL-2.1-or-later */
>  /*
> - * Copyright (C) 2021, Ideas On Board
> + * Copyright (C) 2021-2022, Ideas On Board
>   *
>   * agc.h - RkISP1 AGC/AEC mean-based control algorithm
>   */
> @@ -32,13 +32,15 @@ public:
>  	void process(IPAContext &context, const rkisp1_stat_buffer *stats) override;
>  
>  private:
> -	void computeExposure(IPAContext &Context, double yGain);
> +	void computeExposure(IPAContext &Context, double yGain, double iqMeanGain);
>  	utils::Duration filterExposure(utils::Duration exposureValue);
>  	double estimateLuminance(const rkisp1_cif_isp_ae_stat *ae, double gain);
> +	double measureBrightness(const rkisp1_cif_isp_hist_stat *hist) const;
>  
>  	uint64_t frameCount_;
>  
>  	uint32_t numCells_;
> +	uint32_t numHistBins_;
>  
>  	utils::Duration filteredExposure_;
>  };
> diff --git a/src/ipa/rkisp1/ipa_context.cpp b/src/ipa/rkisp1/ipa_context.cpp
> index 664f572f..790e159b 100644
> --- a/src/ipa/rkisp1/ipa_context.cpp
> +++ b/src/ipa/rkisp1/ipa_context.cpp
> @@ -71,6 +71,9 @@ namespace libcamera::ipa::rkisp1 {
>   * \var IPASessionConfiguration::agc.maxAnalogueGain
>   * \brief Maximum analogue gain supported with the configured sensor
>   *
> + * \var IPASessionConfiguration::agc.measureWindow
> + * \brief AGC measure window
> + *
>   * \var IPASessionConfiguration::hw
>   * \brief RkISP1-specific hardware information
>   *
> diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h
> index 212fa052..35e9b8e5 100644
> --- a/src/ipa/rkisp1/ipa_context.h
> +++ b/src/ipa/rkisp1/ipa_context.h
> @@ -22,6 +22,7 @@ struct IPASessionConfiguration {
>  		utils::Duration maxShutterSpeed;
>  		double minAnalogueGain;
>  		double maxAnalogueGain;
> +		struct rkisp1_cif_isp_window measureWindow;
>  	} agc;
>  
>  	struct {