[libcamera-devel,v3,4/5] ipa: ipu3: Add support for IPU3 AWB algorithm

11779 diff mbox series

Inherit from the Algorithm class to implement basic AWB functions.

Once AWB is done, a color temperature is estimated and a default CCM matrice
is used (yet to be tuned).
Implement a basic "grey-world" AWB algorithm just for demonstration purpose.

BDS output size is passed by the pipeline handler to the IPA.
The best grid is then calculated to maximize the number of pixels taken
into account in each cells.
As commented in the source code, it can be improved, as ithas (at least)
one limitation: if a cell is big (say 128 pixels wide) and indicated as
saturated, it won't be taken into account at all.
Maybe is it possible to have a smaller one, at the cost of a few pixels
to lose. In which case we can center the grid using the x_start and
y_start parameters.

Signed-off-by: Jean-Michel Hautbois <jeanmichel.hautbois@ideasonboard.com>
---
 src/ipa/ipu3/ipu3.cpp     |  74 +++++++++++-
 src/ipa/ipu3/ipu3_awb.cpp | 233 ++++++++++++++++++++++++++++++++++++++
 src/ipa/ipu3/ipu3_awb.h   |  44 +++++++
 src/ipa/ipu3/meson.build  |   1 +
 4 files changed, 346 insertions(+), 6 deletions(-)
 create mode 100644 src/ipa/ipu3/ipu3_awb.cpp
 create mode 100644 src/ipa/ipu3/ipu3_awb.h

Hi Jean-Michel,

Thank you for the patch.

On Mon, Mar 29, 2021 at 09:18:25PM +0200, Jean-Michel Hautbois wrote:
> Inherit from the Algorithm class to implement basic AWB functions.
> 
> Once AWB is done, a color temperature is estimated and a default CCM matrice

s/matrice/matrix/

> is used (yet to be tuned).

See 3/5. The same applies below.

> Implement a basic "grey-world" AWB algorithm just for demonstration purpose.
> 
> BDS output size is passed by the pipeline handler to the IPA.

"The BDS ..."

> The best grid is then calculated to maximize the number of pixels taken
> into account in each cells.
> As commented in the source code, it can be improved, as ithas (at least)

s/ithas/it has/

> one limitation: if a cell is big (say 128 pixels wide) and indicated as
> saturated, it won't be taken into account at all.
> Maybe is it possible to have a smaller one, at the cost of a few pixels

A similar what ?

> to lose. In which case we can center the grid using the x_start and

s/to lose. In which case/to lose, in which case/

> y_start parameters.
>
> Signed-off-by: Jean-Michel Hautbois <jeanmichel.hautbois@ideasonboard.com>
> ---
>  src/ipa/ipu3/ipu3.cpp     |  74 +++++++++++-
>  src/ipa/ipu3/ipu3_awb.cpp | 233 ++++++++++++++++++++++++++++++++++++++
>  src/ipa/ipu3/ipu3_awb.h   |  44 +++++++
>  src/ipa/ipu3/meson.build  |   1 +
>  4 files changed, 346 insertions(+), 6 deletions(-)
>  create mode 100644 src/ipa/ipu3/ipu3_awb.cpp
>  create mode 100644 src/ipa/ipu3/ipu3_awb.h
> 
> diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp
> index 07dbc24a..cc741d69 100644
> --- a/src/ipa/ipu3/ipu3.cpp
> +++ b/src/ipa/ipu3/ipu3.cpp
> @@ -21,6 +21,11 @@
>  #include "libcamera/internal/buffer.h"
>  #include "libcamera/internal/log.h"
>  
> +#include "ipu3_awb.h"
> +
> +static const uint32_t kMaxCellWidthPerSet = 160;
> +static const uint32_t kMaxCellHeightPerSet = 80;

s/const/constexpr/

> +
>  namespace libcamera {
>  
>  LOG_DEFINE_CATEGORY(IPAIPU3)
> @@ -62,8 +67,12 @@ private:
>  	uint32_t minGain_;
>  	uint32_t maxGain_;
>  
> +	/* Interface to the AWB algorithm */
> +	std::unique_ptr<ipa::IPU3Awb> awbAlgo_;

Blank line.

>  	/* Local parameter storage */
> -	ipu3_uapi_params params_;
> +	struct ipu3_uapi_params params_;

The struct should have been in the previous patch. But as that part of
the previous patch should be moved here, that doesn't matter much.

> +
> +	struct ipu3_uapi_grid_config bdsGrid_;
>  };
>  
>  int IPAIPU3::start()
> @@ -73,6 +82,51 @@ int IPAIPU3::start()
>  	return 0;
>  }
>  
> +/* This method calculates a grid for the AWB algorithm in the IPU3 firmware.
> + * Its input it the BDS output size calculated in the imgU.

s/it the/is the/

> + * It is limited for now to the simplest method: find the lesser error
> + * with the width/height and respective log2 width/height of the cells.
> + * \todo smaller cells are better so adapt x_start to lose a bit but have
> + * a better average resolution. If a cell is saturated, it might make a big difference. */

I'm afraid I don't understand this :-(

Please reflow this text, and write the comment as

/*
 * ...
 * ...
 */

> +void IPAIPU3::calculateBdsGrid(const Size &bdsOutputSize)

This function isn't used in this patch, and not even declared in the
class. This will break bisection.

> +{
> +	std::vector<uint32_t> log2WidthVector = { 3, 4, 5, 6, 7 };
> +	std::vector<uint32_t> log2HeightVector = { 3, 4, 5, 6, 7 };
> +	uint32_t minError = std::numeric_limits<uint32_t>::max();
> +	uint32_t bestWidth = 0;
> +	uint32_t bestHeight = 0;
> +	uint32_t bestLog2Width = 0;
> +	uint32_t bestLog2Height = 0;
> +	bdsGrid_ = {};
> +
> +	for (uint32_t i = 0; i < log2WidthVector.size(); ++i) {

The i variable is only used to index log2WidthVector, you can thus use a
range-based loop. Or even better, you can write

	for (unsigned int widthShift = 3; widthShift <= 7; ++widthShift) {

> +		uint32_t width = std::min(kMaxCellWidthPerSet, bdsOutputSize.width >> log2WidthVector[i]);

Line length, and below too.

> +		width = width << log2WidthVector[i];
> +		for (uint32_t j = 0; j < log2HeightVector.size(); ++j) {

Same here.

> +			int32_t height = std::min(kMaxCellHeightPerSet, bdsOutputSize.height >> log2HeightVector[j]);
> +			height = height << log2HeightVector[j];
> +			uint32_t error = std::abs((int)(width - bdsOutputSize.width)) + std::abs((int)(height - bdsOutputSize.height));

C++-style casts, and below too.

> +
> +			if (error > minError)
> +				continue;
> +
> +			minError = error;
> +			bestWidth = width;
> +			bestHeight = height;
> +			bestLog2Width = log2WidthVector[i];
> +			bestLog2Height = log2HeightVector[j];
> +		}
> +	}
> +
> +	bdsGrid_.width = bestWidth >> bestLog2Width;
> +	bdsGrid_.block_width_log2 = bestLog2Width;
> +	bdsGrid_.height = bestHeight >> bestLog2Height;
> +	bdsGrid_.block_height_log2 = bestLog2Height;

Blank line.

> +	LOG(IPAIPU3, Debug) << "Best grid found is: ("
> +			    << (int)bdsGrid_.width << " << " << (int)bdsGrid_.block_width_log2 << ") x ("
> +			    << (int)bdsGrid_.height << "<<" << (int)bdsGrid_.block_height_log2 << ")";
> +}
> +
>  void IPAIPU3::configure(const std::map<uint32_t, ControlInfoMap> &entityControls,
>  			[[maybe_unused]] const Size &bdsOutputSize)

You now make use of this parameter.

>  {
> @@ -103,6 +157,9 @@ void IPAIPU3::configure(const std::map<uint32_t, ControlInfoMap> &entityControls
>  
>  	params_ = {};
>  
> +	awbAlgo_ = std::make_unique<ipa::IPU3Awb>();
> +	awbAlgo_->initialise(params_, bdsOutputSize, bdsGrid_);
> +
>  	setControls(0);
>  }
>  
> @@ -175,10 +232,9 @@ void IPAIPU3::processControls([[maybe_unused]] unsigned int frame,
>  
>  void IPAIPU3::fillParams(unsigned int frame, ipu3_uapi_params *params)
>  {
> -	/* Prepare parameters buffer. */
> -	memset(params, 0, sizeof(*params));
> +	awbAlgo_->updateWbParameters(params_, 1.0);
>  
> -	/* \todo Fill in parameters buffer. */
> +	*params = params_;
>  
>  	ipa::ipu3::IPU3Action op;
>  	op.op = ipa::ipu3::ActionParamFilled;
> @@ -191,8 +247,14 @@ void IPAIPU3::parseStatistics(unsigned int frame,
>  {
>  	ControlList ctrls(controls::controls);
>  
> -	/* \todo React to statistics and update internal state machine. */
> -	/* \todo Add meta-data information to ctrls. */
> +	if (!stats->stats_3a_status.awb_en) {
> +		LOG(IPAIPU3, Error) << "AWB stats are not enabled";

Sounds like something for the awb algorithm to complain about.

> +	} else {
> +		agcAlgo_->process(stats, exposure_, gain_);

There's no agcAlgo_ in this patch, it comes in 5/5.

I'll stop the review of this series here, and resume in v4.

> +		awbAlgo_->calculateWBGains(stats);
> +		if (agcAlgo_->updateControls())
> +			setControls(frame);
> +	}
>  
>  	ipa::ipu3::IPU3Action op;
>  	op.op = ipa::ipu3::ActionMetadataReady;
> diff --git a/src/ipa/ipu3/ipu3_awb.cpp b/src/ipa/ipu3/ipu3_awb.cpp
> new file mode 100644
> index 00000000..c63eb08f
> --- /dev/null
> +++ b/src/ipa/ipu3/ipu3_awb.cpp
> @@ -0,0 +1,233 @@
> +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> +/*
> + * Copyright (C) 2021, Ideas On Board
> + *
> + * ipu3_awb.cpp - AWB control algorithm
> + */
> +#include "ipu3_awb.h"
> +
> +#include <cmath>
> +#include <numeric>
> +#include <unordered_map>
> +
> +#include "libcamera/internal/log.h"
> +
> +namespace libcamera {
> +
> +namespace ipa {
> +
> +LOG_DEFINE_CATEGORY(IPU3Awb)
> +
> +static const struct ipu3_uapi_bnr_static_config imguCssBnrDefaults = {
> +	.wb_gains = { 16, 16, 16, 16 },
> +	.wb_gains_thr = { 255, 255, 255, 255 },
> +	.thr_coeffs = { 1700, 0, 31, 31, 0, 16 },
> +	.thr_ctrl_shd = { 26, 26, 26, 26 },
> +	.opt_center{ -648, 0, -366, 0 },
> +	.lut = {
> +		{ 17, 23, 28, 32, 36, 39, 42, 45,
> +		  48, 51, 53, 55, 58, 60, 62, 64,
> +		  66, 68, 70, 72, 73, 75, 77, 78,
> +		  80, 82, 83, 85, 86, 88, 89, 90 } },
> +	.bp_ctrl = { 20, 0, 1, 40, 0, 6, 0, 6, 0 },
> +	.dn_detect_ctrl{ 9, 3, 4, 0, 8, 0, 1, 1, 1, 1, 0 },
> +	.column_size = 1296,
> +	.opt_center_sqr = { 419904, 133956 },
> +};
> +
> +/* settings for Auto White Balance */
> +static const struct ipu3_uapi_awb_config_s imguCssAwbDefaults = {
> +	.rgbs_thr_gr = 8191,
> +	.rgbs_thr_r = 8191,
> +	.rgbs_thr_gb = 8191,
> +	.rgbs_thr_b = 8191 | IPU3_UAPI_AWB_RGBS_THR_B_EN | IPU3_UAPI_AWB_RGBS_THR_B_INCL_SAT,
> +	.grid = {
> +		.width = 160,
> +		.height = 36,
> +		.block_width_log2 = 3,
> +		.block_height_log2 = 4,
> +		.height_per_slice = 1, /* Overridden by kernel. */
> +		.x_start = 0,
> +		.y_start = 0,
> +		.x_end = 0,
> +		.y_end = 0,
> +	},
> +};
> +
> +static const struct ipu3_uapi_ccm_mat_config imguCssCcmDefault = {
> +	8191, 0, 0, 0,
> +	0, 8191, 0, 0,
> +	0, 0, 8191, 0
> +};
> +
> +IPU3Awb::IPU3Awb()
> +	: Algorithm()
> +{
> +}
> +
> +IPU3Awb::~IPU3Awb()
> +{
> +}
> +
> +void IPU3Awb::initialise(ipu3_uapi_params &params, const Size &bdsOutputSize, struct ipu3_uapi_grid_config &bdsGrid)
> +{
> +	params.use.acc_awb = 1;
> +	params.acc_param.awb.config = imguCssAwbDefaults;
> +
> +	awbGrid_ = bdsGrid;
> +	params.acc_param.awb.config.grid = awbGrid_;
> +
> +	params.use.obgrid = 0;
> +	params.obgrid_param.gr = 20;
> +	params.obgrid_param.r = 28;
> +	params.obgrid_param.b = 28;
> +	params.obgrid_param.gb = 20;
> +
> +	params.use.acc_bnr = 1;
> +	params.acc_param.bnr = imguCssBnrDefaults;
> +	params.acc_param.bnr.opt_center.x_reset = -1 * (bdsOutputSize.width / 2);
> +	params.acc_param.bnr.opt_center.y_reset = -1 * (bdsOutputSize.height / 2);
> +	params.acc_param.bnr.column_size = bdsOutputSize.width;
> +	params.acc_param.bnr.opt_center_sqr.x_sqr_reset = params.acc_param.bnr.opt_center.x_reset * params.acc_param.bnr.opt_center.x_reset;
> +	params.acc_param.bnr.opt_center_sqr.y_sqr_reset = params.acc_param.bnr.opt_center.y_reset * params.acc_param.bnr.opt_center.y_reset;
> +
> +	params.use.acc_ccm = 1;
> +	params.acc_param.ccm = imguCssCcmDefault;
> +
> +	params.use.acc_gamma = 1;
> +	params.acc_param.gamma.gc_ctrl.enable = 1;
> +
> +	params.use.acc_green_disparity = 0;
> +	params.acc_param.green_disparity.gd_black = 2440;
> +	params.acc_param.green_disparity.gd_red = 4;
> +	params.acc_param.green_disparity.gd_blue = 4;
> +	params.acc_param.green_disparity.gd_green = 4;
> +	params.acc_param.green_disparity.gd_shading = 24;
> +	params.acc_param.green_disparity.gd_support = 2;
> +	params.acc_param.green_disparity.gd_clip = 1;
> +	params.acc_param.green_disparity.gd_central_weight = 5;
> +
> +	params.use.acc_cds = 1;
> +	params.acc_param.cds.csc_en = 1;
> +	params.acc_param.cds.uv_bin_output = 0;
> +	params.acc_param.cds.ds_c00 = 0;
> +	params.acc_param.cds.ds_c01 = 1;
> +	params.acc_param.cds.ds_c02 = 1;
> +	params.acc_param.cds.ds_c03 = 0;
> +	params.acc_param.cds.ds_c10 = 0;
> +	params.acc_param.cds.ds_c11 = 1;
> +	params.acc_param.cds.ds_c12 = 1;
> +	params.acc_param.cds.ds_c13 = 0;
> +	params.acc_param.cds.ds_nf = 2;
> +
> +	wbGains_[0] = 16;
> +	wbGains_[1] = 4096;
> +	wbGains_[2] = 4096;
> +	wbGains_[3] = 16;
> +
> +	frame_count_ = 0;
> +}
> +
> +uint32_t IPU3Awb::estimateCCT(uint8_t red, uint8_t green, uint8_t blue)
> +{
> +	double X = (-0.14282) * (red) + (1.54924) * (green) + (-0.95641) * (blue);
> +	double Y = (-0.32466) * (red) + (1.57837) * (green) + (-0.73191) * (blue);
> +	double Z = (-0.68202) * (red) + (0.77073) * (green) + (0.56332) * (blue);
> +
> +	double x = X / (X + Y + Z);
> +	double y = Y / (X + Y + Z);
> +
> +	double n = (x - 0.3320) / (0.1858 - y);
> +	return static_cast<uint32_t>(449 * n * n * n + 3525 * n * n + 6823.3 * n + 5520.33);
> +}
> +
> +double meanValue(std::vector<uint32_t> colorValues)
> +{
> +	uint32_t count = 0;
> +	uint32_t hist[256] = { 0 };
> +	for (uint32_t const &val : colorValues) {
> +		hist[val]++;
> +		count++;
> +	}
> +
> +	double mean = 0.0;
> +	for (uint32_t i = 0; i < 256; i++) {
> +		mean += hist[i] * i;
> +	}
> +	return mean /= count;
> +}
> +
> +void IPU3Awb::calculateWBGains(const ipu3_uapi_stats_3a *stats)
> +{
> +	std::vector<uint32_t> redValues, greenValues, blueValues;
> +	const struct ipu3_uapi_grid_config statsAwbGrid = stats->stats_4a_config.awb_config.grid;
> +	Rectangle awbRegion = { statsAwbGrid.x_start,
> +				statsAwbGrid.y_start,
> +				static_cast<unsigned int>(statsAwbGrid.x_end - statsAwbGrid.x_start) + 1,
> +				static_cast<unsigned int>(statsAwbGrid.y_end - statsAwbGrid.y_start) + 1 };
> +
> +	Point topleft = awbRegion.topLeft();
> +	uint32_t startY = (topleft.y >> awbGrid_.block_height_log2) * awbGrid_.width << awbGrid_.block_width_log2;
> +	uint32_t startX = (topleft.x >> awbGrid_.block_width_log2) << awbGrid_.block_width_log2;
> +	uint32_t endX = (startX + (awbRegion.size().width >> awbGrid_.block_width_log2)) << awbGrid_.block_width_log2;
> +	uint32_t count = 0;
> +	uint32_t i, j;
> +
> +	awbCounted_ = 0;
> +	for (j = (topleft.y >> awbGrid_.block_height_log2);
> +	     j < (topleft.y >> awbGrid_.block_height_log2) + (awbRegion.size().height >> awbGrid_.block_height_log2);
> +	     j++) {
> +		for (i = startX + startY; i < endX + startY; i += 8) {
> +			if (stats->awb_raw_buffer.meta_data[i + 4 + j * awbGrid_.width] == 0) {
> +				greenValues.push_back(stats->awb_raw_buffer.meta_data[i + j * awbGrid_.width]);
> +				redValues.push_back(stats->awb_raw_buffer.meta_data[i + 1 + j * awbGrid_.width]);
> +				blueValues.push_back(stats->awb_raw_buffer.meta_data[i + 2 + j * awbGrid_.width]);
> +				greenValues.push_back(stats->awb_raw_buffer.meta_data[i + 3 + j * awbGrid_.width]);
> +				awbCounted_++;
> +			}
> +			count++;
> +		}
> +	}
> +
> +	double rMean = meanValue(redValues);
> +	double bMean = meanValue(blueValues);
> +	double gMean = meanValue(greenValues);
> +
> +	double rGain = gMean / rMean;
> +	double bGain = gMean / bMean;
> +
> +	wbGains_[0] = 16;
> +	wbGains_[1] = 4096 * rGain;
> +	wbGains_[2] = 4096 * bGain;
> +	wbGains_[3] = 16;
> +
> +	frame_count_++;
> +
> +	cct_ = estimateCCT(rMean, gMean, bMean);
> +}
> +
> +void IPU3Awb::updateWbParameters(ipu3_uapi_params &params, double agcGamma)
> +{
> +	if ((wbGains_[0] == 0) || (wbGains_[1] == 0) || (wbGains_[2] == 0) || (wbGains_[3] == 0)) {
> +		LOG(IPU3Awb, Error) << "Gains can't be 0, check the stats";
> +	} else {
> +		params.acc_param.bnr.wb_gains.gr = wbGains_[0];
> +		params.acc_param.bnr.wb_gains.r = wbGains_[1];
> +		params.acc_param.bnr.wb_gains.b = wbGains_[2];
> +		params.acc_param.bnr.wb_gains.gb = wbGains_[3];
> +
> +		LOG(IPU3Awb, Debug) << "Color temperature estimated: " << cct_
> +				    << " and gamma calculated: " << agcGamma;
> +		params.acc_param.ccm = imguCssCcmDefault;
> +
> +		for (uint32_t i = 0; i < 256; i++) {
> +			double j = i / 255.0;
> +			double gamma = std::pow(j, 1.0 / agcGamma);
> +			params.acc_param.gamma.gc_lut.lut[i] = gamma * 8191;
> +		}
> +	}
> +}
> +
> +} /* namespace ipa */
> +
> +} /* namespace libcamera */
> diff --git a/src/ipa/ipu3/ipu3_awb.h b/src/ipa/ipu3/ipu3_awb.h
> new file mode 100644
> index 00000000..a14401a0
> --- /dev/null
> +++ b/src/ipa/ipu3/ipu3_awb.h
> @@ -0,0 +1,44 @@
> +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> +/*
> + * Copyright (C) 2021, Ideas On Board
> + *
> + * ipu3_awb.h - IPU3 AWB control algorithm
> + */
> +#ifndef __LIBCAMERA_IPU3_AWB_H__
> +#define __LIBCAMERA_IPU3_AWB_H__
> +
> +#include <linux/intel-ipu3.h>
> +
> +#include <libcamera/geometry.h>
> +
> +#include "libipa/algorithm.h"
> +
> +namespace libcamera {
> +
> +namespace ipa {
> +
> +class IPU3Awb : public Algorithm
> +{
> +public:
> +	IPU3Awb();
> +	~IPU3Awb();
> +
> +	void initialise(ipu3_uapi_params &params, const Size &bdsOutputSize, struct ipu3_uapi_grid_config &bdsGrid);
> +	void calculateWBGains(const ipu3_uapi_stats_3a *stats);
> +	void updateWbParameters(ipu3_uapi_params &params, double agcGamma);
> +
> +private:
> +	uint32_t estimateCCT(uint8_t red, uint8_t green, uint8_t blue);
> +
> +	/* WB calculated gains */
> +	uint16_t wbGains_[4];
> +	uint32_t cct_;
> +	uint32_t awbCounted_;
> +	struct ipu3_uapi_grid_config awbGrid_;
> +	uint32_t frame_count_;
> +};
> +
> +} /* namespace ipa */
> +
> +} /* namespace libcamera*/
> +#endif /* __LIBCAMERA_IPU3_AWB_H__ */
> diff --git a/src/ipa/ipu3/meson.build b/src/ipa/ipu3/meson.build
> index 52d98c8e..07a864c8 100644
> --- a/src/ipa/ipu3/meson.build
> +++ b/src/ipa/ipu3/meson.build
> @@ -4,6 +4,7 @@ ipa_name = 'ipa_ipu3'
>  
>  ipu3_ipa_sources = files([
>    'ipu3.cpp',
> +  'ipu3_awb.cpp',
>  ])
>  
>  mod = shared_module(ipa_name,