[libcamera-devel,v4,27/32] ipa: rkisp1: awb: Add support for RGB means
diff mbox series

Message ID 20220908014200.28728-28-laurent.pinchart@ideasonboard.com
State Superseded
Headers show
Series
  • ipa: Frame context queue, IPU3 & RkISP consolidation, and RkISP1 improvements
Related show

Commit Message

Laurent Pinchart Sept. 8, 2022, 1:41 a.m. UTC 
From: Quentin Schulz <quentin.schulz@theobroma-systems.com>

RkISP actually supports two modes for color means, RGB and YCbCr. The
variables where the means are stored are identically named regardless of
the color means mode that's been selected.

Since the gains are computed in RGB mode, a conversion needs to be done
when the mode is YCbCr, which is unnecessary when RGB mode is selected.

This adds support for RGB means mode too, by checking at runtime which
mode is selected at a given time. The default is still set to YCbCr mode
for now.

Cc: Quentin Schulz <foss+libcamera@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
Changes since v1:

- Fix compilation error by storing RGB mode flag in Awb class
- Set ISP AWB configuration based on the selected mode
---
 src/ipa/rkisp1/algorithms/awb.cpp | 130 +++++++++++++++++++-----------
 src/ipa/rkisp1/algorithms/awb.h   |   4 +-
 2 files changed, 86 insertions(+), 48 deletions(-)

Comments

Kieran Bingham Sept. 20, 2022, 11:31 p.m. UTC | #1 
Quoting Laurent Pinchart via libcamera-devel (2022-09-08 02:41:55)
> From: Quentin Schulz <quentin.schulz@theobroma-systems.com>
> 
> RkISP actually supports two modes for color means, RGB and YCbCr. The
> variables where the means are stored are identically named regardless of
> the color means mode that's been selected.
> 
> Since the gains are computed in RGB mode, a conversion needs to be done
> when the mode is YCbCr, which is unnecessary when RGB mode is selected.
> 
> This adds support for RGB means mode too, by checking at runtime which
> mode is selected at a given time. The default is still set to YCbCr mode
> for now.

Interestingly, this sounds like we could change at runtime (with a
capture script?) to change this to see any effect on the image. But such
a control would be quite 'custom' or likely specific to RKISP?

> 
> Cc: Quentin Schulz <foss+libcamera@0leil.net>
> Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> ---
> Changes since v1:
> 
> - Fix compilation error by storing RGB mode flag in Awb class
> - Set ISP AWB configuration based on the selected mode
> ---
>  src/ipa/rkisp1/algorithms/awb.cpp | 130 +++++++++++++++++++-----------
>  src/ipa/rkisp1/algorithms/awb.h   |   4 +-
>  2 files changed, 86 insertions(+), 48 deletions(-)
> 
> diff --git a/src/ipa/rkisp1/algorithms/awb.cpp b/src/ipa/rkisp1/algorithms/awb.cpp
> index b711e93b73ba..2ff67ed98221 100644
> --- a/src/ipa/rkisp1/algorithms/awb.cpp
> +++ b/src/ipa/rkisp1/algorithms/awb.cpp
> @@ -30,6 +30,11 @@ namespace ipa::rkisp1::algorithms {
>  
>  LOG_DEFINE_CATEGORY(RkISP1Awb)
>  
> +Awb::Awb()
> +       : rgbMode_(false)
> +{
> +}
> +
>  /**
>   * \copydoc libcamera::ipa::Algorithm::configure
>   */
> @@ -98,38 +103,60 @@ void Awb::prepare(IPAContext &context, const uint32_t frame,
>         /* Update the gains. */
>         params->module_cfg_update |= RKISP1_CIF_ISP_MODULE_AWB_GAIN;
>  
> -       /* If we already have configured the gains and window, return. */
> +       /* If we already have set the AWB measurement parameters, return. */

"If we have already set the ..."


>         if (frame > 0)
>                 return;
>  
> -       /* Configure the gains to apply. */
> +       rkisp1_cif_isp_awb_meas_config &awb_config = params->meas.awb_meas_config;
> +
> +       /* Configure the measure window for AWB. */
> +       awb_config.awb_wnd = context.configuration.awb.measureWindow;
> +
> +       /* Number of frames to use to estimate the means (0 means 1 frame). */
> +       awb_config.frames = 0;

This sounds like something that could be a tuning file option. (But
that isn't required now I expect).

That said, I think the RGB / YUV mode selection could be a tuning file
option too?

But perhaps that's a 'yak for tomorrow'?


Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>

> +
> +       /* Select RGB or YCbCr means measurement. */
> +       if (rgbMode_) {
> +               awb_config.awb_mode = RKISP1_CIF_ISP_AWB_MODE_RGB;
> +
> +               /*
> +                * For RGB-based measurements, pixels are selected with maximum
> +                * red, green and blue thresholds that are set in the
> +                * awb_ref_cr, awb_min_y and awb_ref_cb respectively. The other
> +                * values are not used, set them to 0.
> +                */
> +               awb_config.awb_ref_cr = 250;
> +               awb_config.min_y = 250;
> +               awb_config.awb_ref_cb = 250;
> +
> +               awb_config.max_y = 0;
> +               awb_config.min_c = 0;
> +               awb_config.max_csum = 0;
> +       } else {
> +               awb_config.awb_mode = RKISP1_CIF_ISP_AWB_MODE_YCBCR;
> +
> +               /* Set the reference Cr and Cb (AWB target) to white. */
> +               awb_config.awb_ref_cb = 128;
> +               awb_config.awb_ref_cr = 128;
> +
> +               /*
> +                * Filter out pixels based on luminance and chrominance values.
> +                * The acceptable luma values are specified as a [16, 250]
> +                * range, while the acceptable chrome values are specified with
> +                * a minimum of 16 and a maximum Cb+Cr sum of 250.
> +                */
> +               awb_config.min_y = 16;
> +               awb_config.max_y = 250;
> +               awb_config.min_c = 16;
> +               awb_config.max_csum = 250;
> +       }
> +
> +       /* Enable the AWB gains. */
>         params->module_en_update |= RKISP1_CIF_ISP_MODULE_AWB_GAIN;
> -       /* Update the ISP to apply the gains configured. */
>         params->module_ens |= RKISP1_CIF_ISP_MODULE_AWB_GAIN;
>  
> -       /* Configure the measure window for AWB. */
> -       params->meas.awb_meas_config.awb_wnd = context.configuration.awb.measureWindow;
> -       /*
> -        * Measure Y, Cr and Cb means.
> -        * \todo RGB is not working, the kernel seems to not configure it ?
> -        */
> -       params->meas.awb_meas_config.awb_mode = RKISP1_CIF_ISP_AWB_MODE_YCBCR;
> -       /* Reference Cr and Cb. */
> -       params->meas.awb_meas_config.awb_ref_cb = 128;
> -       params->meas.awb_meas_config.awb_ref_cr = 128;
> -       /* Y values to include are between min_y and max_y only. */
> -       params->meas.awb_meas_config.min_y = 16;
> -       params->meas.awb_meas_config.max_y = 250;
> -       /* Maximum Cr+Cb value to take into account for awb. */
> -       params->meas.awb_meas_config.max_csum = 250;
> -       /* Minimum Cr and Cb values to take into account. */
> -       params->meas.awb_meas_config.min_c = 16;
> -       /* Number of frames to use to estimate the mean (0 means 1 frame). */
> -       params->meas.awb_meas_config.frames = 0;
> -
> -       /* Update AWB measurement unit configuration. */
> +       /* Update the AWB measurement parameters and enable the AWB module. */
>         params->module_cfg_update |= RKISP1_CIF_ISP_MODULE_AWB;
> -       /* Make sure the ISP is measuring the means for the next frame. */
>         params->module_en_update |= RKISP1_CIF_ISP_MODULE_AWB;
>         params->module_ens |= RKISP1_CIF_ISP_MODULE_AWB;
>  }
> @@ -182,30 +209,39 @@ void Awb::process(IPAContext &context,
>         const rkisp1_cif_isp_stat *params = &stats->params;
>         const rkisp1_cif_isp_awb_stat *awb = &params->awb;
>         IPAActiveState &activeState = context.activeState;
> +       double greenMean;
> +       double redMean;
> +       double blueMean;
>  
> -       /* Get the YCbCr mean values */
> -       double yMean = awb->awb_mean[0].mean_y_or_g;
> -       double crMean = awb->awb_mean[0].mean_cr_or_r;
> -       double cbMean = awb->awb_mean[0].mean_cb_or_b;
> +       if (rgbMode_) {
> +               greenMean = awb->awb_mean[0].mean_y_or_g;
> +               redMean = awb->awb_mean[0].mean_cr_or_r;
> +               blueMean = awb->awb_mean[0].mean_cb_or_b;
> +       } else {
> +               /* Get the YCbCr mean values */
> +               double yMean = awb->awb_mean[0].mean_y_or_g;
> +               double cbMean = awb->awb_mean[0].mean_cb_or_b;
> +               double crMean = awb->awb_mean[0].mean_cr_or_r;
>  
> -       /*
> -        * Convert from YCbCr to RGB.
> -        * The hardware uses the following formulas:
> -        * Y = 16 + 0.2500 R + 0.5000 G + 0.1094 B
> -        * Cb = 128 - 0.1406 R - 0.2969 G + 0.4375 B
> -        * Cr = 128 + 0.4375 R - 0.3750 G - 0.0625 B
> -        *
> -        * The inverse matrix is thus:
> -        * [[1,1636, -0,0623,  1,6008]
> -        *  [1,1636, -0,4045, -0,7949]
> -        *  [1,1636,  1,9912, -0,0250]]
> -        */
> -       yMean -= 16;
> -       cbMean -= 128;
> -       crMean -= 128;
> -       double redMean = 1.1636 * yMean - 0.0623 * cbMean + 1.6008 * crMean;
> -       double greenMean = 1.1636 * yMean - 0.4045 * cbMean - 0.7949 * crMean;
> -       double blueMean = 1.1636 * yMean + 1.9912 * cbMean - 0.0250 * crMean;
> +               /*
> +                * Convert from YCbCr to RGB.
> +                * The hardware uses the following formulas:
> +                * Y = 16 + 0.2500 R + 0.5000 G + 0.1094 B
> +                * Cb = 128 - 0.1406 R - 0.2969 G + 0.4375 B
> +                * Cr = 128 + 0.4375 R - 0.3750 G - 0.0625 B
> +                *
> +                * The inverse matrix is thus:
> +                * [[1,1636, -0,0623,  1,6008]
> +                *  [1,1636, -0,4045, -0,7949]
> +                *  [1,1636,  1,9912, -0,0250]]
> +                */
> +               yMean -= 16;
> +               cbMean -= 128;
> +               crMean -= 128;
> +               redMean = 1.1636 * yMean - 0.0623 * cbMean + 1.6008 * crMean;
> +               greenMean = 1.1636 * yMean - 0.4045 * cbMean - 0.7949 * crMean;
> +               blueMean = 1.1636 * yMean + 1.9912 * cbMean - 0.0250 * crMean;
> +       }
>  
>         /*
>          * The ISP computes the AWB means after applying the colour gains,
> diff --git a/src/ipa/rkisp1/algorithms/awb.h b/src/ipa/rkisp1/algorithms/awb.h
> index 9fd156d8aff9..b1b861d930d1 100644
> --- a/src/ipa/rkisp1/algorithms/awb.h
> +++ b/src/ipa/rkisp1/algorithms/awb.h
> @@ -16,7 +16,7 @@ namespace ipa::rkisp1::algorithms {
>  class Awb : public Algorithm
>  {
>  public:
> -       Awb() = default;
> +       Awb();
>         ~Awb() = default;
>  
>         int configure(IPAContext &context, const IPACameraSensorInfo &configInfo) override;
> @@ -32,6 +32,8 @@ public:
>  
>  private:
>         uint32_t estimateCCT(double red, double green, double blue);
> +
> +       bool rgbMode_;
>  };
>  
>  } /* namespace ipa::rkisp1::algorithms */
> -- 
> Regards,
> 
> Laurent Pinchart
>
Jacopo Mondi Sept. 22, 2022, 10:32 a.m. UTC | #2 
Hi Laurent,

On Thu, Sep 08, 2022 at 04:41:55AM +0300, Laurent Pinchart via libcamera-devel wrote:
> From: Quentin Schulz <quentin.schulz@theobroma-systems.com>
>
> RkISP actually supports two modes for color means, RGB and YCbCr. The
> variables where the means are stored are identically named regardless of
> the color means mode that's been selected.
>
> Since the gains are computed in RGB mode, a conversion needs to be done
> when the mode is YCbCr, which is unnecessary when RGB mode is selected.
>
> This adds support for RGB means mode too, by checking at runtime which
> mode is selected at a given time. The default is still set to YCbCr mode
> for now.
>
> Cc: Quentin Schulz <foss+libcamera@0leil.net>
> Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> ---
> Changes since v1:
>
> - Fix compilation error by storing RGB mode flag in Awb class
> - Set ISP AWB configuration based on the selected mode
> ---
>  src/ipa/rkisp1/algorithms/awb.cpp | 130 +++++++++++++++++++-----------
>  src/ipa/rkisp1/algorithms/awb.h   |   4 +-
>  2 files changed, 86 insertions(+), 48 deletions(-)
>
> diff --git a/src/ipa/rkisp1/algorithms/awb.cpp b/src/ipa/rkisp1/algorithms/awb.cpp
> index b711e93b73ba..2ff67ed98221 100644
> --- a/src/ipa/rkisp1/algorithms/awb.cpp
> +++ b/src/ipa/rkisp1/algorithms/awb.cpp
> @@ -30,6 +30,11 @@ namespace ipa::rkisp1::algorithms {
>
>  LOG_DEFINE_CATEGORY(RkISP1Awb)
>
> +Awb::Awb()
> +	: rgbMode_(false)
> +{
> +}
> +
>  /**
>   * \copydoc libcamera::ipa::Algorithm::configure
>   */
> @@ -98,38 +103,60 @@ void Awb::prepare(IPAContext &context, const uint32_t frame,
>  	/* Update the gains. */
>  	params->module_cfg_update |= RKISP1_CIF_ISP_MODULE_AWB_GAIN;
>
> -	/* If we already have configured the gains and window, return. */
> +	/* If we already have set the AWB measurement parameters, return. */
>  	if (frame > 0)
>  		return;
>
> -	/* Configure the gains to apply. */
> +	rkisp1_cif_isp_awb_meas_config &awb_config = params->meas.awb_meas_config;
> +
> +	/* Configure the measure window for AWB. */
> +	awb_config.awb_wnd = context.configuration.awb.measureWindow;
> +
> +	/* Number of frames to use to estimate the means (0 means 1 frame). */
> +	awb_config.frames = 0;
> +
> +	/* Select RGB or YCbCr means measurement. */
> +	if (rgbMode_) {
> +		awb_config.awb_mode = RKISP1_CIF_ISP_AWB_MODE_RGB;
> +
> +		/*
> +		 * For RGB-based measurements, pixels are selected with maximum
> +		 * red, green and blue thresholds that are set in the
> +		 * awb_ref_cr, awb_min_y and awb_ref_cb respectively. The other
> +		 * values are not used, set them to 0.
> +		 */
> +		awb_config.awb_ref_cr = 250;
> +		awb_config.min_y = 250;
> +		awb_config.awb_ref_cb = 250;
> +
> +		awb_config.max_y = 0;
> +		awb_config.min_c = 0;
> +		awb_config.max_csum = 0;
> +	} else {
> +		awb_config.awb_mode = RKISP1_CIF_ISP_AWB_MODE_YCBCR;
> +
> +		/* Set the reference Cr and Cb (AWB target) to white. */
> +		awb_config.awb_ref_cb = 128;
> +		awb_config.awb_ref_cr = 128;
> +
> +		/*
> +		 * Filter out pixels based on luminance and chrominance values.
> +		 * The acceptable luma values are specified as a [16, 250]
> +		 * range, while the acceptable chrome values are specified with
> +		 * a minimum of 16 and a maximum Cb+Cr sum of 250.
> +		 */
> +		awb_config.min_y = 16;
> +		awb_config.max_y = 250;
> +		awb_config.min_c = 16;
> +		awb_config.max_csum = 250;
> +	}

I have no idea how to verify the above computations, but they match
the existing code, so...

Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>

Thanks
  j

> +
> +	/* Enable the AWB gains. */
>  	params->module_en_update |= RKISP1_CIF_ISP_MODULE_AWB_GAIN;
> -	/* Update the ISP to apply the gains configured. */
>  	params->module_ens |= RKISP1_CIF_ISP_MODULE_AWB_GAIN;
>
> -	/* Configure the measure window for AWB. */
> -	params->meas.awb_meas_config.awb_wnd = context.configuration.awb.measureWindow;
> -	/*
> -	 * Measure Y, Cr and Cb means.
> -	 * \todo RGB is not working, the kernel seems to not configure it ?
> -	 */
> -	params->meas.awb_meas_config.awb_mode = RKISP1_CIF_ISP_AWB_MODE_YCBCR;
> -	/* Reference Cr and Cb. */
> -	params->meas.awb_meas_config.awb_ref_cb = 128;
> -	params->meas.awb_meas_config.awb_ref_cr = 128;
> -	/* Y values to include are between min_y and max_y only. */
> -	params->meas.awb_meas_config.min_y = 16;
> -	params->meas.awb_meas_config.max_y = 250;
> -	/* Maximum Cr+Cb value to take into account for awb. */
> -	params->meas.awb_meas_config.max_csum = 250;
> -	/* Minimum Cr and Cb values to take into account. */
> -	params->meas.awb_meas_config.min_c = 16;
> -	/* Number of frames to use to estimate the mean (0 means 1 frame). */
> -	params->meas.awb_meas_config.frames = 0;
> -
> -	/* Update AWB measurement unit configuration. */
> +	/* Update the AWB measurement parameters and enable the AWB module. */
>  	params->module_cfg_update |= RKISP1_CIF_ISP_MODULE_AWB;
> -	/* Make sure the ISP is measuring the means for the next frame. */
>  	params->module_en_update |= RKISP1_CIF_ISP_MODULE_AWB;
>  	params->module_ens |= RKISP1_CIF_ISP_MODULE_AWB;
>  }
> @@ -182,30 +209,39 @@ void Awb::process(IPAContext &context,
>  	const rkisp1_cif_isp_stat *params = &stats->params;
>  	const rkisp1_cif_isp_awb_stat *awb = &params->awb;
>  	IPAActiveState &activeState = context.activeState;
> +	double greenMean;
> +	double redMean;
> +	double blueMean;
>
> -	/* Get the YCbCr mean values */
> -	double yMean = awb->awb_mean[0].mean_y_or_g;
> -	double crMean = awb->awb_mean[0].mean_cr_or_r;
> -	double cbMean = awb->awb_mean[0].mean_cb_or_b;
> +	if (rgbMode_) {
> +		greenMean = awb->awb_mean[0].mean_y_or_g;
> +		redMean = awb->awb_mean[0].mean_cr_or_r;
> +		blueMean = awb->awb_mean[0].mean_cb_or_b;
> +	} else {
> +		/* Get the YCbCr mean values */
> +		double yMean = awb->awb_mean[0].mean_y_or_g;
> +		double cbMean = awb->awb_mean[0].mean_cb_or_b;
> +		double crMean = awb->awb_mean[0].mean_cr_or_r;
>
> -	/*
> -	 * Convert from YCbCr to RGB.
> -	 * The hardware uses the following formulas:
> -	 * Y = 16 + 0.2500 R + 0.5000 G + 0.1094 B
> -	 * Cb = 128 - 0.1406 R - 0.2969 G + 0.4375 B
> -	 * Cr = 128 + 0.4375 R - 0.3750 G - 0.0625 B
> -	 *
> -	 * The inverse matrix is thus:
> -	 * [[1,1636, -0,0623,  1,6008]
> -	 *  [1,1636, -0,4045, -0,7949]
> -	 *  [1,1636,  1,9912, -0,0250]]
> -	 */
> -	yMean -= 16;
> -	cbMean -= 128;
> -	crMean -= 128;
> -	double redMean = 1.1636 * yMean - 0.0623 * cbMean + 1.6008 * crMean;
> -	double greenMean = 1.1636 * yMean - 0.4045 * cbMean - 0.7949 * crMean;
> -	double blueMean = 1.1636 * yMean + 1.9912 * cbMean - 0.0250 * crMean;
> +		/*
> +		 * Convert from YCbCr to RGB.
> +		 * The hardware uses the following formulas:
> +		 * Y = 16 + 0.2500 R + 0.5000 G + 0.1094 B
> +		 * Cb = 128 - 0.1406 R - 0.2969 G + 0.4375 B
> +		 * Cr = 128 + 0.4375 R - 0.3750 G - 0.0625 B
> +		 *
> +		 * The inverse matrix is thus:
> +		 * [[1,1636, -0,0623,  1,6008]
> +		 *  [1,1636, -0,4045, -0,7949]
> +		 *  [1,1636,  1,9912, -0,0250]]
> +		 */
> +		yMean -= 16;
> +		cbMean -= 128;
> +		crMean -= 128;
> +		redMean = 1.1636 * yMean - 0.0623 * cbMean + 1.6008 * crMean;
> +		greenMean = 1.1636 * yMean - 0.4045 * cbMean - 0.7949 * crMean;
> +		blueMean = 1.1636 * yMean + 1.9912 * cbMean - 0.0250 * crMean;
> +	}
>
>  	/*
>  	 * The ISP computes the AWB means after applying the colour gains,
> diff --git a/src/ipa/rkisp1/algorithms/awb.h b/src/ipa/rkisp1/algorithms/awb.h
> index 9fd156d8aff9..b1b861d930d1 100644
> --- a/src/ipa/rkisp1/algorithms/awb.h
> +++ b/src/ipa/rkisp1/algorithms/awb.h
> @@ -16,7 +16,7 @@ namespace ipa::rkisp1::algorithms {
>  class Awb : public Algorithm
>  {
>  public:
> -	Awb() = default;
> +	Awb();
>  	~Awb() = default;
>
>  	int configure(IPAContext &context, const IPACameraSensorInfo &configInfo) override;
> @@ -32,6 +32,8 @@ public:
>
>  private:
>  	uint32_t estimateCCT(double red, double green, double blue);
> +
> +	bool rgbMode_;
>  };
>
>  } /* namespace ipa::rkisp1::algorithms */
> --
> Regards,
>
> Laurent Pinchart
>
Laurent Pinchart Sept. 22, 2022, 10:24 p.m. UTC | #3 
Hi Kieran,

On Wed, Sep 21, 2022 at 12:31:58AM +0100, Kieran Bingham wrote:
> Quoting Laurent Pinchart via libcamera-devel (2022-09-08 02:41:55)
> > From: Quentin Schulz <quentin.schulz@theobroma-systems.com>
> > 
> > RkISP actually supports two modes for color means, RGB and YCbCr. The
> > variables where the means are stored are identically named regardless of
> > the color means mode that's been selected.
> > 
> > Since the gains are computed in RGB mode, a conversion needs to be done
> > when the mode is YCbCr, which is unnecessary when RGB mode is selected.
> > 
> > This adds support for RGB means mode too, by checking at runtime which
> > mode is selected at a given time. The default is still set to YCbCr mode
> > for now.
> 
> Interestingly, this sounds like we could change at runtime (with a
> capture script?) to change this to see any effect on the image. But such
> a control would be quite 'custom' or likely specific to RKISP?

It would be very custom indeed. I'm not sure what we'll do with this, it
could be set at tuning time, although I have a feeling we'll end up
selecting one mode and sticking with it.

> > Cc: Quentin Schulz <foss+libcamera@0leil.net>
> > Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
> > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > ---
> > Changes since v1:
> > 
> > - Fix compilation error by storing RGB mode flag in Awb class
> > - Set ISP AWB configuration based on the selected mode
> > ---
> >  src/ipa/rkisp1/algorithms/awb.cpp | 130 +++++++++++++++++++-----------
> >  src/ipa/rkisp1/algorithms/awb.h   |   4 +-
> >  2 files changed, 86 insertions(+), 48 deletions(-)
> > 
> > diff --git a/src/ipa/rkisp1/algorithms/awb.cpp b/src/ipa/rkisp1/algorithms/awb.cpp
> > index b711e93b73ba..2ff67ed98221 100644
> > --- a/src/ipa/rkisp1/algorithms/awb.cpp
> > +++ b/src/ipa/rkisp1/algorithms/awb.cpp
> > @@ -30,6 +30,11 @@ namespace ipa::rkisp1::algorithms {
> >  
> >  LOG_DEFINE_CATEGORY(RkISP1Awb)
> >  
> > +Awb::Awb()
> > +       : rgbMode_(false)
> > +{
> > +}
> > +
> >  /**
> >   * \copydoc libcamera::ipa::Algorithm::configure
> >   */
> > @@ -98,38 +103,60 @@ void Awb::prepare(IPAContext &context, const uint32_t frame,
> >         /* Update the gains. */
> >         params->module_cfg_update |= RKISP1_CIF_ISP_MODULE_AWB_GAIN;
> >  
> > -       /* If we already have configured the gains and window, return. */
> > +       /* If we already have set the AWB measurement parameters, return. */
> 
> "If we have already set the ..."
> 
> >         if (frame > 0)
> >                 return;
> >  
> > -       /* Configure the gains to apply. */
> > +       rkisp1_cif_isp_awb_meas_config &awb_config = params->meas.awb_meas_config;
> > +
> > +       /* Configure the measure window for AWB. */
> > +       awb_config.awb_wnd = context.configuration.awb.measureWindow;
> > +
> > +       /* Number of frames to use to estimate the means (0 means 1 frame). */
> > +       awb_config.frames = 0;
> 
> This sounds like something that could be a tuning file option. (But
> that isn't required now I expect).
> 
> That said, I think the RGB / YUV mode selection could be a tuning file
> option too?
> 
> But perhaps that's a 'yak for tomorrow'?

We'll need to experiment with different algorithms.

> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> 
> > +
> > +       /* Select RGB or YCbCr means measurement. */
> > +       if (rgbMode_) {
> > +               awb_config.awb_mode = RKISP1_CIF_ISP_AWB_MODE_RGB;
> > +
> > +               /*
> > +                * For RGB-based measurements, pixels are selected with maximum
> > +                * red, green and blue thresholds that are set in the
> > +                * awb_ref_cr, awb_min_y and awb_ref_cb respectively. The other
> > +                * values are not used, set them to 0.
> > +                */
> > +               awb_config.awb_ref_cr = 250;
> > +               awb_config.min_y = 250;
> > +               awb_config.awb_ref_cb = 250;
> > +
> > +               awb_config.max_y = 0;
> > +               awb_config.min_c = 0;
> > +               awb_config.max_csum = 0;
> > +       } else {
> > +               awb_config.awb_mode = RKISP1_CIF_ISP_AWB_MODE_YCBCR;
> > +
> > +               /* Set the reference Cr and Cb (AWB target) to white. */
> > +               awb_config.awb_ref_cb = 128;
> > +               awb_config.awb_ref_cr = 128;
> > +
> > +               /*
> > +                * Filter out pixels based on luminance and chrominance values.
> > +                * The acceptable luma values are specified as a [16, 250]
> > +                * range, while the acceptable chrome values are specified with
> > +                * a minimum of 16 and a maximum Cb+Cr sum of 250.
> > +                */
> > +               awb_config.min_y = 16;
> > +               awb_config.max_y = 250;
> > +               awb_config.min_c = 16;
> > +               awb_config.max_csum = 250;
> > +       }
> > +
> > +       /* Enable the AWB gains. */
> >         params->module_en_update |= RKISP1_CIF_ISP_MODULE_AWB_GAIN;
> > -       /* Update the ISP to apply the gains configured. */
> >         params->module_ens |= RKISP1_CIF_ISP_MODULE_AWB_GAIN;
> >  
> > -       /* Configure the measure window for AWB. */
> > -       params->meas.awb_meas_config.awb_wnd = context.configuration.awb.measureWindow;
> > -       /*
> > -        * Measure Y, Cr and Cb means.
> > -        * \todo RGB is not working, the kernel seems to not configure it ?
> > -        */
> > -       params->meas.awb_meas_config.awb_mode = RKISP1_CIF_ISP_AWB_MODE_YCBCR;
> > -       /* Reference Cr and Cb. */
> > -       params->meas.awb_meas_config.awb_ref_cb = 128;
> > -       params->meas.awb_meas_config.awb_ref_cr = 128;
> > -       /* Y values to include are between min_y and max_y only. */
> > -       params->meas.awb_meas_config.min_y = 16;
> > -       params->meas.awb_meas_config.max_y = 250;
> > -       /* Maximum Cr+Cb value to take into account for awb. */
> > -       params->meas.awb_meas_config.max_csum = 250;
> > -       /* Minimum Cr and Cb values to take into account. */
> > -       params->meas.awb_meas_config.min_c = 16;
> > -       /* Number of frames to use to estimate the mean (0 means 1 frame). */
> > -       params->meas.awb_meas_config.frames = 0;
> > -
> > -       /* Update AWB measurement unit configuration. */
> > +       /* Update the AWB measurement parameters and enable the AWB module. */
> >         params->module_cfg_update |= RKISP1_CIF_ISP_MODULE_AWB;
> > -       /* Make sure the ISP is measuring the means for the next frame. */
> >         params->module_en_update |= RKISP1_CIF_ISP_MODULE_AWB;
> >         params->module_ens |= RKISP1_CIF_ISP_MODULE_AWB;
> >  }
> > @@ -182,30 +209,39 @@ void Awb::process(IPAContext &context,
> >         const rkisp1_cif_isp_stat *params = &stats->params;
> >         const rkisp1_cif_isp_awb_stat *awb = &params->awb;
> >         IPAActiveState &activeState = context.activeState;
> > +       double greenMean;
> > +       double redMean;
> > +       double blueMean;
> >  
> > -       /* Get the YCbCr mean values */
> > -       double yMean = awb->awb_mean[0].mean_y_or_g;
> > -       double crMean = awb->awb_mean[0].mean_cr_or_r;
> > -       double cbMean = awb->awb_mean[0].mean_cb_or_b;
> > +       if (rgbMode_) {
> > +               greenMean = awb->awb_mean[0].mean_y_or_g;
> > +               redMean = awb->awb_mean[0].mean_cr_or_r;
> > +               blueMean = awb->awb_mean[0].mean_cb_or_b;
> > +       } else {
> > +               /* Get the YCbCr mean values */
> > +               double yMean = awb->awb_mean[0].mean_y_or_g;
> > +               double cbMean = awb->awb_mean[0].mean_cb_or_b;
> > +               double crMean = awb->awb_mean[0].mean_cr_or_r;
> >  
> > -       /*
> > -        * Convert from YCbCr to RGB.
> > -        * The hardware uses the following formulas:
> > -        * Y = 16 + 0.2500 R + 0.5000 G + 0.1094 B
> > -        * Cb = 128 - 0.1406 R - 0.2969 G + 0.4375 B
> > -        * Cr = 128 + 0.4375 R - 0.3750 G - 0.0625 B
> > -        *
> > -        * The inverse matrix is thus:
> > -        * [[1,1636, -0,0623,  1,6008]
> > -        *  [1,1636, -0,4045, -0,7949]
> > -        *  [1,1636,  1,9912, -0,0250]]
> > -        */
> > -       yMean -= 16;
> > -       cbMean -= 128;
> > -       crMean -= 128;
> > -       double redMean = 1.1636 * yMean - 0.0623 * cbMean + 1.6008 * crMean;
> > -       double greenMean = 1.1636 * yMean - 0.4045 * cbMean - 0.7949 * crMean;
> > -       double blueMean = 1.1636 * yMean + 1.9912 * cbMean - 0.0250 * crMean;
> > +               /*
> > +                * Convert from YCbCr to RGB.
> > +                * The hardware uses the following formulas:
> > +                * Y = 16 + 0.2500 R + 0.5000 G + 0.1094 B
> > +                * Cb = 128 - 0.1406 R - 0.2969 G + 0.4375 B
> > +                * Cr = 128 + 0.4375 R - 0.3750 G - 0.0625 B
> > +                *
> > +                * The inverse matrix is thus:
> > +                * [[1,1636, -0,0623,  1,6008]
> > +                *  [1,1636, -0,4045, -0,7949]
> > +                *  [1,1636,  1,9912, -0,0250]]
> > +                */
> > +               yMean -= 16;
> > +               cbMean -= 128;
> > +               crMean -= 128;
> > +               redMean = 1.1636 * yMean - 0.0623 * cbMean + 1.6008 * crMean;
> > +               greenMean = 1.1636 * yMean - 0.4045 * cbMean - 0.7949 * crMean;
> > +               blueMean = 1.1636 * yMean + 1.9912 * cbMean - 0.0250 * crMean;
> > +       }
> >  
> >         /*
> >          * The ISP computes the AWB means after applying the colour gains,
> > diff --git a/src/ipa/rkisp1/algorithms/awb.h b/src/ipa/rkisp1/algorithms/awb.h
> > index 9fd156d8aff9..b1b861d930d1 100644
> > --- a/src/ipa/rkisp1/algorithms/awb.h
> > +++ b/src/ipa/rkisp1/algorithms/awb.h
> > @@ -16,7 +16,7 @@ namespace ipa::rkisp1::algorithms {
> >  class Awb : public Algorithm
> >  {
> >  public:
> > -       Awb() = default;
> > +       Awb();
> >         ~Awb() = default;
> >  
> >         int configure(IPAContext &context, const IPACameraSensorInfo &configInfo) override;
> > @@ -32,6 +32,8 @@ public:
> >  
> >  private:
> >         uint32_t estimateCCT(double red, double green, double blue);
> > +
> > +       bool rgbMode_;
> >  };
> >  
> >  } /* namespace ipa::rkisp1::algorithms */

Patch
diff mbox series 

diff --git a/src/ipa/rkisp1/algorithms/awb.cpp b/src/ipa/rkisp1/algorithms/awb.cpp
index b711e93b73ba..2ff67ed98221 100644
--- a/src/ipa/rkisp1/algorithms/awb.cpp
+++ b/src/ipa/rkisp1/algorithms/awb.cpp
@@ -30,6 +30,11 @@  namespace ipa::rkisp1::algorithms {
 
 LOG_DEFINE_CATEGORY(RkISP1Awb)
 
+Awb::Awb()
+	: rgbMode_(false)
+{
+}
+
 /**
  * \copydoc libcamera::ipa::Algorithm::configure
  */
@@ -98,38 +103,60 @@  void Awb::prepare(IPAContext &context, const uint32_t frame,
 	/* Update the gains. */
 	params->module_cfg_update |= RKISP1_CIF_ISP_MODULE_AWB_GAIN;
 
-	/* If we already have configured the gains and window, return. */
+	/* If we already have set the AWB measurement parameters, return. */
 	if (frame > 0)
 		return;
 
-	/* Configure the gains to apply. */
+	rkisp1_cif_isp_awb_meas_config &awb_config = params->meas.awb_meas_config;
+
+	/* Configure the measure window for AWB. */
+	awb_config.awb_wnd = context.configuration.awb.measureWindow;
+
+	/* Number of frames to use to estimate the means (0 means 1 frame). */
+	awb_config.frames = 0;
+
+	/* Select RGB or YCbCr means measurement. */
+	if (rgbMode_) {
+		awb_config.awb_mode = RKISP1_CIF_ISP_AWB_MODE_RGB;
+
+		/*
+		 * For RGB-based measurements, pixels are selected with maximum
+		 * red, green and blue thresholds that are set in the
+		 * awb_ref_cr, awb_min_y and awb_ref_cb respectively. The other
+		 * values are not used, set them to 0.
+		 */
+		awb_config.awb_ref_cr = 250;
+		awb_config.min_y = 250;
+		awb_config.awb_ref_cb = 250;
+
+		awb_config.max_y = 0;
+		awb_config.min_c = 0;
+		awb_config.max_csum = 0;
+	} else {
+		awb_config.awb_mode = RKISP1_CIF_ISP_AWB_MODE_YCBCR;
+
+		/* Set the reference Cr and Cb (AWB target) to white. */
+		awb_config.awb_ref_cb = 128;
+		awb_config.awb_ref_cr = 128;
+
+		/*
+		 * Filter out pixels based on luminance and chrominance values.
+		 * The acceptable luma values are specified as a [16, 250]
+		 * range, while the acceptable chrome values are specified with
+		 * a minimum of 16 and a maximum Cb+Cr sum of 250.
+		 */
+		awb_config.min_y = 16;
+		awb_config.max_y = 250;
+		awb_config.min_c = 16;
+		awb_config.max_csum = 250;
+	}
+
+	/* Enable the AWB gains. */
 	params->module_en_update |= RKISP1_CIF_ISP_MODULE_AWB_GAIN;
-	/* Update the ISP to apply the gains configured. */
 	params->module_ens |= RKISP1_CIF_ISP_MODULE_AWB_GAIN;
 
-	/* Configure the measure window for AWB. */
-	params->meas.awb_meas_config.awb_wnd = context.configuration.awb.measureWindow;
-	/*
-	 * Measure Y, Cr and Cb means.
-	 * \todo RGB is not working, the kernel seems to not configure it ?
-	 */
-	params->meas.awb_meas_config.awb_mode = RKISP1_CIF_ISP_AWB_MODE_YCBCR;
-	/* Reference Cr and Cb. */
-	params->meas.awb_meas_config.awb_ref_cb = 128;
-	params->meas.awb_meas_config.awb_ref_cr = 128;
-	/* Y values to include are between min_y and max_y only. */
-	params->meas.awb_meas_config.min_y = 16;
-	params->meas.awb_meas_config.max_y = 250;
-	/* Maximum Cr+Cb value to take into account for awb. */
-	params->meas.awb_meas_config.max_csum = 250;
-	/* Minimum Cr and Cb values to take into account. */
-	params->meas.awb_meas_config.min_c = 16;
-	/* Number of frames to use to estimate the mean (0 means 1 frame). */
-	params->meas.awb_meas_config.frames = 0;
-
-	/* Update AWB measurement unit configuration. */
+	/* Update the AWB measurement parameters and enable the AWB module. */
 	params->module_cfg_update |= RKISP1_CIF_ISP_MODULE_AWB;
-	/* Make sure the ISP is measuring the means for the next frame. */
 	params->module_en_update |= RKISP1_CIF_ISP_MODULE_AWB;
 	params->module_ens |= RKISP1_CIF_ISP_MODULE_AWB;
 }
@@ -182,30 +209,39 @@  void Awb::process(IPAContext &context,
 	const rkisp1_cif_isp_stat *params = &stats->params;
 	const rkisp1_cif_isp_awb_stat *awb = &params->awb;
 	IPAActiveState &activeState = context.activeState;
+	double greenMean;
+	double redMean;
+	double blueMean;
 
-	/* Get the YCbCr mean values */
-	double yMean = awb->awb_mean[0].mean_y_or_g;
-	double crMean = awb->awb_mean[0].mean_cr_or_r;
-	double cbMean = awb->awb_mean[0].mean_cb_or_b;
+	if (rgbMode_) {
+		greenMean = awb->awb_mean[0].mean_y_or_g;
+		redMean = awb->awb_mean[0].mean_cr_or_r;
+		blueMean = awb->awb_mean[0].mean_cb_or_b;
+	} else {
+		/* Get the YCbCr mean values */
+		double yMean = awb->awb_mean[0].mean_y_or_g;
+		double cbMean = awb->awb_mean[0].mean_cb_or_b;
+		double crMean = awb->awb_mean[0].mean_cr_or_r;
 
-	/*
-	 * Convert from YCbCr to RGB.
-	 * The hardware uses the following formulas:
-	 * Y = 16 + 0.2500 R + 0.5000 G + 0.1094 B
-	 * Cb = 128 - 0.1406 R - 0.2969 G + 0.4375 B
-	 * Cr = 128 + 0.4375 R - 0.3750 G - 0.0625 B
-	 *
-	 * The inverse matrix is thus:
-	 * [[1,1636, -0,0623,  1,6008]
-	 *  [1,1636, -0,4045, -0,7949]
-	 *  [1,1636,  1,9912, -0,0250]]
-	 */
-	yMean -= 16;
-	cbMean -= 128;
-	crMean -= 128;
-	double redMean = 1.1636 * yMean - 0.0623 * cbMean + 1.6008 * crMean;
-	double greenMean = 1.1636 * yMean - 0.4045 * cbMean - 0.7949 * crMean;
-	double blueMean = 1.1636 * yMean + 1.9912 * cbMean - 0.0250 * crMean;
+		/*
+		 * Convert from YCbCr to RGB.
+		 * The hardware uses the following formulas:
+		 * Y = 16 + 0.2500 R + 0.5000 G + 0.1094 B
+		 * Cb = 128 - 0.1406 R - 0.2969 G + 0.4375 B
+		 * Cr = 128 + 0.4375 R - 0.3750 G - 0.0625 B
+		 *
+		 * The inverse matrix is thus:
+		 * [[1,1636, -0,0623,  1,6008]
+		 *  [1,1636, -0,4045, -0,7949]
+		 *  [1,1636,  1,9912, -0,0250]]
+		 */
+		yMean -= 16;
+		cbMean -= 128;
+		crMean -= 128;
+		redMean = 1.1636 * yMean - 0.0623 * cbMean + 1.6008 * crMean;
+		greenMean = 1.1636 * yMean - 0.4045 * cbMean - 0.7949 * crMean;
+		blueMean = 1.1636 * yMean + 1.9912 * cbMean - 0.0250 * crMean;
+	}
 
 	/*
 	 * The ISP computes the AWB means after applying the colour gains,
diff --git a/src/ipa/rkisp1/algorithms/awb.h b/src/ipa/rkisp1/algorithms/awb.h
index 9fd156d8aff9..b1b861d930d1 100644
--- a/src/ipa/rkisp1/algorithms/awb.h
+++ b/src/ipa/rkisp1/algorithms/awb.h
@@ -16,7 +16,7 @@  namespace ipa::rkisp1::algorithms {
 class Awb : public Algorithm
 {
 public:
-	Awb() = default;
+	Awb();
 	~Awb() = default;
 
 	int configure(IPAContext &context, const IPACameraSensorInfo &configInfo) override;
@@ -32,6 +32,8 @@  public:
 
 private:
 	uint32_t estimateCCT(double red, double green, double blue);
+
+	bool rgbMode_;
 };
 
 } /* namespace ipa::rkisp1::algorithms */