[v2] libcamera: software_isp: Exclude saturated pixels from colour sums
diff mbox series

Message ID 20261026173433.69339-1-robert.mader@collabora.com
State Superseded
Headers show
Series
  • [v2] libcamera: software_isp: Exclude saturated pixels from colour sums
Related show

Commit Message

Robert Mader Oct. 26, 2026, 5:34 p.m. UTC
From: Martin Neiva de Carvalho <martincarvalho@gmail.com>

The grey world estimate takes the sums of the colour channels over the
frame and assumes the scene averages to grey. Saturated pixels - both fully
and partially - break that: whatever colour a pixel was, once one or more
channels clip it carries wrong information about the illuminant. It is
however the brightest kind of pixel there is, and the sums are weighted
by brightness, so it carries a large share of the vote.

Measured on a scene with a sunlit white wall filling part of the frame,
9% of the photosites were clipped and, being at full scale, accounted
for roughly a third of the total sums. The estimate was dragged towards
neutral and left a visible cyan cast: surfaces that should have been
neutral came out with a linear R/G of 0.855 and B/G of 0.942. Leaving
the saturated pixels out brings those to 0.944 and 0.986.

Keep a separate count of the pixels that did contribute, because the
black level the IPA subtracts from the sums is a per-pixel offset and
would otherwise be scaled by the wrong number.

Signed-off-by: Martin Neiva de Carvalho <martincarvalho@gmail.com>
Tested-by: Tobias Predel <tobias.predel@gmail.com>
Tested-by: Geir Isene <geir@isene.com>
Reviewed-by: Hans de Goede <johannes.goede@oss.qualcomm.com>
Reviewed-by: Milan Zamazal <mzamazal@redhat.com>
Tested-by: Robert Mader <robert.mader@collabora.com>

---

Changes in V2:
 - Split out from the series "softisp: Five fixes found on a camera with no
   hardware ISP" (https://patchwork.libcamera.org/cover/28173/)
 - Apply suggestions by Hans and Milan to comments and the commit
   message - no changes to the code
 - Collect tags
---
 .../internal/software_isp/swisp_stats.h       |  8 ++++
 src/ipa/softisp/algorithms/awb.cpp            | 15 +++++--
 src/libcamera/software_isp/swstats_cpu.cpp    | 40 ++++++++++++++-----
 3 files changed, 48 insertions(+), 15 deletions(-)

Comments

Laurent Pinchart Oct. 28, 2026, 9:46 a.m. UTC | #1
On Mon, Oct 26, 2026 at 06:34:33PM +0100, Robert Mader wrote:
> From: Martin Neiva de Carvalho <martincarvalho@gmail.com>
> 
> The grey world estimate takes the sums of the colour channels over the
> frame and assumes the scene averages to grey. Saturated pixels - both fully
> and partially - break that: whatever colour a pixel was, once one or more
> channels clip it carries wrong information about the illuminant. It is
> however the brightest kind of pixel there is, and the sums are weighted
> by brightness, so it carries a large share of the vote.
> 
> Measured on a scene with a sunlit white wall filling part of the frame,
> 9% of the photosites were clipped and, being at full scale, accounted
> for roughly a third of the total sums. The estimate was dragged towards
> neutral and left a visible cyan cast: surfaces that should have been
> neutral came out with a linear R/G of 0.855 and B/G of 0.942. Leaving
> the saturated pixels out brings those to 0.944 and 0.986.
> 
> Keep a separate count of the pixels that did contribute, because the
> black level the IPA subtracts from the sums is a per-pixel offset and
> would otherwise be scaled by the wrong number.

While the code looks OK, the commit message makes little sense.

We're seeing an increase in the number of LLM-generated patches that
show little to no real understanding by the submitter of the issues they
address. They require a disproportionate amount of work from
maintainers. This is not sustainable, neither in libcamera nor in FOSS
in general.

I want to discourage this type of behaviour. As the code changes below
fix a real issue and are easy to review, I will post a new version of
this patch with manually written commit messages and comments, and strip
original authorship. Meat proxies do not deserve credit.

> Signed-off-by: Martin Neiva de Carvalho <martincarvalho@gmail.com>
> Tested-by: Tobias Predel <tobias.predel@gmail.com>
> Tested-by: Geir Isene <geir@isene.com>
> Reviewed-by: Hans de Goede <johannes.goede@oss.qualcomm.com>
> Reviewed-by: Milan Zamazal <mzamazal@redhat.com>
> Tested-by: Robert Mader <robert.mader@collabora.com>
> 
> ---
> 
> Changes in V2:
>  - Split out from the series "softisp: Five fixes found on a camera with no
>    hardware ISP" (https://patchwork.libcamera.org/cover/28173/)
>  - Apply suggestions by Hans and Milan to comments and the commit
>    message - no changes to the code
>  - Collect tags
> ---
>  .../internal/software_isp/swisp_stats.h       |  8 ++++
>  src/ipa/softisp/algorithms/awb.cpp            | 15 +++++--
>  src/libcamera/software_isp/swstats_cpu.cpp    | 40 ++++++++++++++-----
>  3 files changed, 48 insertions(+), 15 deletions(-)
> 
> diff --git a/include/libcamera/internal/software_isp/swisp_stats.h b/include/libcamera/internal/software_isp/swisp_stats.h
> index d9d0d9be84a6..aa315f6daecd 100644
> --- a/include/libcamera/internal/software_isp/swisp_stats.h
> +++ b/include/libcamera/internal/software_isp/swisp_stats.h
> @@ -31,6 +31,14 @@ struct SwIspStats {
>  	 * \brief Sums of colour channels of all the sampled pixels
>  	 */
>  	RGB<uint64_t> sum_;
> +	/**
> +	 * \brief Number of sampled pixels that contributed to sum_
> +	 *
> +	 * Saturated pixels are left out of the sums, so this can be smaller
> +	 * than the number of pixels sampled. It is needed to subtract the black
> +	 * level from the sums, which is a per-pixel offset.
> +	 */
> +	uint64_t sumSamples_;
>  	/**
>  	 * \brief Number of bins in the yHistogram
>  	 */
> diff --git a/src/ipa/softisp/algorithms/awb.cpp b/src/ipa/softisp/algorithms/awb.cpp
> index 55fd326fc182..b48441987a02 100644
> --- a/src/ipa/softisp/algorithms/awb.cpp
> +++ b/src/ipa/softisp/algorithms/awb.cpp
> @@ -7,7 +7,6 @@
>  
>  #include "awb.h"
>  
> -#include <numeric>
>  #include <stdint.h>
>  
>  #include <libcamera/base/log.h>
> @@ -118,16 +117,24 @@ SoftIspAwbStats Awb::calculateRgbMeans(IPAContext &context,
>  	if (!stats->valid)
>  		return {};
>  
> -	const SwIspStats::Histogram &histogram = stats->yHistogram;
>  	const uint8_t blackLevel = context.activeState.blc.level;
>  
> +	/*
> +	 * Saturated pixels are not counted in the sums, so the number of
> +	 * contributing pixels comes from the statistics rather than from the
> +	 * histogram, which counts every pixel sampled.
> +	 */
> +	const uint64_t nPixels = stats->sumSamples_;
> +
> +	/* Nothing but saturation was sampled, so there is nothing to say. */
> +	if (!nPixels)
> +		return {};
> +
>  	/*
>  	 * Black level must be subtracted to get the correct AWB ratios, they
>  	 * would be off if they were computed from the whole brightness range
>  	 * rather than from the sensor range.
>  	 */
> -	const uint64_t nPixels = std::accumulate(
> -		histogram.begin(), histogram.end(), uint64_t(0));
>  	const uint64_t offset = blackLevel * nPixels;
>  	const uint64_t minValid = 1;
>  
> diff --git a/src/libcamera/software_isp/swstats_cpu.cpp b/src/libcamera/software_isp/swstats_cpu.cpp
> index 7fb77ce7ddd6..7ae6ce4a5e93 100644
> --- a/src/libcamera/software_isp/swstats_cpu.cpp
> +++ b/src/libcamera/software_isp/swstats_cpu.cpp
> @@ -167,6 +167,14 @@ SwStatsCpu::SwStatsCpu(const CameraManager &cm)
>  			<< "Failed to create shared memory for statistics";
>  }
>  
> +/*
> + * Pixels with any colour channel at or above this value are treated as
> + * saturated. Fully saturated pixels don't represent the values for white
> + * balance computation correctly; nearly saturated pixels may provide inaccurate
> + * values from hardware.
> + */
> +static constexpr unsigned int kSaturationThreshold = 248;
> +
>  static constexpr unsigned int kRedYMul = 77; /* 0.299 * 256 */
>  static constexpr unsigned int kGreenYMul = 150; /* 0.587 * 256 */
>  static constexpr unsigned int kBlueYMul = 29; /* 0.114 * 256 */
> @@ -177,22 +185,29 @@ static constexpr unsigned int kBlueYMul = 29; /* 0.114 * 256 */
>                                            \
>  	uint64_t sumR = 0;                \
>  	uint64_t sumG = 0;                \
> -	uint64_t sumB = 0;
> -
> -#define SWSTATS_ACCUMULATE_LINE_STATS(div) \
> -	sumR += r;                         \
> -	sumG += g;                         \
> -	sumB += b;                         \
> -                                           \
> -	yVal = r * kRedYMul;               \
> -	yVal += g * kGreenYMul;            \
> -	yVal += b * kBlueYMul;             \
> +	uint64_t sumB = 0;                \
> +	uint64_t nSamples = 0;
> +
> +#define SWSTATS_ACCUMULATE_LINE_STATS(div)      \
> +	if (r < kSaturationThreshold * (div) && \
> +	    g < kSaturationThreshold * (div) && \
> +	    b < kSaturationThreshold * (div)) { \
> +		sumR += r;                      \
> +		sumG += g;                      \
> +		sumB += b;                      \
> +		nSamples++;                     \
> +	}                                       \
> +                                                \
> +	yVal = r * kRedYMul;                    \
> +	yVal += g * kGreenYMul;                 \
> +	yVal += b * kBlueYMul;                  \
>  	stats.yHistogram[yVal * SwIspStats::kYHistogramSize / (256 * 256 * (div))]++;
>  
>  #define SWSTATS_FINISH_LINE_STATS() \
>  	stats.sum_.r() += sumR;     \
>  	stats.sum_.g() += sumG;     \
> -	stats.sum_.b() += sumB;
> +	stats.sum_.b() += sumB;     \
> +	stats.sumSamples_ += nSamples;
>  
>  void SwStatsCpu::statsBGGR8Line0(const uint8_t *src[], SwIspStats &stats)
>  {
> @@ -391,6 +406,7 @@ void SwStatsCpu::startFrame(uint32_t frame)
>  
>  	for (auto &s : stats_) {
>  		s.sum_ = RGB<uint64_t>({ 0, 0, 0 });
> +		s.sumSamples_ = 0;
>  		s.yHistogram.fill(0);
>  	}
>  }
> @@ -408,9 +424,11 @@ void SwStatsCpu::finishFrame(uint32_t frame, uint32_t bufferId)
>  
>  	if (valid) {
>  		sharedStats_->sum_ = RGB<uint64_t>({ 0, 0, 0 });
> +		sharedStats_->sumSamples_ = 0;
>  		sharedStats_->yHistogram.fill(0);
>  		for (const auto &s : stats_) {
>  			sharedStats_->sum_ += s.sum_;
> +			sharedStats_->sumSamples_ += s.sumSamples_;
>  			for (unsigned int j = 0; j < SwIspStats::kYHistogramSize; j++)
>  				sharedStats_->yHistogram[j] += s.yHistogram[j];
>  		}

Patch
diff mbox series

diff --git a/include/libcamera/internal/software_isp/swisp_stats.h b/include/libcamera/internal/software_isp/swisp_stats.h
index d9d0d9be84a6..aa315f6daecd 100644
--- a/include/libcamera/internal/software_isp/swisp_stats.h
+++ b/include/libcamera/internal/software_isp/swisp_stats.h
@@ -31,6 +31,14 @@  struct SwIspStats {
 	 * \brief Sums of colour channels of all the sampled pixels
 	 */
 	RGB<uint64_t> sum_;
+	/**
+	 * \brief Number of sampled pixels that contributed to sum_
+	 *
+	 * Saturated pixels are left out of the sums, so this can be smaller
+	 * than the number of pixels sampled. It is needed to subtract the black
+	 * level from the sums, which is a per-pixel offset.
+	 */
+	uint64_t sumSamples_;
 	/**
 	 * \brief Number of bins in the yHistogram
 	 */
diff --git a/src/ipa/softisp/algorithms/awb.cpp b/src/ipa/softisp/algorithms/awb.cpp
index 55fd326fc182..b48441987a02 100644
--- a/src/ipa/softisp/algorithms/awb.cpp
+++ b/src/ipa/softisp/algorithms/awb.cpp
@@ -7,7 +7,6 @@ 
 
 #include "awb.h"
 
-#include <numeric>
 #include <stdint.h>
 
 #include <libcamera/base/log.h>
@@ -118,16 +117,24 @@  SoftIspAwbStats Awb::calculateRgbMeans(IPAContext &context,
 	if (!stats->valid)
 		return {};
 
-	const SwIspStats::Histogram &histogram = stats->yHistogram;
 	const uint8_t blackLevel = context.activeState.blc.level;
 
+	/*
+	 * Saturated pixels are not counted in the sums, so the number of
+	 * contributing pixels comes from the statistics rather than from the
+	 * histogram, which counts every pixel sampled.
+	 */
+	const uint64_t nPixels = stats->sumSamples_;
+
+	/* Nothing but saturation was sampled, so there is nothing to say. */
+	if (!nPixels)
+		return {};
+
 	/*
 	 * Black level must be subtracted to get the correct AWB ratios, they
 	 * would be off if they were computed from the whole brightness range
 	 * rather than from the sensor range.
 	 */
-	const uint64_t nPixels = std::accumulate(
-		histogram.begin(), histogram.end(), uint64_t(0));
 	const uint64_t offset = blackLevel * nPixels;
 	const uint64_t minValid = 1;
 
diff --git a/src/libcamera/software_isp/swstats_cpu.cpp b/src/libcamera/software_isp/swstats_cpu.cpp
index 7fb77ce7ddd6..7ae6ce4a5e93 100644
--- a/src/libcamera/software_isp/swstats_cpu.cpp
+++ b/src/libcamera/software_isp/swstats_cpu.cpp
@@ -167,6 +167,14 @@  SwStatsCpu::SwStatsCpu(const CameraManager &cm)
 			<< "Failed to create shared memory for statistics";
 }
 
+/*
+ * Pixels with any colour channel at or above this value are treated as
+ * saturated. Fully saturated pixels don't represent the values for white
+ * balance computation correctly; nearly saturated pixels may provide inaccurate
+ * values from hardware.
+ */
+static constexpr unsigned int kSaturationThreshold = 248;
+
 static constexpr unsigned int kRedYMul = 77; /* 0.299 * 256 */
 static constexpr unsigned int kGreenYMul = 150; /* 0.587 * 256 */
 static constexpr unsigned int kBlueYMul = 29; /* 0.114 * 256 */
@@ -177,22 +185,29 @@  static constexpr unsigned int kBlueYMul = 29; /* 0.114 * 256 */
                                           \
 	uint64_t sumR = 0;                \
 	uint64_t sumG = 0;                \
-	uint64_t sumB = 0;
-
-#define SWSTATS_ACCUMULATE_LINE_STATS(div) \
-	sumR += r;                         \
-	sumG += g;                         \
-	sumB += b;                         \
-                                           \
-	yVal = r * kRedYMul;               \
-	yVal += g * kGreenYMul;            \
-	yVal += b * kBlueYMul;             \
+	uint64_t sumB = 0;                \
+	uint64_t nSamples = 0;
+
+#define SWSTATS_ACCUMULATE_LINE_STATS(div)      \
+	if (r < kSaturationThreshold * (div) && \
+	    g < kSaturationThreshold * (div) && \
+	    b < kSaturationThreshold * (div)) { \
+		sumR += r;                      \
+		sumG += g;                      \
+		sumB += b;                      \
+		nSamples++;                     \
+	}                                       \
+                                                \
+	yVal = r * kRedYMul;                    \
+	yVal += g * kGreenYMul;                 \
+	yVal += b * kBlueYMul;                  \
 	stats.yHistogram[yVal * SwIspStats::kYHistogramSize / (256 * 256 * (div))]++;
 
 #define SWSTATS_FINISH_LINE_STATS() \
 	stats.sum_.r() += sumR;     \
 	stats.sum_.g() += sumG;     \
-	stats.sum_.b() += sumB;
+	stats.sum_.b() += sumB;     \
+	stats.sumSamples_ += nSamples;
 
 void SwStatsCpu::statsBGGR8Line0(const uint8_t *src[], SwIspStats &stats)
 {
@@ -391,6 +406,7 @@  void SwStatsCpu::startFrame(uint32_t frame)
 
 	for (auto &s : stats_) {
 		s.sum_ = RGB<uint64_t>({ 0, 0, 0 });
+		s.sumSamples_ = 0;
 		s.yHistogram.fill(0);
 	}
 }
@@ -408,9 +424,11 @@  void SwStatsCpu::finishFrame(uint32_t frame, uint32_t bufferId)
 
 	if (valid) {
 		sharedStats_->sum_ = RGB<uint64_t>({ 0, 0, 0 });
+		sharedStats_->sumSamples_ = 0;
 		sharedStats_->yHistogram.fill(0);
 		for (const auto &s : stats_) {
 			sharedStats_->sum_ += s.sum_;
+			sharedStats_->sumSamples_ += s.sumSamples_;
 			for (unsigned int j = 0; j < SwIspStats::kYHistogramSize; j++)
 				sharedStats_->yHistogram[j] += s.yHistogram[j];
 		}