| Message ID | 20261026173433.69339-1-robert.mader@collabora.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series |
|
| Related | show |
On Mon, Oct 26, 2026 at 06:34:33PM +0100, Robert Mader wrote: > From: Martin Neiva de Carvalho <martincarvalho@gmail.com> > > The grey world estimate takes the sums of the colour channels over the > frame and assumes the scene averages to grey. Saturated pixels - both fully > and partially - break that: whatever colour a pixel was, once one or more > channels clip it carries wrong information about the illuminant. It is > however the brightest kind of pixel there is, and the sums are weighted > by brightness, so it carries a large share of the vote. > > Measured on a scene with a sunlit white wall filling part of the frame, > 9% of the photosites were clipped and, being at full scale, accounted > for roughly a third of the total sums. The estimate was dragged towards > neutral and left a visible cyan cast: surfaces that should have been > neutral came out with a linear R/G of 0.855 and B/G of 0.942. Leaving > the saturated pixels out brings those to 0.944 and 0.986. > > Keep a separate count of the pixels that did contribute, because the > black level the IPA subtracts from the sums is a per-pixel offset and > would otherwise be scaled by the wrong number. While the code looks OK, the commit message makes little sense. We're seeing an increase in the number of LLM-generated patches that show little to no real understanding by the submitter of the issues they address. They require a disproportionate amount of work from maintainers. This is not sustainable, neither in libcamera nor in FOSS in general. I want to discourage this type of behaviour. As the code changes below fix a real issue and are easy to review, I will post a new version of this patch with manually written commit messages and comments, and strip original authorship. Meat proxies do not deserve credit. > Signed-off-by: Martin Neiva de Carvalho <martincarvalho@gmail.com> > Tested-by: Tobias Predel <tobias.predel@gmail.com> > Tested-by: Geir Isene <geir@isene.com> > Reviewed-by: Hans de Goede <johannes.goede@oss.qualcomm.com> > Reviewed-by: Milan Zamazal <mzamazal@redhat.com> > Tested-by: Robert Mader <robert.mader@collabora.com> > > --- > > Changes in V2: > - Split out from the series "softisp: Five fixes found on a camera with no > hardware ISP" (https://patchwork.libcamera.org/cover/28173/) > - Apply suggestions by Hans and Milan to comments and the commit > message - no changes to the code > - Collect tags > --- > .../internal/software_isp/swisp_stats.h | 8 ++++ > src/ipa/softisp/algorithms/awb.cpp | 15 +++++-- > src/libcamera/software_isp/swstats_cpu.cpp | 40 ++++++++++++++----- > 3 files changed, 48 insertions(+), 15 deletions(-) > > diff --git a/include/libcamera/internal/software_isp/swisp_stats.h b/include/libcamera/internal/software_isp/swisp_stats.h > index d9d0d9be84a6..aa315f6daecd 100644 > --- a/include/libcamera/internal/software_isp/swisp_stats.h > +++ b/include/libcamera/internal/software_isp/swisp_stats.h > @@ -31,6 +31,14 @@ struct SwIspStats { > * \brief Sums of colour channels of all the sampled pixels > */ > RGB<uint64_t> sum_; > + /** > + * \brief Number of sampled pixels that contributed to sum_ > + * > + * Saturated pixels are left out of the sums, so this can be smaller > + * than the number of pixels sampled. It is needed to subtract the black > + * level from the sums, which is a per-pixel offset. > + */ > + uint64_t sumSamples_; > /** > * \brief Number of bins in the yHistogram > */ > diff --git a/src/ipa/softisp/algorithms/awb.cpp b/src/ipa/softisp/algorithms/awb.cpp > index 55fd326fc182..b48441987a02 100644 > --- a/src/ipa/softisp/algorithms/awb.cpp > +++ b/src/ipa/softisp/algorithms/awb.cpp > @@ -7,7 +7,6 @@ > > #include "awb.h" > > -#include <numeric> > #include <stdint.h> > > #include <libcamera/base/log.h> > @@ -118,16 +117,24 @@ SoftIspAwbStats Awb::calculateRgbMeans(IPAContext &context, > if (!stats->valid) > return {}; > > - const SwIspStats::Histogram &histogram = stats->yHistogram; > const uint8_t blackLevel = context.activeState.blc.level; > > + /* > + * Saturated pixels are not counted in the sums, so the number of > + * contributing pixels comes from the statistics rather than from the > + * histogram, which counts every pixel sampled. > + */ > + const uint64_t nPixels = stats->sumSamples_; > + > + /* Nothing but saturation was sampled, so there is nothing to say. */ > + if (!nPixels) > + return {}; > + > /* > * Black level must be subtracted to get the correct AWB ratios, they > * would be off if they were computed from the whole brightness range > * rather than from the sensor range. > */ > - const uint64_t nPixels = std::accumulate( > - histogram.begin(), histogram.end(), uint64_t(0)); > const uint64_t offset = blackLevel * nPixels; > const uint64_t minValid = 1; > > diff --git a/src/libcamera/software_isp/swstats_cpu.cpp b/src/libcamera/software_isp/swstats_cpu.cpp > index 7fb77ce7ddd6..7ae6ce4a5e93 100644 > --- a/src/libcamera/software_isp/swstats_cpu.cpp > +++ b/src/libcamera/software_isp/swstats_cpu.cpp > @@ -167,6 +167,14 @@ SwStatsCpu::SwStatsCpu(const CameraManager &cm) > << "Failed to create shared memory for statistics"; > } > > +/* > + * Pixels with any colour channel at or above this value are treated as > + * saturated. Fully saturated pixels don't represent the values for white > + * balance computation correctly; nearly saturated pixels may provide inaccurate > + * values from hardware. > + */ > +static constexpr unsigned int kSaturationThreshold = 248; > + > static constexpr unsigned int kRedYMul = 77; /* 0.299 * 256 */ > static constexpr unsigned int kGreenYMul = 150; /* 0.587 * 256 */ > static constexpr unsigned int kBlueYMul = 29; /* 0.114 * 256 */ > @@ -177,22 +185,29 @@ static constexpr unsigned int kBlueYMul = 29; /* 0.114 * 256 */ > \ > uint64_t sumR = 0; \ > uint64_t sumG = 0; \ > - uint64_t sumB = 0; > - > -#define SWSTATS_ACCUMULATE_LINE_STATS(div) \ > - sumR += r; \ > - sumG += g; \ > - sumB += b; \ > - \ > - yVal = r * kRedYMul; \ > - yVal += g * kGreenYMul; \ > - yVal += b * kBlueYMul; \ > + uint64_t sumB = 0; \ > + uint64_t nSamples = 0; > + > +#define SWSTATS_ACCUMULATE_LINE_STATS(div) \ > + if (r < kSaturationThreshold * (div) && \ > + g < kSaturationThreshold * (div) && \ > + b < kSaturationThreshold * (div)) { \ > + sumR += r; \ > + sumG += g; \ > + sumB += b; \ > + nSamples++; \ > + } \ > + \ > + yVal = r * kRedYMul; \ > + yVal += g * kGreenYMul; \ > + yVal += b * kBlueYMul; \ > stats.yHistogram[yVal * SwIspStats::kYHistogramSize / (256 * 256 * (div))]++; > > #define SWSTATS_FINISH_LINE_STATS() \ > stats.sum_.r() += sumR; \ > stats.sum_.g() += sumG; \ > - stats.sum_.b() += sumB; > + stats.sum_.b() += sumB; \ > + stats.sumSamples_ += nSamples; > > void SwStatsCpu::statsBGGR8Line0(const uint8_t *src[], SwIspStats &stats) > { > @@ -391,6 +406,7 @@ void SwStatsCpu::startFrame(uint32_t frame) > > for (auto &s : stats_) { > s.sum_ = RGB<uint64_t>({ 0, 0, 0 }); > + s.sumSamples_ = 0; > s.yHistogram.fill(0); > } > } > @@ -408,9 +424,11 @@ void SwStatsCpu::finishFrame(uint32_t frame, uint32_t bufferId) > > if (valid) { > sharedStats_->sum_ = RGB<uint64_t>({ 0, 0, 0 }); > + sharedStats_->sumSamples_ = 0; > sharedStats_->yHistogram.fill(0); > for (const auto &s : stats_) { > sharedStats_->sum_ += s.sum_; > + sharedStats_->sumSamples_ += s.sumSamples_; > for (unsigned int j = 0; j < SwIspStats::kYHistogramSize; j++) > sharedStats_->yHistogram[j] += s.yHistogram[j]; > }
diff --git a/include/libcamera/internal/software_isp/swisp_stats.h b/include/libcamera/internal/software_isp/swisp_stats.h index d9d0d9be84a6..aa315f6daecd 100644 --- a/include/libcamera/internal/software_isp/swisp_stats.h +++ b/include/libcamera/internal/software_isp/swisp_stats.h @@ -31,6 +31,14 @@ struct SwIspStats { * \brief Sums of colour channels of all the sampled pixels */ RGB<uint64_t> sum_; + /** + * \brief Number of sampled pixels that contributed to sum_ + * + * Saturated pixels are left out of the sums, so this can be smaller + * than the number of pixels sampled. It is needed to subtract the black + * level from the sums, which is a per-pixel offset. + */ + uint64_t sumSamples_; /** * \brief Number of bins in the yHistogram */ diff --git a/src/ipa/softisp/algorithms/awb.cpp b/src/ipa/softisp/algorithms/awb.cpp index 55fd326fc182..b48441987a02 100644 --- a/src/ipa/softisp/algorithms/awb.cpp +++ b/src/ipa/softisp/algorithms/awb.cpp @@ -7,7 +7,6 @@ #include "awb.h" -#include <numeric> #include <stdint.h> #include <libcamera/base/log.h> @@ -118,16 +117,24 @@ SoftIspAwbStats Awb::calculateRgbMeans(IPAContext &context, if (!stats->valid) return {}; - const SwIspStats::Histogram &histogram = stats->yHistogram; const uint8_t blackLevel = context.activeState.blc.level; + /* + * Saturated pixels are not counted in the sums, so the number of + * contributing pixels comes from the statistics rather than from the + * histogram, which counts every pixel sampled. + */ + const uint64_t nPixels = stats->sumSamples_; + + /* Nothing but saturation was sampled, so there is nothing to say. */ + if (!nPixels) + return {}; + /* * Black level must be subtracted to get the correct AWB ratios, they * would be off if they were computed from the whole brightness range * rather than from the sensor range. */ - const uint64_t nPixels = std::accumulate( - histogram.begin(), histogram.end(), uint64_t(0)); const uint64_t offset = blackLevel * nPixels; const uint64_t minValid = 1; diff --git a/src/libcamera/software_isp/swstats_cpu.cpp b/src/libcamera/software_isp/swstats_cpu.cpp index 7fb77ce7ddd6..7ae6ce4a5e93 100644 --- a/src/libcamera/software_isp/swstats_cpu.cpp +++ b/src/libcamera/software_isp/swstats_cpu.cpp @@ -167,6 +167,14 @@ SwStatsCpu::SwStatsCpu(const CameraManager &cm) << "Failed to create shared memory for statistics"; } +/* + * Pixels with any colour channel at or above this value are treated as + * saturated. Fully saturated pixels don't represent the values for white + * balance computation correctly; nearly saturated pixels may provide inaccurate + * values from hardware. + */ +static constexpr unsigned int kSaturationThreshold = 248; + static constexpr unsigned int kRedYMul = 77; /* 0.299 * 256 */ static constexpr unsigned int kGreenYMul = 150; /* 0.587 * 256 */ static constexpr unsigned int kBlueYMul = 29; /* 0.114 * 256 */ @@ -177,22 +185,29 @@ static constexpr unsigned int kBlueYMul = 29; /* 0.114 * 256 */ \ uint64_t sumR = 0; \ uint64_t sumG = 0; \ - uint64_t sumB = 0; - -#define SWSTATS_ACCUMULATE_LINE_STATS(div) \ - sumR += r; \ - sumG += g; \ - sumB += b; \ - \ - yVal = r * kRedYMul; \ - yVal += g * kGreenYMul; \ - yVal += b * kBlueYMul; \ + uint64_t sumB = 0; \ + uint64_t nSamples = 0; + +#define SWSTATS_ACCUMULATE_LINE_STATS(div) \ + if (r < kSaturationThreshold * (div) && \ + g < kSaturationThreshold * (div) && \ + b < kSaturationThreshold * (div)) { \ + sumR += r; \ + sumG += g; \ + sumB += b; \ + nSamples++; \ + } \ + \ + yVal = r * kRedYMul; \ + yVal += g * kGreenYMul; \ + yVal += b * kBlueYMul; \ stats.yHistogram[yVal * SwIspStats::kYHistogramSize / (256 * 256 * (div))]++; #define SWSTATS_FINISH_LINE_STATS() \ stats.sum_.r() += sumR; \ stats.sum_.g() += sumG; \ - stats.sum_.b() += sumB; + stats.sum_.b() += sumB; \ + stats.sumSamples_ += nSamples; void SwStatsCpu::statsBGGR8Line0(const uint8_t *src[], SwIspStats &stats) { @@ -391,6 +406,7 @@ void SwStatsCpu::startFrame(uint32_t frame) for (auto &s : stats_) { s.sum_ = RGB<uint64_t>({ 0, 0, 0 }); + s.sumSamples_ = 0; s.yHistogram.fill(0); } } @@ -408,9 +424,11 @@ void SwStatsCpu::finishFrame(uint32_t frame, uint32_t bufferId) if (valid) { sharedStats_->sum_ = RGB<uint64_t>({ 0, 0, 0 }); + sharedStats_->sumSamples_ = 0; sharedStats_->yHistogram.fill(0); for (const auto &s : stats_) { sharedStats_->sum_ += s.sum_; + sharedStats_->sumSamples_ += s.sumSamples_; for (unsigned int j = 0; j < SwIspStats::kYHistogramSize; j++) sharedStats_->yHistogram[j] += s.yHistogram[j]; }