[v6,00/10] Software ISP support for CCM

22728 mbox series

This series implements application of color correction matrices in
software ISP.

Support for color temperature is added to auto white balance algorithm,
this is needed to obtain the right CCM.  A new Ccm algorithm is added to
determine the matrix.  Both roughly follow what the hardware pipelines
do.

Lut algorithm is modified to incorporate the CCM and perform some
precomputations in the form of 4 lookup tables (one per each matrix
column and one for gamma).

A tricky part is to support both CCM and the original (no-CCM)
transformations.  CCM obviously adds a significant overhead; per-frame
time increases by ~45% on Debix Model A and ~85% on TI AM69 SK.  This
means CCM must be optional, which is determined by presence of Ccm
algorithm in the tuning file.

The performance critical debayering code must not include extra
conditionals and must work efficiently for both the cases.  The macros
in debayering code are refactored and restructured for this.  An added
bonus is that this also removes some annoying code duplication.  And
some more, partially already present, templating is used to select
between CCM and non-CCM processing.

Performance is much influenced by the data structures used in
debayering.  I tried to define something reasonable and performed only
basic performance testing.  Suggestions for possible improvements are
welcome.  There is already one from Hans about avoiding using std::clamp
in debayering but I don’t have a working patch for this yet.

When trying to use the following imx219 color correction matrix from rpi
subtree, I get quite noticeable color casts:

 - Ccm:
     ccms:
       - ct: 2860
         ccm: [ 2.12089, -0.52461, -0.59629,
                -0.85342, 2.80445, -0.95103,
                -0.26897, -1.14788, 2.41685 ]
       - ct: 2960
         ccm: [ 2.26962, -0.54174, -0.72789,
                -0.77008, 2.60271, -0.83262,
                -0.26036, -1.51254, 2.77289 ]
       - ct: 3603
         ccm: [ 2.18644, -0.66148, -0.52496,
                -0.77828, 2.69474, -0.91645,
                -0.25239, -0.83059, 2.08298 ]
       - ct: 4650
         ccm: [ 2.18174, -0.70887, -0.47287,
                -0.70196, 2.76426, -1.06231,
                -0.25157, -0.71978, 1.97135 ]
       - ct: 5858
         ccm: [ 2.32392, -0.88421, -0.43971,
                -0.63821, 2.58348, -0.94527,
                -0.28541, -0.54112, 1.82653 ]
       - ct: 7580
         ccm: [ 2.21175, -0.53242, -0.67933,
                -0.57875, 3.07922, -1.50047,
                -0.27709, -0.73338, 2.01048 ]

One of the possible explanations could be that the software ISP AWB
(used to determine the color temperature) is no way accurate and
produces some (different) color casts too.  But it can also be caused by
something else.

Changes in v6:
- A transposed CCM tables access was used in debayering, fixed.
  Unfortunately this slows down the processing by almost 10% in my
  environment, apparently due to worse locality.
- When red-blue swapping is applied in debayering, `r' and `b' items in
  the CCM lookup tables must be swapped too, fixed.
- The multiplication is now CCM * RGB rather than RGB * CCM; this
  matches the representation of CCMs in the tuning files.
- CCM and gain value type changed from double to float to be consistent
  with the rkisp1 pipeline and controls::ColourCorrectionMatrix.
- Some code simplifications suggested by Laurent.
- Comments added to lookup tables declarations.

Changes in v5:
- Local variable to store temperature removed; the context variable used
  directly instead.
- Patch to use RGB<double> instead of a custom struct to store gains
  added.
- Superfluous include of stdlib.h removed from ccm.cpp.
- Error returned from Ccm::init when there is no CCM defined.
- Ccm::kTemperatureThreshold moved to a constexpr global variable.
- `const' added to const variables.
- Separate color lookup tables for CCM / simple rather than unions.
- ccmEnabled passed as a normal rather than template argument to
  setDebayerFunctions.
- ccmEnabled is determined in init rather than in configure.

Changes in v4:
- utils::abs_diff used instead of the ugly typecasting.
- A missing argument type added to Debayer::configure docstring.
- CCM and related processing is disabled when tuning data is not
  available.
- Unneeded initialization of context_.activeState.ccm.enabled to the
  default value removed.

Changes in v3:
- Rebased on master.
- AWB gains applied.
- ccmEnabled docstring moved to the right patch.
- Small notes about std::variant and about std::clamp added to the
  commit message of the last patch.

Changes in v2:
- Fix of the gamma table index upper boundary.
- Fix of red<->blue being temporarily swapped in an intermediate patch.
- Fix of CCM transformation when swapRedBlueGains_ is true.
- A related minor lookup data structures rearrangement, with an
  unintended but welcome effect of ~10% speedup with CCM in my
  environment. 

Milan Zamazal (10):
  libcamera: software_isp: Determine color temperature
  libcamera: software_isp: Use RGB type to represent gains
  libcamera: software_isp: Store color temperature to metadata
  libcamera: software_isp: lut: Remove maybe_unused on a used argument
  libcamera: software_isp: Use common code to store debayered pixels
  libcamera: software_isp: Use a macro to assign debayering methods
  libcamera: software_isp: Add CCM algorithm
  libcamera: software_isp: Add an example CCM to uncalibrated.yaml
  libcamera: software_isp: Track whether CCM is enabled
  libcamera: software_isp: Apply CCM in debayering

 .../internal/software_isp/debayer_params.h    |  34 +++-
 .../internal/software_isp/software_isp.h      |   1 +
 include/libcamera/ipa/soft.mojom              |   2 +-
 src/ipa/simple/algorithms/awb.cpp             |  29 +++-
 src/ipa/simple/algorithms/ccm.cpp             |  76 +++++++++
 src/ipa/simple/algorithms/ccm.h               |  43 +++++
 src/ipa/simple/algorithms/lut.cpp             |  63 ++++---
 src/ipa/simple/algorithms/lut.h               |   1 +
 src/ipa/simple/algorithms/meson.build         |   1 +
 src/ipa/simple/data/uncalibrated.yaml         |   9 +
 src/ipa/simple/ipa_context.h                  |  21 ++-
 src/ipa/simple/soft_simple.cpp                |   8 +-
 src/libcamera/software_isp/debayer.cpp        |  55 +++++-
 src/libcamera/software_isp/debayer.h          |   3 +-
 src/libcamera/software_isp/debayer_cpu.cpp    | 158 +++++++++++-------
 src/libcamera/software_isp/debayer_cpu.h      |  39 +++--
 src/libcamera/software_isp/software_isp.cpp   |   5 +-
 17 files changed, 427 insertions(+), 121 deletions(-)
 create mode 100644 src/ipa/simple/algorithms/ccm.cpp
 create mode 100644 src/ipa/simple/algorithms/ccm.h