[v3,0/9] Software ISP support for CCM

22276 mbox series

This series implements application of color correction matrices in
software ISP.

Support for color temperature is added to auto white balance algorithm,
this is needed to obtain the right CCM.  A new Ccm algorithm is added to
determine the matrix.  Both roughly follow what the hardware pipelines
do.

Lut algorithm is modified to incorporate the CCM and perform some
precomputations in the form of 10 lookup tables (one per each matrix
element and one for gamma).

A tricky part is to support both CCM and the original (no-CCM)
transformations.  CCM obviously adds a significant overhead; per-frame
time increases by ~45% on Debix Model A and ~85% on TI AM69 SK.  This
means CCM must be optional, which is determined by presence of Ccm
algorithm in the tuning file.

The performance critical debayering code must not include extra
conditionals and must work efficiently for both the cases.  The macros
in debayering code are refactored and restructured for this.  An added
bonus is that this also removes some annoying code duplication.  And
some more, partially already present, templating is used to select
between CCM and non-CCM processing.

Performance is much influenced by the data structures used in
debayering.  I tried to define something reasonable and performed only
basic performance testing.  Suggestions for possible improvements are
welcome.  There is already one from Hans about avoiding using std::clamp
in debayering but I don’t have a working patch for this yet.

When trying to use the following imx219 color correction matrix from rpi
subtree, I get quite noticeable color casts:

 - Ccm:
     ccms:
       - ct: 2860
         ccm: [ 2.12089, -0.52461, -0.59629,
                -0.85342, 2.80445, -0.95103,
                -0.26897, -1.14788, 2.41685 ]
       - ct: 2960
         ccm: [ 2.26962, -0.54174, -0.72789,
                -0.77008, 2.60271, -0.83262,
                -0.26036, -1.51254, 2.77289 ]
       - ct: 3603
         ccm: [ 2.18644, -0.66148, -0.52496,
                -0.77828, 2.69474, -0.91645,
                -0.25239, -0.83059, 2.08298 ]
       - ct: 4650
         ccm: [ 2.18174, -0.70887, -0.47287,
                -0.70196, 2.76426, -1.06231,
                -0.25157, -0.71978, 1.97135 ]
       - ct: 5858
         ccm: [ 2.32392, -0.88421, -0.43971,
                -0.63821, 2.58348, -0.94527,
                -0.28541, -0.54112, 1.82653 ]
       - ct: 7580
         ccm: [ 2.21175, -0.53242, -0.67933,
                -0.57875, 3.07922, -1.50047,
                -0.27709, -0.73338, 2.01048 ]

One of the possible explanations could be that the software ISP AWB
(used to determine the color temperature) is no way accurate and
produces some (different) color casts too.  But it can also be caused by
some bug or misunderstanding, independent testing is welcome.

Changes in v3:
- Rebased on master.
- AWB gains applied.
- ccmEnabled docstring moved to the right patch.
- Small notes about std::variant and about std::clamp added to the
  commit message of the last patch.

Changes in v2:
- Fix of the gamma table index upper boundary.
- Fix of red<->blue being temporarily swapped in an intermediate patch.
- Fix of CCM transformation when swapRedBlueGains_ is true.
- A related minor lookup data structures rearrangement, with an
  unintended but welcome effect of ~10% speedup with CCM in my
  environment. 

Milan Zamazal (9):
  libcamera: software_isp: Determine color temperature
  libcamera: software_isp: Store color temperature to metadata
  libcamera: software_isp: lut: Remove maybe_unused on a used argument
  libcamera: software_isp: Use common code to store debayered pixels
  libcamera: software_isp: Use a macro to assign debayering methods
  libcamera: software_isp: Add CCM algorithm
  libcamera: software_isp: Add an example CCM to uncalibrated.yaml
  libcamera: software_isp: Track whether CCM is enabled
  libcamera: software_isp: Apply CCM in debayering

 .../internal/software_isp/debayer_params.h    |  20 ++-
 include/libcamera/ipa/soft.mojom              |   2 +-
 src/ipa/simple/algorithms/awb.cpp             |  18 ++-
 src/ipa/simple/algorithms/ccm.cpp             |  86 ++++++++++++
 src/ipa/simple/algorithms/ccm.h               |  46 ++++++
 src/ipa/simple/algorithms/lut.cpp             |  67 ++++++---
 src/ipa/simple/algorithms/lut.h               |   1 +
 src/ipa/simple/algorithms/meson.build         |   1 +
 src/ipa/simple/data/uncalibrated.yaml         |   7 +
 src/ipa/simple/ipa_context.h                  |  24 +++-
 src/ipa/simple/soft_simple.cpp                |   8 +-
 src/libcamera/software_isp/debayer.cpp        |  53 ++++++-
 src/libcamera/software_isp/debayer.h          |   3 +-
 src/libcamera/software_isp/debayer_cpu.cpp    | 132 ++++++++++--------
 src/libcamera/software_isp/debayer_cpu.h      |  31 ++--
 src/libcamera/software_isp/software_isp.cpp   |  11 +-
 16 files changed, 400 insertions(+), 110 deletions(-)
 create mode 100644 src/ipa/simple/algorithms/ccm.cpp
 create mode 100644 src/ipa/simple/algorithms/ccm.h