[v4,00/23] Add GLES 2.0 GPUISP to libcamera

25124 mbox series

This version 4:

- Drops AWB since the CCM contains it already
- Includes Gamma
- Includes Contrast - testable via camshark
- Includes Saturation - testable via camshark
- Includes a scaler from Robert
- Includes synch changes from Robert
- Includes all feedback incorporated from Pavel
- Generates a default 65k CCM if none is supplied
- Various Doxygen torments fixed along the way
- And is the "top half" of the precursor series as the GPUISP
  series becomes 44 patches long this is an unreasonable number
  to merge in one go.

- Full testable branch
Link: https://gitlab.freedesktop.org/camera/libcamera-softisp/-/tree/v0.5.2-gpuisp-v4e?ref_type=heads

- The first part of the series is in the precurso here
Link: https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/v0.5.2-gpuisp-v4e-split

That precursor is just a tag about half way through the integrated series.

This version 3:

- Adds AWB to the debayer routine as calculated by the IPA thread

- Implements ~ all of the feedback from Barnabas quicker to mention
  what hasn't been done.
  a) A comment about member initialisation in eGL.cpp
     code I wrote to make constructor init common seemed to negate
     that ask.
  b) meson dependency checks for egl.
     I remember struggling with this earlier on in development.
     I will certainly try to do this for a v4 so its more
     pending a try as opposed to not indended to be done.

- Incorporates various fixes from Robert Mader
  When to sync removing tearing for Milan
  Some error checking that although Robert didn't mention in his
  feedback were in his patches so I stole that code. Thanks.

- Also worth mentioning Robert identified a permissions fix
  that pipewire would need for eGL to work in libcamera with pipewire
  published that fix and got it merged too.

  Owe you a beer for that one.

- Is rebased on tip-of-tree

- Currently the documentation checks for the various classes
  don't pass but that is easy enough to fix in a V4.

- In line with our discussions gpuisp is now the default instead of cpuisp.

- Since its only the documentation checks that are pending I thought
  rather than delay further it was time to publish the series without
  and see if anything major gets snagged.

v2:

This version 2 is an incomplete update with-respect-to previous comment
feedback, which ordinarily I would not publish however, given OSSEU is
starting on Monday and we have talk about this topic, in addition to some
pretty good progress in the interregnum I thought a v2 would be
appropriate.

- V2 drops use of GBM surface in favour of generating a framebuffer from
  the dma-buf handle, called render-to-texture.

  The conversion from GBM surface + memcpy() including the associated cache
  invalidate has a dramatic effect on GPUISP performance.

  Some rough stats for a Qualcomm sm8250 "kona" device with an imx517
  sensor @ 4048 x 3040 ABRG8888 - debug builds

    CPUISP + CCM:
      2 FPS CPU usage > 100% single core pulls about 9 watts

    GPUISP v1 + CCM:
      14 FPS - power not measured

    GPUISP v2 + CCM:
      30 FPS - sensor linerate - CPU usage ~ 70 % pulling 8 Watts.

  Milan Zamal has reported a TI AM69 + imx219 - unknown resolution

    CPUISP 4 FPS
    GPUISP v2 - 2 or 3 FPS
    GPUISP v2 - 15 FPS =3D=3D sensor linerate

  In other words for these boards we can hit linerate with GPUISP + 3A +
  CCM.

- Drop GBM surface rendering
- Drop swapbuffers
- Use eglCreateImageKHR to directly render into the output dma-buf buffer
  eglCreateImageKHR lets you specify the FOURCC of the texture which means
  we can create the texture in the uncompressed target output pixel format
  we want.
- Fix stride calculation to 256 bytes
  Laurent and Maxime explained to me about GPU stride alignments being
  tribal wisdom and that 256 bytes is a good cross-platform value.
  This helped to get the render-to-texture command right.
- A synchronous blocking wait is used to ensure GPU operations have
  completed. Laurent wants this to be made async.
  At the moment its not clear to me the eglWaitSyncKHR is really required
  and in any case doesn't seem to have any performance impact.
  But this part is still TBD - I've included the sync wait for simplicity
  and safety.
- A Debayer::stop() method has been introduced to ensure we call
  eglDestroySyncKHR when the eGL context is valid, as opposed to in the
  callchain of destructors triggering eGL::~eGL();
- stats move constructor call chain dropped - Branabas
- Incorporates Milan's area-of-interest constraint for Bayer stats
  i.e. squashes his v3 update into debayer_egl.cpp directly
- Moves ALIGN_TO into a common area to facilitate its reuse in
  egl.cpp
- Rebases on 0.5.2

- There are a number of known checks failing on the CI loop right now

Link to v1: https://lists.libcamera.org/pipermail/libcamera-devel/2025-June=
/050692.html

v1:
This series introduces a GLES 2.0 GPU ISP to libcamera.

We have had extensive discussions, meetings and collaborative discussions
about this topic over the last year or so.

As an overview we want to start to move as much processing of software_isp
into the GPU as possible. This is especially advantageous when we are
talking about processing a framebuffer's worth of pixels as quickly as
possible.

The decision to use GLES 2.0 instead of say Vulcan stems from a desire to
support as much in the way of older hardware as possible and the fact we
already have upstream GLES 2.0 fragment shaders to do debayer.

Generally the approach is

- Move the fragment shaders out of qcam and into a common location
- Update the existing SoftwareISP Debayer/DebayerCPU pair to facilitate
  addition of a new class DebayerEGL.
- Introduce that class
- Then do progressive change of the shaders and DebayerEGL class to make
  the modifications as transparent as possible in the git log.
- Reuse as much of the SoftIPA data-structures and logic as possible.
- Consume the data from SoftIPA in the Debayer Shaders so that CPUISP and
  GPUISP give similar - hopefully the same results but with GPUISP going
  faster.

In order to get untiled and uncompressed pixel data out of the GPU
framebuffer we need to tell the GPU how to store the data it is writing to
that framebuffer. GPUs can store their framebuffer data in tiled or even
compressed formats which is why the naive approach of running your fragment
shader and then using glReadPixels(GL_RGBA); will be horrendously slow as
glReadPixels must convert from the internal GPU format to the requested
output format - an operation that for me takes ~ 10 milliseconds per frame.

Instead we get the GPU to store its data as ARGB8888 swap buffers and
memcpy() from the swapped buffer to our output frame. Right now this series
supports 32 bit output formats only.

The memcpy() also entails flushing the cache of the target buffer as per
the terms of the dma-buf software contract.

This leads us onto the main outstanding TODOs

- 24 bit GBM buffer support leading
- 24 bit output framebuffer support
- Surfaceless GBM and eGL context with no swapbuffer
- Render to texture
  If we render directly to a buffer provided to the GPU the output
  buffer we will not need to memcpy() to the output buffer
  nor will we need to invalidate the output buffer cache.
- eglCreateImageKHR for the texture upload.

This list is of the colour "make it go faster" not "make it work" which is
why we are moving to start to submit a v1 for discussion in the full
realisation it will have to go through several cycles of review giving us
the opportunity to fix:

- Doxygen is missing for new classes and methods
- Some of the pipelines don't complete in gitlab
- 24 bit output seems doable before merge
- Render to texture perhaps even too

For me on my Qualcomm hardware GPUISP works very well I get 30fps in qcam
with about 75% CPU usage versus > 100% - cam goes faster which to me
implies a good bit of time is being consumed in qcam itself.

The series starts out with fixes and updates from Hans and finishes it out
with shader modifications from Milan both of whom along with Kieran,
Laurent and Maxime I'd like to thank for being some helpful and patient.

Bryan O'Donoghue (21):
  libcamera: software_isp: gbm: Add a GBM helper class for GPU surface
    access
  libcamera: software_isp: Make isStandardBayerOrder static
  libcamera: software_isp: egl: Add a eGL base helper class
  libcamera: shaders: Use highp not mediump for float precision
  libcamera: shaders: Extend debayer shaders to apply RGB gain values on
    output
  libcamera: shaders: Extend bayer shaders to support swapping R and B
    on output
  libcamera: shaders: Add support for black level compenstation
  libcamera: shaders: Add support for Gamma
  libcamera: shaders: Add support for contrast
  libcamera: software_isp: debayer_egl: Add an eGL debayer class
  libcamera: software_isp: debayer_egl: Make DebayerEGL an environment
    option
  libcamera: software_isp: debayer_egl: Make gpuisp default softisp mode
  libcamera: software_isp: debayer_cpu: Make getInputConfig and
    getOutputConfig static
  libcamera: software_isp: Add a gpuisp todo list
  libcamera: software_isp: lut: Change default Gamma to 1.0/2.2
  ipa: Add a new Algorithm::init() to support self-initalising
    algorithms
  libcamera: software_isp: Implement a static init() routine
  ipa: simple: Add a flag to indicate gpuIspEnabled
  ipa: libipa: module: Add createSelfEnumeratingAlgorithm
  ipa: software_isp: Call createSelfEnumeratingAlgorithm() to statically
    instantiate CCM algo
  libcamera: software_isp: lut:  Skip calculation lookup tables if
    gpuIspEnabled is true.

Milan Zamazal (2):
  libcamera: shaders: Rename bayer_8 to bayer_unpacked
  libcamera: software_isp: GPU support for unpacked 10/12-bit formats

 include/libcamera/internal/egl.h              | 412 +++++++++++
 include/libcamera/internal/gbm.h              |  84 +++
 include/libcamera/internal/meson.build        |   1 +
 include/libcamera/internal/shaders/RGB.frag   |   2 +-
 .../internal/shaders/YUV_2_planes.frag        |   2 +-
 .../internal/shaders/YUV_3_planes.frag        |   2 +-
 .../internal/shaders/YUV_packed.frag          |   2 +-
 .../internal/shaders/bayer_1x_packed.frag     |  89 ++-
 .../{bayer_8.frag => bayer_unpacked.frag}     | 105 ++-
 .../{bayer_8.vert => bayer_unpacked.vert}     |   0
 .../libcamera/internal/shaders/meson.build    |   4 +-
 include/libcamera/ipa/soft.mojom              |   2 +-
 src/apps/qcam/assets/shader/shaders.qrc       |   4 +-
 src/apps/qcam/viewfinder_gl.cpp               |  16 +-
 src/ipa/libipa/algorithm.cpp                  |  13 +-
 src/ipa/libipa/algorithm.h                    |   5 +
 src/ipa/libipa/module.h                       |  41 ++
 src/ipa/simple/algorithms/ccm.cpp             |  18 +
 src/ipa/simple/algorithms/ccm.h               |   1 +
 src/ipa/simple/algorithms/lut.cpp             |  72 +-
 src/ipa/simple/ipa_context.h                  |   1 +
 src/ipa/simple/soft_simple.cpp                |  13 +-
 src/libcamera/egl.cpp                         | 436 ++++++++++++
 src/libcamera/gbm.cpp                         |  61 ++
 src/libcamera/meson.build                     |  34 +
 src/libcamera/software_isp/debayer.h          |   2 +-
 src/libcamera/software_isp/debayer_cpu.h      |   4 +-
 src/libcamera/software_isp/debayer_egl.cpp    | 668 ++++++++++++++++++
 src/libcamera/software_isp/debayer_egl.h      | 177 +++++
 src/libcamera/software_isp/gpuisp-todo.txt    |  83 +++
 src/libcamera/software_isp/meson.build        |   8 +
 src/libcamera/software_isp/software_isp.cpp   |  35 +-
 32 files changed, 2334 insertions(+), 63 deletions(-)
 create mode 100644 include/libcamera/internal/egl.h
 create mode 100644 include/libcamera/internal/gbm.h
 rename include/libcamera/internal/shaders/{bayer_8.frag => bayer_unpacked.frag} (50%)
 rename include/libcamera/internal/shaders/{bayer_8.vert => bayer_unpacked.vert} (100%)
 create mode 100644 src/libcamera/egl.cpp
 create mode 100644 src/libcamera/gbm.cpp
 create mode 100644 src/libcamera/software_isp/debayer_egl.cpp
 create mode 100644 src/libcamera/software_isp/debayer_egl.h
 create mode 100644 src/libcamera/software_isp/gpuisp-todo.txt