[RFC,v2,00/26] gpuisp multipass + fixes
mbox series

Message ID 20261118095735.170587-1-bryan.odonoghue@linaro.org
Headers show
Series
  • gpuisp multipass + fixes
Related show

Message

Bryan O'Donoghue Nov. 18, 2026, 9:57 a.m. UTC
This series is a bit more advanced now just rebased on 0.8 it tackles a
number of open issues.

- First we have an ongoing problem of having individual shaders for
  the packed and unpacked debayer path.
  There is no real reason for this except "that is how it was done" and
  "changing the shaders is difficult"

  Or at least that's been my excuse.

  Unification of the two shader paths is a no-brainer. You don't maintain
  parallel almost identical functions in a single file, so you shouldn't
  maintain parallel virtually identical shaders either.

- The second problem tackled is "we don't understand the logic of the
  shaders well".

  Fixes end up being posted for the demosiac failures which the authors
  can't fully explain ostensibly because of the LLM input but really IMO
  because what is going on with the shaders was already complex and not
  well understood.

  So we have a cognition bug.

- And we have a general desire to extend out the gpuisp to make it more
  composable - possible to have real hardware ISP pipelines invoke the GPU
  to do additional work.

  This last problem isn't really addressed in this series but the series
  takes a step in that direction.

A series of classes get renamed to reflect the fact we are moving, have
moved beyond simple debayering - thus the namespace of the software should
reflect the logic and intent of what is going on. We aren't debayering only
so lets stop calling it "debayer_egl.cpp" etc.

A number of demosiac bugs are fixed in this series in a deliberate order.

A) The fact that we have two parallel shaders doing almost
   exactly the same job.

   Since we are doing two passes this means our first pass vertex/fragment
   shader pair outputs a standard frame buffer for the second pass to
   consume.

B) The second pass is a unified demosiac pass for the packed and unpacked
   path.

C) The cognition bug - the readability of the code is addressed.

D) Selecting the correct center byte is fixed.

   It is not apparent on real hardware that we readily select the wrong
   center byte but, testing on Google Chrome's swift shader which is a
   pure software implementation - we can quickly see that the sampled
   center-byte is incorrect.

   This is because of conversion from pixel co-ordinates and glsl
   texture2D() co-ordinates.

   texture2D() wants co-ordinates in the range 0.0f..1.0f.

   The details are in patch # 23.

E) Confusion between two different ways of specifying the texel
   to sample using both the line+texture and a passed xCoord/YCoord

   The simple solution when you strip all of the technical analysis away is
   to not do that. Have a single source of truth.

So D and E answer the question "why do multi-pass gpuisp just fix bugs
first". Replicating the fixes to two different shaders is just
proliferation of our technical debt.

Once this series lands we are in pretty good shape to tackle GPU stats but,
before adding new functionality I really wanted to drill down on bugs.

I'm doing my part:

https://www.youtube.com/watch?v=81M62lI4mhc

Bryan O'Donoghue (25):
  libcamera: software_isp: Rename Bayer classes to SoftwareIspInstance
  libcamera: software_isp: gpu: rename debayerGPU to processGPU
  libcamera: software_isp: gpu_pipeline_shader_pass: Add base class
    GpuPipelineShaderPass
  libcamera: software_isp: gpu_pipeline_shader_pass: Add
    GpuPipelineShaderPassDemosiac
  libcamera: software_isp: gpu: Switch to using GpuIspShaderPassDemosiac
  libcamera: software_isp: gpu: Make Rectangle window_ a local variable
    in configure()
  libcamera: software_isp: gpu_pipeline_shader_pass: Move common
    attribute and uniform variables to base shader class
  libcamera: software_isp: gpu_pipeline_shader_pass: Move common shader
    selection logic into base class in new method initShaders()
  libcamera: shaders: Split packed and unpacked demosiac up
  libcamera: shaders: bayer_glr16_to_rgba.frag: Use bilinear filtering
  libcamera: shaders: bayer_{1x_packed|unpacked}_to_glr16f.frag: Add
    blacklevel correction to bayer domain shaders
  libcamera: software_isp: gpu: Add GpuIspShaderPassBlcNormalise
  libcamera: software_isp: egl: Extend eGL::createTexture2D to
    understand floats
  libcamera: software_isp: egl: Use Texture Unit 3 for final output
    texture
  libcamera: software_isp: egl: Add Ping/Pong buffers with start/stop
    bindings only
  libcamera: software_isp: gpu: Include GpuIspShaderPassBlcNormalise in
    init sequence
  libcamera: software_isp: gpu: Swtich to two pass logic
  libcamera: shaders: bayer_glr16_to_rgba.frag: Make fetch() into a
    function
  libcamera: software_isp: gpu_pipeline_shader_pass_demosiac: Switch to
    unified fragment shader
  libcamera: shaders: bayer_glr16_to_rgba.frag: Sample pixels into a
    structure
  libcamera: shaders: bayer_glr16_to_rgba.frag: Move debayer logic to a
    routine
  libcamera: software_isp: egl: Move scaling to the second stage
  libcamera: shaders: bayer_1x_packed_to_blc_glr16f.frag: Sample inputs
    at the texel centers
  libcamera: shaders: bayer_glr16_to_rgba.frag: Use center.zw to derive
    3x3 texel center
  libcamera: software_isp: gpu_pipeline_shader_pass_demosiac: Remove
    unnecessary tex_step input

Ray Strode (1):
  libcamera: software_isp: gpu_pipeline_shader_pass_blc_normalise: Fix
    texture sampling steps

 .../internal/software_isp/software_isp.h      |   4 +-
 .../internal/software_isp/swstats_gpu.h       |  30 +
 include/linux/v4l2-isp.h                      | 102 +++
 src/libcamera/egl.cpp                         |  21 +-
 .../shaders/bayer_1x_packed_to_glr16f.frag    |  97 +++
 .../shaders/bayer_glr16f_to_rgba.frag         | 221 +++++
 src/libcamera/shaders/bayer_unpacked.vert     |  14 -
 .../shaders/bayer_unpacked_to_glr16f.frag     |  50 ++
 src/libcamera/shaders/meson.build             |   3 +
 src/libcamera/software_isp/debayer_egl.cpp    | 752 ------------------
 .../software_isp/gpu_pipeline_shader_pass.cpp | 179 +++++
 .../software_isp/gpu_pipeline_shader_pass.h   | 105 +++
 ...gpu_pipeline_shader_pass_blc_normalise.cpp | 209 +++++
 .../gpu_pipeline_shader_pass_blc_normalise.h  |  52 ++
 .../gpu_pipeline_shader_pass_demosiac.cpp     | 253 ++++++
 .../gpu_pipeline_shader_pass_demosiac.h       |  68 ++
 src/libcamera/software_isp/meson.build        |   9 +-
 src/libcamera/software_isp/software_isp.cpp   |  58 +-
 ...{debayer.cpp => software_isp_instance.cpp} |  88 +-
 .../{debayer.h => software_isp_instance.h}    |   8 +-
 ..._cpu.cpp => software_isp_instance_cpu.cpp} | 200 ++---
 ...ayer_cpu.h => software_isp_instance_cpu.h} |  16 +-
 .../software_isp_instance_gpu.cpp             | 465 +++++++++++
 ...ayer_egl.h => software_isp_instance_gpu.h} |  63 +-
 src/libcamera/software_isp/swstats_gpu.cpp    |  69 ++
 25 files changed, 2128 insertions(+), 1008 deletions(-)
 create mode 100644 include/libcamera/internal/software_isp/swstats_gpu.h
 create mode 100644 include/linux/v4l2-isp.h
 create mode 100644 src/libcamera/shaders/bayer_1x_packed_to_glr16f.frag
 create mode 100644 src/libcamera/shaders/bayer_glr16f_to_rgba.frag
 create mode 100644 src/libcamera/shaders/bayer_unpacked_to_glr16f.frag
 delete mode 100644 src/libcamera/software_isp/debayer_egl.cpp
 create mode 100644 src/libcamera/software_isp/gpu_pipeline_shader_pass.cpp
 create mode 100644 src/libcamera/software_isp/gpu_pipeline_shader_pass.h
 create mode 100644 src/libcamera/software_isp/gpu_pipeline_shader_pass_blc_normalise.cpp
 create mode 100644 src/libcamera/software_isp/gpu_pipeline_shader_pass_blc_normalise.h
 create mode 100644 src/libcamera/software_isp/gpu_pipeline_shader_pass_demosiac.cpp
 create mode 100644 src/libcamera/software_isp/gpu_pipeline_shader_pass_demosiac.h
 rename src/libcamera/software_isp/{debayer.cpp => software_isp_instance.cpp} (72%)
 rename src/libcamera/software_isp/{debayer.h => software_isp_instance.h} (93%)
 rename src/libcamera/software_isp/{debayer_cpu.cpp => software_isp_instance_cpu.cpp} (81%)
 rename src/libcamera/software_isp/{debayer_cpu.h => software_isp_instance_cpu.h} (93%)
 create mode 100644 src/libcamera/software_isp/software_isp_instance_gpu.cpp
 rename src/libcamera/software_isp/{debayer_egl.h => software_isp_instance_gpu.h} (66%)
 create mode 100644 src/libcamera/software_isp/swstats_gpu.cpp