[v1,0/3] RPi: prepare for "pixel peeping" IPAs
mbox series

Message ID 20261118140719.46849-1-david.plowman@raspberrypi.com
Headers show
Series
  • RPi: prepare for "pixel peeping" IPAs
Related show

Message

David Plowman Nov. 18, 2026, 1:15 p.m. UTC
Hi everyone

This is a small preparatory set of patches for "pixel peeping" IPAs,
meaning IPAs that want to have a look at some of the raw pixel data.

There are 3 patches:

1. Pass the PixelFormat to the IPAs.

I didn't turn the PixelFormat into a first class mojom object as it
would have required me to write more code (!), but I could do that if
we thought others might want to do the same at some point, or if
anyone generally felt strongly about it.

2. Memory map raw buffers and track their fds.

Naush - I've done this a bit differently to your original version,
mostly to keep the extra mapping calls out of the platform files
(vc4.cpp and pisp.cpp), and only in the shared files. Perhaps you
could check you're still happy with this...

3. Pass raw buffer to IPAs.

This is just an update to our prepare() methods to accept the raw
buffer, even though no one is using it yet. I wanted to keep all this
fluff out of the previous commit to make that one easier to digest.

And in case folks are curious, one of the first possible IPAs on my
hitlist might be a global motion detection algorithm.

Thanks!

David

David Plowman (3):
  pipeline: rpi: Pass the raw buffer pixel format to the IPA
  ipa: rpi: Memory map raw stream buffers
  ipa: rpi: Pass the raw stream buffer to the control algorithms

 include/libcamera/ipa/raspberrypi.mojom       |  2 +
 src/ipa/rpi/common/ipa_base.cpp               | 16 ++++---
 src/ipa/rpi/common/ipa_base.h                 |  2 +
 src/ipa/rpi/controller/algorithm.cpp          |  3 +-
 src/ipa/rpi/controller/algorithm.h            |  2 +-
 src/ipa/rpi/controller/camera_mode.h          |  3 ++
 src/ipa/rpi/controller/controller.cpp         |  4 +-
 src/ipa/rpi/controller/controller.h           | 11 ++++-
 src/ipa/rpi/controller/rpi/af.cpp             |  3 +-
 src/ipa/rpi/controller/rpi/af.h               |  2 +-
 src/ipa/rpi/controller/rpi/agc.cpp            |  3 +-
 src/ipa/rpi/controller/rpi/agc.h              |  2 +-
 src/ipa/rpi/controller/rpi/alsc.cpp           |  3 +-
 src/ipa/rpi/controller/rpi/alsc.h             |  2 +-
 src/ipa/rpi/controller/rpi/awb.cpp            |  3 +-
 src/ipa/rpi/controller/rpi/awb.h              |  2 +-
 src/ipa/rpi/controller/rpi/black_level.cpp    |  3 +-
 src/ipa/rpi/controller/rpi/black_level.h      |  2 +-
 src/ipa/rpi/controller/rpi/cac.cpp            |  3 +-
 src/ipa/rpi/controller/rpi/cac.h              |  2 +-
 src/ipa/rpi/controller/rpi/ccm.cpp            |  3 +-
 src/ipa/rpi/controller/rpi/ccm.h              |  2 +-
 src/ipa/rpi/controller/rpi/contrast.cpp       |  3 +-
 src/ipa/rpi/controller/rpi/contrast.h         |  2 +-
 src/ipa/rpi/controller/rpi/decompand.cpp      |  3 +-
 src/ipa/rpi/controller/rpi/decompand.h        |  2 +-
 src/ipa/rpi/controller/rpi/denoise.cpp        |  3 +-
 src/ipa/rpi/controller/rpi/denoise.h          |  2 +-
 src/ipa/rpi/controller/rpi/dpc.cpp            |  3 +-
 src/ipa/rpi/controller/rpi/dpc.h              |  2 +-
 src/ipa/rpi/controller/rpi/geq.cpp            |  3 +-
 src/ipa/rpi/controller/rpi/geq.h              |  2 +-
 src/ipa/rpi/controller/rpi/hdr.cpp            |  3 +-
 src/ipa/rpi/controller/rpi/hdr.h              |  2 +-
 src/ipa/rpi/controller/rpi/lux.cpp            |  3 +-
 src/ipa/rpi/controller/rpi/lux.h              |  2 +-
 src/ipa/rpi/controller/rpi/noise.cpp          |  3 +-
 src/ipa/rpi/controller/rpi/noise.h            |  2 +-
 src/ipa/rpi/controller/rpi/saturation.cpp     |  3 +-
 src/ipa/rpi/controller/rpi/saturation.h       |  2 +-
 src/ipa/rpi/controller/rpi/sdn.cpp            |  3 +-
 src/ipa/rpi/controller/rpi/sdn.h              |  2 +-
 src/ipa/rpi/controller/rpi/sharpen.cpp        |  3 +-
 src/ipa/rpi/controller/rpi/sharpen.h          |  2 +-
 src/ipa/rpi/controller/rpi/tonemap.cpp        |  3 +-
 src/ipa/rpi/controller/rpi/tonemap.h          |  2 +-
 .../pipeline/rpi/common/pipeline_base.cpp     | 44 ++++++++++++++++++-
 .../pipeline/rpi/common/pipeline_base.h       |  1 +
 48 files changed, 133 insertions(+), 50 deletions(-)

Comments

Laurent Pinchart Nov. 18, 2026, 2:21 p.m. UTC | #1
Hi David,

On Wed, Nov 18, 2026 at 01:15:16PM +0000, David Plowman wrote:
> Hi everyone
> 
> This is a small preparatory set of patches for "pixel peeping" IPAs,
> meaning IPAs that want to have a look at some of the raw pixel data.
> 
> There are 3 patches:
> 
> 1. Pass the PixelFormat to the IPAs.
> 
> I didn't turn the PixelFormat into a first class mojom object as it
> would have required me to write more code (!), but I could do that if
> we thought others might want to do the same at some point, or if
> anyone generally felt strongly about it.
> 
> 2. Memory map raw buffers and track their fds.
> 
> Naush - I've done this a bit differently to your original version,
> mostly to keep the extra mapping calls out of the platform files
> (vc4.cpp and pisp.cpp), and only in the shared files. Perhaps you
> could check you're still happy with this...
> 
> 3. Pass raw buffer to IPAs.
> 
> This is just an update to our prepare() methods to accept the raw
> buffer, even though no one is using it yet. I wanted to keep all this
> fluff out of the previous commit to make that one easier to digest.
> 
> And in case folks are curious, one of the first possible IPAs on my
> hitlist might be a global motion detection algorithm.

The CPU-based soft ISP performs both pixel processing and statistics
calculation in the pipeline handler, and GPU acceleration is also
handled there. Is there something fundamentally different in motion
detection to require processing on the IPA module side ?

> David Plowman (3):
>   pipeline: rpi: Pass the raw buffer pixel format to the IPA
>   ipa: rpi: Memory map raw stream buffers
>   ipa: rpi: Pass the raw stream buffer to the control algorithms
> 
>  include/libcamera/ipa/raspberrypi.mojom       |  2 +
>  src/ipa/rpi/common/ipa_base.cpp               | 16 ++++---
>  src/ipa/rpi/common/ipa_base.h                 |  2 +
>  src/ipa/rpi/controller/algorithm.cpp          |  3 +-
>  src/ipa/rpi/controller/algorithm.h            |  2 +-
>  src/ipa/rpi/controller/camera_mode.h          |  3 ++
>  src/ipa/rpi/controller/controller.cpp         |  4 +-
>  src/ipa/rpi/controller/controller.h           | 11 ++++-
>  src/ipa/rpi/controller/rpi/af.cpp             |  3 +-
>  src/ipa/rpi/controller/rpi/af.h               |  2 +-
>  src/ipa/rpi/controller/rpi/agc.cpp            |  3 +-
>  src/ipa/rpi/controller/rpi/agc.h              |  2 +-
>  src/ipa/rpi/controller/rpi/alsc.cpp           |  3 +-
>  src/ipa/rpi/controller/rpi/alsc.h             |  2 +-
>  src/ipa/rpi/controller/rpi/awb.cpp            |  3 +-
>  src/ipa/rpi/controller/rpi/awb.h              |  2 +-
>  src/ipa/rpi/controller/rpi/black_level.cpp    |  3 +-
>  src/ipa/rpi/controller/rpi/black_level.h      |  2 +-
>  src/ipa/rpi/controller/rpi/cac.cpp            |  3 +-
>  src/ipa/rpi/controller/rpi/cac.h              |  2 +-
>  src/ipa/rpi/controller/rpi/ccm.cpp            |  3 +-
>  src/ipa/rpi/controller/rpi/ccm.h              |  2 +-
>  src/ipa/rpi/controller/rpi/contrast.cpp       |  3 +-
>  src/ipa/rpi/controller/rpi/contrast.h         |  2 +-
>  src/ipa/rpi/controller/rpi/decompand.cpp      |  3 +-
>  src/ipa/rpi/controller/rpi/decompand.h        |  2 +-
>  src/ipa/rpi/controller/rpi/denoise.cpp        |  3 +-
>  src/ipa/rpi/controller/rpi/denoise.h          |  2 +-
>  src/ipa/rpi/controller/rpi/dpc.cpp            |  3 +-
>  src/ipa/rpi/controller/rpi/dpc.h              |  2 +-
>  src/ipa/rpi/controller/rpi/geq.cpp            |  3 +-
>  src/ipa/rpi/controller/rpi/geq.h              |  2 +-
>  src/ipa/rpi/controller/rpi/hdr.cpp            |  3 +-
>  src/ipa/rpi/controller/rpi/hdr.h              |  2 +-
>  src/ipa/rpi/controller/rpi/lux.cpp            |  3 +-
>  src/ipa/rpi/controller/rpi/lux.h              |  2 +-
>  src/ipa/rpi/controller/rpi/noise.cpp          |  3 +-
>  src/ipa/rpi/controller/rpi/noise.h            |  2 +-
>  src/ipa/rpi/controller/rpi/saturation.cpp     |  3 +-
>  src/ipa/rpi/controller/rpi/saturation.h       |  2 +-
>  src/ipa/rpi/controller/rpi/sdn.cpp            |  3 +-
>  src/ipa/rpi/controller/rpi/sdn.h              |  2 +-
>  src/ipa/rpi/controller/rpi/sharpen.cpp        |  3 +-
>  src/ipa/rpi/controller/rpi/sharpen.h          |  2 +-
>  src/ipa/rpi/controller/rpi/tonemap.cpp        |  3 +-
>  src/ipa/rpi/controller/rpi/tonemap.h          |  2 +-
>  .../pipeline/rpi/common/pipeline_base.cpp     | 44 ++++++++++++++++++-
>  .../pipeline/rpi/common/pipeline_base.h       |  1 +
>  48 files changed, 133 insertions(+), 50 deletions(-)
> 
> -- 
> 2.47.3
>
David Plowman Nov. 18, 2026, 2:55 p.m. UTC | #2
Hi Laurent

On Wed, 18 Nov 2026 at 14:21, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi David,
>
> On Wed, Nov 18, 2026 at 01:15:16PM +0000, David Plowman wrote:
> > Hi everyone
> >
> > This is a small preparatory set of patches for "pixel peeping" IPAs,
> > meaning IPAs that want to have a look at some of the raw pixel data.
> >
> > There are 3 patches:
> >
> > 1. Pass the PixelFormat to the IPAs.
> >
> > I didn't turn the PixelFormat into a first class mojom object as it
> > would have required me to write more code (!), but I could do that if
> > we thought others might want to do the same at some point, or if
> > anyone generally felt strongly about it.
> >
> > 2. Memory map raw buffers and track their fds.
> >
> > Naush - I've done this a bit differently to your original version,
> > mostly to keep the extra mapping calls out of the platform files
> > (vc4.cpp and pisp.cpp), and only in the shared files. Perhaps you
> > could check you're still happy with this...
> >
> > 3. Pass raw buffer to IPAs.
> >
> > This is just an update to our prepare() methods to accept the raw
> > buffer, even though no one is using it yet. I wanted to keep all this
> > fluff out of the previous commit to make that one easier to digest.
> >
> > And in case folks are curious, one of the first possible IPAs on my
> > hitlist might be a global motion detection algorithm.
>
> The CPU-based soft ISP performs both pixel processing and statistics
> calculation in the pipeline handler, and GPU acceleration is also
> handled there. Is there something fundamentally different in motion
> detection to require processing on the IPA module side ?

Pragmatically, I think we can expect to have different algorithms and
tuning parameters for them for different sensors and different use
cases. The information produced by such algorithms might be of
interest to other IPAs, for example, so it all fits very naturally
there using all the mechanisms that we already have.

More abstractly, I think would could make arguments about the soft ISP
really being the core part of its pipeline handler, so it doesn't make
sense to push it somewhere else as if it were an accessory or optional
function. However, I feel that these "image processing algorithms"
fall much more into that category, so treating them as our other IPAs
is reasonable as well.

David

>
> > David Plowman (3):
> >   pipeline: rpi: Pass the raw buffer pixel format to the IPA
> >   ipa: rpi: Memory map raw stream buffers
> >   ipa: rpi: Pass the raw stream buffer to the control algorithms
> >
> >  include/libcamera/ipa/raspberrypi.mojom       |  2 +
> >  src/ipa/rpi/common/ipa_base.cpp               | 16 ++++---
> >  src/ipa/rpi/common/ipa_base.h                 |  2 +
> >  src/ipa/rpi/controller/algorithm.cpp          |  3 +-
> >  src/ipa/rpi/controller/algorithm.h            |  2 +-
> >  src/ipa/rpi/controller/camera_mode.h          |  3 ++
> >  src/ipa/rpi/controller/controller.cpp         |  4 +-
> >  src/ipa/rpi/controller/controller.h           | 11 ++++-
> >  src/ipa/rpi/controller/rpi/af.cpp             |  3 +-
> >  src/ipa/rpi/controller/rpi/af.h               |  2 +-
> >  src/ipa/rpi/controller/rpi/agc.cpp            |  3 +-
> >  src/ipa/rpi/controller/rpi/agc.h              |  2 +-
> >  src/ipa/rpi/controller/rpi/alsc.cpp           |  3 +-
> >  src/ipa/rpi/controller/rpi/alsc.h             |  2 +-
> >  src/ipa/rpi/controller/rpi/awb.cpp            |  3 +-
> >  src/ipa/rpi/controller/rpi/awb.h              |  2 +-
> >  src/ipa/rpi/controller/rpi/black_level.cpp    |  3 +-
> >  src/ipa/rpi/controller/rpi/black_level.h      |  2 +-
> >  src/ipa/rpi/controller/rpi/cac.cpp            |  3 +-
> >  src/ipa/rpi/controller/rpi/cac.h              |  2 +-
> >  src/ipa/rpi/controller/rpi/ccm.cpp            |  3 +-
> >  src/ipa/rpi/controller/rpi/ccm.h              |  2 +-
> >  src/ipa/rpi/controller/rpi/contrast.cpp       |  3 +-
> >  src/ipa/rpi/controller/rpi/contrast.h         |  2 +-
> >  src/ipa/rpi/controller/rpi/decompand.cpp      |  3 +-
> >  src/ipa/rpi/controller/rpi/decompand.h        |  2 +-
> >  src/ipa/rpi/controller/rpi/denoise.cpp        |  3 +-
> >  src/ipa/rpi/controller/rpi/denoise.h          |  2 +-
> >  src/ipa/rpi/controller/rpi/dpc.cpp            |  3 +-
> >  src/ipa/rpi/controller/rpi/dpc.h              |  2 +-
> >  src/ipa/rpi/controller/rpi/geq.cpp            |  3 +-
> >  src/ipa/rpi/controller/rpi/geq.h              |  2 +-
> >  src/ipa/rpi/controller/rpi/hdr.cpp            |  3 +-
> >  src/ipa/rpi/controller/rpi/hdr.h              |  2 +-
> >  src/ipa/rpi/controller/rpi/lux.cpp            |  3 +-
> >  src/ipa/rpi/controller/rpi/lux.h              |  2 +-
> >  src/ipa/rpi/controller/rpi/noise.cpp          |  3 +-
> >  src/ipa/rpi/controller/rpi/noise.h            |  2 +-
> >  src/ipa/rpi/controller/rpi/saturation.cpp     |  3 +-
> >  src/ipa/rpi/controller/rpi/saturation.h       |  2 +-
> >  src/ipa/rpi/controller/rpi/sdn.cpp            |  3 +-
> >  src/ipa/rpi/controller/rpi/sdn.h              |  2 +-
> >  src/ipa/rpi/controller/rpi/sharpen.cpp        |  3 +-
> >  src/ipa/rpi/controller/rpi/sharpen.h          |  2 +-
> >  src/ipa/rpi/controller/rpi/tonemap.cpp        |  3 +-
> >  src/ipa/rpi/controller/rpi/tonemap.h          |  2 +-
> >  .../pipeline/rpi/common/pipeline_base.cpp     | 44 ++++++++++++++++++-
> >  .../pipeline/rpi/common/pipeline_base.h       |  1 +
> >  48 files changed, 133 insertions(+), 50 deletions(-)
> >
> > --
> > 2.47.3
> >
>
> --
> Regards,
>
> Laurent Pinchart