[v3,00/16] libcamera: introduce Software ISP and Software IPA
mbox series

Message ID 20240214170122.60754-1-hdegoede@redhat.com
Headers show
Series
  • libcamera: introduce Software ISP and Software IPA
Related show

Message

Hans de Goede Feb. 14, 2024, 5:01 p.m. UTC 
Hi All,

Here is v3 of the patch-set to add Software ISP support
to libcamera / to the simple pipeline-handler.

Changes in v3 vs v2:
- The Software ISP now is always build when building the simple pipeline
  handler, no more -Dpipelines=simple/simple. Instead whether the SoftISP
  is used or not is based on the media-controller driver. For now it is
  only enabled for the "qcom-camss" driver (Andrey)
- SoftISP factory has been removed, there is a just a single SoftwareISP class now
- Fix the multi-threading issues (Andrey)
- Fully document the SharedMemObject and DmaHeap classes (Andrey)
- Drop SwStats base (integrate into SwStatsCpu class)
- Move headers for classes only used by the SoftISP into src/libcamera/software_isp
- Move SwStats / SwDebayer docs into .cpp and extend it
- Rename many foo_bar symbols to fooBar
- Add constexpr kFooBar values for varies hardcoded sizes like
  the yHistoGram having 16 bins and the gammalookuptable having 1024 entries
- Make startFrame() and finishFrame() normal methods instead of
  using function pointers for these
- Document how/why an array of src pointers is passed to
  the debayer functions
- 12bpp unpacked raw bayer input support (Kieran)
- Add "Software ISP benchmarking" doc

Changes in v2 vs v1:
- Integrated Dennis, Martti and Toon's auto-exposure algorithm
  based on the paper which they found which gives us a nice
  relatively simple AEC + AGC algorithm
  TODO: Add link to paper to source + commit-message
- Integrated Dennis' doxygen comments for all new classes, no more warnings!
- Move AWB gain calculations to the IPA (Andrey)
- Added 8 bpp and 10 bpp unpacked raw bayer input support (Hans)
- Use dma-buf for the output FrameBuffer-s (Andrey)
- Memcpy data from input FrameBuffers to a heap buffer on a line by line
  basis to speedup debayering from uncached mem (Hans)
- This addresses all "open issues" from the v1 posting

Changes in v1 vs RFC-v2:
- Add and use SwStats[Cpu] and Debayer[Cpu] classes
- Rename linaro soft-IPA and soft-ISP implementations to simple following
  the simple pipeline hander
- Integrate Pavel's swstats and debayer improvements

This has been tested on:
- Qualcomm RB5 board with a mezzanine board equipped with RPi camera v2 (Andrey)
- Lenovo x13s, sc8280xp (Bryan)
- Pinephone (Pavel)
- Debix Model A (Milan)
- Lenovo Thinkpad Yoga X1 yoga gen7/8, Alder Lake/Raptor Lake IPU6EP +
  ov2740 10bpp packed + unpacked (Dennis / Hans)
- Dell Latitude 9420, Tiger Lake IPU6 + ov01a1s (RGBI) sensor
  (Martti, requires IGIG_GBGR_IGIG_GRGB patches on top)

git branch for this v3 patch set:
https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v06

git branch for v2 of the patch set:
https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v05

git branch for v1 of the patch set:
https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v04

Old RFC-v2 cover-letter with small updates:

Here is an implementation of Software ISP and Software IPA
which provide debayering and implementation of image processing
algorithms for systems without a hardware ISP, or in cases when
there are no public drivers for the hardware ISP present in the
system.

The Software IPA has auto exposure/AGC and AWB. For the AGC
the analogue gain control of the camera sensor is used (if
available).

The 6th patch renames some variables in the simple pipeline
handler for the Software ISP to use the same buffer handling
code as the Converter currently does. This lets one to
avoid adding extra code to the pipeline handler, but also
makes the Software ISP and the Converter mutually exclusive.

The Software ISP / IPA are intended to be used with the simple
pipeline handler. The pipeline handler doesn't interact with
the Software IPA directly - the Software IPA is hidden behind
the Software ISP interface. To use the Software ISP the build
must be configured with -Dpipelines=simple -Dipas=simple .

This patch set uses SharedMemObject class used by the RPi pipeline
handler - the second patch in the series moves the header file
to a common directory.

This patch set has been tested on Qualcomm RB5 board with
a mezzanine board equipped with RPi camera v2 (not the
standard RB5 camera mezzanine).

git branch for the RFC-v2 patch set:
https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v03

RFC-v1 of the patch set:
https://patchwork.libcamera.org/cover/19262/

Changes in RFC v2 vs RFC v1:
- patches are restructured and reordered
- the Software IPA is hidden behind the Software ISP interface. The
  pipeline handler doesn't interact with the Software IPA directly
- instantiation of the Software ISP / IPA is configurable (at build
  time)
- comment explaining the implementation limitations is added to
  SwIspLinaro::IspWorker::debayerRaw10P()

Regards,

Hans



Andrei Konovalov (1):
  libcamera: shared_mem_object: reorganize the code and document the
    SharedMemObject class

Andrey Konovalov (8):
  libcamera: pipeline: simple: fix size adjustment in validate()
  libcamera: internal: Move dma_heaps.[h,cpp] to common directories
  libcamera: dma_heaps: extend DmaHeap class to support system heap
  libcamera: internal: Move SharedMemObject class to a common directory
  libcamera: ipa: add Soft IPA
  libcamera: introduce SoftwareIsp
  libcamera: pipeline: simple: rename converterBuffers_ and related vars
  libcamera: pipeline: simple: enable use of Soft ISP and Soft IPA

Hans de Goede (7):
  libcamera: software_isp: Add SwStatsCpu class
  libcamera: software_isp: Add Debayer base class
  libcamera: software_isp: Add DebayerCpu class
  libcamera: swstats_cpu: Add support for 8, 10 and 12 bpp unpacked
    bayer input
  libcamera: debayer_cpu: Add support for 8, 10 and 12 bpp unpacked
    bayer input
  libcamera: debayer_cpu: Add BGR888 output support
  libcamera: Add "Software ISP benchmarking" documentation

 Documentation/Doxyfile.in                     |   1 +
 Documentation/index.rst                       |   1 +
 Documentation/meson.build                     |   1 +
 Documentation/software-isp-benchmarking.rst   |  82 ++
 .../libcamera/internal}/dma_heaps.h           |  14 +-
 include/libcamera/internal/meson.build        |   3 +
 .../libcamera/internal}/shared_mem_object.h   |  94 ++-
 .../internal/software_isp/debayer_params.h    |  48 ++
 .../internal/software_isp/meson.build         |   7 +
 .../internal/software_isp/software_isp.h      |  98 +++
 .../internal/software_isp/swisp_stats.h       |  38 +
 include/libcamera/ipa/meson.build             |   1 +
 include/libcamera/ipa/soft.mojom              |  28 +
 meson_options.txt                             |   2 +-
 src/ipa/simple/data/meson.build               |   9 +
 src/ipa/simple/data/soft.conf                 |   3 +
 src/ipa/simple/meson.build                    |  25 +
 src/ipa/simple/soft_simple.cpp                | 308 +++++++
 src/libcamera/dma_heaps.cpp                   | 164 ++++
 src/libcamera/meson.build                     |   3 +
 src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp  |  90 --
 src/libcamera/pipeline/rpi/vc4/meson.build    |   1 -
 src/libcamera/pipeline/rpi/vc4/vc4.cpp        |   5 +-
 src/libcamera/pipeline/simple/simple.cpp      | 232 ++++--
 src/libcamera/shared_mem_object.cpp           | 191 +++++
 src/libcamera/software_isp/debayer.cpp        |  29 +
 src/libcamera/software_isp/debayer.h          | 104 +++
 src/libcamera/software_isp/debayer_cpu.cpp    | 779 ++++++++++++++++++
 src/libcamera/software_isp/debayer_cpu.h      | 157 ++++
 src/libcamera/software_isp/meson.build        |   8 +
 src/libcamera/software_isp/software_isp.cpp   | 349 ++++++++
 src/libcamera/software_isp/swstats_cpu.cpp    | 336 ++++++++
 src/libcamera/software_isp/swstats_cpu.h      | 167 ++++
 33 files changed, 3186 insertions(+), 192 deletions(-)
 create mode 100644 Documentation/software-isp-benchmarking.rst
 rename {src/libcamera/pipeline/rpi/vc4 => include/libcamera/internal}/dma_heaps.h (65%)
 rename {src/libcamera/pipeline/rpi/common => include/libcamera/internal}/shared_mem_object.h (66%)
 create mode 100644 include/libcamera/internal/software_isp/debayer_params.h
 create mode 100644 include/libcamera/internal/software_isp/meson.build
 create mode 100644 include/libcamera/internal/software_isp/software_isp.h
 create mode 100644 include/libcamera/internal/software_isp/swisp_stats.h
 create mode 100644 include/libcamera/ipa/soft.mojom
 create mode 100644 src/ipa/simple/data/meson.build
 create mode 100644 src/ipa/simple/data/soft.conf
 create mode 100644 src/ipa/simple/meson.build
 create mode 100644 src/ipa/simple/soft_simple.cpp
 create mode 100644 src/libcamera/dma_heaps.cpp
 delete mode 100644 src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp
 create mode 100644 src/libcamera/shared_mem_object.cpp
 create mode 100644 src/libcamera/software_isp/debayer.cpp
 create mode 100644 src/libcamera/software_isp/debayer.h
 create mode 100644 src/libcamera/software_isp/debayer_cpu.cpp
 create mode 100644 src/libcamera/software_isp/debayer_cpu.h
 create mode 100644 src/libcamera/software_isp/meson.build
 create mode 100644 src/libcamera/software_isp/software_isp.cpp
 create mode 100644 src/libcamera/software_isp/swstats_cpu.cpp
 create mode 100644 src/libcamera/software_isp/swstats_cpu.h

Comments

Hans de Goede Feb. 14, 2024, 5:03 p.m. UTC | #1 
Hi All,

On 2/14/24 18:01, Hans de Goede wrote:
> Hi All,
> 
> Here is v3 of the patch-set to add Software ISP support
> to libcamera / to the simple pipeline-handler.

Note as Kieran pointed out in an offlist email, the new CI has
already caught a couple of issues with this v3:

Unit test configuration failed to build:
 - https://gitlab.freedesktop.org/camera/libcamera-softisp/-/jobs/55028866#L357

Debian 10 has an issue with some of the documentation additions:
 - https://gitlab.freedesktop.org/camera/libcamera-softisp/-/jobs/55028856#L439

Clang picks up warnings in the debayer_cpu implementation:
 - https://gitlab.freedesktop.org/camera/libcamera-softisp/-/jobs/55028858#L508

Clang picks up warnings in the swstats implementation:
 - https://gitlab.freedesktop.org/camera/libcamera-softisp/-/jobs/55028858#L549

I'll start looking into these next week at the earliest, feel
free to beat me to it :)

Regards,

Hans









> Changes in v3 vs v2:
> - The Software ISP now is always build when building the simple pipeline
>   handler, no more -Dpipelines=simple/simple. Instead whether the SoftISP
>   is used or not is based on the media-controller driver. For now it is
>   only enabled for the "qcom-camss" driver (Andrey)
> - SoftISP factory has been removed, there is a just a single SoftwareISP class now
> - Fix the multi-threading issues (Andrey)
> - Fully document the SharedMemObject and DmaHeap classes (Andrey)
> - Drop SwStats base (integrate into SwStatsCpu class)
> - Move headers for classes only used by the SoftISP into src/libcamera/software_isp
> - Move SwStats / SwDebayer docs into .cpp and extend it
> - Rename many foo_bar symbols to fooBar
> - Add constexpr kFooBar values for varies hardcoded sizes like
>   the yHistoGram having 16 bins and the gammalookuptable having 1024 entries
> - Make startFrame() and finishFrame() normal methods instead of
>   using function pointers for these
> - Document how/why an array of src pointers is passed to
>   the debayer functions
> - 12bpp unpacked raw bayer input support (Kieran)
> - Add "Software ISP benchmarking" doc
> 
> Changes in v2 vs v1:
> - Integrated Dennis, Martti and Toon's auto-exposure algorithm
>   based on the paper which they found which gives us a nice
>   relatively simple AEC + AGC algorithm
>   TODO: Add link to paper to source + commit-message
> - Integrated Dennis' doxygen comments for all new classes, no more warnings!
> - Move AWB gain calculations to the IPA (Andrey)
> - Added 8 bpp and 10 bpp unpacked raw bayer input support (Hans)
> - Use dma-buf for the output FrameBuffer-s (Andrey)
> - Memcpy data from input FrameBuffers to a heap buffer on a line by line
>   basis to speedup debayering from uncached mem (Hans)
> - This addresses all "open issues" from the v1 posting
> 
> Changes in v1 vs RFC-v2:
> - Add and use SwStats[Cpu] and Debayer[Cpu] classes
> - Rename linaro soft-IPA and soft-ISP implementations to simple following
>   the simple pipeline hander
> - Integrate Pavel's swstats and debayer improvements
> 
> This has been tested on:
> - Qualcomm RB5 board with a mezzanine board equipped with RPi camera v2 (Andrey)
> - Lenovo x13s, sc8280xp (Bryan)
> - Pinephone (Pavel)
> - Debix Model A (Milan)
> - Lenovo Thinkpad Yoga X1 yoga gen7/8, Alder Lake/Raptor Lake IPU6EP +
>   ov2740 10bpp packed + unpacked (Dennis / Hans)
> - Dell Latitude 9420, Tiger Lake IPU6 + ov01a1s (RGBI) sensor
>   (Martti, requires IGIG_GBGR_IGIG_GRGB patches on top)
> 
> git branch for this v3 patch set:
> https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v06
> 
> git branch for v2 of the patch set:
> https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v05
> 
> git branch for v1 of the patch set:
> https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v04
> 
> Old RFC-v2 cover-letter with small updates:
> 
> Here is an implementation of Software ISP and Software IPA
> which provide debayering and implementation of image processing
> algorithms for systems without a hardware ISP, or in cases when
> there are no public drivers for the hardware ISP present in the
> system.
> 
> The Software IPA has auto exposure/AGC and AWB. For the AGC
> the analogue gain control of the camera sensor is used (if
> available).
> 
> The 6th patch renames some variables in the simple pipeline
> handler for the Software ISP to use the same buffer handling
> code as the Converter currently does. This lets one to
> avoid adding extra code to the pipeline handler, but also
> makes the Software ISP and the Converter mutually exclusive.
> 
> The Software ISP / IPA are intended to be used with the simple
> pipeline handler. The pipeline handler doesn't interact with
> the Software IPA directly - the Software IPA is hidden behind
> the Software ISP interface. To use the Software ISP the build
> must be configured with -Dpipelines=simple -Dipas=simple .
> 
> This patch set uses SharedMemObject class used by the RPi pipeline
> handler - the second patch in the series moves the header file
> to a common directory.
> 
> This patch set has been tested on Qualcomm RB5 board with
> a mezzanine board equipped with RPi camera v2 (not the
> standard RB5 camera mezzanine).
> 
> git branch for the RFC-v2 patch set:
> https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v03
> 
> RFC-v1 of the patch set:
> https://patchwork.libcamera.org/cover/19262/
> 
> Changes in RFC v2 vs RFC v1:
> - patches are restructured and reordered
> - the Software IPA is hidden behind the Software ISP interface. The
>   pipeline handler doesn't interact with the Software IPA directly
> - instantiation of the Software ISP / IPA is configurable (at build
>   time)
> - comment explaining the implementation limitations is added to
>   SwIspLinaro::IspWorker::debayerRaw10P()
> 
> Regards,
> 
> Hans
> 
> 
> 
> Andrei Konovalov (1):
>   libcamera: shared_mem_object: reorganize the code and document the
>     SharedMemObject class
> 
> Andrey Konovalov (8):
>   libcamera: pipeline: simple: fix size adjustment in validate()
>   libcamera: internal: Move dma_heaps.[h,cpp] to common directories
>   libcamera: dma_heaps: extend DmaHeap class to support system heap
>   libcamera: internal: Move SharedMemObject class to a common directory
>   libcamera: ipa: add Soft IPA
>   libcamera: introduce SoftwareIsp
>   libcamera: pipeline: simple: rename converterBuffers_ and related vars
>   libcamera: pipeline: simple: enable use of Soft ISP and Soft IPA
> 
> Hans de Goede (7):
>   libcamera: software_isp: Add SwStatsCpu class
>   libcamera: software_isp: Add Debayer base class
>   libcamera: software_isp: Add DebayerCpu class
>   libcamera: swstats_cpu: Add support for 8, 10 and 12 bpp unpacked
>     bayer input
>   libcamera: debayer_cpu: Add support for 8, 10 and 12 bpp unpacked
>     bayer input
>   libcamera: debayer_cpu: Add BGR888 output support
>   libcamera: Add "Software ISP benchmarking" documentation
> 
>  Documentation/Doxyfile.in                     |   1 +
>  Documentation/index.rst                       |   1 +
>  Documentation/meson.build                     |   1 +
>  Documentation/software-isp-benchmarking.rst   |  82 ++
>  .../libcamera/internal}/dma_heaps.h           |  14 +-
>  include/libcamera/internal/meson.build        |   3 +
>  .../libcamera/internal}/shared_mem_object.h   |  94 ++-
>  .../internal/software_isp/debayer_params.h    |  48 ++
>  .../internal/software_isp/meson.build         |   7 +
>  .../internal/software_isp/software_isp.h      |  98 +++
>  .../internal/software_isp/swisp_stats.h       |  38 +
>  include/libcamera/ipa/meson.build             |   1 +
>  include/libcamera/ipa/soft.mojom              |  28 +
>  meson_options.txt                             |   2 +-
>  src/ipa/simple/data/meson.build               |   9 +
>  src/ipa/simple/data/soft.conf                 |   3 +
>  src/ipa/simple/meson.build                    |  25 +
>  src/ipa/simple/soft_simple.cpp                | 308 +++++++
>  src/libcamera/dma_heaps.cpp                   | 164 ++++
>  src/libcamera/meson.build                     |   3 +
>  src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp  |  90 --
>  src/libcamera/pipeline/rpi/vc4/meson.build    |   1 -
>  src/libcamera/pipeline/rpi/vc4/vc4.cpp        |   5 +-
>  src/libcamera/pipeline/simple/simple.cpp      | 232 ++++--
>  src/libcamera/shared_mem_object.cpp           | 191 +++++
>  src/libcamera/software_isp/debayer.cpp        |  29 +
>  src/libcamera/software_isp/debayer.h          | 104 +++
>  src/libcamera/software_isp/debayer_cpu.cpp    | 779 ++++++++++++++++++
>  src/libcamera/software_isp/debayer_cpu.h      | 157 ++++
>  src/libcamera/software_isp/meson.build        |   8 +
>  src/libcamera/software_isp/software_isp.cpp   | 349 ++++++++
>  src/libcamera/software_isp/swstats_cpu.cpp    | 336 ++++++++
>  src/libcamera/software_isp/swstats_cpu.h      | 167 ++++
>  33 files changed, 3186 insertions(+), 192 deletions(-)
>  create mode 100644 Documentation/software-isp-benchmarking.rst
>  rename {src/libcamera/pipeline/rpi/vc4 => include/libcamera/internal}/dma_heaps.h (65%)
>  rename {src/libcamera/pipeline/rpi/common => include/libcamera/internal}/shared_mem_object.h (66%)
>  create mode 100644 include/libcamera/internal/software_isp/debayer_params.h
>  create mode 100644 include/libcamera/internal/software_isp/meson.build
>  create mode 100644 include/libcamera/internal/software_isp/software_isp.h
>  create mode 100644 include/libcamera/internal/software_isp/swisp_stats.h
>  create mode 100644 include/libcamera/ipa/soft.mojom
>  create mode 100644 src/ipa/simple/data/meson.build
>  create mode 100644 src/ipa/simple/data/soft.conf
>  create mode 100644 src/ipa/simple/meson.build
>  create mode 100644 src/ipa/simple/soft_simple.cpp
>  create mode 100644 src/libcamera/dma_heaps.cpp
>  delete mode 100644 src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp
>  create mode 100644 src/libcamera/shared_mem_object.cpp
>  create mode 100644 src/libcamera/software_isp/debayer.cpp
>  create mode 100644 src/libcamera/software_isp/debayer.h
>  create mode 100644 src/libcamera/software_isp/debayer_cpu.cpp
>  create mode 100644 src/libcamera/software_isp/debayer_cpu.h
>  create mode 100644 src/libcamera/software_isp/meson.build
>  create mode 100644 src/libcamera/software_isp/software_isp.cpp
>  create mode 100644 src/libcamera/software_isp/swstats_cpu.cpp
>  create mode 100644 src/libcamera/software_isp/swstats_cpu.h
>
Hans de Goede Feb. 14, 2024, 5:26 p.m. UTC | #2 
Hi,

On 2/14/24 18:03, Hans de Goede wrote:
> Hi All,
> 
> On 2/14/24 18:01, Hans de Goede wrote:
>> Hi All,
>>
>> Here is v3 of the patch-set to add Software ISP support
>> to libcamera / to the simple pipeline-handler.
> 
> Note as Kieran pointed out in an offlist email, the new CI has
> already caught a couple of issues with this v3:
> 
> Unit test configuration failed to build:
>  - https://gitlab.freedesktop.org/camera/libcamera-softisp/-/jobs/55028866#L357

Ok, so I took a quick peek at this one and the issue is
that the simple pipeline handler will not build unless
-Dipas=simple is passed when configuring.

The missing header here is autogenerated.

I see that the ipu3 pipeline handler also unconditionally
includes the auto-generated:

#include <libcamera/ipa/ipu3_ipa_interface.h>
#include <libcamera/ipa/ipu3_ipa_proxy.h>

headers, so it will show similar erros when using -Dpipelines=ipu3
without -Dipas=ipu3 .

So I wonder how to fix this. Is there some meson magic
somewhere to automatically enable the ipu3 ipa when the
ipu3 pipeline is enabled which we can copy for the simple
pipeline ?

Regards,

Hans
Bryan O'Donoghue Feb. 15, 2024, 12:14 p.m. UTC | #3 
On 14/02/2024 5:01 p.m., Hans de Goede wrote:
> Hi All,
> 
> Here is v3 of the patch-set to add Software ISP support
> to libcamera / to the simple pipeline-handler.
> 
> Changes in v3 vs v2:
> - The Software ISP now is always build when building the simple pipeline
>    handler, no more -Dpipelines=simple/simple. Instead whether the SoftISP
>    is used or not is based on the media-controller driver. For now it is
>    only enabled for the "qcom-camss" driver (Andrey)
> - SoftISP factory has been removed, there is a just a single SoftwareISP class now
> - Fix the multi-threading issues (Andrey)
> - Fully document the SharedMemObject and DmaHeap classes (Andrey)
> - Drop SwStats base (integrate into SwStatsCpu class)
> - Move headers for classes only used by the SoftISP into src/libcamera/software_isp
> - Move SwStats / SwDebayer docs into .cpp and extend it
> - Rename many foo_bar symbols to fooBar
> - Add constexpr kFooBar values for varies hardcoded sizes like
>    the yHistoGram having 16 bins and the gammalookuptable having 1024 entries
> - Make startFrame() and finishFrame() normal methods instead of
>    using function pointers for these
> - Document how/why an array of src pointers is passed to
>    the debayer functions
> - 12bpp unpacked raw bayer input support (Kieran)
> - Add "Software ISP benchmarking" doc

Tested-by: Bryan O'Donoghue <bryan.odonoghue@linaro.org> # lenovo x13s
Milan Zamazal Feb. 16, 2024, 12:24 p.m. UTC | #4 
Hans de Goede <hdegoede@redhat.com> writes:

> Hi All,
>
> Here is v3 of the patch-set to add Software ISP support
> to libcamera / to the simple pipeline-handler.

Hi Hans,

thank you, great work by you and others involved.  I have commented on all what
I thought I had something to say about, tell me in case you need more review on
something from me.

Regards,
Milan

> Changes in v3 vs v2:
> - The Software ISP now is always build when building the simple pipeline
>   handler, no more -Dpipelines=simple/simple. Instead whether the SoftISP
>   is used or not is based on the media-controller driver. For now it is
>   only enabled for the "qcom-camss" driver (Andrey)
> - SoftISP factory has been removed, there is a just a single SoftwareISP class now
> - Fix the multi-threading issues (Andrey)
> - Fully document the SharedMemObject and DmaHeap classes (Andrey)
> - Drop SwStats base (integrate into SwStatsCpu class)
> - Move headers for classes only used by the SoftISP into src/libcamera/software_isp
> - Move SwStats / SwDebayer docs into .cpp and extend it
> - Rename many foo_bar symbols to fooBar
> - Add constexpr kFooBar values for varies hardcoded sizes like
>   the yHistoGram having 16 bins and the gammalookuptable having 1024 entries
> - Make startFrame() and finishFrame() normal methods instead of
>   using function pointers for these
> - Document how/why an array of src pointers is passed to
>   the debayer functions
> - 12bpp unpacked raw bayer input support (Kieran)
> - Add "Software ISP benchmarking" doc
>
> Changes in v2 vs v1:
> - Integrated Dennis, Martti and Toon's auto-exposure algorithm
>   based on the paper which they found which gives us a nice
>   relatively simple AEC + AGC algorithm
>   TODO: Add link to paper to source + commit-message
> - Integrated Dennis' doxygen comments for all new classes, no more warnings!
> - Move AWB gain calculations to the IPA (Andrey)
> - Added 8 bpp and 10 bpp unpacked raw bayer input support (Hans)
> - Use dma-buf for the output FrameBuffer-s (Andrey)
> - Memcpy data from input FrameBuffers to a heap buffer on a line by line
>   basis to speedup debayering from uncached mem (Hans)
> - This addresses all "open issues" from the v1 posting
>
> Changes in v1 vs RFC-v2:
> - Add and use SwStats[Cpu] and Debayer[Cpu] classes
> - Rename linaro soft-IPA and soft-ISP implementations to simple following
>   the simple pipeline hander
> - Integrate Pavel's swstats and debayer improvements
>
> This has been tested on:
> - Qualcomm RB5 board with a mezzanine board equipped with RPi camera v2 (Andrey)
> - Lenovo x13s, sc8280xp (Bryan)
> - Pinephone (Pavel)
> - Debix Model A (Milan)
> - Lenovo Thinkpad Yoga X1 yoga gen7/8, Alder Lake/Raptor Lake IPU6EP +
>   ov2740 10bpp packed + unpacked (Dennis / Hans)
> - Dell Latitude 9420, Tiger Lake IPU6 + ov01a1s (RGBI) sensor
>   (Martti, requires IGIG_GBGR_IGIG_GRGB patches on top)
>
> git branch for this v3 patch set:
> https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v06
>
> git branch for v2 of the patch set:
> https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v05
>
> git branch for v1 of the patch set:
> https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v04
>
> Old RFC-v2 cover-letter with small updates:
>
> Here is an implementation of Software ISP and Software IPA
> which provide debayering and implementation of image processing
> algorithms for systems without a hardware ISP, or in cases when
> there are no public drivers for the hardware ISP present in the
> system.
>
> The Software IPA has auto exposure/AGC and AWB. For the AGC
> the analogue gain control of the camera sensor is used (if
> available).
>
> The 6th patch renames some variables in the simple pipeline
> handler for the Software ISP to use the same buffer handling
> code as the Converter currently does. This lets one to
> avoid adding extra code to the pipeline handler, but also
> makes the Software ISP and the Converter mutually exclusive.
>
> The Software ISP / IPA are intended to be used with the simple
> pipeline handler. The pipeline handler doesn't interact with
> the Software IPA directly - the Software IPA is hidden behind
> the Software ISP interface. To use the Software ISP the build
> must be configured with -Dpipelines=simple -Dipas=simple .
>
> This patch set uses SharedMemObject class used by the RPi pipeline
> handler - the second patch in the series moves the header file
> to a common directory.
>
> This patch set has been tested on Qualcomm RB5 board with
> a mezzanine board equipped with RPi camera v2 (not the
> standard RB5 camera mezzanine).
>
> git branch for the RFC-v2 patch set:
> https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v03
>
> RFC-v1 of the patch set:
> https://patchwork.libcamera.org/cover/19262/
>
> Changes in RFC v2 vs RFC v1:
> - patches are restructured and reordered
> - the Software IPA is hidden behind the Software ISP interface. The
>   pipeline handler doesn't interact with the Software IPA directly
> - instantiation of the Software ISP / IPA is configurable (at build
>   time)
> - comment explaining the implementation limitations is added to
>   SwIspLinaro::IspWorker::debayerRaw10P()
>
> Regards,
>
> Hans
>
>
>
> Andrei Konovalov (1):
>   libcamera: shared_mem_object: reorganize the code and document the
>     SharedMemObject class
>
> Andrey Konovalov (8):
>   libcamera: pipeline: simple: fix size adjustment in validate()
>   libcamera: internal: Move dma_heaps.[h,cpp] to common directories
>   libcamera: dma_heaps: extend DmaHeap class to support system heap
>   libcamera: internal: Move SharedMemObject class to a common directory
>   libcamera: ipa: add Soft IPA
>   libcamera: introduce SoftwareIsp
>   libcamera: pipeline: simple: rename converterBuffers_ and related vars
>   libcamera: pipeline: simple: enable use of Soft ISP and Soft IPA
>
> Hans de Goede (7):
>   libcamera: software_isp: Add SwStatsCpu class
>   libcamera: software_isp: Add Debayer base class
>   libcamera: software_isp: Add DebayerCpu class
>   libcamera: swstats_cpu: Add support for 8, 10 and 12 bpp unpacked
>     bayer input
>   libcamera: debayer_cpu: Add support for 8, 10 and 12 bpp unpacked
>     bayer input
>   libcamera: debayer_cpu: Add BGR888 output support
>   libcamera: Add "Software ISP benchmarking" documentation
>
>  Documentation/Doxyfile.in                     |   1 +
>  Documentation/index.rst                       |   1 +
>  Documentation/meson.build                     |   1 +
>  Documentation/software-isp-benchmarking.rst   |  82 ++
>  .../libcamera/internal}/dma_heaps.h           |  14 +-
>  include/libcamera/internal/meson.build        |   3 +
>  .../libcamera/internal}/shared_mem_object.h   |  94 ++-
>  .../internal/software_isp/debayer_params.h    |  48 ++
>  .../internal/software_isp/meson.build         |   7 +
>  .../internal/software_isp/software_isp.h      |  98 +++
>  .../internal/software_isp/swisp_stats.h       |  38 +
>  include/libcamera/ipa/meson.build             |   1 +
>  include/libcamera/ipa/soft.mojom              |  28 +
>  meson_options.txt                             |   2 +-
>  src/ipa/simple/data/meson.build               |   9 +
>  src/ipa/simple/data/soft.conf                 |   3 +
>  src/ipa/simple/meson.build                    |  25 +
>  src/ipa/simple/soft_simple.cpp                | 308 +++++++
>  src/libcamera/dma_heaps.cpp                   | 164 ++++
>  src/libcamera/meson.build                     |   3 +
>  src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp  |  90 --
>  src/libcamera/pipeline/rpi/vc4/meson.build    |   1 -
>  src/libcamera/pipeline/rpi/vc4/vc4.cpp        |   5 +-
>  src/libcamera/pipeline/simple/simple.cpp      | 232 ++++--
>  src/libcamera/shared_mem_object.cpp           | 191 +++++
>  src/libcamera/software_isp/debayer.cpp        |  29 +
>  src/libcamera/software_isp/debayer.h          | 104 +++
>  src/libcamera/software_isp/debayer_cpu.cpp    | 779 ++++++++++++++++++
>  src/libcamera/software_isp/debayer_cpu.h      | 157 ++++
>  src/libcamera/software_isp/meson.build        |   8 +
>  src/libcamera/software_isp/software_isp.cpp   | 349 ++++++++
>  src/libcamera/software_isp/swstats_cpu.cpp    | 336 ++++++++
>  src/libcamera/software_isp/swstats_cpu.h      | 167 ++++
>  33 files changed, 3186 insertions(+), 192 deletions(-)
>  create mode 100644 Documentation/software-isp-benchmarking.rst
>  rename {src/libcamera/pipeline/rpi/vc4 => include/libcamera/internal}/dma_heaps.h (65%)
>  rename {src/libcamera/pipeline/rpi/common => include/libcamera/internal}/shared_mem_object.h (66%)
>  create mode 100644 include/libcamera/internal/software_isp/debayer_params.h
>  create mode 100644 include/libcamera/internal/software_isp/meson.build
>  create mode 100644 include/libcamera/internal/software_isp/software_isp.h
>  create mode 100644 include/libcamera/internal/software_isp/swisp_stats.h
>  create mode 100644 include/libcamera/ipa/soft.mojom
>  create mode 100644 src/ipa/simple/data/meson.build
>  create mode 100644 src/ipa/simple/data/soft.conf
>  create mode 100644 src/ipa/simple/meson.build
>  create mode 100644 src/ipa/simple/soft_simple.cpp
>  create mode 100644 src/libcamera/dma_heaps.cpp
>  delete mode 100644 src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp
>  create mode 100644 src/libcamera/shared_mem_object.cpp
>  create mode 100644 src/libcamera/software_isp/debayer.cpp
>  create mode 100644 src/libcamera/software_isp/debayer.h
>  create mode 100644 src/libcamera/software_isp/debayer_cpu.cpp
>  create mode 100644 src/libcamera/software_isp/debayer_cpu.h
>  create mode 100644 src/libcamera/software_isp/meson.build
>  create mode 100644 src/libcamera/software_isp/software_isp.cpp
>  create mode 100644 src/libcamera/software_isp/swstats_cpu.cpp
>  create mode 100644 src/libcamera/software_isp/swstats_cpu.h