[libcamera-devel,v3,00/13] Add Bayer format support for RkISP1

17663 mbox series

Hello,

This patch series implements support for raw Bayer capture in the RkISP1
pipeline handler and IPA module. I have taken over Florian's v2,
addressed my review comments, and added further fixes and improvements.

The RkISP1 can capture raw frames by bypassing the ISP, feeding the
input data directly to the resizer (which has to be bypassed as well,
with a 1:1 scaling ratio). In this mode, the ISP doesn't consume
parameters or produce statistics, which prevents most algorithms from
running. The notable exception is the AGC algorithm which still needs to
run in order to set manual exposure time and analogue gain (automatic
mode is not available).

The series contains a few drive-by fixes and cleanups. Patch 01/13 is
one of those, it reorders the algorithm operations in the .h and .cpp
files. If it gets accepted, I will likely merge it sooner than later and
separately from the rest of the series as it has a high risk of causing
conflicts. Patch 02/13 is another small cleanup in the RkISP1 AGC.

Patch 03/13 starts adding support for raw capture by addressing the IPA
side first, allowing algorithms to report if they support raw capture,
and skipping the ones that don't at runtime when capturing raw frames.
The next patch, 04/13, adds support for manual gain and exposure time in
the AGC algorithm. I have picked it up from the list (posted as "[PATCH
v3] ipa: rkisp1: Add support for manual gain and exposure"), in order to
implement raw support in AGC on top in patch 05/13.

With patches 06/13 to 08/13 we get back on the libcamera side, with
more drive-by improvements that turn StreamRole into a scoped enum and
return a unique pointer from PipelineHandler::generateConfiguration().
These could also be merged independently from the rest of the series.

Patches 09/13 to 11/13 then prepare the RkISP1 pipeline handler. Patch
09/13 is a small refactoring to prepare for the larger changes, patch
10/13 an improvement previously posted to the list (as "[RFC PATCH v2]
pipeline: rkisp1: Query the driver for formats") that I found useful,
and patch 11/13 a bug fix.

Finally, patches 12/13 and 13/13 implement raw capture support in the
pipeline handler. I have split this into runtime support (12/13) and
configuration support (13/13) to ease review. Patch 13/13 could possibly
be further split into three patches for configure(), validate() and
generateConfiguration(), but that's more complicated and I think that,
while relatively big, 13/13 is still reviewable.

The result has been tested on an NXP i.MX8MP with an IMX219 camera
sensor. I haven't noticed any regression with non-raw capture, and have
successfully captured raw frames in different resolutions by specifying
either a raw role or a raw pixel format on the cam command line.

Florian Sylvestre (2):
  pipeline: rkisp1: Support raw Bayer capture at runtime
  pipeline: rkisp1: Support raw Bayer capture configuration

Laurent Pinchart (9):
  ipa: Sort algorithm operations based on calling order
  ipa: rkisp1: Move shutter speed and analogue gain limits from agc to
    sensor
  ipa: rkisp1: Support raw capture in IPA operations
  ipa: rkisp1: agc: Support raw capture
  libcamera: stream: Add operator<<() to print StreamRole as a string
  libcamera: stream: Turn StreamRole into scoped enumeration
  libcamera: pipeline_handler: Return unique_ptr from
    generateConfiguration()
  pipeline: rkisp1: Pass info pointer to tryCompleteRequest()
  pipeline: rkisp1: Fix stream size validation

Paul Elder (2):
  ipa: rkisp1: Add support for manual gain and exposure
  pipeline: rkisp1: Query the driver for formats

 include/libcamera/internal/pipeline_handler.h |   2 +-
 include/libcamera/ipa/rkisp1.mojom            |   2 +-
 include/libcamera/stream.h                    |   5 +-
 src/apps/lc-compliance/capture_test.cpp       |  17 +-
 src/gstreamer/gstlibcamerapad.cpp             |  20 +-
 src/ipa/ipu3/algorithms/af.cpp                |  32 +-
 src/ipa/ipu3/algorithms/awb.cpp               | 166 +++++-----
 src/ipa/libipa/algorithm.cpp                  |  34 +-
 src/ipa/libipa/algorithm.h                    |  14 +-
 src/ipa/rkisp1/algorithms/agc.cpp             | 184 +++++++----
 src/ipa/rkisp1/algorithms/agc.h               |   6 +
 src/ipa/rkisp1/algorithms/algorithm.h         |  12 +-
 src/ipa/rkisp1/algorithms/awb.cpp             |  94 +++---
 src/ipa/rkisp1/algorithms/awb.h               |   6 +-
 src/ipa/rkisp1/ipa_context.cpp                |  33 +-
 src/ipa/rkisp1/ipa_context.h                  |  40 ++-
 src/ipa/rkisp1/rkisp1.cpp                     |  69 +++-
 src/libcamera/camera.cpp                      |   8 +-
 src/libcamera/pipeline/ipu3/ipu3.cpp          |  10 +-
 .../pipeline/raspberrypi/raspberrypi.cpp      |  12 +-
 src/libcamera/pipeline/rkisp1/rkisp1.cpp      | 309 ++++++++++++------
 src/libcamera/pipeline/rkisp1/rkisp1_path.cpp | 254 ++++++++++++--
 src/libcamera/pipeline/rkisp1/rkisp1_path.h   |  15 +-
 src/libcamera/pipeline/simple/simple.cpp      |  12 +-
 src/libcamera/pipeline/uvcvideo/uvcvideo.cpp  |   8 +-
 src/libcamera/pipeline/vimc/vimc.cpp          |   8 +-
 src/libcamera/pipeline_handler.cpp            |   3 +-
 src/libcamera/stream.cpp                      |  19 ++
 28 files changed, 957 insertions(+), 437 deletions(-)


base-commit: f9c859343936db876f13d550ff9adca96a9ef528