[libcamera-devel,v4,00/31] libcamera: ipu3: Add ImgU support + multiple streams

746 mbox series

Hello,
I know, 31 patches, it's bad, but please don't run away. Not yet at least.

This series combines a few things, I decided to keep it together as one part
is based on the other, but they could be actually considered separate.

The series includes:
- Support for IPU3 ImgU unit (patches 01->15)
  I have sent v1 some time ago, v2 not too long ago and an updated version
  called v3-part which included and updated version of the first 4 patches only.
  I have updated the v3-part patches with:
  - Remove all formats not supported by the IPU3 (only keep 10-bit RAW Bayer)
  - Standardized printout of formats to remove space wherever possible
  - Add Kieran's tags
  Patches from 04->15 are similar to the ones in v2 with small mostly cosmetic
  updates. Please review these ones if you've not tackled v2 already.

- Support for multiple stream capture in IPU3 pipeline handler (16->26)
  This part contains a few small changes I consider for inclusion for the
  Request class and a few I would like to receive feedback for the Pipeline
  Handler base class to add pre/post memory allocation hooks (marked as RFC).
  All other patches extends the IPU3 pipeline handler to work with both output
  and viewfinder, with the quirk that I need to operate the viewfinder and the
  stat node even when not required as otherwise capture operations stall.

- Hacks on the cam application to support capture of multiple streams (26->31)
  Mostly hacks to enable my testing, with the very bad ones marked as HACK and
  the remaining ones I hope might be useful to Niklas to extend format parsing
  to support multiple streams.

Tested on Soraka capturing output, viewfinder and output+viewfinder at the
same time in 2560x1920, 1920x1080 and 640x480 resolutions.

A few points still bother me:
- viewfinder and stat nodes needs to be operated anyhow, otherwise ImgU does
  work and stall.
- It is not yet clear to me how to calculate the sizes in the capture pipeline
  based on the requested output resolutions. Upstream pointed us to an XML
  file with all sizes hardcoded, which is an unacceptable solution as sizes
  have to be calculated dynamically based on requested stream configurations.
- When operating on one of the two ImgU pipes, if the other one is still linked
  from a previous capture operation, the whole ImgU unit stall. This is maybe
  expected, but I have not found yet a reasonable way to handle this.
  Please see 15/31 (marked as RFC for this reason) where I try to explain the
  issue in a lengthy comment.

I'll do more testing on the above points and get in contact with IPU3 upstream
driver developers soon to have clarifications, in the meantime, feedbacks are
welcome.

Thanks
   j

Jacopo Mondi (31):
  libcamera: formats: Add toString() methods
  libcamera: formats: Define FormatEnum type
  libcamera: ipu3: Make sure sensor provides a compatible format
  libcamera: ipu3: Set stream configuration from sensor
  libcamera: ipu3: Initialize and configure CIO2
  libcamera: ipu3: Initialize and configure ImgUs
  libcamera: ipu3: Propagate image format
  libcamera: ipu3: Implement camera start/stop
  libcamera: ipu3: Implement buffer allocation
  libcamera: ipu3: Implement buffer release
  libcamera: ipu3: Queue requests to the pipeline
  libcamera: ipu3: Connect CIO2 and ImgU bufferReady signals
  libcamera: ipu3: Use NV12 as default image format
  HACK: Soraka: Limit resolution to 2560x1920
  RFC: libcamera: ipu3: Enable ImgU media links
  libcamera: request: Store the streams in the request
  libcamera: request: Add camera() getter method
  libcamera: ipu3: Create camera with 2 streams
  libcamera: ipu3: Configure 'output' and 'viewfinder'
  libcamera: ipu3: Add multiple stream memory management
  RFC: libcamera: pipeline_handlers: Add preAllocateBuffers
  RFC: libcamera: pipeline_handlers: Add postFreeBuffers
  libcamera: ipu3: Enable media links conditionally
  libcamera: ipu3: Queue request for multiple streams
  libcamera: ipu3: Connect viewfinder's BufferReady signal
  HACK: cam: Allow camera with multiple streams
  src: cam: Create requests with multiple streams
  src: cam: Add output name to frame writer
  src: cam: Handle multiple streams in request complete
  HACK: src: cam: Play with streams combinations
  HACK: cam: Add options to select the stream to use

 include/libcamera/request.h              |    3 +
 src/cam/buffer_writer.cpp                |    4 +-
 src/cam/buffer_writer.h                  |    2 +-
 src/cam/main.cpp                         |  132 ++-
 src/libcamera/camera.cpp                 |   12 +-
 src/libcamera/formats.cpp                |   27 +
 src/libcamera/include/formats.h          |   22 +
 src/libcamera/include/pipeline_handler.h |    9 +
 src/libcamera/include/v4l2_device.h      |    2 +
 src/libcamera/include/v4l2_subdevice.h   |    6 +-
 src/libcamera/meson.build                |    1 +
 src/libcamera/pipeline/ipu3/ipu3.cpp     | 1383 +++++++++++++++++++---
 src/libcamera/pipeline_handler.cpp       |   40 +-
 src/libcamera/request.cpp                |   23 +
 src/libcamera/v4l2_device.cpp            |   18 +
 src/libcamera/v4l2_subdevice.cpp         |   23 +-
 16 files changed, 1478 insertions(+), 229 deletions(-)
 create mode 100644 src/libcamera/formats.cpp
 create mode 100644 src/libcamera/include/formats.h

--
2.21.0