[libcamera-devel,v3,00/30] libcamera: Handle fallout of FrameBuffer offset support

13666 mbox series

Hello everybody,

This patch series started as an investigation of a qcam failure with
IPU3 after the merge of the FrameBuffer offset support. While a hack in
qcam would be possible, I decided to instead address the core issue and
fix it in V4L2VideoDevice.

Compared to v2, the series now includes fixes the Android HAL (cam and
qcam where already included in v2, and the libcamera core was addressed
in v1).

I have tested this series with the IPU3 (NV12), VIMC (ARGB8888) and UVC
(MJPEG and YUYV) pipeline handlers using cam, qcam, libcamerasrc and
v4l2-compat, and with the Raspberry Pi (ARGB8888, NV12 and YUV420)
pipeline handler using cam and qcam only. (On a side note, the qcam
format conversion code doesn't take strides into account correctly,
patches will follow.)

Umang, you have been kind enough to test the Android HAL with the IPU3
pipeline handler on v2, I would be grateful if you could test this
version too.

The most important change is in patches 13/30 and 14/30, which translate
between V4L2 buffers and libcamera FrameBuffer to handle the case where
a multi-planar frame buffer is used with the V4L2 single-planar API.
It's working more or less by chance at the moment (except in qcam where
it's broken, and possibly in other places I haven't tested). Patches
01/30 to 12/30 are cleanups and additions to prepare for the work in
V4L2VideoDevice, and patch 15/30 is a small cleanup on top. Patches
16/30 and 17/30 then improve the FrameBuffer class API as a cleanup.

Patches 18/30 to 30/30 fix issues in the Android HAL, v4l2-compat, cam
and qcam. Worth being noted is patch 21-30 that introduces an Image
class shared by cam and qcam. The class duplicates the MappedFrameBuffer
implementation private to libcamera. I've tried to rework
MappedFrameBuffer into something I would be happy to see in the public
API, but failed to do so in a reasonable amount of time, and I didn't
want to delay this important regression fix.

This series doesn't break any unit test, as vimc doesn't support NV12.
Addition of NV12 support to the vimc kernel driver would be very nice,
in order to test multi-planar support in our unit tests. Volunteers are
welcome ;-)

Laurent Pinchart (30):
  libcamera: base: utils: Use size_t for index in utils::enumerate()
  libcamera: file_descriptor: Add a function to retrieve the inode
  libcamera: v4l2_videodevice: Drop toV4L2PixelFormat()
  libcamera: Use V4L2PixelFormat::fromPixelFormat()
  libcamera: formats: Move plane info structure to PixelFormatInfo
  libcamera: formats: Add planeSize() helpers to PixelFormatInfo
  libcamera: formats: Support V4L2 non-contiguous formats
  libcamera: framebuffer: Move planes check to constructor
  libcamera: framebuffer: Add a function to check if planes are
    contiguous
  libcamera: v4l2_videodevice: Cache PixelFormatInfo
  libcamera: v4l2_videodevice: Document plane handling in createBuffer()
  libcamera: v4l2_videodevice: Take stride into account to compute
    offsets
  libcamera: v4l2_videodevice: Coalesce planes when queuing buffer
  libcamera: v4l2_videodevice: Split planes when dequeuing buffer
  libcamera: v4l2_videodevice: Use utils::enumerate()
  libcamera: framebuffer: Allocate metadata planes at construction time
  libcamera: framebuffer: Prevent modifying the number of metadata
    planes
  android: camera_device: Don't assume all planes use the same fd
  android: jpeg: Use stride instead of image width for line address
  android: jpeg: Support multi-planar buffers
  cam: Add Image class
  cam: file_sink: Use Image class to access pixel data
  cam: drm: Support per-plane stride values
  cam: drm: Set per-plane offsets when creating DRM frame buffer
  cam: drm: Avoid importing the same dmabuf multiple times
  qcam: Print bytesused for all planes
  qcam: Use Image class to access pixel data
  qcam: viewfinder_gl: Support multi-planar buffers
  qcam: viewfinder_qt: Support multi-planar buffers
  v4l2: Sum bytesused for all planes when dequeuing buffer

 include/libcamera/base/utils.h                |   4 +-
 include/libcamera/file_descriptor.h           |   3 +
 include/libcamera/framebuffer.h               |  19 +-
 include/libcamera/internal/formats.h          |  22 +-
 include/libcamera/internal/framebuffer.h      |   2 +
 include/libcamera/internal/v4l2_pixelformat.h |   2 +-
 include/libcamera/internal/v4l2_videodevice.h |   3 +-
 src/android/camera_device.cpp                 |  25 +-
 src/android/jpeg/encoder_libjpeg.cpp          |  19 +-
 src/android/jpeg/encoder_libjpeg.h            |   8 +-
 src/android/jpeg/post_processor_jpeg.cpp      |   2 +-
 src/android/jpeg/thumbnailer.cpp              |   4 +-
 src/android/mm/generic_camera_buffer.cpp      |  11 +-
 src/android/yuv/post_processor_yuv.cpp        |  10 +-
 src/cam/camera_session.cpp                    |   4 +-
 src/cam/drm.cpp                               |  38 +-
 src/cam/drm.h                                 |   7 +-
 src/cam/file_sink.cpp                         |  44 +--
 src/cam/file_sink.h                           |   6 +-
 src/cam/image.cpp                             | 107 +++++
 src/cam/image.h                               |  52 +++
 src/cam/kms_sink.cpp                          |  28 +-
 src/cam/meson.build                           |   1 +
 src/libcamera/file_descriptor.cpp             |  26 ++
 src/libcamera/formats.cpp                     | 373 ++++++++++++++----
 src/libcamera/framebuffer.cpp                 |  57 ++-
 src/libcamera/pipeline/ipu3/cio2.cpp          |   4 +-
 src/libcamera/pipeline/ipu3/imgu.cpp          |   2 +-
 .../pipeline/raspberrypi/raspberrypi.cpp      |   8 +-
 src/libcamera/pipeline/rkisp1/rkisp1_path.cpp |   6 +-
 src/libcamera/pipeline/simple/converter.cpp   |   8 +-
 src/libcamera/pipeline/simple/simple.cpp      |   4 +-
 src/libcamera/pipeline/uvcvideo/uvcvideo.cpp  |   6 +-
 src/libcamera/pipeline/vimc/vimc.cpp          |   8 +-
 src/libcamera/v4l2_pixelformat.cpp            |  11 +-
 src/libcamera/v4l2_videodevice.cpp            | 201 +++++++---
 src/qcam/format_converter.cpp                 |  18 +-
 src/qcam/format_converter.h                   |   9 +-
 src/qcam/main_window.cpp                      |  38 +-
 src/qcam/main_window.h                        |   4 +-
 src/qcam/meson.build                          |   1 +
 src/qcam/viewfinder.h                         |   6 +-
 src/qcam/viewfinder_gl.cpp                    |  45 +--
 src/qcam/viewfinder_gl.h                      |   4 +-
 src/qcam/viewfinder_qt.cpp                    |  20 +-
 src/qcam/viewfinder_qt.h                      |   2 +-
 src/v4l2/v4l2_camera_proxy.cpp                |  16 +-
 test/libtest/buffer_source.cpp                |   3 +-
 test/utils.cpp                                |  10 +-
 49 files changed, 939 insertions(+), 372 deletions(-)
 create mode 100644 src/cam/image.cpp
 create mode 100644 src/cam/image.h