[libcamera-devel,RFC,v1,00/12] libcamera: Handle fallout of FrameBuffer offset support

Message ID	20210902042303.2254-1-laurent.pinchart@ideasonboard.com
Headers	show Return-Path: <libcamera-devel-bounces@lists.libcamera.org> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com> To: libcamera-devel@lists.libcamera.org Date: Thu, 2 Sep 2021 07:22:51 +0300 Message-Id: <20210902042303.2254-1-laurent.pinchart@ideasonboard.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [libcamera-devel] [RFC PATCH v1 00/12] libcamera: Handle fallout of FrameBuffer offset support Precedence: list Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" <libcamera-devel-bounces@lists.libcamera.org>
Series	libcamera: Handle fallout of FrameBuffer offset support
Related	show [libcamera-devel,RFC,v1,00/12] libcamera: Handle fallout of FrameBuffer offset support [libcamera-devel,RFC,v1,01/12] libcamera: base: utils: Use size_t for index in utils::enumerate() [libcamera-devel,RFC,v1,02/12] libcamera: file_descriptor: Add a function to retrieve the inode [libcamera-devel,RFC,v1,05/12] libcamera: framebuffer: Move planes check to constructor [libcamera-devel,RFC,v1,12/12] libcamera: v4l2_videodevice: Use utils::enumerate()

Laurent Pinchart Sept. 2, 2021, 4:22 a.m. UTC

Hello everybody,

This patch series started as an investigation of a qcam failure with
IPU3 after the merge of the FrameBuffer offset support. While a hack in
qcam would be possible, I decided to instead address the core issue and
fix it in V4L2VideoDevice.

The series only covers the changes needed in the libcamera core. qcam is
still broken, and this actually introduces a breakage with cam as well.
I wouldn't be surprised if the adaptation layers (Android, GStreamer and
V4L2) were also broken somehow by this series when the camera produces
multi-planar formats. I will address that as a next step, but I'd like
feedback on the changes in the core first as they will have an impact on
the applications and adaptation layers.

The most important change is in patches 10/12 and 11/12, which translate
between V4L2 buffers and libcamera FrameBuffer to handle the case where
a multi-planar frame buffer is used with the V4L2 single-planar API.
It's working more or less by chance at the moment (except in qcam where
it's broken, and possibly in other places I haven't tested). Patches
01/12 to 09/12 are cleanups and additions to prepare for the work in
V4L2VideoDevice, and patch 12/12 is a small cleanup on top.

This series doesn't break any unit test, as vimc doesn't support NV12.
Addition of NV12 support to the vimc kernel driver would be very nice,
in order to test multi-planar support in our unit tests. Volunteers are
welcome ;-)

Laurent Pinchart (12):
  libcamera: base: utils: Use size_t for index in utils::enumerate()
  libcamera: file_descriptor: Add a function to retrieve the inode
  libcamera: formats: Add planeSize() helpers to PixelFormatInfo
  libcamera: formats: Support V4L2 non-contiguous formats
  libcamera: framebuffer: Move planes check to constructor
  libcamera: framebuffer: Add a function to check if planes are
    contiguous
  libcamera: framebuffer: Allocate metadata planes at construction time
  libcamera: framebuffer: Prevent modifying the number of metadata
    planes
  libcamera: v4l2_videodevice: Cache PixelFormatInfo
  libcamera: v4l2_videodevice: Coalesce planes when queuing buffer
  libcamera: v4l2_videodevice: Split planes when dequeuing buffer
  libcamera: v4l2_videodevice: Use utils::enumerate()

 include/libcamera/base/utils.h                |   4 +-
 include/libcamera/file_descriptor.h           |   3 +
 include/libcamera/framebuffer.h               |  19 +-
 include/libcamera/internal/formats.h          |   6 +-
 include/libcamera/internal/framebuffer.h      |   2 +
 include/libcamera/internal/v4l2_videodevice.h |   1 +
 src/android/mm/generic_camera_buffer.cpp      |  11 +-
 src/android/yuv/post_processor_yuv.cpp        |  10 +-
 src/cam/camera_session.cpp                    |   4 +-
 src/cam/file_sink.cpp                         |   2 +-
 src/libcamera/file_descriptor.cpp             |  22 ++
 src/libcamera/formats.cpp                     | 223 ++++++++++++------
 src/libcamera/framebuffer.cpp                 |  60 ++++-
 src/libcamera/pipeline/ipu3/cio2.cpp          |   2 +-
 src/libcamera/v4l2_pixelformat.cpp            |  11 +-
 src/libcamera/v4l2_videodevice.cpp            | 148 +++++++++---
 src/qcam/main_window.cpp                      |   2 +-
 src/qcam/viewfinder_gl.cpp                    |   2 +-
 src/qcam/viewfinder_qt.cpp                    |   2 +-
 src/v4l2/v4l2_camera_proxy.cpp                |   8 +-
 test/utils.cpp                                |  10 +-
 21 files changed, 402 insertions(+), 150 deletions(-)

Kieran Bingham Sept. 2, 2021, 8:49 a.m. UTC | #1

On 02/09/2021 05:22, Laurent Pinchart wrote:
> Add two helpers functions to the PixelFormatInfo class to compute the
> byte size of a given plane, taking the frame size, the stride, the
> alignement constraints and the vertical subsampling into account.

s/alignement/alignment/


> Use the new functions through the code base to replace manual
> implementations.


\o/ that makes me happy.

Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>

> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> ---
>  include/libcamera/internal/formats.h     |  4 ++
>  src/android/mm/generic_camera_buffer.cpp | 11 +---
>  src/android/yuv/post_processor_yuv.cpp   | 10 ++-
>  src/libcamera/formats.cpp                | 82 ++++++++++++++++++++----
>  src/libcamera/v4l2_videodevice.cpp       |  6 +-
>  5 files changed, 79 insertions(+), 34 deletions(-)
> 
> diff --git a/include/libcamera/internal/formats.h b/include/libcamera/internal/formats.h
> index 51a8a6b8b0ae..8f314e99889b 100644
> --- a/include/libcamera/internal/formats.h
> +++ b/include/libcamera/internal/formats.h
> @@ -42,6 +42,10 @@ public:
>  
>  	unsigned int stride(unsigned int width, unsigned int plane,
>  			    unsigned int align = 1) const;
> +	unsigned int planeSize(const Size &size, unsigned int plane,
> +			       unsigned int align = 1) const;
> +	unsigned int planeSize(unsigned int height, unsigned int plane,
> +			       unsigned int stride) const;
>  	unsigned int frameSize(const Size &size, unsigned int align = 1) const;
>  	unsigned int frameSize(const Size &size,
>  			       const std::array<unsigned int, 3> &strides) const;
> diff --git a/src/android/mm/generic_camera_buffer.cpp b/src/android/mm/generic_camera_buffer.cpp
> index 22efc4d4b13a..93aa5821e470 100644
> --- a/src/android/mm/generic_camera_buffer.cpp
> +++ b/src/android/mm/generic_camera_buffer.cpp
> @@ -108,16 +108,9 @@ CameraBuffer::Private::Private([[maybe_unused]] CameraBuffer *cameraBuffer,
>  
>  	unsigned int offset = 0;
>  	for (unsigned int i = 0; i < numPlanes; ++i) {
> -		/*
> -		 * \todo Remove if this plane size computation function is
> -		 * added to PixelFormatInfo.
> -		 */
> -		const unsigned int vertSubSample = info.planes[i].verticalSubSampling;
> -		const unsigned int stride = info.stride(size.width, i, 1u);
> -		const unsigned int planeSize =
> -			stride * ((size.height + vertSubSample - 1) / vertSubSample);
> +		const unsigned int planeSize = info.planeSize(size, i);
>  
> -		planeInfo_[i].stride = stride;
> +		planeInfo_[i].stride = info.stride(size.width, i, 1u);
>  		planeInfo_[i].offset = offset;
>  		planeInfo_[i].size = planeSize;
>  
> diff --git a/src/android/yuv/post_processor_yuv.cpp b/src/android/yuv/post_processor_yuv.cpp
> index 6952fc38b0ef..7b3b49609cb1 100644
> --- a/src/android/yuv/post_processor_yuv.cpp
> +++ b/src/android/yuv/post_processor_yuv.cpp
> @@ -134,11 +134,9 @@ void PostProcessorYuv::calculateLengths(const StreamConfiguration &inCfg,
>  		sourceStride_[i] = inCfg.stride;
>  		destinationStride_[i] = nv12Info.stride(destinationSize_.width, i, 1);
>  
> -		const unsigned int vertSubSample =
> -			nv12Info.planes[i].verticalSubSampling;
> -		sourceLength_[i] = sourceStride_[i] *
> -			((sourceSize_.height + vertSubSample - 1) / vertSubSample);
> -		destinationLength_[i] = destinationStride_[i] *
> -			((destinationSize_.height + vertSubSample - 1) / vertSubSample);
> +		sourceLength_[i] = nv12Info.planeSize(sourceSize_.height, i,
> +						      sourceStride_[i]);
> +		destinationLength_[i] = nv12Info.planeSize(destinationSize_.height, i,
> +							   destinationStride_[i]);
>  	}
>  }
> diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
> index 603d88619fe0..76be93bc1c5c 100644
> --- a/src/libcamera/formats.cpp
> +++ b/src/libcamera/formats.cpp
> @@ -11,6 +11,7 @@
>  #include <errno.h>
>  
>  #include <libcamera/base/log.h>
> +#include <libcamera/base/utils.h>
>  
>  #include <libcamera/formats.h>
>  
> @@ -801,32 +802,85 @@ unsigned int PixelFormatInfo::stride(unsigned int width, unsigned int plane,
>  }
>  
>  /**
> - * \brief Compute the number of bytes necessary to store a frame
> + * \brief Compute the number of bytes necessary to store a plane of a frame
>   * \param[in] size The size of the frame, in pixels
> + * \param[in] plane The plane index
>   * \param[in] align The stride alignment, in bytes (1 for default alignment)
>   *
> - * The frame is computed by adding the product of the line stride and the frame
> - * height for all planes, taking subsampling and other format characteristics
> - * into account. Additional stride alignment constraints may be specified
> - * through the \a align parameter, and will apply to all planes. For more
> - * complex stride constraints, use the frameSize() overloaded version that takes
> - * an array of stride values.
> + * The plane size is computed by multiplying the line stride and the frame
> + * height, taking subsampling and other format characteristics into account.
> + * Stride alignment constraints may be specified through the \a align parameter.
>   *
>   * \sa stride()
>   *
> + * \return The number of bytes necessary to store the plane, or 0 if the
> + * PixelFormatInfo instance is not valid or the plane number isn't valid for the
> + * format
> + */
> +unsigned int PixelFormatInfo::planeSize(const Size &size, unsigned int plane,
> +					unsigned int align) const
> +{
> +	unsigned int stride = PixelFormatInfo::stride(size.width, plane, align);
> +	if (!stride)
> +		return 0;
> +
> +	unsigned int vertSubSample = planes[plane].verticalSubSampling;
> +	if (!vertSubSample)
> +		return 0;
> +
> +	/* stride * ceil(height / verticalSubSampling) */
> +	return stride * ((size.height + vertSubSample - 1) / vertSubSample);
> +}
> +
> +/**
> + * \brief Compute the number of bytes necessary to store a plane of a frame
> + * \param[in] height The height of the frame, in pixels
> + * \param[in] plane The plane index
> + * \param[in] stride The plane stride, in bytes
> + *
> + * The plane size is computed by multiplying the line stride and the frame
> + * height, taking subsampling and other format characteristics into account.
> + * Stride alignment constraints may be specified through the \a align parameter.
> + *
> + * \return The number of bytes necessary to store the plane, or 0 if the
> + * PixelFormatInfo instance is not valid or the plane number isn't valid for the
> + * format
> + */
> +unsigned int PixelFormatInfo::planeSize(unsigned int height, unsigned int plane,
> +					unsigned int stride) const
> +{
> +	unsigned int vertSubSample = planes[plane].verticalSubSampling;
> +	if (!vertSubSample)
> +		return 0;
> +
> +	/* stride * ceil(height / verticalSubSampling) */
> +	return stride * ((height + vertSubSample - 1) / vertSubSample);
> +}
> +
> +/**
> + * \brief Compute the number of bytes necessary to store a frame
> + * \param[in] size The size of the frame, in pixels
> + * \param[in] align The stride alignment, in bytes (1 for default alignment)
> + *
> + * The frame size is computed by adding the size of all planes, as computed by
> + * planeSize(), using the specified alignment constraints for all planes. For
> + * more complex stride constraints, use the frameSize() overloaded version that
> + * takes an array of stride values.
> + *
> + * \sa planeSize()
> + *
>   * \return The number of bytes necessary to store the frame, or 0 if the
>   * PixelFormatInfo instance is not valid
>   */
>  unsigned int PixelFormatInfo::frameSize(const Size &size, unsigned int align) const
>  {
> -	/* stride * ceil(height / verticalSubSampling) */
>  	unsigned int sum = 0;
> -	for (unsigned int i = 0; i < 3; i++) {
> -		unsigned int vertSubSample = planes[i].verticalSubSampling;
> -		if (!vertSubSample)
> -			continue;
> -		sum += stride(size.width, i, align)
> -		     * ((size.height + vertSubSample - 1) / vertSubSample);
> +
> +	for (const auto &[i, plane] : utils::enumerate(planes)) {
> +		if (plane.bytesPerGroup == 0)
> +			break;
> +
> +		sum += planeSize(size, i, align);
>  	}
>  
>  	return sum;
> diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> index 4e1c2b7cef5e..adabd4720668 100644
> --- a/src/libcamera/v4l2_videodevice.cpp
> +++ b/src/libcamera/v4l2_videodevice.cpp
> @@ -1337,11 +1337,7 @@ std::unique_ptr<FrameBuffer> V4L2VideoDevice::createBuffer(unsigned int index)
>  			planes[i].offset = offset;
>  
>  			/* \todo Take the V4L2 stride into account */
> -			const unsigned int vertSubSample =
> -				info.planes[i].verticalSubSampling;
> -			planes[i].length =
> -				info.stride(format_.size.width, i, 1u) *
> -				((format_.size.height + vertSubSample - 1) / vertSubSample);
> +			planes[i].length = info.planeSize(format_.size, i);
>  			offset += planes[i].length;
>  		}
>  	}
>

Kieran Bingham Sept. 2, 2021, 9:23 a.m. UTC | #2

On 02/09/2021 05:22, Laurent Pinchart wrote:
> V4L2 describes multi-planar formats with different 4CCs depending on
> whether or not the planes are stored contiguously in memory. Support
> this when translating between PixelFormat and V4L2PixelFormat.
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> ---
>  include/libcamera/internal/formats.h |   2 +-
>  src/libcamera/formats.cpp            | 141 ++++++++++++++++-----------
>  src/libcamera/pipeline/ipu3/cio2.cpp |   2 +-
>  src/libcamera/v4l2_pixelformat.cpp   |  11 ++-
>  src/v4l2/v4l2_camera_proxy.cpp       |   6 +-
>  5 files changed, 99 insertions(+), 63 deletions(-)
> 
> diff --git a/include/libcamera/internal/formats.h b/include/libcamera/internal/formats.h
> index 8f314e99889b..f9ce5f0834fa 100644
> --- a/include/libcamera/internal/formats.h
> +++ b/include/libcamera/internal/formats.h
> @@ -55,7 +55,7 @@ public:
>  	/* \todo Add support for non-contiguous memory planes */
>  	const char *name;
>  	PixelFormat format;
> -	V4L2PixelFormat v4l2Format;
> +	std::array<V4L2PixelFormat, 2> v4l2Format;

As discussed below, I'm curious if a struct would be clearer here and
help prevent bugs later...

	struct {
		V4L2PixelFormat planar;
		V4L2PixelFormat multiplanar;
	} v4l2Format;


>  	unsigned int bitsPerPixel;
>  	enum ColourEncoding colourEncoding;
>  	bool packed;
> diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
> index 76be93bc1c5c..ce7c5a2d267e 100644
> --- a/src/libcamera/formats.cpp
> +++ b/src/libcamera/formats.cpp
> @@ -56,7 +56,14 @@ LOG_DEFINE_CATEGORY(Formats)
>   * \brief The PixelFormat described by this instance
>   *
>   * \var PixelFormatInfo::v4l2Format
> - * \brief The V4L2 pixel format corresponding to the PixelFormat
> + * \brief The V4L2 pixel formats corresponding to the PixelFormat
> + *
> + * Multiple V4L2 formats may exist for one PixelFormat when the format uses
> + * multiple planes, as V4L2 defines separate 4CCs for contiguous and separate
> + * planes formats. The two entries in the array store the contiguous and
> + * non-contiguous V4L2 formats respectively. If the PixelFormat isn't a
> + * multiplanar format, or if no corresponding non-contiguous V4L2 format
> + * exists, the second entry is invalid.
>   *
>   * \var PixelFormatInfo::bitsPerPixel
>   * \brief The average number of bits per pixel
> @@ -149,7 +156,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::RGB565, {
>  		.name = "RGB565",
>  		.format = formats::RGB565,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGB565),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGB565), },
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> @@ -159,7 +166,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::RGB565_BE, {
>  		.name = "RGB565_BE",
>  		.format = formats::RGB565_BE,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGB565X),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGB565X), },
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> @@ -169,7 +176,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::BGR888, {
>  		.name = "BGR888",
>  		.format = formats::BGR888,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGB24),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGB24), },
>  		.bitsPerPixel = 24,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> @@ -179,7 +186,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::RGB888, {
>  		.name = "RGB888",
>  		.format = formats::RGB888,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_BGR24),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_BGR24), },
>  		.bitsPerPixel = 24,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> @@ -189,7 +196,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::XRGB8888, {
>  		.name = "XRGB8888",
>  		.format = formats::XRGB8888,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_XBGR32),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_XBGR32), },
>  		.bitsPerPixel = 32,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> @@ -199,7 +206,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::XBGR8888, {
>  		.name = "XBGR8888",
>  		.format = formats::XBGR8888,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGBX32),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGBX32), },
>  		.bitsPerPixel = 32,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> @@ -209,7 +216,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::BGRX8888, {
>  		.name = "BGRX8888",
>  		.format = formats::BGRX8888,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_XRGB32),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_XRGB32), },
>  		.bitsPerPixel = 32,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> @@ -219,7 +226,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::ABGR8888, {
>  		.name = "ABGR8888",
>  		.format = formats::ABGR8888,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGBA32),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGBA32), },
>  		.bitsPerPixel = 32,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> @@ -229,7 +236,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::ARGB8888, {
>  		.name = "ARGB8888",
>  		.format = formats::ARGB8888,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_ABGR32),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_ABGR32), },
>  		.bitsPerPixel = 32,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> @@ -239,7 +246,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::BGRA8888, {
>  		.name = "BGRA8888",
>  		.format = formats::BGRA8888,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_ARGB32),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_ARGB32), },
>  		.bitsPerPixel = 32,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> @@ -249,7 +256,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::RGBA8888, {
>  		.name = "RGBA8888",
>  		.format = formats::RGBA8888,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_BGRA32),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_BGRA32), },
>  		.bitsPerPixel = 32,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> @@ -261,7 +268,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::YUYV, {
>  		.name = "YUYV",
>  		.format = formats::YUYV,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YUYV),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_YUYV), },
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> @@ -271,7 +278,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::YVYU, {
>  		.name = "YVYU",
>  		.format = formats::YVYU,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YVYU),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_YVYU), },
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> @@ -281,7 +288,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::UYVY, {
>  		.name = "UYVY",
>  		.format = formats::UYVY,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_UYVY),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_UYVY), },
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> @@ -291,7 +298,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::VYUY, {
>  		.name = "VYUY",
>  		.format = formats::VYUY,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_VYUY),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_VYUY), },
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> @@ -303,7 +310,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::NV12, {
>  		.name = "NV12",
>  		.format = formats::NV12,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV12),
> +		.v4l2Format = {
> +			V4L2PixelFormat(V4L2_PIX_FMT_NV12),
> +			V4L2PixelFormat(V4L2_PIX_FMT_NV12M),
> +		},
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> @@ -313,7 +323,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::NV21, {
>  		.name = "NV21",
>  		.format = formats::NV21,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV21),
> +		.v4l2Format = {
> +			V4L2PixelFormat(V4L2_PIX_FMT_NV21),
> +			V4L2PixelFormat(V4L2_PIX_FMT_NV21M),
> +		},
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> @@ -323,7 +336,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::NV16, {
>  		.name = "NV16",
>  		.format = formats::NV16,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV16),
> +		.v4l2Format = {
> +			V4L2PixelFormat(V4L2_PIX_FMT_NV16),
> +			V4L2PixelFormat(V4L2_PIX_FMT_NV16M),
> +		},
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> @@ -333,7 +349,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::NV61, {
>  		.name = "NV61",
>  		.format = formats::NV61,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV61),
> +		.v4l2Format = {
> +			V4L2PixelFormat(V4L2_PIX_FMT_NV61),
> +			V4L2PixelFormat(V4L2_PIX_FMT_NV61M),
> +		},
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> @@ -343,7 +362,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::NV24, {
>  		.name = "NV24",
>  		.format = formats::NV24,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV24),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_NV24), },
>  		.bitsPerPixel = 24,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> @@ -353,7 +372,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::NV42, {
>  		.name = "NV42",
>  		.format = formats::NV42,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV42),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_NV42), },
>  		.bitsPerPixel = 24,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> @@ -363,7 +382,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::YUV420, {
>  		.name = "YUV420",
>  		.format = formats::YUV420,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YUV420),
> +		.v4l2Format = {
> +			V4L2PixelFormat(V4L2_PIX_FMT_YUV420),
> +			V4L2PixelFormat(V4L2_PIX_FMT_YUV420M),
> +		},
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> @@ -373,7 +395,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::YVU420, {
>  		.name = "YVU420",
>  		.format = formats::YVU420,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YVU420),
> +		.v4l2Format = {
> +			V4L2PixelFormat(V4L2_PIX_FMT_YVU420),
> +			V4L2PixelFormat(V4L2_PIX_FMT_YVU420M),
> +		},
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> @@ -383,7 +408,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::YUV422, {
>  		.name = "YUV422",
>  		.format = formats::YUV422,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YUV422P),
> +		.v4l2Format = {
> +			V4L2PixelFormat(V4L2_PIX_FMT_YUV422P),
> +			V4L2PixelFormat(V4L2_PIX_FMT_YUV422M),
> +		},
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> @@ -395,7 +423,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::R8, {
>  		.name = "R8",
>  		.format = formats::R8,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_GREY),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_GREY), },
>  		.bitsPerPixel = 8,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> @@ -407,7 +435,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SBGGR8, {
>  		.name = "SBGGR8",
>  		.format = formats::SBGGR8,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR8),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR8), },
>  		.bitsPerPixel = 8,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> @@ -417,7 +445,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SGBRG8, {
>  		.name = "SGBRG8",
>  		.format = formats::SGBRG8,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG8),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG8), },
>  		.bitsPerPixel = 8,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> @@ -427,7 +455,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SGRBG8, {
>  		.name = "SGRBG8",
>  		.format = formats::SGRBG8,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG8),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG8), },
>  		.bitsPerPixel = 8,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> @@ -437,7 +465,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SRGGB8, {
>  		.name = "SRGGB8",
>  		.format = formats::SRGGB8,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB8),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB8), },
>  		.bitsPerPixel = 8,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> @@ -447,7 +475,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SBGGR10, {
>  		.name = "SBGGR10",
>  		.format = formats::SBGGR10,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10), },
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> @@ -457,7 +485,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SGBRG10, {
>  		.name = "SGBRG10",
>  		.format = formats::SGBRG10,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10), },
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> @@ -467,7 +495,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SGRBG10, {
>  		.name = "SGRBG10",
>  		.format = formats::SGRBG10,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10), },
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> @@ -477,7 +505,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SRGGB10, {
>  		.name = "SRGGB10",
>  		.format = formats::SRGGB10,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10), },
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> @@ -487,7 +515,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SBGGR10_CSI2P, {
>  		.name = "SBGGR10_CSI2P",
>  		.format = formats::SBGGR10_CSI2P,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10P),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10P), },
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> @@ -497,7 +525,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SGBRG10_CSI2P, {
>  		.name = "SGBRG10_CSI2P",
>  		.format = formats::SGBRG10_CSI2P,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10P),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10P), },
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> @@ -507,7 +535,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SGRBG10_CSI2P, {
>  		.name = "SGRBG10_CSI2P",
>  		.format = formats::SGRBG10_CSI2P,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10P),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10P), },
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> @@ -517,7 +545,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SRGGB10_CSI2P, {
>  		.name = "SRGGB10_CSI2P",
>  		.format = formats::SRGGB10_CSI2P,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10P),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10P), },
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> @@ -527,7 +555,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SBGGR12, {
>  		.name = "SBGGR12",
>  		.format = formats::SBGGR12,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12), },
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> @@ -537,7 +565,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SGBRG12, {
>  		.name = "SGBRG12",
>  		.format = formats::SGBRG12,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12), },
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> @@ -547,7 +575,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SGRBG12, {
>  		.name = "SGRBG12",
>  		.format = formats::SGRBG12,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12), },
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> @@ -557,7 +585,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SRGGB12, {
>  		.name = "SRGGB12",
>  		.format = formats::SRGGB12,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12), },
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> @@ -567,7 +595,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SBGGR12_CSI2P, {
>  		.name = "SBGGR12_CSI2P",
>  		.format = formats::SBGGR12_CSI2P,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12P),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12P), },
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> @@ -577,7 +605,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SGBRG12_CSI2P, {
>  		.name = "SGBRG12_CSI2P",
>  		.format = formats::SGBRG12_CSI2P,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12P),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12P), },
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> @@ -587,7 +615,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SGRBG12_CSI2P, {
>  		.name = "SGRBG12_CSI2P",
>  		.format = formats::SGRBG12_CSI2P,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12P),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12P), },
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> @@ -597,7 +625,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SRGGB12_CSI2P, {
>  		.name = "SRGGB12_CSI2P",
>  		.format = formats::SRGGB12_CSI2P,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12P),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12P), },
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> @@ -607,7 +635,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SBGGR16, {
>  		.name = "SBGGR16",
>  		.format = formats::SBGGR16,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR16),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR16), },
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> @@ -617,7 +645,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SGBRG16, {
>  		.name = "SGBRG16",
>  		.format = formats::SGBRG16,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG16),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG16), },
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> @@ -627,7 +655,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SGRBG16, {
>  		.name = "SGRBG16",
>  		.format = formats::SGRBG16,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG16),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG16), },
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> @@ -637,7 +665,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SRGGB16, {
>  		.name = "SRGGB16",
>  		.format = formats::SRGGB16,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB16),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB16), },
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> @@ -647,7 +675,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SBGGR10_IPU3, {
>  		.name = "SBGGR10_IPU3",
>  		.format = formats::SBGGR10_IPU3,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SBGGR10),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SBGGR10), },
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> @@ -658,7 +686,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SGBRG10_IPU3, {
>  		.name = "SGBRG10_IPU3",
>  		.format = formats::SGBRG10_IPU3,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGBRG10),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGBRG10), },
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> @@ -668,7 +696,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SGRBG10_IPU3, {
>  		.name = "SGRBG10_IPU3",
>  		.format = formats::SGRBG10_IPU3,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGRBG10),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGRBG10), },
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> @@ -678,7 +706,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::SRGGB10_IPU3, {
>  		.name = "SRGGB10_IPU3",
>  		.format = formats::SRGGB10_IPU3,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SRGGB10),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SRGGB10), },
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> @@ -690,7 +718,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	{ formats::MJPEG, {
>  		.name = "MJPEG",
>  		.format = formats::MJPEG,
> -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_MJPEG),
> +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_MJPEG), },
>  		.bitsPerPixel = 0,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> @@ -736,7 +764,8 @@ const PixelFormatInfo &PixelFormatInfo::info(const V4L2PixelFormat &format)
>  {
>  	const auto &info = std::find_if(pixelFormatInfo.begin(), pixelFormatInfo.end(),
>  					[format](auto pair) {
> -						return pair.second.v4l2Format == format;
> +						return pair.second.v4l2Format[0] == format ||
> +						       pair.second.v4l2Format[1] == format;
>  					});
>  	if (info == pixelFormatInfo.end())
>  		return pixelFormatInfoInvalid;
> diff --git a/src/libcamera/pipeline/ipu3/cio2.cpp b/src/libcamera/pipeline/ipu3/cio2.cpp
> index 9cedcb5b2879..5a9cffc80c8d 100644
> --- a/src/libcamera/pipeline/ipu3/cio2.cpp
> +++ b/src/libcamera/pipeline/ipu3/cio2.cpp
> @@ -205,7 +205,7 @@ int CIO2Device::configure(const Size &size, V4L2DeviceFormat *outputFormat)
>  
>  	const PixelFormatInfo &info = PixelFormatInfo::info(itInfo->second);
>  
> -	outputFormat->fourcc = info.v4l2Format;
> +	outputFormat->fourcc = info.v4l2Format[0];
>  	outputFormat->size = sensorFormat.size;
>  	outputFormat->planesCount = 1;
>  
> diff --git a/src/libcamera/v4l2_pixelformat.cpp b/src/libcamera/v4l2_pixelformat.cpp
> index 93fc4446cc64..93ead8928ed7 100644
> --- a/src/libcamera/v4l2_pixelformat.cpp
> +++ b/src/libcamera/v4l2_pixelformat.cpp
> @@ -67,12 +67,19 @@ const std::map<V4L2PixelFormat, PixelFormat> vpf2pf{
>  
>  	/* YUV planar formats. */
>  	{ V4L2PixelFormat(V4L2_PIX_FMT_NV16), formats::NV16 },
> +	{ V4L2PixelFormat(V4L2_PIX_FMT_NV16M), formats::NV16 },
>  	{ V4L2PixelFormat(V4L2_PIX_FMT_NV61), formats::NV61 },
> +	{ V4L2PixelFormat(V4L2_PIX_FMT_NV61M), formats::NV61 },
>  	{ V4L2PixelFormat(V4L2_PIX_FMT_NV12), formats::NV12 },
> +	{ V4L2PixelFormat(V4L2_PIX_FMT_NV12M), formats::NV12 },
>  	{ V4L2PixelFormat(V4L2_PIX_FMT_NV21), formats::NV21 },
> +	{ V4L2PixelFormat(V4L2_PIX_FMT_NV21M), formats::NV21 },
>  	{ V4L2PixelFormat(V4L2_PIX_FMT_YUV420), formats::YUV420 },
> +	{ V4L2PixelFormat(V4L2_PIX_FMT_YUV420M), formats::YUV420 },
>  	{ V4L2PixelFormat(V4L2_PIX_FMT_YVU420), formats::YVU420 },
> +	{ V4L2PixelFormat(V4L2_PIX_FMT_YVU420M), formats::YVU420 },
>  	{ V4L2PixelFormat(V4L2_PIX_FMT_YUV422P), formats::YUV422 },
> +	{ V4L2PixelFormat(V4L2_PIX_FMT_YUV422M), formats::YUV422 },

I'm worried that we're losing information here.
But presumably it's handled by whether we use mplane or not on the
device...?


>  
>  	/* Greyscale formats. */
>  	{ V4L2PixelFormat(V4L2_PIX_FMT_GREY), formats::R8 },
> @@ -202,13 +209,13 @@ PixelFormat V4L2PixelFormat::toPixelFormat() const
>   * \return The V4L2PixelFormat corresponding to \a pixelFormat
>   */
>  V4L2PixelFormat V4L2PixelFormat::fromPixelFormat(const PixelFormat &pixelFormat,
> -						 [[maybe_unused]] bool multiplanar)
> +						 bool multiplanar)
>  {
>  	const PixelFormatInfo &info = PixelFormatInfo::info(pixelFormat);
>  	if (!info.isValid())
>  		return V4L2PixelFormat();
>  
> -	return info.v4l2Format;
> +	return info.v4l2Format[multiplanar ? 1 : 0];

I was going to say, a struct with named fields of 'planar', and
'multiplanar' might be a better, though I like the simplicity of this bit.



>  }
>  
>  } /* namespace libcamera */
> diff --git a/src/v4l2/v4l2_camera_proxy.cpp b/src/v4l2/v4l2_camera_proxy.cpp
> index 7682c4bddf90..8d8ee395954f 100644
> --- a/src/v4l2/v4l2_camera_proxy.cpp
> +++ b/src/v4l2/v4l2_camera_proxy.cpp
> @@ -169,7 +169,7 @@ void V4L2CameraProxy::setFmtFromConfig(const StreamConfiguration &streamConfig)
>  
>  	v4l2PixFormat_.width        = size.width;
>  	v4l2PixFormat_.height       = size.height;
> -	v4l2PixFormat_.pixelformat  = info.v4l2Format;
> +	v4l2PixFormat_.pixelformat  = info.v4l2Format[0];
>  	v4l2PixFormat_.field        = V4L2_FIELD_NONE;
>  	v4l2PixFormat_.bytesperline = streamConfig.stride;
>  	v4l2PixFormat_.sizeimage    = streamConfig.frameSize;
> @@ -276,7 +276,7 @@ int V4L2CameraProxy::vidioc_enum_fmt(V4L2CameraFile *file, struct v4l2_fmtdesc *
>  	/* \todo Add map from format to description. */
>  	utils::strlcpy(reinterpret_cast<char *>(arg->description),
>  		       "Video Format Description", sizeof(arg->description));
> -	arg->pixelformat = PixelFormatInfo::info(format).v4l2Format;
> +	arg->pixelformat = PixelFormatInfo::info(format).v4l2Format[0];
>  
>  	memset(arg->reserved, 0, sizeof(arg->reserved));
>  
> @@ -315,7 +315,7 @@ int V4L2CameraProxy::tryFormat(struct v4l2_format *arg)
>  
>  	arg->fmt.pix.width        = config.size.width;
>  	arg->fmt.pix.height       = config.size.height;
> -	arg->fmt.pix.pixelformat  = info.v4l2Format;
> +	arg->fmt.pix.pixelformat  = info.v4l2Format[0];

But for occasions like here, and in the CIO2
	arg->fmt.pix.pixelformat  = info.v4l2Format.planar;

Would be far more descriptive over which one is being chosen, and might
help make it easier to spot issues when debugging multiplanar format
bugs ...

>  	arg->fmt.pix.field        = V4L2_FIELD_NONE;
>  	arg->fmt.pix.bytesperline = config.stride;
>  	arg->fmt.pix.sizeimage    = config.frameSize;
>

Kieran Bingham Sept. 2, 2021, 9:43 a.m. UTC | #3

On 02/09/2021 05:22, Laurent Pinchart wrote:
> Multi-planar frame buffers can store their planes contiguously in
> memory, or split them in discontiguous memory areas. Add a private
> function to check in which of these two categories the frame buffer
> belongs. This will be used to correctly handle the differences between
> the V4L2 single and multi planar APIs.
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> ---
>  include/libcamera/internal/framebuffer.h |  2 ++
>  src/libcamera/framebuffer.cpp            | 45 +++++++++++++++++++++++-
>  2 files changed, 46 insertions(+), 1 deletion(-)
> 
> diff --git a/include/libcamera/internal/framebuffer.h b/include/libcamera/internal/framebuffer.h
> index 606aed2b4782..cd33c295466e 100644
> --- a/include/libcamera/internal/framebuffer.h
> +++ b/include/libcamera/internal/framebuffer.h
> @@ -21,9 +21,11 @@ public:
>  	Private();
>  
>  	void setRequest(Request *request) { request_ = request; }
> +	bool isContiguous() const { return isContiguous_; }
>  
>  private:
>  	Request *request_;
> +	bool isContiguous_;
>  };
>  
>  } /* namespace libcamera */
> diff --git a/src/libcamera/framebuffer.cpp b/src/libcamera/framebuffer.cpp
> index 53ef89bf458f..99265b44da43 100644
> --- a/src/libcamera/framebuffer.cpp
> +++ b/src/libcamera/framebuffer.cpp
> @@ -106,7 +106,7 @@ LOG_DEFINE_CATEGORY(Buffer)
>   */
>  
>  FrameBuffer::Private::Private()
> -	: request_(nullptr)
> +	: request_(nullptr), isContiguous_(true)
>  {
>  }
>  
> @@ -120,6 +120,17 @@ FrameBuffer::Private::Private()
>   * handlers, it is called by the pipeline handlers themselves.
>   */
>  
> +/**
> + * \fn FrameBuffer::Private::isContiguous()
> + * \brief Check if the frame buffer stores planes contiguously in memory
> + *
> + * Multi-planar frame buffers can store their planes contiguously in memory, or
> + * split them in discontiguous memory areas. This function checks in which of

'split them into discontiguous'

> + * these two categories the frame buffer belongs.
> + *
> + * \return True if the planes are stored contiguously in memory, false otherwise
> + */
> +
>  /**
>   * \class FrameBuffer
>   * \brief Frame buffer data and its associated dynamic metadata
> @@ -199,9 +210,41 @@ FrameBuffer::FrameBuffer(const std::vector<Plane> &planes, unsigned int cookie)
>  	: Extensible(std::make_unique<Private>()), planes_(planes),
>  	  cookie_(cookie)
>  {
> +	unsigned int offset = 0;
> +
>  	/* \todo Remove the assertions after sufficient testing */
>  	for (const auto &plane : planes_)
>  		ASSERT(plane.offset != Plane::kInvalidOffset);
> +
> +	bool isContiguous = true;
> +	ino_t inode = 0;
> +
> +	for (const auto &plane : planes_) {
> +		if (plane.offset != offset) {
> +			isContiguous = false;
> +			break;
> +		}
> +
> +		/*
> +		 * Two different dmabuf file descriptors may still refer to the
> +		 * same dmabuf instance. Check this using inodes.

Going back to the FileDescriptor::inode() extension, if that cached the
inode (it can't change) on either first call, or construction, couldn't
this whole check be simplified to:

		if (plane.fd.inode() != planes_[0].fd.inode()) {
			isContiguous = false;
			break;
		}

> +		 */
> +		if (plane.fd.fd() != planes_[0].fd.fd()) {
> +			if (!inode)
> +				inode = planes_[0].fd.inode();
> +			if (plane.fd.inode() != inode) {
> +				isContiguous = false;
> +				break;
> +			}
> +		}
> +
> +		offset += plane.length;
> +	}
> +
> +	LOG(Buffer, Debug)
> +		<< "Buffer is " << (isContiguous ? "not " : "") << "contiguous";
> +
> +	_d()->isContiguous_ = isContiguous;
>  }
>  
>  /**
>

Kieran Bingham Sept. 2, 2021, 9:54 a.m. UTC | #4

On 02/09/2021 05:22, Laurent Pinchart wrote:
> The metadata planes are allocated by V4L2VideoDevice when dequeuing a
> buffer. This causes the metadata planes to only be allocated after a
> buffer gets dequeued, and doesn't provide any strong guarantee that
> their number matches the number of FrameBuffer planes. The lack of this
> invariant makes the FrameBuffer class fragile.
> 
> As a first step towards fixing this, allocate the metadata planes when
> the FrameBuffer is constructed. The FrameMetadata API should be further
> improved by preventing a change in the number of planes.
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> ---
>  src/libcamera/framebuffer.cpp      |  2 ++
>  src/libcamera/v4l2_videodevice.cpp | 12 +++++++++---
>  2 files changed, 11 insertions(+), 3 deletions(-)
> 
> diff --git a/src/libcamera/framebuffer.cpp b/src/libcamera/framebuffer.cpp
> index 99265b44da43..c1529dfb0ad1 100644
> --- a/src/libcamera/framebuffer.cpp
> +++ b/src/libcamera/framebuffer.cpp
> @@ -210,6 +210,8 @@ FrameBuffer::FrameBuffer(const std::vector<Plane> &planes, unsigned int cookie)
>  	: Extensible(std::make_unique<Private>()), planes_(planes),
>  	  cookie_(cookie)
>  {
> +	metadata_.planes.resize(planes_.size());
> +
>  	unsigned int offset = 0;
>  
>  	/* \todo Remove the assertions after sufficient testing */
> diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> index adabd4720668..a51971879e75 100644
> --- a/src/libcamera/v4l2_videodevice.cpp
> +++ b/src/libcamera/v4l2_videodevice.cpp
> @@ -1586,12 +1586,18 @@ FrameBuffer *V4L2VideoDevice::dequeueBuffer()
>  	buffer->metadata_.timestamp = buf.timestamp.tv_sec * 1000000000ULL
>  				    + buf.timestamp.tv_usec * 1000ULL;
>  
> -	buffer->metadata_.planes.clear();
>  	if (multiPlanar) {
> +		if (buf.length > buffer->metadata_.planes.size()) {
> +			LOG(V4L2, Error)

Are we going to 'leak' queued FrameBuffers at this point?
Is there a way to prevent this occurring at queue time rather than dequeue?

In fact, Can it even happen? or should it be an ASSERT/Fatal?


> +				<< "Invalid number of planes (" << buf.length
> +				<< " != " << buffer->metadata_.planes.size() << ")";
> +			return nullptr;
> +		}
> +
>  		for (unsigned int nplane = 0; nplane < buf.length; nplane++)
> -			buffer->metadata_.planes.push_back({ planes[nplane].bytesused });
> +			buffer->metadata_.planes[nplane].bytesused = planes[nplane].bytesused;
>  	} else {
> -		buffer->metadata_.planes.push_back({ buf.bytesused });
> +		buffer->metadata_.planes[0].bytesused = buf.bytesused;
>  	}
>  
>  	return buffer;
>

Kieran Bingham Sept. 2, 2021, 10:03 a.m. UTC | #5

On 02/09/2021 05:22, Laurent Pinchart wrote:
> The number of metadata planes should always match the number of frame
> buffer planes. Enforce this by making the vector private and providing
> accessor functions.
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> ---
>  include/libcamera/framebuffer.h    | 10 +++++++++-
>  src/cam/camera_session.cpp         |  4 ++--
>  src/cam/file_sink.cpp              |  2 +-
>  src/libcamera/framebuffer.cpp      | 12 +++++++++---
>  src/libcamera/v4l2_videodevice.cpp | 14 +++++++-------
>  src/qcam/main_window.cpp           |  2 +-
>  src/qcam/viewfinder_gl.cpp         |  2 +-
>  src/qcam/viewfinder_qt.cpp         |  2 +-
>  src/v4l2/v4l2_camera_proxy.cpp     |  2 +-
>  9 files changed, 32 insertions(+), 18 deletions(-)
> 
> diff --git a/include/libcamera/framebuffer.h b/include/libcamera/framebuffer.h
> index fd68ed0a139d..7f2f176af691 100644
> --- a/include/libcamera/framebuffer.h
> +++ b/include/libcamera/framebuffer.h
> @@ -13,6 +13,7 @@
>  #include <vector>
>  
>  #include <libcamera/base/class.h>
> +#include <libcamera/base/span.h>
>  
>  #include <libcamera/file_descriptor.h>
>  
> @@ -34,7 +35,14 @@ struct FrameMetadata {
>  	Status status;
>  	unsigned int sequence;
>  	uint64_t timestamp;
> -	std::vector<Plane> planes;
> +
> +	Span<Plane> planes() { return planes_; }
> +	Span<const Plane> planes() const { return planes_; }

Returning a span here is nice.

This likely causes compile breakage for any external app.

I know we're not ABI/API stable, but I wonder if we should highlight
when we do cause breakage somehow, perhaps in the commit message at
least, as we already have external users who we might want to notify.


> +
> +private:
> +	friend class FrameBuffer;
> +
> +	std::vector<Plane> planes_;
>  };
>  
>  class FrameBuffer final : public Extensible
> diff --git a/src/cam/camera_session.cpp b/src/cam/camera_session.cpp
> index 60d640f2b15c..32a373a99b72 100644
> --- a/src/cam/camera_session.cpp
> +++ b/src/cam/camera_session.cpp
> @@ -374,9 +374,9 @@ void CameraSession::processRequest(Request *request)
>  		     << " bytesused: ";
>  
>  		unsigned int nplane = 0;
> -		for (const FrameMetadata::Plane &plane : metadata.planes) {
> +		for (const FrameMetadata::Plane &plane : metadata.planes()) {
>  			info << plane.bytesused;
> -			if (++nplane < metadata.planes.size())
> +			if (++nplane < metadata.planes().size())
>  				info << "/";
>  		}
>  	}
> diff --git a/src/cam/file_sink.cpp b/src/cam/file_sink.cpp
> index 0b529e3eb767..0fc7d621f50b 100644
> --- a/src/cam/file_sink.cpp
> +++ b/src/cam/file_sink.cpp
> @@ -110,7 +110,7 @@ void FileSink::writeBuffer(const Stream *stream, FrameBuffer *buffer)
>  
>  	for (unsigned int i = 0; i < buffer->planes().size(); ++i) {
>  		const FrameBuffer::Plane &plane = buffer->planes()[i];
> -		const FrameMetadata::Plane &meta = buffer->metadata().planes[i];
> +		const FrameMetadata::Plane &meta = buffer->metadata().planes()[i];
>  
>  		uint8_t *data = planeData_[&plane];
>  		unsigned int length = std::min(meta.bytesused, plane.length);
> diff --git a/src/libcamera/framebuffer.cpp b/src/libcamera/framebuffer.cpp
> index c1529dfb0ad1..83cf7b83d182 100644
> --- a/src/libcamera/framebuffer.cpp
> +++ b/src/libcamera/framebuffer.cpp
> @@ -91,8 +91,14 @@ LOG_DEFINE_CATEGORY(Buffer)
>   */
>  
>  /**
> - * \var FrameMetadata::planes
> - * \brief Array of per-plane metadata
> + * \fn FrameMetadata::planes()
> + * \copydoc FrameMetadata::planes() const
> + */
> +
> +/**
> + * \fn FrameMetadata::planes() const
> + * \brief Retrieve the array of per-plane metadata
> + * \return The array of per-plane metadata
>   */
>  
>  /**
> @@ -210,7 +216,7 @@ FrameBuffer::FrameBuffer(const std::vector<Plane> &planes, unsigned int cookie)
>  	: Extensible(std::make_unique<Private>()), planes_(planes),
>  	  cookie_(cookie)
>  {
> -	metadata_.planes.resize(planes_.size());
> +	metadata_.planes_.resize(planes_.size());
>  
>  	unsigned int offset = 0;
>  
> diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> index a51971879e75..82ddaed3656f 100644
> --- a/src/libcamera/v4l2_videodevice.cpp
> +++ b/src/libcamera/v4l2_videodevice.cpp
> @@ -1485,14 +1485,14 @@ int V4L2VideoDevice::queueBuffer(FrameBuffer *buffer)
>  
>  		if (multiPlanar) {
>  			unsigned int nplane = 0;
> -			for (const FrameMetadata::Plane &plane : metadata.planes) {
> +			for (const FrameMetadata::Plane &plane : metadata.planes()) {
>  				v4l2Planes[nplane].bytesused = plane.bytesused;
>  				v4l2Planes[nplane].length = buffer->planes()[nplane].length;
>  				nplane++;
>  			}
>  		} else {
> -			if (metadata.planes.size())
> -				buf.bytesused = metadata.planes[0].bytesused;
> +			if (metadata.planes().size())
> +				buf.bytesused = metadata.planes()[0].bytesused;
>  		}
>  
>  		buf.sequence = metadata.sequence;
> @@ -1587,17 +1587,17 @@ FrameBuffer *V4L2VideoDevice::dequeueBuffer()
>  				    + buf.timestamp.tv_usec * 1000ULL;
>  
>  	if (multiPlanar) {
> -		if (buf.length > buffer->metadata_.planes.size()) {
> +		if (buf.length > buffer->metadata_.planes().size()) {
>  			LOG(V4L2, Error)
>  				<< "Invalid number of planes (" << buf.length
> -				<< " != " << buffer->metadata_.planes.size() << ")";
> +				<< " != " << buffer->metadata_.planes().size() << ")";
>  			return nullptr;
>  		}
>  
>  		for (unsigned int nplane = 0; nplane < buf.length; nplane++)
> -			buffer->metadata_.planes[nplane].bytesused = planes[nplane].bytesused;
> +			buffer->metadata_.planes()[nplane].bytesused = planes[nplane].bytesused;
>  	} else {
> -		buffer->metadata_.planes[0].bytesused = buf.bytesused;
> +		buffer->metadata_.planes()[0].bytesused = buf.bytesused;
>  	}
>  
>  	return buffer;
> diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp
> index 1536b2b5bd66..ac853e360aea 100644
> --- a/src/qcam/main_window.cpp
> +++ b/src/qcam/main_window.cpp
> @@ -756,7 +756,7 @@ void MainWindow::processViewfinder(FrameBuffer *buffer)
>  
>  	qDebug().noquote()
>  		<< QString("seq: %1").arg(metadata.sequence, 6, 10, QLatin1Char('0'))
> -		<< "bytesused:" << metadata.planes[0].bytesused
> +		<< "bytesused:" << metadata.planes()[0].bytesused
>  		<< "timestamp:" << metadata.timestamp
>  		<< "fps:" << Qt::fixed << qSetRealNumberPrecision(2) << fps;
>  
> diff --git a/src/qcam/viewfinder_gl.cpp b/src/qcam/viewfinder_gl.cpp
> index 40226601f9fd..d2ef036974f4 100644
> --- a/src/qcam/viewfinder_gl.cpp
> +++ b/src/qcam/viewfinder_gl.cpp
> @@ -125,7 +125,7 @@ void ViewFinderGL::render(libcamera::FrameBuffer *buffer,
>  	/*
>  	 * \todo Get the stride from the buffer instead of computing it naively
>  	 */
> -	stride_ = buffer->metadata().planes[0].bytesused / size_.height();
> +	stride_ = buffer->metadata().planes()[0].bytesused / size_.height();

Can this be obtained from the PixelFormatInfo now ?
or do we still not expose that to applications...

Anyway, that's likely a change on top even if we could to solve the \todo.

Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>


>  	update();
>  	buffer_ = buffer;
>  }
> diff --git a/src/qcam/viewfinder_qt.cpp b/src/qcam/viewfinder_qt.cpp
> index efa1d412584b..a0bf99b0b522 100644
> --- a/src/qcam/viewfinder_qt.cpp
> +++ b/src/qcam/viewfinder_qt.cpp
> @@ -87,7 +87,7 @@ void ViewFinderQt::render(libcamera::FrameBuffer *buffer,
>  	}
>  
>  	unsigned char *memory = mem.data();
> -	size_t size = buffer->metadata().planes[0].bytesused;
> +	size_t size = buffer->metadata().planes()[0].bytesused;
>  
>  	{
>  		QMutexLocker locker(&mutex_);
> diff --git a/src/v4l2/v4l2_camera_proxy.cpp b/src/v4l2/v4l2_camera_proxy.cpp
> index 8d8ee395954f..68e47ee81834 100644
> --- a/src/v4l2/v4l2_camera_proxy.cpp
> +++ b/src/v4l2/v4l2_camera_proxy.cpp
> @@ -212,7 +212,7 @@ void V4L2CameraProxy::updateBuffers()
>  
>  		switch (fmd.status) {
>  		case FrameMetadata::FrameSuccess:
> -			buf.bytesused = fmd.planes[0].bytesused;
> +			buf.bytesused = fmd.planes()[0].bytesused;
>  			buf.field = V4L2_FIELD_NONE;
>  			buf.timestamp.tv_sec = fmd.timestamp / 1000000000;
>  			buf.timestamp.tv_usec = fmd.timestamp % 1000000;
>

Kieran Bingham Sept. 2, 2021, 10:21 a.m. UTC | #6

On 02/09/2021 05:23, Laurent Pinchart wrote:
> Cache the PixelFormatInfo instead of looking it up in every call to
> createBuffer(). This prepare for usage of the info in queueBuffer(), to

s/prepare/prepares/

> avoid a looking every time a buffer is queued.
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> ---
>  include/libcamera/internal/v4l2_videodevice.h |  1 +
>  src/libcamera/v4l2_videodevice.cpp            | 19 +++++++++++--------
>  2 files changed, 12 insertions(+), 8 deletions(-)
> 
> diff --git a/include/libcamera/internal/v4l2_videodevice.h b/include/libcamera/internal/v4l2_videodevice.h
> index 7a145f608a5b..6096cd609b97 100644
> --- a/include/libcamera/internal/v4l2_videodevice.h
> +++ b/include/libcamera/internal/v4l2_videodevice.h
> @@ -243,6 +243,7 @@ private:
>  
>  	V4L2Capability caps_;
>  	V4L2DeviceFormat format_;
> +	const PixelFormatInfo *formatInfo_;
>  
>  	enum v4l2_buf_type bufferType_;
>  	enum v4l2_memory memoryType_;
> diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> index 82ddaed3656f..2d9a94c3c974 100644
> --- a/src/libcamera/v4l2_videodevice.cpp
> +++ b/src/libcamera/v4l2_videodevice.cpp
> @@ -482,8 +482,8 @@ const std::string V4L2DeviceFormat::toString() const
>   * \param[in] deviceNode The file-system path to the video device node
>   */
>  V4L2VideoDevice::V4L2VideoDevice(const std::string &deviceNode)
> -	: V4L2Device(deviceNode), cache_(nullptr), fdBufferNotifier_(nullptr),
> -	  streaming_(false)
> +	: V4L2Device(deviceNode), formatInfo_(nullptr), cache_(nullptr),
> +	  fdBufferNotifier_(nullptr), streaming_(false)
>  {
>  	/*
>  	 * We default to an MMAP based CAPTURE video device, however this will
> @@ -586,6 +586,7 @@ int V4L2VideoDevice::open()
>  		return ret;
>  	}
>  
> +	formatInfo_ = &PixelFormatInfo::info(format_.fourcc);

I shudder a little seeing code hugging the return like that ;-)

But it's purely subjective, same below of course.


>  	return 0;
>  }
>  
> @@ -681,6 +682,7 @@ int V4L2VideoDevice::open(int handle, enum v4l2_buf_type type)
>  		return ret;
>  	}
>  
> +	formatInfo_ = &PixelFormatInfo::info(format_.fourcc);
>  	return 0;
>  }
>  
> @@ -695,6 +697,8 @@ void V4L2VideoDevice::close()
>  	releaseBuffers();
>  	delete fdBufferNotifier_;
>  
> +	formatInfo_ = nullptr;
> +
>  	V4L2Device::close();
>  }
>  
> @@ -787,6 +791,7 @@ int V4L2VideoDevice::setFormat(V4L2DeviceFormat *format)
>  		return ret;
>  
>  	format_ = *format;
> +	formatInfo_ = &PixelFormatInfo::info(format_.fourcc);

Looks like we were already using that pattern though ;-)


>  	return 0;
>  }
>  
> @@ -1325,19 +1330,17 @@ std::unique_ptr<FrameBuffer> V4L2VideoDevice::createBuffer(unsigned int index)
>  		planes.push_back(std::move(plane));
>  	}
>  
> -	const auto &info = PixelFormatInfo::info(format_.fourcc);
> -	if (info.isValid() && info.numPlanes() != numPlanes) {
> +	if (formatInfo_->isValid() && formatInfo_->numPlanes() != numPlanes) {
>  		ASSERT(numPlanes == 1u);

Not for this patch, but I guess this assert needed a
  \todo Multiplanar support

around it?

However, this patch is just for caching the formatInfo, which seems
reasonable so

Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>

> -		const size_t numColorPlanes = info.numPlanes();
> -		planes.resize(numColorPlanes);
> +		planes.resize(formatInfo_->numPlanes());
>  		const FileDescriptor &fd = planes[0].fd;
>  		size_t offset = 0;
> -		for (size_t i = 0; i < numColorPlanes; ++i) {
> +		for (size_t i = 0; i < planes.size(); ++i) {
>  			planes[i].fd = fd;
>  			planes[i].offset = offset;
>  
>  			/* \todo Take the V4L2 stride into account */
> -			planes[i].length = info.planeSize(format_.size, i);
> +			planes[i].length = formatInfo_->planeSize(format_.size, i);
>  			offset += planes[i].length;
>  		}
>  	}
>

Kieran Bingham Sept. 2, 2021, 11:17 a.m. UTC | #7

On 02/09/2021 05:23, Laurent Pinchart wrote:
> When dequeueing a buffer from a V4L2VideoDevice, the number of planes in
> the FrameBuffer may not match the number of V4L2 buffer planes if the
> PixelFormat is multi-planar (has multiple colour planes) and the V4L2
> format is single-planar (has a single buffer plane). In this case, we
> need to split the single V4L2 buffer plane into FrameBuffer planes. Do
> so, and add checks to reject invalid V4L2 buffers in case of a driver
> issue.
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> ---
>  src/libcamera/v4l2_videodevice.cpp | 54 ++++++++++++++++++++++++------
>  1 file changed, 43 insertions(+), 11 deletions(-)
> 
> diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> index 79cb792117d5..b80b9038a914 100644
> --- a/src/libcamera/v4l2_videodevice.cpp
> +++ b/src/libcamera/v4l2_videodevice.cpp
> @@ -7,6 +7,7 @@
>  
>  #include "libcamera/internal/v4l2_videodevice.h"
>  
> +#include <algorithm>
>  #include <array>
>  #include <fcntl.h>
>  #include <iomanip>
> @@ -1637,18 +1638,49 @@ FrameBuffer *V4L2VideoDevice::dequeueBuffer()
>  	buffer->metadata_.timestamp = buf.timestamp.tv_sec * 1000000000ULL
>  				    + buf.timestamp.tv_usec * 1000ULL;
>  
> -	if (multiPlanar) {
> -		if (buf.length > buffer->metadata_.planes().size()) {
> -			LOG(V4L2, Error)
> -				<< "Invalid number of planes (" << buf.length
> -				<< " != " << buffer->metadata_.planes().size() << ")";
> -			return nullptr;
> +	if (V4L2_TYPE_IS_CAPTURE(buf.type)) {

Should we swap this around and return the buffer early to lower the
indents below?

 if (!V4L2_TYPE_IS_CAPTURE(buf.type)) {
	/* Output buffer is completed */
	return buffer;
 }


> +		unsigned int numV4l2Planes = multiPlanar ? buf.length : 1;
> +		FrameMetadata &metadata = buffer->metadata_;
> +
> +		if (numV4l2Planes != metadata.planes().size()) {
> +			/*
> +			 * If we have a multi-planar buffer with a V4L2
> +			 * single-planar format, split the V4L2 buffer across
> +			 * the buffer planes. Only the last plane may have less
> +			 * bytes used than its length.
> +			 */
> +			if (numV4l2Planes != 1) {
> +				LOG(V4L2, Error)
> +					<< "Invalid number of planes (" << numV4l2Planes
> +					<< " != " << metadata.planes().size() << ")";
> +				return nullptr;
> +			}
> +
> +			unsigned int bytesused = multiPlanar ? planes[0].bytesused
> +					       : buf.bytesused;
> +
> +			for (auto [i, plane] : utils::enumerate(buffer->planes())) {
> +				if (!bytesused) {
> +					LOG(V4L2, Error)
> +						<< "Dequeued buffer is too small";

Presumably if we got here - it's likely something (hardware) already
overwrote memory it likely didn't own?

> +					return nullptr;

I'm still very weary of returning nullptrs on dequeue when a buffer is
removed from queuedBuffers_


Wouldn't it be better to return the buffer, but mark it as corrupted or
invalid or such ?


> +				}
> +
> +				metadata.planes()[i].bytesused =
> +					std::min(plane.length, bytesused);
> +				bytesused -= metadata.planes()[i].bytesused;
> +			}
> +		} else if (multiPlanar) {
> +			/*
> +			 * If we use the multi-planar API, fill in the planes.
> +			 * The number of planes in the frame buffer and in the
> +			 * V4L2 buffer is guaranteed to be equal at this point.
> +			 */
> +			for (unsigned int i = 0; i < numV4l2Planes; ++i)
> +				metadata.planes()[i].bytesused = planes[i].bytesused;
> +		} else {
> +			metadata.planes()[0].bytesused = buf.bytesused;
>  		}
> -
> -		for (unsigned int nplane = 0; nplane < buf.length; nplane++)
> -			buffer->metadata_.planes()[nplane].bytesused = planes[nplane].bytesused;
> -	} else {
> -		buffer->metadata_.planes()[0].bytesused = buf.bytesused;
>  	}
>  
>  	return buffer;
>

Kieran Bingham Sept. 2, 2021, 11:25 a.m. UTC | #8

On 02/09/2021 05:23, Laurent Pinchart wrote:
> When queueing a buffer to a V4L2VideoDevice, the number of planes in the
> FrameBuffer may not match the number of V4L2 buffer planes if the
> PixelFormat is multi-planar (has multiple colour planes) and the V4L2
> format is single-planar (has a single buffer plane). In this case, we
> need to coalesce all FrameBuffer planes into a single V4L2 buffer plane.
> Do so, and add validity checks to reject frame buffers that can't be
> described using a single V4L2 buffer plane.


Yikes ;-) lets see...

> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> ---
>  src/libcamera/v4l2_videodevice.cpp | 68 +++++++++++++++++++++++++-----
>  1 file changed, 58 insertions(+), 10 deletions(-)
> 
> diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> index 2d9a94c3c974..79cb792117d5 100644
> --- a/src/libcamera/v4l2_videodevice.cpp
> +++ b/src/libcamera/v4l2_videodevice.cpp
> @@ -22,10 +22,12 @@
>  
>  #include <libcamera/base/event_notifier.h>
>  #include <libcamera/base/log.h>
> +#include <libcamera/base/utils.h>
>  
>  #include <libcamera/file_descriptor.h>
>  
>  #include "libcamera/internal/formats.h"
> +#include "libcamera/internal/framebuffer.h"
>  #include "libcamera/internal/media_device.h"
>  #include "libcamera/internal/media_object.h"
>  
> @@ -1468,10 +1470,21 @@ int V4L2VideoDevice::queueBuffer(FrameBuffer *buffer)
>  
>  	bool multiPlanar = V4L2_TYPE_IS_MULTIPLANAR(buf.type);
>  	const std::vector<FrameBuffer::Plane> &planes = buffer->planes();
> +	unsigned int numV4l2Planes = format_.fourcc == formatInfo_->v4l2Format[0]
> +				   ? 1 : planes.size();

Aha, ok so that's how we recover the planar/multiplanar information I
was concerned we had lost earlier.

I wonder if we should set/cache numV4l2Planes_ when we set formatInfo_
as they are so closely paired .. But it's a trivial operation here.



> +
> +	/*
> +	 * If the frame buffer has multiple planes and the V4L2 format requires
> +	 * contiguous planes, ensure that's the case.
> +	 */
> +	if (planes.size() != numV4l2Planes && !buffer->_d()->isContiguous()) {
> +		LOG(V4L2, Error) << "Device format requires contiguous buffer";
> +		return -EINVAL;
> +	}
>  
>  	if (buf.memory == V4L2_MEMORY_DMABUF) {
>  		if (multiPlanar) {
> -			for (unsigned int p = 0; p < planes.size(); ++p)
> +			for (unsigned int p = 0; p < numV4l2Planes; ++p)
>  				v4l2Planes[p].m.fd = planes[p].fd.fd();
>  		} else {
>  			buf.m.fd = planes[0].fd.fd();
> @@ -1479,23 +1492,58 @@ int V4L2VideoDevice::queueBuffer(FrameBuffer *buffer)
>  	}
>  
>  	if (multiPlanar) {
> -		buf.length = planes.size();
> +		buf.length = numV4l2Planes;
>  		buf.m.planes = v4l2Planes;
>  	}
>  
>  	if (V4L2_TYPE_IS_OUTPUT(buf.type)) {
>  		const FrameMetadata &metadata = buffer->metadata();
>  
> -		if (multiPlanar) {
> -			unsigned int nplane = 0;
> -			for (const FrameMetadata::Plane &plane : metadata.planes()) {
> -				v4l2Planes[nplane].bytesused = plane.bytesused;
> -				v4l2Planes[nplane].length = buffer->planes()[nplane].length;
> -				nplane++;
> +		if (numV4l2Planes != planes.size()) {
> +			/*
> +			 * If we have a multi-planar buffer with a V4L2
> +			 * single-planar format, coalesce all planes. The length
> +			 * and number of bytes used may only differ in the last
> +			 * plane as any other situation can't be represented.
> +			 */
> +			unsigned int bytesused = 0;
> +			unsigned int length = 0;
> +
> +			for (auto [i, plane] : utils::enumerate(planes)) {
> +				bytesused += metadata.planes()[i].bytesused;
> +				length += plane.length;
> +
> +				if (i != planes.size() - 1 && bytesused != length) {

Ah, i almost thought that was wrong, but it's
   "If we're not the last plane, and the plane is not fully used..."

So it's fine.


> +					LOG(V4L2, Error)
> +						<< "Holes in multi-planar buffer not supported";
> +					return -EINVAL;
> +				}
> +			}
> +
> +			if (multiPlanar) {
> +				v4l2Planes[0].bytesused = bytesused;
> +				v4l2Planes[0].length = length;
> +			} else {
> +				buf.bytesused = bytesused;
> +				buf.length = length;
> +			}
> +		} else if (multiPlanar) {
> +			/*
> +			 * If we use the multi-planar API, fill in the planes.
> +			 * The number of planes in the frame buffer and in the
> +			 * V4L2 buffer is guaranteed to be equal at this point.
> +			 */
> +			for (auto [i, plane] : utils::enumerate(planes)) {
> +				v4l2Planes[i].bytesused = metadata.planes()[i].bytesused;
> +				v4l2Planes[i].length = plane.length;
>  			}
>  		} else {
> -			if (metadata.planes().size())

Guaranteeing the planes are there is much nicer here.

Phew, this patch 'looks' a lot more complex than it actually is,

So I think ... this is fine.

Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>

> -				buf.bytesused = metadata.planes()[0].bytesused;
> +			/*
> +			 * Single-planar API with a single plane in the buffer
> +			 * is trivial to handle.
> +			 */
> +			buf.bytesused = metadata.planes()[0].bytesused;
> +			buf.length = planes[0].length;
>  		}
>  
>  		buf.sequence = metadata.sequence;
>

Laurent Pinchart Sept. 2, 2021, 6:07 p.m. UTC | #9

Hi Kieran,

On Thu, Sep 02, 2021 at 10:43:56AM +0100, Kieran Bingham wrote:
> On 02/09/2021 05:22, Laurent Pinchart wrote:
> > Multi-planar frame buffers can store their planes contiguously in
> > memory, or split them in discontiguous memory areas. Add a private
> > function to check in which of these two categories the frame buffer
> > belongs. This will be used to correctly handle the differences between
> > the V4L2 single and multi planar APIs.
> > 
> > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > ---
> >  include/libcamera/internal/framebuffer.h |  2 ++
> >  src/libcamera/framebuffer.cpp            | 45 +++++++++++++++++++++++-
> >  2 files changed, 46 insertions(+), 1 deletion(-)
> > 
> > diff --git a/include/libcamera/internal/framebuffer.h b/include/libcamera/internal/framebuffer.h
> > index 606aed2b4782..cd33c295466e 100644
> > --- a/include/libcamera/internal/framebuffer.h
> > +++ b/include/libcamera/internal/framebuffer.h
> > @@ -21,9 +21,11 @@ public:
> >  	Private();
> >  
> >  	void setRequest(Request *request) { request_ = request; }
> > +	bool isContiguous() const { return isContiguous_; }
> >  
> >  private:
> >  	Request *request_;
> > +	bool isContiguous_;
> >  };
> >  
> >  } /* namespace libcamera */
> > diff --git a/src/libcamera/framebuffer.cpp b/src/libcamera/framebuffer.cpp
> > index 53ef89bf458f..99265b44da43 100644
> > --- a/src/libcamera/framebuffer.cpp
> > +++ b/src/libcamera/framebuffer.cpp
> > @@ -106,7 +106,7 @@ LOG_DEFINE_CATEGORY(Buffer)
> >   */
> >  
> >  FrameBuffer::Private::Private()
> > -	: request_(nullptr)
> > +	: request_(nullptr), isContiguous_(true)
> >  {
> >  }
> >  
> > @@ -120,6 +120,17 @@ FrameBuffer::Private::Private()
> >   * handlers, it is called by the pipeline handlers themselves.
> >   */
> >  
> > +/**
> > + * \fn FrameBuffer::Private::isContiguous()
> > + * \brief Check if the frame buffer stores planes contiguously in memory
> > + *
> > + * Multi-planar frame buffers can store their planes contiguously in memory, or
> > + * split them in discontiguous memory areas. This function checks in which of
> 
> 'split them into discontiguous'
> 
> > + * these two categories the frame buffer belongs.
> > + *
> > + * \return True if the planes are stored contiguously in memory, false otherwise
> > + */
> > +
> >  /**
> >   * \class FrameBuffer
> >   * \brief Frame buffer data and its associated dynamic metadata
> > @@ -199,9 +210,41 @@ FrameBuffer::FrameBuffer(const std::vector<Plane> &planes, unsigned int cookie)
> >  	: Extensible(std::make_unique<Private>()), planes_(planes),
> >  	  cookie_(cookie)
> >  {
> > +	unsigned int offset = 0;
> > +
> >  	/* \todo Remove the assertions after sufficient testing */
> >  	for (const auto &plane : planes_)
> >  		ASSERT(plane.offset != Plane::kInvalidOffset);
> > +
> > +	bool isContiguous = true;
> > +	ino_t inode = 0;
> > +
> > +	for (const auto &plane : planes_) {
> > +		if (plane.offset != offset) {
> > +			isContiguous = false;
> > +			break;
> > +		}
> > +
> > +		/*
> > +		 * Two different dmabuf file descriptors may still refer to the
> > +		 * same dmabuf instance. Check this using inodes.
> 
> Going back to the FileDescriptor::inode() extension, if that cached the
> inode (it can't change) on either first call, or construction, couldn't
> this whole check be simplified to:
> 
> 		if (plane.fd.inode() != planes_[0].fd.inode()) {
> 			isContiguous = false;
> 			break;
> 		}

This would call fstat() every time, while the fd comparison here is
meant to skip the fstat() calls in case the numerical fd match. We could
do

		if (plane.fd != planes_[0].fd)

if we introduced an operator==(), but as explained in a reply to the
patch that adds inode(), I'd like to handle that later, to avoid
blocking this series.

> > +		 */
> > +		if (plane.fd.fd() != planes_[0].fd.fd()) {
> > +			if (!inode)
> > +				inode = planes_[0].fd.inode();
> > +			if (plane.fd.inode() != inode) {
> > +				isContiguous = false;
> > +				break;
> > +			}
> > +		}
> > +
> > +		offset += plane.length;
> > +	}
> > +
> > +	LOG(Buffer, Debug)
> > +		<< "Buffer is " << (isContiguous ? "not " : "") << "contiguous";
> > +
> > +	_d()->isContiguous_ = isContiguous;
> >  }
> >  
> >  /**

Laurent Pinchart Sept. 2, 2021, 6:11 p.m. UTC | #10

Hi Kieran,

On Thu, Sep 02, 2021 at 10:54:19AM +0100, Kieran Bingham wrote:
> On 02/09/2021 05:22, Laurent Pinchart wrote:
> > The metadata planes are allocated by V4L2VideoDevice when dequeuing a
> > buffer. This causes the metadata planes to only be allocated after a
> > buffer gets dequeued, and doesn't provide any strong guarantee that
> > their number matches the number of FrameBuffer planes. The lack of this
> > invariant makes the FrameBuffer class fragile.
> > 
> > As a first step towards fixing this, allocate the metadata planes when
> > the FrameBuffer is constructed. The FrameMetadata API should be further
> > improved by preventing a change in the number of planes.
> > 
> > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > ---
> >  src/libcamera/framebuffer.cpp      |  2 ++
> >  src/libcamera/v4l2_videodevice.cpp | 12 +++++++++---
> >  2 files changed, 11 insertions(+), 3 deletions(-)
> > 
> > diff --git a/src/libcamera/framebuffer.cpp b/src/libcamera/framebuffer.cpp
> > index 99265b44da43..c1529dfb0ad1 100644
> > --- a/src/libcamera/framebuffer.cpp
> > +++ b/src/libcamera/framebuffer.cpp
> > @@ -210,6 +210,8 @@ FrameBuffer::FrameBuffer(const std::vector<Plane> &planes, unsigned int cookie)
> >  	: Extensible(std::make_unique<Private>()), planes_(planes),
> >  	  cookie_(cookie)
> >  {
> > +	metadata_.planes.resize(planes_.size());
> > +
> >  	unsigned int offset = 0;
> >  
> >  	/* \todo Remove the assertions after sufficient testing */
> > diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> > index adabd4720668..a51971879e75 100644
> > --- a/src/libcamera/v4l2_videodevice.cpp
> > +++ b/src/libcamera/v4l2_videodevice.cpp
> > @@ -1586,12 +1586,18 @@ FrameBuffer *V4L2VideoDevice::dequeueBuffer()
> >  	buffer->metadata_.timestamp = buf.timestamp.tv_sec * 1000000000ULL
> >  				    + buf.timestamp.tv_usec * 1000ULL;
> >  
> > -	buffer->metadata_.planes.clear();
> >  	if (multiPlanar) {
> > +		if (buf.length > buffer->metadata_.planes.size()) {
> > +			LOG(V4L2, Error)
> 
> Are we going to 'leak' queued FrameBuffers at this point?
> Is there a way to prevent this occurring at queue time rather than dequeue?
> 
> In fact, Can it even happen? or should it be an ASSERT/Fatal?

It's not supposed to happen, I'd say it's a kernel bug in that case. But
it's a good point, I think we can return the buffer, and mark its status
as erroneous. I'll do so.

> > +				<< "Invalid number of planes (" << buf.length
> > +				<< " != " << buffer->metadata_.planes.size() << ")";
> > +			return nullptr;
> > +		}
> > +
> >  		for (unsigned int nplane = 0; nplane < buf.length; nplane++)
> > -			buffer->metadata_.planes.push_back({ planes[nplane].bytesused });
> > +			buffer->metadata_.planes[nplane].bytesused = planes[nplane].bytesused;
> >  	} else {
> > -		buffer->metadata_.planes.push_back({ buf.bytesused });
> > +		buffer->metadata_.planes[0].bytesused = buf.bytesused;
> >  	}
> >  
> >  	return buffer;
> >

Laurent Pinchart Sept. 2, 2021, 6:20 p.m. UTC | #11

Hi Kieran,

On Thu, Sep 02, 2021 at 11:03:47AM +0100, Kieran Bingham wrote:
> On 02/09/2021 05:22, Laurent Pinchart wrote:
> > The number of metadata planes should always match the number of frame
> > buffer planes. Enforce this by making the vector private and providing
> > accessor functions.
> > 
> > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > ---
> >  include/libcamera/framebuffer.h    | 10 +++++++++-
> >  src/cam/camera_session.cpp         |  4 ++--
> >  src/cam/file_sink.cpp              |  2 +-
> >  src/libcamera/framebuffer.cpp      | 12 +++++++++---
> >  src/libcamera/v4l2_videodevice.cpp | 14 +++++++-------
> >  src/qcam/main_window.cpp           |  2 +-
> >  src/qcam/viewfinder_gl.cpp         |  2 +-
> >  src/qcam/viewfinder_qt.cpp         |  2 +-
> >  src/v4l2/v4l2_camera_proxy.cpp     |  2 +-
> >  9 files changed, 32 insertions(+), 18 deletions(-)
> > 
> > diff --git a/include/libcamera/framebuffer.h b/include/libcamera/framebuffer.h
> > index fd68ed0a139d..7f2f176af691 100644
> > --- a/include/libcamera/framebuffer.h
> > +++ b/include/libcamera/framebuffer.h
> > @@ -13,6 +13,7 @@
> >  #include <vector>
> >  
> >  #include <libcamera/base/class.h>
> > +#include <libcamera/base/span.h>
> >  
> >  #include <libcamera/file_descriptor.h>
> >  
> > @@ -34,7 +35,14 @@ struct FrameMetadata {
> >  	Status status;
> >  	unsigned int sequence;
> >  	uint64_t timestamp;
> > -	std::vector<Plane> planes;
> > +
> > +	Span<Plane> planes() { return planes_; }
> > +	Span<const Plane> planes() const { return planes_; }
> 
> Returning a span here is nice.
> 
> This likely causes compile breakage for any external app.
> 
> I know we're not ABI/API stable, but I wonder if we should highlight
> when we do cause breakage somehow, perhaps in the commit message at
> least, as we already have external users who we might want to notify.

Most changes to public header files will be ABI/API breakages at this
point, and users will certainly notice when they get a compilation
failure :-) What value do you think this would bring, who would read
those messages ? I think it could be different for things that change
the ABI without breaking compilation and that would require more than a
recompilation to fix, there a warning seems to have more value, but I'm
also sure we'll forget from time to time, if not most of the time.

> > +
> > +private:
> > +	friend class FrameBuffer;
> > +
> > +	std::vector<Plane> planes_;
> >  };
> >  
> >  class FrameBuffer final : public Extensible
> > diff --git a/src/cam/camera_session.cpp b/src/cam/camera_session.cpp
> > index 60d640f2b15c..32a373a99b72 100644
> > --- a/src/cam/camera_session.cpp
> > +++ b/src/cam/camera_session.cpp
> > @@ -374,9 +374,9 @@ void CameraSession::processRequest(Request *request)
> >  		     << " bytesused: ";
> >  
> >  		unsigned int nplane = 0;
> > -		for (const FrameMetadata::Plane &plane : metadata.planes) {
> > +		for (const FrameMetadata::Plane &plane : metadata.planes()) {
> >  			info << plane.bytesused;
> > -			if (++nplane < metadata.planes.size())
> > +			if (++nplane < metadata.planes().size())
> >  				info << "/";
> >  		}
> >  	}
> > diff --git a/src/cam/file_sink.cpp b/src/cam/file_sink.cpp
> > index 0b529e3eb767..0fc7d621f50b 100644
> > --- a/src/cam/file_sink.cpp
> > +++ b/src/cam/file_sink.cpp
> > @@ -110,7 +110,7 @@ void FileSink::writeBuffer(const Stream *stream, FrameBuffer *buffer)
> >  
> >  	for (unsigned int i = 0; i < buffer->planes().size(); ++i) {
> >  		const FrameBuffer::Plane &plane = buffer->planes()[i];
> > -		const FrameMetadata::Plane &meta = buffer->metadata().planes[i];
> > +		const FrameMetadata::Plane &meta = buffer->metadata().planes()[i];
> >  
> >  		uint8_t *data = planeData_[&plane];
> >  		unsigned int length = std::min(meta.bytesused, plane.length);
> > diff --git a/src/libcamera/framebuffer.cpp b/src/libcamera/framebuffer.cpp
> > index c1529dfb0ad1..83cf7b83d182 100644
> > --- a/src/libcamera/framebuffer.cpp
> > +++ b/src/libcamera/framebuffer.cpp
> > @@ -91,8 +91,14 @@ LOG_DEFINE_CATEGORY(Buffer)
> >   */
> >  
> >  /**
> > - * \var FrameMetadata::planes
> > - * \brief Array of per-plane metadata
> > + * \fn FrameMetadata::planes()
> > + * \copydoc FrameMetadata::planes() const
> > + */
> > +
> > +/**
> > + * \fn FrameMetadata::planes() const
> > + * \brief Retrieve the array of per-plane metadata
> > + * \return The array of per-plane metadata
> >   */
> >  
> >  /**
> > @@ -210,7 +216,7 @@ FrameBuffer::FrameBuffer(const std::vector<Plane> &planes, unsigned int cookie)
> >  	: Extensible(std::make_unique<Private>()), planes_(planes),
> >  	  cookie_(cookie)
> >  {
> > -	metadata_.planes.resize(planes_.size());
> > +	metadata_.planes_.resize(planes_.size());
> >  
> >  	unsigned int offset = 0;
> >  
> > diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> > index a51971879e75..82ddaed3656f 100644
> > --- a/src/libcamera/v4l2_videodevice.cpp
> > +++ b/src/libcamera/v4l2_videodevice.cpp
> > @@ -1485,14 +1485,14 @@ int V4L2VideoDevice::queueBuffer(FrameBuffer *buffer)
> >  
> >  		if (multiPlanar) {
> >  			unsigned int nplane = 0;
> > -			for (const FrameMetadata::Plane &plane : metadata.planes) {
> > +			for (const FrameMetadata::Plane &plane : metadata.planes()) {
> >  				v4l2Planes[nplane].bytesused = plane.bytesused;
> >  				v4l2Planes[nplane].length = buffer->planes()[nplane].length;
> >  				nplane++;
> >  			}
> >  		} else {
> > -			if (metadata.planes.size())
> > -				buf.bytesused = metadata.planes[0].bytesused;
> > +			if (metadata.planes().size())
> > +				buf.bytesused = metadata.planes()[0].bytesused;
> >  		}
> >  
> >  		buf.sequence = metadata.sequence;
> > @@ -1587,17 +1587,17 @@ FrameBuffer *V4L2VideoDevice::dequeueBuffer()
> >  				    + buf.timestamp.tv_usec * 1000ULL;
> >  
> >  	if (multiPlanar) {
> > -		if (buf.length > buffer->metadata_.planes.size()) {
> > +		if (buf.length > buffer->metadata_.planes().size()) {
> >  			LOG(V4L2, Error)
> >  				<< "Invalid number of planes (" << buf.length
> > -				<< " != " << buffer->metadata_.planes.size() << ")";
> > +				<< " != " << buffer->metadata_.planes().size() << ")";
> >  			return nullptr;
> >  		}
> >  
> >  		for (unsigned int nplane = 0; nplane < buf.length; nplane++)
> > -			buffer->metadata_.planes[nplane].bytesused = planes[nplane].bytesused;
> > +			buffer->metadata_.planes()[nplane].bytesused = planes[nplane].bytesused;
> >  	} else {
> > -		buffer->metadata_.planes[0].bytesused = buf.bytesused;
> > +		buffer->metadata_.planes()[0].bytesused = buf.bytesused;
> >  	}
> >  
> >  	return buffer;
> > diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp
> > index 1536b2b5bd66..ac853e360aea 100644
> > --- a/src/qcam/main_window.cpp
> > +++ b/src/qcam/main_window.cpp
> > @@ -756,7 +756,7 @@ void MainWindow::processViewfinder(FrameBuffer *buffer)
> >  
> >  	qDebug().noquote()
> >  		<< QString("seq: %1").arg(metadata.sequence, 6, 10, QLatin1Char('0'))
> > -		<< "bytesused:" << metadata.planes[0].bytesused
> > +		<< "bytesused:" << metadata.planes()[0].bytesused
> >  		<< "timestamp:" << metadata.timestamp
> >  		<< "fps:" << Qt::fixed << qSetRealNumberPrecision(2) << fps;
> >  
> > diff --git a/src/qcam/viewfinder_gl.cpp b/src/qcam/viewfinder_gl.cpp
> > index 40226601f9fd..d2ef036974f4 100644
> > --- a/src/qcam/viewfinder_gl.cpp
> > +++ b/src/qcam/viewfinder_gl.cpp
> > @@ -125,7 +125,7 @@ void ViewFinderGL::render(libcamera::FrameBuffer *buffer,
> >  	/*
> >  	 * \todo Get the stride from the buffer instead of computing it naively
> >  	 */
> > -	stride_ = buffer->metadata().planes[0].bytesused / size_.height();
> > +	stride_ = buffer->metadata().planes()[0].bytesused / size_.height();
> 
> Can this be obtained from the PixelFormatInfo now ?
> or do we still not expose that to applications...

PixelFormatInfo is still internal. I'm considering exposing it, but then
it would be good to move the V4L2-specific part somewhere else.

> Anyway, that's likely a change on top even if we could to solve the \todo.
> 
> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> 
> >  	update();
> >  	buffer_ = buffer;
> >  }
> > diff --git a/src/qcam/viewfinder_qt.cpp b/src/qcam/viewfinder_qt.cpp
> > index efa1d412584b..a0bf99b0b522 100644
> > --- a/src/qcam/viewfinder_qt.cpp
> > +++ b/src/qcam/viewfinder_qt.cpp
> > @@ -87,7 +87,7 @@ void ViewFinderQt::render(libcamera::FrameBuffer *buffer,
> >  	}
> >  
> >  	unsigned char *memory = mem.data();
> > -	size_t size = buffer->metadata().planes[0].bytesused;
> > +	size_t size = buffer->metadata().planes()[0].bytesused;
> >  
> >  	{
> >  		QMutexLocker locker(&mutex_);
> > diff --git a/src/v4l2/v4l2_camera_proxy.cpp b/src/v4l2/v4l2_camera_proxy.cpp
> > index 8d8ee395954f..68e47ee81834 100644
> > --- a/src/v4l2/v4l2_camera_proxy.cpp
> > +++ b/src/v4l2/v4l2_camera_proxy.cpp
> > @@ -212,7 +212,7 @@ void V4L2CameraProxy::updateBuffers()
> >  
> >  		switch (fmd.status) {
> >  		case FrameMetadata::FrameSuccess:
> > -			buf.bytesused = fmd.planes[0].bytesused;
> > +			buf.bytesused = fmd.planes()[0].bytesused;
> >  			buf.field = V4L2_FIELD_NONE;
> >  			buf.timestamp.tv_sec = fmd.timestamp / 1000000000;
> >  			buf.timestamp.tv_usec = fmd.timestamp % 1000000;

Laurent Pinchart Sept. 2, 2021, 6:26 p.m. UTC | #12

Hi Kieran,

On Thu, Sep 02, 2021 at 11:21:29AM +0100, Kieran Bingham wrote:
> On 02/09/2021 05:23, Laurent Pinchart wrote:
> > Cache the PixelFormatInfo instead of looking it up in every call to
> > createBuffer(). This prepare for usage of the info in queueBuffer(), to
> 
> s/prepare/prepares/
> 
> > avoid a looking every time a buffer is queued.
> > 
> > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > ---
> >  include/libcamera/internal/v4l2_videodevice.h |  1 +
> >  src/libcamera/v4l2_videodevice.cpp            | 19 +++++++++++--------
> >  2 files changed, 12 insertions(+), 8 deletions(-)
> > 
> > diff --git a/include/libcamera/internal/v4l2_videodevice.h b/include/libcamera/internal/v4l2_videodevice.h
> > index 7a145f608a5b..6096cd609b97 100644
> > --- a/include/libcamera/internal/v4l2_videodevice.h
> > +++ b/include/libcamera/internal/v4l2_videodevice.h
> > @@ -243,6 +243,7 @@ private:
> >  
> >  	V4L2Capability caps_;
> >  	V4L2DeviceFormat format_;
> > +	const PixelFormatInfo *formatInfo_;
> >  
> >  	enum v4l2_buf_type bufferType_;
> >  	enum v4l2_memory memoryType_;
> > diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> > index 82ddaed3656f..2d9a94c3c974 100644
> > --- a/src/libcamera/v4l2_videodevice.cpp
> > +++ b/src/libcamera/v4l2_videodevice.cpp
> > @@ -482,8 +482,8 @@ const std::string V4L2DeviceFormat::toString() const
> >   * \param[in] deviceNode The file-system path to the video device node
> >   */
> >  V4L2VideoDevice::V4L2VideoDevice(const std::string &deviceNode)
> > -	: V4L2Device(deviceNode), cache_(nullptr), fdBufferNotifier_(nullptr),
> > -	  streaming_(false)
> > +	: V4L2Device(deviceNode), formatInfo_(nullptr), cache_(nullptr),
> > +	  fdBufferNotifier_(nullptr), streaming_(false)
> >  {
> >  	/*
> >  	 * We default to an MMAP based CAPTURE video device, however this will
> > @@ -586,6 +586,7 @@ int V4L2VideoDevice::open()
> >  		return ret;
> >  	}
> >  
> > +	formatInfo_ = &PixelFormatInfo::info(format_.fourcc);
> 
> I shudder a little seeing code hugging the return like that ;-)
> 
> But it's purely subjective, same below of course.

We have various patterns indeed :-) I haven't been able to establish yet
what my mental process to deal with these are, sometimes it bothers me
too, but not always.

> >  	return 0;
> >  }
> >  
> > @@ -681,6 +682,7 @@ int V4L2VideoDevice::open(int handle, enum v4l2_buf_type type)
> >  		return ret;
> >  	}
> >  
> > +	formatInfo_ = &PixelFormatInfo::info(format_.fourcc);
> >  	return 0;
> >  }
> >  
> > @@ -695,6 +697,8 @@ void V4L2VideoDevice::close()
> >  	releaseBuffers();
> >  	delete fdBufferNotifier_;
> >  
> > +	formatInfo_ = nullptr;
> > +
> >  	V4L2Device::close();
> >  }
> >  
> > @@ -787,6 +791,7 @@ int V4L2VideoDevice::setFormat(V4L2DeviceFormat *format)
> >  		return ret;
> >  
> >  	format_ = *format;
> > +	formatInfo_ = &PixelFormatInfo::info(format_.fourcc);
> 
> Looks like we were already using that pattern though ;-)
> 
> >  	return 0;
> >  }
> >  
> > @@ -1325,19 +1330,17 @@ std::unique_ptr<FrameBuffer> V4L2VideoDevice::createBuffer(unsigned int index)
> >  		planes.push_back(std::move(plane));
> >  	}
> >  
> > -	const auto &info = PixelFormatInfo::info(format_.fourcc);
> > -	if (info.isValid() && info.numPlanes() != numPlanes) {
> > +	if (formatInfo_->isValid() && formatInfo_->numPlanes() != numPlanes) {
> >  		ASSERT(numPlanes == 1u);
> 
> Not for this patch, but I guess this assert needed a
>   \todo Multiplanar support
> 
> around it?

No, when formatInfo_->numPlanes() != numPlanes, it means that we have a
multi-planar FrameBuffer with a single-planar V4L2 buffer format (e.g.
V4L2_PIX_FMT_NV12 as opposed to V4L2_PIX_FMT_NV12M). numPlanes must be 1
in that case, otherwise it's a kernel bug.

> However, this patch is just for caching the formatInfo, which seems
> reasonable so
> 
> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> 
> > -		const size_t numColorPlanes = info.numPlanes();
> > -		planes.resize(numColorPlanes);
> > +		planes.resize(formatInfo_->numPlanes());
> >  		const FileDescriptor &fd = planes[0].fd;
> >  		size_t offset = 0;
> > -		for (size_t i = 0; i < numColorPlanes; ++i) {
> > +		for (size_t i = 0; i < planes.size(); ++i) {
> >  			planes[i].fd = fd;
> >  			planes[i].offset = offset;
> >  
> >  			/* \todo Take the V4L2 stride into account */
> > -			planes[i].length = info.planeSize(format_.size, i);
> > +			planes[i].length = formatInfo_->planeSize(format_.size, i);
> >  			offset += planes[i].length;
> >  		}
> >  	}

Laurent Pinchart Sept. 2, 2021, 6:31 p.m. UTC | #13

Hi Kieran,

On Thu, Sep 02, 2021 at 12:17:31PM +0100, Kieran Bingham wrote:
> On 02/09/2021 05:23, Laurent Pinchart wrote:
> > When dequeueing a buffer from a V4L2VideoDevice, the number of planes in
> > the FrameBuffer may not match the number of V4L2 buffer planes if the
> > PixelFormat is multi-planar (has multiple colour planes) and the V4L2
> > format is single-planar (has a single buffer plane). In this case, we
> > need to split the single V4L2 buffer plane into FrameBuffer planes. Do
> > so, and add checks to reject invalid V4L2 buffers in case of a driver
> > issue.
> > 
> > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > ---
> >  src/libcamera/v4l2_videodevice.cpp | 54 ++++++++++++++++++++++++------
> >  1 file changed, 43 insertions(+), 11 deletions(-)
> > 
> > diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> > index 79cb792117d5..b80b9038a914 100644
> > --- a/src/libcamera/v4l2_videodevice.cpp
> > +++ b/src/libcamera/v4l2_videodevice.cpp
> > @@ -7,6 +7,7 @@
> >  
> >  #include "libcamera/internal/v4l2_videodevice.h"
> >  
> > +#include <algorithm>
> >  #include <array>
> >  #include <fcntl.h>
> >  #include <iomanip>
> > @@ -1637,18 +1638,49 @@ FrameBuffer *V4L2VideoDevice::dequeueBuffer()
> >  	buffer->metadata_.timestamp = buf.timestamp.tv_sec * 1000000000ULL
> >  				    + buf.timestamp.tv_usec * 1000ULL;
> >  
> > -	if (multiPlanar) {
> > -		if (buf.length > buffer->metadata_.planes().size()) {
> > -			LOG(V4L2, Error)
> > -				<< "Invalid number of planes (" << buf.length
> > -				<< " != " << buffer->metadata_.planes().size() << ")";
> > -			return nullptr;
> > +	if (V4L2_TYPE_IS_CAPTURE(buf.type)) {
> 
> Should we swap this around and return the buffer early to lower the
> indents below?
> 
>  if (!V4L2_TYPE_IS_CAPTURE(buf.type)) {
> 	/* Output buffer is completed */
> 	return buffer;
>  }

Yes, good point.

> > +		unsigned int numV4l2Planes = multiPlanar ? buf.length : 1;
> > +		FrameMetadata &metadata = buffer->metadata_;
> > +
> > +		if (numV4l2Planes != metadata.planes().size()) {
> > +			/*
> > +			 * If we have a multi-planar buffer with a V4L2
> > +			 * single-planar format, split the V4L2 buffer across
> > +			 * the buffer planes. Only the last plane may have less
> > +			 * bytes used than its length.
> > +			 */
> > +			if (numV4l2Planes != 1) {
> > +				LOG(V4L2, Error)
> > +					<< "Invalid number of planes (" << numV4l2Planes
> > +					<< " != " << metadata.planes().size() << ")";
> > +				return nullptr;
> > +			}
> > +
> > +			unsigned int bytesused = multiPlanar ? planes[0].bytesused
> > +					       : buf.bytesused;
> > +
> > +			for (auto [i, plane] : utils::enumerate(buffer->planes())) {
> > +				if (!bytesused) {
> > +					LOG(V4L2, Error)
> > +						<< "Dequeued buffer is too small";
> 
> Presumably if we got here - it's likely something (hardware) already
> overwrote memory it likely didn't own?

It likely means something bad happened, yes. It's a bit of defensive
programming, to output a clear error message if the kernel does
something bad, but also to possibly catch bugs in the implementation of
this function.

> > +					return nullptr;
> 
> I'm still very weary of returning nullptrs on dequeue when a buffer is
> removed from queuedBuffers_
> 
> 
> Wouldn't it be better to return the buffer, but mark it as corrupted or
> invalid or such ?

Yes, I'll do that.

> > +				}
> > +
> > +				metadata.planes()[i].bytesused =
> > +					std::min(plane.length, bytesused);
> > +				bytesused -= metadata.planes()[i].bytesused;
> > +			}
> > +		} else if (multiPlanar) {
> > +			/*
> > +			 * If we use the multi-planar API, fill in the planes.
> > +			 * The number of planes in the frame buffer and in the
> > +			 * V4L2 buffer is guaranteed to be equal at this point.
> > +			 */
> > +			for (unsigned int i = 0; i < numV4l2Planes; ++i)
> > +				metadata.planes()[i].bytesused = planes[i].bytesused;
> > +		} else {
> > +			metadata.planes()[0].bytesused = buf.bytesused;
> >  		}
> > -
> > -		for (unsigned int nplane = 0; nplane < buf.length; nplane++)
> > -			buffer->metadata_.planes()[nplane].bytesused = planes[nplane].bytesused;
> > -	} else {
> > -		buffer->metadata_.planes()[0].bytesused = buf.bytesused;
> >  	}
> >  
> >  	return buffer;
> >

Laurent Pinchart Sept. 2, 2021, 6:36 p.m. UTC | #14

Hi Kieran,

On Thu, Sep 02, 2021 at 10:23:48AM +0100, Kieran Bingham wrote:
> On 02/09/2021 05:22, Laurent Pinchart wrote:
> > V4L2 describes multi-planar formats with different 4CCs depending on
> > whether or not the planes are stored contiguously in memory. Support
> > this when translating between PixelFormat and V4L2PixelFormat.
> > 
> > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > ---
> >  include/libcamera/internal/formats.h |   2 +-
> >  src/libcamera/formats.cpp            | 141 ++++++++++++++++-----------
> >  src/libcamera/pipeline/ipu3/cio2.cpp |   2 +-
> >  src/libcamera/v4l2_pixelformat.cpp   |  11 ++-
> >  src/v4l2/v4l2_camera_proxy.cpp       |   6 +-
> >  5 files changed, 99 insertions(+), 63 deletions(-)
> > 
> > diff --git a/include/libcamera/internal/formats.h b/include/libcamera/internal/formats.h
> > index 8f314e99889b..f9ce5f0834fa 100644
> > --- a/include/libcamera/internal/formats.h
> > +++ b/include/libcamera/internal/formats.h
> > @@ -55,7 +55,7 @@ public:
> >  	/* \todo Add support for non-contiguous memory planes */
> >  	const char *name;
> >  	PixelFormat format;
> > -	V4L2PixelFormat v4l2Format;
> > +	std::array<V4L2PixelFormat, 2> v4l2Format;
> 
> As discussed below, I'm curious if a struct would be clearer here and
> help prevent bugs later...
> 
> 	struct {
> 		V4L2PixelFormat planar;
> 		V4L2PixelFormat multiplanar;
> 	} v4l2Format;
> 
> >  	unsigned int bitsPerPixel;
> >  	enum ColourEncoding colourEncoding;
> >  	bool packed;
> > diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
> > index 76be93bc1c5c..ce7c5a2d267e 100644
> > --- a/src/libcamera/formats.cpp
> > +++ b/src/libcamera/formats.cpp
> > @@ -56,7 +56,14 @@ LOG_DEFINE_CATEGORY(Formats)
> >   * \brief The PixelFormat described by this instance
> >   *
> >   * \var PixelFormatInfo::v4l2Format
> > - * \brief The V4L2 pixel format corresponding to the PixelFormat
> > + * \brief The V4L2 pixel formats corresponding to the PixelFormat
> > + *
> > + * Multiple V4L2 formats may exist for one PixelFormat when the format uses
> > + * multiple planes, as V4L2 defines separate 4CCs for contiguous and separate
> > + * planes formats. The two entries in the array store the contiguous and
> > + * non-contiguous V4L2 formats respectively. If the PixelFormat isn't a
> > + * multiplanar format, or if no corresponding non-contiguous V4L2 format
> > + * exists, the second entry is invalid.
> >   *
> >   * \var PixelFormatInfo::bitsPerPixel
> >   * \brief The average number of bits per pixel
> > @@ -149,7 +156,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::RGB565, {
> >  		.name = "RGB565",
> >  		.format = formats::RGB565,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGB565),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGB565), },
> >  		.bitsPerPixel = 16,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> >  		.packed = false,
> > @@ -159,7 +166,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::RGB565_BE, {
> >  		.name = "RGB565_BE",
> >  		.format = formats::RGB565_BE,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGB565X),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGB565X), },
> >  		.bitsPerPixel = 16,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> >  		.packed = false,
> > @@ -169,7 +176,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::BGR888, {
> >  		.name = "BGR888",
> >  		.format = formats::BGR888,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGB24),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGB24), },
> >  		.bitsPerPixel = 24,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> >  		.packed = false,
> > @@ -179,7 +186,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::RGB888, {
> >  		.name = "RGB888",
> >  		.format = formats::RGB888,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_BGR24),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_BGR24), },
> >  		.bitsPerPixel = 24,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> >  		.packed = false,
> > @@ -189,7 +196,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::XRGB8888, {
> >  		.name = "XRGB8888",
> >  		.format = formats::XRGB8888,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_XBGR32),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_XBGR32), },
> >  		.bitsPerPixel = 32,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> >  		.packed = false,
> > @@ -199,7 +206,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::XBGR8888, {
> >  		.name = "XBGR8888",
> >  		.format = formats::XBGR8888,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGBX32),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGBX32), },
> >  		.bitsPerPixel = 32,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> >  		.packed = false,
> > @@ -209,7 +216,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::BGRX8888, {
> >  		.name = "BGRX8888",
> >  		.format = formats::BGRX8888,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_XRGB32),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_XRGB32), },
> >  		.bitsPerPixel = 32,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> >  		.packed = false,
> > @@ -219,7 +226,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::ABGR8888, {
> >  		.name = "ABGR8888",
> >  		.format = formats::ABGR8888,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGBA32),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGBA32), },
> >  		.bitsPerPixel = 32,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> >  		.packed = false,
> > @@ -229,7 +236,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::ARGB8888, {
> >  		.name = "ARGB8888",
> >  		.format = formats::ARGB8888,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_ABGR32),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_ABGR32), },
> >  		.bitsPerPixel = 32,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> >  		.packed = false,
> > @@ -239,7 +246,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::BGRA8888, {
> >  		.name = "BGRA8888",
> >  		.format = formats::BGRA8888,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_ARGB32),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_ARGB32), },
> >  		.bitsPerPixel = 32,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> >  		.packed = false,
> > @@ -249,7 +256,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::RGBA8888, {
> >  		.name = "RGBA8888",
> >  		.format = formats::RGBA8888,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_BGRA32),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_BGRA32), },
> >  		.bitsPerPixel = 32,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> >  		.packed = false,
> > @@ -261,7 +268,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::YUYV, {
> >  		.name = "YUYV",
> >  		.format = formats::YUYV,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YUYV),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_YUYV), },
> >  		.bitsPerPixel = 16,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> >  		.packed = false,
> > @@ -271,7 +278,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::YVYU, {
> >  		.name = "YVYU",
> >  		.format = formats::YVYU,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YVYU),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_YVYU), },
> >  		.bitsPerPixel = 16,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> >  		.packed = false,
> > @@ -281,7 +288,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::UYVY, {
> >  		.name = "UYVY",
> >  		.format = formats::UYVY,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_UYVY),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_UYVY), },
> >  		.bitsPerPixel = 16,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> >  		.packed = false,
> > @@ -291,7 +298,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::VYUY, {
> >  		.name = "VYUY",
> >  		.format = formats::VYUY,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_VYUY),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_VYUY), },
> >  		.bitsPerPixel = 16,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> >  		.packed = false,
> > @@ -303,7 +310,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::NV12, {
> >  		.name = "NV12",
> >  		.format = formats::NV12,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV12),
> > +		.v4l2Format = {
> > +			V4L2PixelFormat(V4L2_PIX_FMT_NV12),
> > +			V4L2PixelFormat(V4L2_PIX_FMT_NV12M),
> > +		},
> >  		.bitsPerPixel = 12,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> >  		.packed = false,
> > @@ -313,7 +323,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::NV21, {
> >  		.name = "NV21",
> >  		.format = formats::NV21,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV21),
> > +		.v4l2Format = {
> > +			V4L2PixelFormat(V4L2_PIX_FMT_NV21),
> > +			V4L2PixelFormat(V4L2_PIX_FMT_NV21M),
> > +		},
> >  		.bitsPerPixel = 12,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> >  		.packed = false,
> > @@ -323,7 +336,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::NV16, {
> >  		.name = "NV16",
> >  		.format = formats::NV16,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV16),
> > +		.v4l2Format = {
> > +			V4L2PixelFormat(V4L2_PIX_FMT_NV16),
> > +			V4L2PixelFormat(V4L2_PIX_FMT_NV16M),
> > +		},
> >  		.bitsPerPixel = 16,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> >  		.packed = false,
> > @@ -333,7 +349,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::NV61, {
> >  		.name = "NV61",
> >  		.format = formats::NV61,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV61),
> > +		.v4l2Format = {
> > +			V4L2PixelFormat(V4L2_PIX_FMT_NV61),
> > +			V4L2PixelFormat(V4L2_PIX_FMT_NV61M),
> > +		},
> >  		.bitsPerPixel = 16,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> >  		.packed = false,
> > @@ -343,7 +362,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::NV24, {
> >  		.name = "NV24",
> >  		.format = formats::NV24,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV24),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_NV24), },
> >  		.bitsPerPixel = 24,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> >  		.packed = false,
> > @@ -353,7 +372,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::NV42, {
> >  		.name = "NV42",
> >  		.format = formats::NV42,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV42),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_NV42), },
> >  		.bitsPerPixel = 24,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> >  		.packed = false,
> > @@ -363,7 +382,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::YUV420, {
> >  		.name = "YUV420",
> >  		.format = formats::YUV420,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YUV420),
> > +		.v4l2Format = {
> > +			V4L2PixelFormat(V4L2_PIX_FMT_YUV420),
> > +			V4L2PixelFormat(V4L2_PIX_FMT_YUV420M),
> > +		},
> >  		.bitsPerPixel = 12,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> >  		.packed = false,
> > @@ -373,7 +395,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::YVU420, {
> >  		.name = "YVU420",
> >  		.format = formats::YVU420,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YVU420),
> > +		.v4l2Format = {
> > +			V4L2PixelFormat(V4L2_PIX_FMT_YVU420),
> > +			V4L2PixelFormat(V4L2_PIX_FMT_YVU420M),
> > +		},
> >  		.bitsPerPixel = 12,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> >  		.packed = false,
> > @@ -383,7 +408,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::YUV422, {
> >  		.name = "YUV422",
> >  		.format = formats::YUV422,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YUV422P),
> > +		.v4l2Format = {
> > +			V4L2PixelFormat(V4L2_PIX_FMT_YUV422P),
> > +			V4L2PixelFormat(V4L2_PIX_FMT_YUV422M),
> > +		},
> >  		.bitsPerPixel = 16,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> >  		.packed = false,
> > @@ -395,7 +423,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::R8, {
> >  		.name = "R8",
> >  		.format = formats::R8,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_GREY),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_GREY), },
> >  		.bitsPerPixel = 8,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> >  		.packed = false,
> > @@ -407,7 +435,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SBGGR8, {
> >  		.name = "SBGGR8",
> >  		.format = formats::SBGGR8,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR8),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR8), },
> >  		.bitsPerPixel = 8,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = false,
> > @@ -417,7 +445,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SGBRG8, {
> >  		.name = "SGBRG8",
> >  		.format = formats::SGBRG8,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG8),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG8), },
> >  		.bitsPerPixel = 8,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = false,
> > @@ -427,7 +455,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SGRBG8, {
> >  		.name = "SGRBG8",
> >  		.format = formats::SGRBG8,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG8),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG8), },
> >  		.bitsPerPixel = 8,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = false,
> > @@ -437,7 +465,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SRGGB8, {
> >  		.name = "SRGGB8",
> >  		.format = formats::SRGGB8,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB8),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB8), },
> >  		.bitsPerPixel = 8,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = false,
> > @@ -447,7 +475,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SBGGR10, {
> >  		.name = "SBGGR10",
> >  		.format = formats::SBGGR10,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10), },
> >  		.bitsPerPixel = 10,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = false,
> > @@ -457,7 +485,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SGBRG10, {
> >  		.name = "SGBRG10",
> >  		.format = formats::SGBRG10,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10), },
> >  		.bitsPerPixel = 10,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = false,
> > @@ -467,7 +495,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SGRBG10, {
> >  		.name = "SGRBG10",
> >  		.format = formats::SGRBG10,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10), },
> >  		.bitsPerPixel = 10,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = false,
> > @@ -477,7 +505,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SRGGB10, {
> >  		.name = "SRGGB10",
> >  		.format = formats::SRGGB10,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10), },
> >  		.bitsPerPixel = 10,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = false,
> > @@ -487,7 +515,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SBGGR10_CSI2P, {
> >  		.name = "SBGGR10_CSI2P",
> >  		.format = formats::SBGGR10_CSI2P,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10P),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10P), },
> >  		.bitsPerPixel = 10,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = true,
> > @@ -497,7 +525,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SGBRG10_CSI2P, {
> >  		.name = "SGBRG10_CSI2P",
> >  		.format = formats::SGBRG10_CSI2P,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10P),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10P), },
> >  		.bitsPerPixel = 10,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = true,
> > @@ -507,7 +535,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SGRBG10_CSI2P, {
> >  		.name = "SGRBG10_CSI2P",
> >  		.format = formats::SGRBG10_CSI2P,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10P),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10P), },
> >  		.bitsPerPixel = 10,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = true,
> > @@ -517,7 +545,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SRGGB10_CSI2P, {
> >  		.name = "SRGGB10_CSI2P",
> >  		.format = formats::SRGGB10_CSI2P,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10P),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10P), },
> >  		.bitsPerPixel = 10,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = true,
> > @@ -527,7 +555,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SBGGR12, {
> >  		.name = "SBGGR12",
> >  		.format = formats::SBGGR12,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12), },
> >  		.bitsPerPixel = 12,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = false,
> > @@ -537,7 +565,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SGBRG12, {
> >  		.name = "SGBRG12",
> >  		.format = formats::SGBRG12,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12), },
> >  		.bitsPerPixel = 12,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = false,
> > @@ -547,7 +575,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SGRBG12, {
> >  		.name = "SGRBG12",
> >  		.format = formats::SGRBG12,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12), },
> >  		.bitsPerPixel = 12,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = false,
> > @@ -557,7 +585,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SRGGB12, {
> >  		.name = "SRGGB12",
> >  		.format = formats::SRGGB12,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12), },
> >  		.bitsPerPixel = 12,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = false,
> > @@ -567,7 +595,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SBGGR12_CSI2P, {
> >  		.name = "SBGGR12_CSI2P",
> >  		.format = formats::SBGGR12_CSI2P,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12P),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12P), },
> >  		.bitsPerPixel = 12,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = true,
> > @@ -577,7 +605,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SGBRG12_CSI2P, {
> >  		.name = "SGBRG12_CSI2P",
> >  		.format = formats::SGBRG12_CSI2P,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12P),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12P), },
> >  		.bitsPerPixel = 12,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = true,
> > @@ -587,7 +615,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SGRBG12_CSI2P, {
> >  		.name = "SGRBG12_CSI2P",
> >  		.format = formats::SGRBG12_CSI2P,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12P),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12P), },
> >  		.bitsPerPixel = 12,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = true,
> > @@ -597,7 +625,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SRGGB12_CSI2P, {
> >  		.name = "SRGGB12_CSI2P",
> >  		.format = formats::SRGGB12_CSI2P,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12P),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12P), },
> >  		.bitsPerPixel = 12,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = true,
> > @@ -607,7 +635,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SBGGR16, {
> >  		.name = "SBGGR16",
> >  		.format = formats::SBGGR16,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR16),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR16), },
> >  		.bitsPerPixel = 16,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = false,
> > @@ -617,7 +645,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SGBRG16, {
> >  		.name = "SGBRG16",
> >  		.format = formats::SGBRG16,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG16),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG16), },
> >  		.bitsPerPixel = 16,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = false,
> > @@ -627,7 +655,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SGRBG16, {
> >  		.name = "SGRBG16",
> >  		.format = formats::SGRBG16,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG16),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG16), },
> >  		.bitsPerPixel = 16,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = false,
> > @@ -637,7 +665,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SRGGB16, {
> >  		.name = "SRGGB16",
> >  		.format = formats::SRGGB16,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB16),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB16), },
> >  		.bitsPerPixel = 16,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = false,
> > @@ -647,7 +675,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SBGGR10_IPU3, {
> >  		.name = "SBGGR10_IPU3",
> >  		.format = formats::SBGGR10_IPU3,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SBGGR10),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SBGGR10), },
> >  		.bitsPerPixel = 10,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = true,
> > @@ -658,7 +686,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SGBRG10_IPU3, {
> >  		.name = "SGBRG10_IPU3",
> >  		.format = formats::SGBRG10_IPU3,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGBRG10),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGBRG10), },
> >  		.bitsPerPixel = 10,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = true,
> > @@ -668,7 +696,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SGRBG10_IPU3, {
> >  		.name = "SGRBG10_IPU3",
> >  		.format = formats::SGRBG10_IPU3,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGRBG10),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGRBG10), },
> >  		.bitsPerPixel = 10,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = true,
> > @@ -678,7 +706,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::SRGGB10_IPU3, {
> >  		.name = "SRGGB10_IPU3",
> >  		.format = formats::SRGGB10_IPU3,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SRGGB10),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SRGGB10), },
> >  		.bitsPerPixel = 10,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> >  		.packed = true,
> > @@ -690,7 +718,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	{ formats::MJPEG, {
> >  		.name = "MJPEG",
> >  		.format = formats::MJPEG,
> > -		.v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_MJPEG),
> > +		.v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_MJPEG), },
> >  		.bitsPerPixel = 0,
> >  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> >  		.packed = false,
> > @@ -736,7 +764,8 @@ const PixelFormatInfo &PixelFormatInfo::info(const V4L2PixelFormat &format)
> >  {
> >  	const auto &info = std::find_if(pixelFormatInfo.begin(), pixelFormatInfo.end(),
> >  					[format](auto pair) {
> > -						return pair.second.v4l2Format == format;
> > +						return pair.second.v4l2Format[0] == format ||
> > +						       pair.second.v4l2Format[1] == format;
> >  					});
> >  	if (info == pixelFormatInfo.end())
> >  		return pixelFormatInfoInvalid;
> > diff --git a/src/libcamera/pipeline/ipu3/cio2.cpp b/src/libcamera/pipeline/ipu3/cio2.cpp
> > index 9cedcb5b2879..5a9cffc80c8d 100644
> > --- a/src/libcamera/pipeline/ipu3/cio2.cpp
> > +++ b/src/libcamera/pipeline/ipu3/cio2.cpp
> > @@ -205,7 +205,7 @@ int CIO2Device::configure(const Size &size, V4L2DeviceFormat *outputFormat)
> >  
> >  	const PixelFormatInfo &info = PixelFormatInfo::info(itInfo->second);
> >  
> > -	outputFormat->fourcc = info.v4l2Format;
> > +	outputFormat->fourcc = info.v4l2Format[0];
> >  	outputFormat->size = sensorFormat.size;
> >  	outputFormat->planesCount = 1;
> >  
> > diff --git a/src/libcamera/v4l2_pixelformat.cpp b/src/libcamera/v4l2_pixelformat.cpp
> > index 93fc4446cc64..93ead8928ed7 100644
> > --- a/src/libcamera/v4l2_pixelformat.cpp
> > +++ b/src/libcamera/v4l2_pixelformat.cpp
> > @@ -67,12 +67,19 @@ const std::map<V4L2PixelFormat, PixelFormat> vpf2pf{
> >  
> >  	/* YUV planar formats. */
> >  	{ V4L2PixelFormat(V4L2_PIX_FMT_NV16), formats::NV16 },
> > +	{ V4L2PixelFormat(V4L2_PIX_FMT_NV16M), formats::NV16 },
> >  	{ V4L2PixelFormat(V4L2_PIX_FMT_NV61), formats::NV61 },
> > +	{ V4L2PixelFormat(V4L2_PIX_FMT_NV61M), formats::NV61 },
> >  	{ V4L2PixelFormat(V4L2_PIX_FMT_NV12), formats::NV12 },
> > +	{ V4L2PixelFormat(V4L2_PIX_FMT_NV12M), formats::NV12 },
> >  	{ V4L2PixelFormat(V4L2_PIX_FMT_NV21), formats::NV21 },
> > +	{ V4L2PixelFormat(V4L2_PIX_FMT_NV21M), formats::NV21 },
> >  	{ V4L2PixelFormat(V4L2_PIX_FMT_YUV420), formats::YUV420 },
> > +	{ V4L2PixelFormat(V4L2_PIX_FMT_YUV420M), formats::YUV420 },
> >  	{ V4L2PixelFormat(V4L2_PIX_FMT_YVU420), formats::YVU420 },
> > +	{ V4L2PixelFormat(V4L2_PIX_FMT_YVU420M), formats::YVU420 },
> >  	{ V4L2PixelFormat(V4L2_PIX_FMT_YUV422P), formats::YUV422 },
> > +	{ V4L2PixelFormat(V4L2_PIX_FMT_YUV422M), formats::YUV422 },
> 
> I'm worried that we're losing information here.
> But presumably it's handled by whether we use mplane or not on the
> device...?

We don't expose to applications whether the camera requires contiguous
buffers or can support non-contiguous buffers. We don't otherwise lose
information, as we pick a specific V4L2 format from the PixelFormat,
based on an explicit selection of multiplanar or singleplanar V4L2
format by the pipeline handler.

> >  	/* Greyscale formats. */
> >  	{ V4L2PixelFormat(V4L2_PIX_FMT_GREY), formats::R8 },
> > @@ -202,13 +209,13 @@ PixelFormat V4L2PixelFormat::toPixelFormat() const
> >   * \return The V4L2PixelFormat corresponding to \a pixelFormat
> >   */
> >  V4L2PixelFormat V4L2PixelFormat::fromPixelFormat(const PixelFormat &pixelFormat,
> > -						 [[maybe_unused]] bool multiplanar)
> > +						 bool multiplanar)
> >  {
> >  	const PixelFormatInfo &info = PixelFormatInfo::info(pixelFormat);
> >  	if (!info.isValid())
> >  		return V4L2PixelFormat();
> >  
> > -	return info.v4l2Format;
> > +	return info.v4l2Format[multiplanar ? 1 : 0];
> 
> I was going to say, a struct with named fields of 'planar', and
> 'multiplanar' might be a better, though I like the simplicity of this bit.

Would be easy to rewrite though.

> >  }
> >  
> >  } /* namespace libcamera */
> > diff --git a/src/v4l2/v4l2_camera_proxy.cpp b/src/v4l2/v4l2_camera_proxy.cpp
> > index 7682c4bddf90..8d8ee395954f 100644
> > --- a/src/v4l2/v4l2_camera_proxy.cpp
> > +++ b/src/v4l2/v4l2_camera_proxy.cpp
> > @@ -169,7 +169,7 @@ void V4L2CameraProxy::setFmtFromConfig(const StreamConfiguration &streamConfig)
> >  
> >  	v4l2PixFormat_.width        = size.width;
> >  	v4l2PixFormat_.height       = size.height;
> > -	v4l2PixFormat_.pixelformat  = info.v4l2Format;
> > +	v4l2PixFormat_.pixelformat  = info.v4l2Format[0];
> >  	v4l2PixFormat_.field        = V4L2_FIELD_NONE;
> >  	v4l2PixFormat_.bytesperline = streamConfig.stride;
> >  	v4l2PixFormat_.sizeimage    = streamConfig.frameSize;
> > @@ -276,7 +276,7 @@ int V4L2CameraProxy::vidioc_enum_fmt(V4L2CameraFile *file, struct v4l2_fmtdesc *
> >  	/* \todo Add map from format to description. */
> >  	utils::strlcpy(reinterpret_cast<char *>(arg->description),
> >  		       "Video Format Description", sizeof(arg->description));
> > -	arg->pixelformat = PixelFormatInfo::info(format).v4l2Format;
> > +	arg->pixelformat = PixelFormatInfo::info(format).v4l2Format[0];
> >  
> >  	memset(arg->reserved, 0, sizeof(arg->reserved));
> >  
> > @@ -315,7 +315,7 @@ int V4L2CameraProxy::tryFormat(struct v4l2_format *arg)
> >  
> >  	arg->fmt.pix.width        = config.size.width;
> >  	arg->fmt.pix.height       = config.size.height;
> > -	arg->fmt.pix.pixelformat  = info.v4l2Format;
> > +	arg->fmt.pix.pixelformat  = info.v4l2Format[0];
> 
> But for occasions like here, and in the CIO2
> 	arg->fmt.pix.pixelformat  = info.v4l2Format.planar;
> 
> Would be far more descriptive over which one is being chosen, and might
> help make it easier to spot issues when debugging multiplanar format
> bugs ...

I agree. I don't like the names "planar" and "multiplanar" though, as
that's a bit ambiguous. V4L2 did a really bad job when it comes to
naming here. I'll try to think of better names after sleeping over it,
but please feel free to suggest alternatives :-) Bonus points of they're
short and have the same length :-)

> >  	arg->fmt.pix.field        = V4L2_FIELD_NONE;
> >  	arg->fmt.pix.bytesperline = config.stride;
> >  	arg->fmt.pix.sizeimage    = config.frameSize;
> >

Kieran Bingham Sept. 3, 2021, 8:24 a.m. UTC | #15

On 02/09/2021 19:07, Laurent Pinchart wrote:
> Hi Kieran,
> 
> On Thu, Sep 02, 2021 at 10:43:56AM +0100, Kieran Bingham wrote:
>> On 02/09/2021 05:22, Laurent Pinchart wrote:
>>> Multi-planar frame buffers can store their planes contiguously in
>>> memory, or split them in discontiguous memory areas. Add a private
>>> function to check in which of these two categories the frame buffer
>>> belongs. This will be used to correctly handle the differences between
>>> the V4L2 single and multi planar APIs.
>>>
>>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>>> ---
>>>  include/libcamera/internal/framebuffer.h |  2 ++
>>>  src/libcamera/framebuffer.cpp            | 45 +++++++++++++++++++++++-
>>>  2 files changed, 46 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/include/libcamera/internal/framebuffer.h b/include/libcamera/internal/framebuffer.h
>>> index 606aed2b4782..cd33c295466e 100644
>>> --- a/include/libcamera/internal/framebuffer.h
>>> +++ b/include/libcamera/internal/framebuffer.h
>>> @@ -21,9 +21,11 @@ public:
>>>  	Private();
>>>  
>>>  	void setRequest(Request *request) { request_ = request; }
>>> +	bool isContiguous() const { return isContiguous_; }
>>>  
>>>  private:
>>>  	Request *request_;
>>> +	bool isContiguous_;
>>>  };
>>>  
>>>  } /* namespace libcamera */
>>> diff --git a/src/libcamera/framebuffer.cpp b/src/libcamera/framebuffer.cpp
>>> index 53ef89bf458f..99265b44da43 100644
>>> --- a/src/libcamera/framebuffer.cpp
>>> +++ b/src/libcamera/framebuffer.cpp
>>> @@ -106,7 +106,7 @@ LOG_DEFINE_CATEGORY(Buffer)
>>>   */
>>>  
>>>  FrameBuffer::Private::Private()
>>> -	: request_(nullptr)
>>> +	: request_(nullptr), isContiguous_(true)
>>>  {
>>>  }
>>>  
>>> @@ -120,6 +120,17 @@ FrameBuffer::Private::Private()
>>>   * handlers, it is called by the pipeline handlers themselves.
>>>   */
>>>  
>>> +/**
>>> + * \fn FrameBuffer::Private::isContiguous()
>>> + * \brief Check if the frame buffer stores planes contiguously in memory
>>> + *
>>> + * Multi-planar frame buffers can store their planes contiguously in memory, or
>>> + * split them in discontiguous memory areas. This function checks in which of
>>
>> 'split them into discontiguous'
>>
>>> + * these two categories the frame buffer belongs.
>>> + *
>>> + * \return True if the planes are stored contiguously in memory, false otherwise
>>> + */
>>> +
>>>  /**
>>>   * \class FrameBuffer
>>>   * \brief Frame buffer data and its associated dynamic metadata
>>> @@ -199,9 +210,41 @@ FrameBuffer::FrameBuffer(const std::vector<Plane> &planes, unsigned int cookie)
>>>  	: Extensible(std::make_unique<Private>()), planes_(planes),
>>>  	  cookie_(cookie)
>>>  {
>>> +	unsigned int offset = 0;
>>> +
>>>  	/* \todo Remove the assertions after sufficient testing */
>>>  	for (const auto &plane : planes_)
>>>  		ASSERT(plane.offset != Plane::kInvalidOffset);
>>> +
>>> +	bool isContiguous = true;
>>> +	ino_t inode = 0;
>>> +
>>> +	for (const auto &plane : planes_) {
>>> +		if (plane.offset != offset) {
>>> +			isContiguous = false;
>>> +			break;
>>> +		}
>>> +
>>> +		/*
>>> +		 * Two different dmabuf file descriptors may still refer to the
>>> +		 * same dmabuf instance. Check this using inodes.
>>
>> Going back to the FileDescriptor::inode() extension, if that cached the
>> inode (it can't change) on either first call, or construction, couldn't
>> this whole check be simplified to:
>>
>> 		if (plane.fd.inode() != planes_[0].fd.inode()) {
>> 			isContiguous = false;
>> 			break;
>> 		}
> 
> This would call fstat() every time, while the fd comparison here is
> meant to skip the fstat() calls in case the numerical fd match. We could
> do

Did you miss the part where I said "If we cache the inode in
FileDescriptor"?

My intention was that caching there would mean we can simplify
comparisons here, without incurring repeated fstat calls.


> 		if (plane.fd != planes_[0].fd)
> 
> if we introduced an operator==(), but as explained in a reply to the
> patch that adds inode(), I'd like to handle that later, to avoid
> blocking this series.

Ok, I still don't see the harm in caching the inode within
FileDescriptor(), it can't change once read the first time.

But, it can be deferred:

Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>


>>> +		 */
>>> +		if (plane.fd.fd() != planes_[0].fd.fd()) {
>>> +			if (!inode)
>>> +				inode = planes_[0].fd.inode();
>>> +			if (plane.fd.inode() != inode) {
>>> +				isContiguous = false;
>>> +				break;
>>> +			}
>>> +		}
>>> +
>>> +		offset += plane.length;
>>> +	}
>>> +
>>> +	LOG(Buffer, Debug)
>>> +		<< "Buffer is " << (isContiguous ? "not " : "") << "contiguous";
>>> +
>>> +	_d()->isContiguous_ = isContiguous;
>>>  }
>>>  
>>>  /**
>

Kieran Bingham Sept. 3, 2021, 8:26 a.m. UTC | #16

On 02/09/2021 19:20, Laurent Pinchart wrote:
> Hi Kieran,
> 
> On Thu, Sep 02, 2021 at 11:03:47AM +0100, Kieran Bingham wrote:
>> On 02/09/2021 05:22, Laurent Pinchart wrote:
>>> The number of metadata planes should always match the number of frame
>>> buffer planes. Enforce this by making the vector private and providing
>>> accessor functions.
>>>
>>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>>> ---
>>>  include/libcamera/framebuffer.h    | 10 +++++++++-
>>>  src/cam/camera_session.cpp         |  4 ++--
>>>  src/cam/file_sink.cpp              |  2 +-
>>>  src/libcamera/framebuffer.cpp      | 12 +++++++++---
>>>  src/libcamera/v4l2_videodevice.cpp | 14 +++++++-------
>>>  src/qcam/main_window.cpp           |  2 +-
>>>  src/qcam/viewfinder_gl.cpp         |  2 +-
>>>  src/qcam/viewfinder_qt.cpp         |  2 +-
>>>  src/v4l2/v4l2_camera_proxy.cpp     |  2 +-
>>>  9 files changed, 32 insertions(+), 18 deletions(-)
>>>
>>> diff --git a/include/libcamera/framebuffer.h b/include/libcamera/framebuffer.h
>>> index fd68ed0a139d..7f2f176af691 100644
>>> --- a/include/libcamera/framebuffer.h
>>> +++ b/include/libcamera/framebuffer.h
>>> @@ -13,6 +13,7 @@
>>>  #include <vector>
>>>  
>>>  #include <libcamera/base/class.h>
>>> +#include <libcamera/base/span.h>
>>>  
>>>  #include <libcamera/file_descriptor.h>
>>>  
>>> @@ -34,7 +35,14 @@ struct FrameMetadata {
>>>  	Status status;
>>>  	unsigned int sequence;
>>>  	uint64_t timestamp;
>>> -	std::vector<Plane> planes;
>>> +
>>> +	Span<Plane> planes() { return planes_; }
>>> +	Span<const Plane> planes() const { return planes_; }
>>
>> Returning a span here is nice.
>>
>> This likely causes compile breakage for any external app.
>>
>> I know we're not ABI/API stable, but I wonder if we should highlight
>> when we do cause breakage somehow, perhaps in the commit message at
>> least, as we already have external users who we might want to notify.
> 
> Most changes to public header files will be ABI/API breakages at this
> point, and users will certainly notice when they get a compilation
> failure :-) What value do you think this would bring, who would read
> those messages ? I think it could be different for things that change
> the ABI without breaking compilation and that would require more than a
> recompilation to fix, there a warning seems to have more value, but I'm
> also sure we'll forget from time to time, if not most of the time.

Because I think we need to start being more aware of when we introduce
both ABI and API breakages.

At least making it explicitly noted in the commit message states that we
were aware of the breakage at that point.


>>> +
>>> +private:
>>> +	friend class FrameBuffer;
>>> +
>>> +	std::vector<Plane> planes_;
>>>  };
>>>  
>>>  class FrameBuffer final : public Extensible
>>> diff --git a/src/cam/camera_session.cpp b/src/cam/camera_session.cpp
>>> index 60d640f2b15c..32a373a99b72 100644
>>> --- a/src/cam/camera_session.cpp
>>> +++ b/src/cam/camera_session.cpp
>>> @@ -374,9 +374,9 @@ void CameraSession::processRequest(Request *request)
>>>  		     << " bytesused: ";
>>>  
>>>  		unsigned int nplane = 0;
>>> -		for (const FrameMetadata::Plane &plane : metadata.planes) {
>>> +		for (const FrameMetadata::Plane &plane : metadata.planes()) {
>>>  			info << plane.bytesused;
>>> -			if (++nplane < metadata.planes.size())
>>> +			if (++nplane < metadata.planes().size())
>>>  				info << "/";
>>>  		}
>>>  	}
>>> diff --git a/src/cam/file_sink.cpp b/src/cam/file_sink.cpp
>>> index 0b529e3eb767..0fc7d621f50b 100644
>>> --- a/src/cam/file_sink.cpp
>>> +++ b/src/cam/file_sink.cpp
>>> @@ -110,7 +110,7 @@ void FileSink::writeBuffer(const Stream *stream, FrameBuffer *buffer)
>>>  
>>>  	for (unsigned int i = 0; i < buffer->planes().size(); ++i) {
>>>  		const FrameBuffer::Plane &plane = buffer->planes()[i];
>>> -		const FrameMetadata::Plane &meta = buffer->metadata().planes[i];
>>> +		const FrameMetadata::Plane &meta = buffer->metadata().planes()[i];
>>>  
>>>  		uint8_t *data = planeData_[&plane];
>>>  		unsigned int length = std::min(meta.bytesused, plane.length);
>>> diff --git a/src/libcamera/framebuffer.cpp b/src/libcamera/framebuffer.cpp
>>> index c1529dfb0ad1..83cf7b83d182 100644
>>> --- a/src/libcamera/framebuffer.cpp
>>> +++ b/src/libcamera/framebuffer.cpp
>>> @@ -91,8 +91,14 @@ LOG_DEFINE_CATEGORY(Buffer)
>>>   */
>>>  
>>>  /**
>>> - * \var FrameMetadata::planes
>>> - * \brief Array of per-plane metadata
>>> + * \fn FrameMetadata::planes()
>>> + * \copydoc FrameMetadata::planes() const
>>> + */
>>> +
>>> +/**
>>> + * \fn FrameMetadata::planes() const
>>> + * \brief Retrieve the array of per-plane metadata
>>> + * \return The array of per-plane metadata
>>>   */
>>>  
>>>  /**
>>> @@ -210,7 +216,7 @@ FrameBuffer::FrameBuffer(const std::vector<Plane> &planes, unsigned int cookie)
>>>  	: Extensible(std::make_unique<Private>()), planes_(planes),
>>>  	  cookie_(cookie)
>>>  {
>>> -	metadata_.planes.resize(planes_.size());
>>> +	metadata_.planes_.resize(planes_.size());
>>>  
>>>  	unsigned int offset = 0;
>>>  
>>> diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
>>> index a51971879e75..82ddaed3656f 100644
>>> --- a/src/libcamera/v4l2_videodevice.cpp
>>> +++ b/src/libcamera/v4l2_videodevice.cpp
>>> @@ -1485,14 +1485,14 @@ int V4L2VideoDevice::queueBuffer(FrameBuffer *buffer)
>>>  
>>>  		if (multiPlanar) {
>>>  			unsigned int nplane = 0;
>>> -			for (const FrameMetadata::Plane &plane : metadata.planes) {
>>> +			for (const FrameMetadata::Plane &plane : metadata.planes()) {
>>>  				v4l2Planes[nplane].bytesused = plane.bytesused;
>>>  				v4l2Planes[nplane].length = buffer->planes()[nplane].length;
>>>  				nplane++;
>>>  			}
>>>  		} else {
>>> -			if (metadata.planes.size())
>>> -				buf.bytesused = metadata.planes[0].bytesused;
>>> +			if (metadata.planes().size())
>>> +				buf.bytesused = metadata.planes()[0].bytesused;
>>>  		}
>>>  
>>>  		buf.sequence = metadata.sequence;
>>> @@ -1587,17 +1587,17 @@ FrameBuffer *V4L2VideoDevice::dequeueBuffer()
>>>  				    + buf.timestamp.tv_usec * 1000ULL;
>>>  
>>>  	if (multiPlanar) {
>>> -		if (buf.length > buffer->metadata_.planes.size()) {
>>> +		if (buf.length > buffer->metadata_.planes().size()) {
>>>  			LOG(V4L2, Error)
>>>  				<< "Invalid number of planes (" << buf.length
>>> -				<< " != " << buffer->metadata_.planes.size() << ")";
>>> +				<< " != " << buffer->metadata_.planes().size() << ")";
>>>  			return nullptr;
>>>  		}
>>>  
>>>  		for (unsigned int nplane = 0; nplane < buf.length; nplane++)
>>> -			buffer->metadata_.planes[nplane].bytesused = planes[nplane].bytesused;
>>> +			buffer->metadata_.planes()[nplane].bytesused = planes[nplane].bytesused;
>>>  	} else {
>>> -		buffer->metadata_.planes[0].bytesused = buf.bytesused;
>>> +		buffer->metadata_.planes()[0].bytesused = buf.bytesused;
>>>  	}
>>>  
>>>  	return buffer;
>>> diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp
>>> index 1536b2b5bd66..ac853e360aea 100644
>>> --- a/src/qcam/main_window.cpp
>>> +++ b/src/qcam/main_window.cpp
>>> @@ -756,7 +756,7 @@ void MainWindow::processViewfinder(FrameBuffer *buffer)
>>>  
>>>  	qDebug().noquote()
>>>  		<< QString("seq: %1").arg(metadata.sequence, 6, 10, QLatin1Char('0'))
>>> -		<< "bytesused:" << metadata.planes[0].bytesused
>>> +		<< "bytesused:" << metadata.planes()[0].bytesused
>>>  		<< "timestamp:" << metadata.timestamp
>>>  		<< "fps:" << Qt::fixed << qSetRealNumberPrecision(2) << fps;
>>>  
>>> diff --git a/src/qcam/viewfinder_gl.cpp b/src/qcam/viewfinder_gl.cpp
>>> index 40226601f9fd..d2ef036974f4 100644
>>> --- a/src/qcam/viewfinder_gl.cpp
>>> +++ b/src/qcam/viewfinder_gl.cpp
>>> @@ -125,7 +125,7 @@ void ViewFinderGL::render(libcamera::FrameBuffer *buffer,
>>>  	/*
>>>  	 * \todo Get the stride from the buffer instead of computing it naively
>>>  	 */
>>> -	stride_ = buffer->metadata().planes[0].bytesused / size_.height();
>>> +	stride_ = buffer->metadata().planes()[0].bytesused / size_.height();
>>
>> Can this be obtained from the PixelFormatInfo now ?
>> or do we still not expose that to applications...
> 
> PixelFormatInfo is still internal. I'm considering exposing it, but then
> it would be good to move the V4L2-specific part somewhere else.
> 
>> Anyway, that's likely a change on top even if we could to solve the \todo.
>>
>> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
>>
>>>  	update();
>>>  	buffer_ = buffer;
>>>  }
>>> diff --git a/src/qcam/viewfinder_qt.cpp b/src/qcam/viewfinder_qt.cpp
>>> index efa1d412584b..a0bf99b0b522 100644
>>> --- a/src/qcam/viewfinder_qt.cpp
>>> +++ b/src/qcam/viewfinder_qt.cpp
>>> @@ -87,7 +87,7 @@ void ViewFinderQt::render(libcamera::FrameBuffer *buffer,
>>>  	}
>>>  
>>>  	unsigned char *memory = mem.data();
>>> -	size_t size = buffer->metadata().planes[0].bytesused;
>>> +	size_t size = buffer->metadata().planes()[0].bytesused;
>>>  
>>>  	{
>>>  		QMutexLocker locker(&mutex_);
>>> diff --git a/src/v4l2/v4l2_camera_proxy.cpp b/src/v4l2/v4l2_camera_proxy.cpp
>>> index 8d8ee395954f..68e47ee81834 100644
>>> --- a/src/v4l2/v4l2_camera_proxy.cpp
>>> +++ b/src/v4l2/v4l2_camera_proxy.cpp
>>> @@ -212,7 +212,7 @@ void V4L2CameraProxy::updateBuffers()
>>>  
>>>  		switch (fmd.status) {
>>>  		case FrameMetadata::FrameSuccess:
>>> -			buf.bytesused = fmd.planes[0].bytesused;
>>> +			buf.bytesused = fmd.planes()[0].bytesused;
>>>  			buf.field = V4L2_FIELD_NONE;
>>>  			buf.timestamp.tv_sec = fmd.timestamp / 1000000000;
>>>  			buf.timestamp.tv_usec = fmd.timestamp % 1000000;
>

Kieran Bingham Sept. 3, 2021, 8:29 a.m. UTC | #17

On 02/09/2021 19:26, Laurent Pinchart wrote:
> Hi Kieran,
> 
> On Thu, Sep 02, 2021 at 11:21:29AM +0100, Kieran Bingham wrote:
>> On 02/09/2021 05:23, Laurent Pinchart wrote:
>>> Cache the PixelFormatInfo instead of looking it up in every call to
>>> createBuffer(). This prepare for usage of the info in queueBuffer(), to
>>
>> s/prepare/prepares/
>>
>>> avoid a looking every time a buffer is queued.
>>>
>>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>>> ---
>>>  include/libcamera/internal/v4l2_videodevice.h |  1 +
>>>  src/libcamera/v4l2_videodevice.cpp            | 19 +++++++++++--------
>>>  2 files changed, 12 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/include/libcamera/internal/v4l2_videodevice.h b/include/libcamera/internal/v4l2_videodevice.h
>>> index 7a145f608a5b..6096cd609b97 100644
>>> --- a/include/libcamera/internal/v4l2_videodevice.h
>>> +++ b/include/libcamera/internal/v4l2_videodevice.h
>>> @@ -243,6 +243,7 @@ private:
>>>  
>>>  	V4L2Capability caps_;
>>>  	V4L2DeviceFormat format_;
>>> +	const PixelFormatInfo *formatInfo_;
>>>  
>>>  	enum v4l2_buf_type bufferType_;
>>>  	enum v4l2_memory memoryType_;
>>> diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
>>> index 82ddaed3656f..2d9a94c3c974 100644
>>> --- a/src/libcamera/v4l2_videodevice.cpp
>>> +++ b/src/libcamera/v4l2_videodevice.cpp
>>> @@ -482,8 +482,8 @@ const std::string V4L2DeviceFormat::toString() const
>>>   * \param[in] deviceNode The file-system path to the video device node
>>>   */
>>>  V4L2VideoDevice::V4L2VideoDevice(const std::string &deviceNode)
>>> -	: V4L2Device(deviceNode), cache_(nullptr), fdBufferNotifier_(nullptr),
>>> -	  streaming_(false)
>>> +	: V4L2Device(deviceNode), formatInfo_(nullptr), cache_(nullptr),
>>> +	  fdBufferNotifier_(nullptr), streaming_(false)
>>>  {
>>>  	/*
>>>  	 * We default to an MMAP based CAPTURE video device, however this will
>>> @@ -586,6 +586,7 @@ int V4L2VideoDevice::open()
>>>  		return ret;
>>>  	}
>>>  
>>> +	formatInfo_ = &PixelFormatInfo::info(format_.fourcc);
>>
>> I shudder a little seeing code hugging the return like that ;-)
>>
>> But it's purely subjective, same below of course.
> 
> We have various patterns indeed :-) I haven't been able to establish yet
> what my mental process to deal with these are, sometimes it bothers me
> too, but not always.
> 
>>>  	return 0;
>>>  }
>>>  
>>> @@ -681,6 +682,7 @@ int V4L2VideoDevice::open(int handle, enum v4l2_buf_type type)
>>>  		return ret;
>>>  	}
>>>  
>>> +	formatInfo_ = &PixelFormatInfo::info(format_.fourcc);
>>>  	return 0;
>>>  }
>>>  
>>> @@ -695,6 +697,8 @@ void V4L2VideoDevice::close()
>>>  	releaseBuffers();
>>>  	delete fdBufferNotifier_;
>>>  
>>> +	formatInfo_ = nullptr;
>>> +
>>>  	V4L2Device::close();
>>>  }
>>>  
>>> @@ -787,6 +791,7 @@ int V4L2VideoDevice::setFormat(V4L2DeviceFormat *format)
>>>  		return ret;
>>>  
>>>  	format_ = *format;
>>> +	formatInfo_ = &PixelFormatInfo::info(format_.fourcc);
>>
>> Looks like we were already using that pattern though ;-)
>>
>>>  	return 0;
>>>  }
>>>  
>>> @@ -1325,19 +1330,17 @@ std::unique_ptr<FrameBuffer> V4L2VideoDevice::createBuffer(unsigned int index)
>>>  		planes.push_back(std::move(plane));
>>>  	}
>>>  
>>> -	const auto &info = PixelFormatInfo::info(format_.fourcc);
>>> -	if (info.isValid() && info.numPlanes() != numPlanes) {
>>> +	if (formatInfo_->isValid() && formatInfo_->numPlanes() != numPlanes) {
>>>  		ASSERT(numPlanes == 1u);
>>
>> Not for this patch, but I guess this assert needed a
>>   \todo Multiplanar support
>>
>> around it?
> 
> No, when formatInfo_->numPlanes() != numPlanes, it means that we have a
> multi-planar FrameBuffer with a single-planar V4L2 buffer format (e.g.
> V4L2_PIX_FMT_NV12 as opposed to V4L2_PIX_FMT_NV12M). numPlanes must be 1
> in that case, otherwise it's a kernel bug.

As you're updating the conditional there anyway, could you add a short
brief to explain that before the ASSERT please?

I don't think that extra context is clear from just the if statement.
(Perhaps there is more context outside of the hunk maybe, but it's not
clear from just this diff).

> 
>> However, this patch is just for caching the formatInfo, which seems
>> reasonable so
>>
>> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
>>
>>> -		const size_t numColorPlanes = info.numPlanes();
>>> -		planes.resize(numColorPlanes);
>>> +		planes.resize(formatInfo_->numPlanes());
>>>  		const FileDescriptor &fd = planes[0].fd;
>>>  		size_t offset = 0;
>>> -		for (size_t i = 0; i < numColorPlanes; ++i) {
>>> +		for (size_t i = 0; i < planes.size(); ++i) {
>>>  			planes[i].fd = fd;
>>>  			planes[i].offset = offset;
>>>  
>>>  			/* \todo Take the V4L2 stride into account */
>>> -			planes[i].length = info.planeSize(format_.size, i);
>>> +			planes[i].length = formatInfo_->planeSize(format_.size, i);
>>>  			offset += planes[i].length;
>>>  		}
>>>  	}
>

Laurent Pinchart Sept. 3, 2021, 8:39 a.m. UTC | #18

Hi Kieran,

On Fri, Sep 03, 2021 at 09:24:02AM +0100, Kieran Bingham wrote:
> On 02/09/2021 19:07, Laurent Pinchart wrote:
> > On Thu, Sep 02, 2021 at 10:43:56AM +0100, Kieran Bingham wrote:
> >> On 02/09/2021 05:22, Laurent Pinchart wrote:
> >>> Multi-planar frame buffers can store their planes contiguously in
> >>> memory, or split them in discontiguous memory areas. Add a private
> >>> function to check in which of these two categories the frame buffer
> >>> belongs. This will be used to correctly handle the differences between
> >>> the V4L2 single and multi planar APIs.
> >>>
> >>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> >>> ---
> >>>  include/libcamera/internal/framebuffer.h |  2 ++
> >>>  src/libcamera/framebuffer.cpp            | 45 +++++++++++++++++++++++-
> >>>  2 files changed, 46 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/include/libcamera/internal/framebuffer.h b/include/libcamera/internal/framebuffer.h
> >>> index 606aed2b4782..cd33c295466e 100644
> >>> --- a/include/libcamera/internal/framebuffer.h
> >>> +++ b/include/libcamera/internal/framebuffer.h
> >>> @@ -21,9 +21,11 @@ public:
> >>>  	Private();
> >>>  
> >>>  	void setRequest(Request *request) { request_ = request; }
> >>> +	bool isContiguous() const { return isContiguous_; }
> >>>  
> >>>  private:
> >>>  	Request *request_;
> >>> +	bool isContiguous_;
> >>>  };
> >>>  
> >>>  } /* namespace libcamera */
> >>> diff --git a/src/libcamera/framebuffer.cpp b/src/libcamera/framebuffer.cpp
> >>> index 53ef89bf458f..99265b44da43 100644
> >>> --- a/src/libcamera/framebuffer.cpp
> >>> +++ b/src/libcamera/framebuffer.cpp
> >>> @@ -106,7 +106,7 @@ LOG_DEFINE_CATEGORY(Buffer)
> >>>   */
> >>>  
> >>>  FrameBuffer::Private::Private()
> >>> -	: request_(nullptr)
> >>> +	: request_(nullptr), isContiguous_(true)
> >>>  {
> >>>  }
> >>>  
> >>> @@ -120,6 +120,17 @@ FrameBuffer::Private::Private()
> >>>   * handlers, it is called by the pipeline handlers themselves.
> >>>   */
> >>>  
> >>> +/**
> >>> + * \fn FrameBuffer::Private::isContiguous()
> >>> + * \brief Check if the frame buffer stores planes contiguously in memory
> >>> + *
> >>> + * Multi-planar frame buffers can store their planes contiguously in memory, or
> >>> + * split them in discontiguous memory areas. This function checks in which of
> >>
> >> 'split them into discontiguous'
> >>
> >>> + * these two categories the frame buffer belongs.
> >>> + *
> >>> + * \return True if the planes are stored contiguously in memory, false otherwise
> >>> + */
> >>> +
> >>>  /**
> >>>   * \class FrameBuffer
> >>>   * \brief Frame buffer data and its associated dynamic metadata
> >>> @@ -199,9 +210,41 @@ FrameBuffer::FrameBuffer(const std::vector<Plane> &planes, unsigned int cookie)
> >>>  	: Extensible(std::make_unique<Private>()), planes_(planes),
> >>>  	  cookie_(cookie)
> >>>  {
> >>> +	unsigned int offset = 0;
> >>> +
> >>>  	/* \todo Remove the assertions after sufficient testing */
> >>>  	for (const auto &plane : planes_)
> >>>  		ASSERT(plane.offset != Plane::kInvalidOffset);
> >>> +
> >>> +	bool isContiguous = true;
> >>> +	ino_t inode = 0;
> >>> +
> >>> +	for (const auto &plane : planes_) {
> >>> +		if (plane.offset != offset) {
> >>> +			isContiguous = false;
> >>> +			break;
> >>> +		}
> >>> +
> >>> +		/*
> >>> +		 * Two different dmabuf file descriptors may still refer to the
> >>> +		 * same dmabuf instance. Check this using inodes.
> >>
> >> Going back to the FileDescriptor::inode() extension, if that cached the
> >> inode (it can't change) on either first call, or construction, couldn't
> >> this whole check be simplified to:
> >>
> >> 		if (plane.fd.inode() != planes_[0].fd.inode()) {
> >> 			isContiguous = false;
> >> 			break;
> >> 		}
> > 
> > This would call fstat() every time, while the fd comparison here is
> > meant to skip the fstat() calls in case the numerical fd match. We could
> > do
> 
> Did you miss the part where I said "If we cache the inode in
> FileDescriptor"?

No I didn't :-)

> My intention was that caching there would mean we can simplify
> comparisons here, without incurring repeated fstat calls.

If the numerical fds are identical, there's no need to compare the
inodes, and we can avoid calling fstat() at all.

> > 		if (plane.fd != planes_[0].fd)
> > 
> > if we introduced an operator==(), but as explained in a reply to the
> > patch that adds inode(), I'd like to handle that later, to avoid
> > blocking this series.
> 
> Ok, I still don't see the harm in caching the inode within
> FileDescriptor(), it can't change once read the first time.

What we'd need to do in operator==() is to first compare the numerical
fds, and only compare the inodes if the fds differ. Technically that's
not an issue, but it requires deciding what the semantics of equality is
as explaining in a separate e-mail, and that's the part that I'd like to
defer to avoid delaying the offset fixes.

> But, it can be deferred:
> 
> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> 
> >>> +		 */
> >>> +		if (plane.fd.fd() != planes_[0].fd.fd()) {
> >>> +			if (!inode)
> >>> +				inode = planes_[0].fd.inode();
> >>> +			if (plane.fd.inode() != inode) {
> >>> +				isContiguous = false;
> >>> +				break;
> >>> +			}
> >>> +		}
> >>> +
> >>> +		offset += plane.length;
> >>> +	}
> >>> +
> >>> +	LOG(Buffer, Debug)
> >>> +		<< "Buffer is " << (isContiguous ? "not " : "") << "contiguous";
> >>> +
> >>> +	_d()->isContiguous_ = isContiguous;
> >>>  }
> >>>  
> >>>  /**

Laurent Pinchart Sept. 3, 2021, 8:48 a.m. UTC | #19

Hi Kieran,

On Fri, Sep 03, 2021 at 09:29:40AM +0100, Kieran Bingham wrote:
> On 02/09/2021 19:26, Laurent Pinchart wrote:
> > On Thu, Sep 02, 2021 at 11:21:29AM +0100, Kieran Bingham wrote:
> >> On 02/09/2021 05:23, Laurent Pinchart wrote:
> >>> Cache the PixelFormatInfo instead of looking it up in every call to
> >>> createBuffer(). This prepare for usage of the info in queueBuffer(), to
> >>
> >> s/prepare/prepares/
> >>
> >>> avoid a looking every time a buffer is queued.
> >>>
> >>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> >>> ---
> >>>  include/libcamera/internal/v4l2_videodevice.h |  1 +
> >>>  src/libcamera/v4l2_videodevice.cpp            | 19 +++++++++++--------
> >>>  2 files changed, 12 insertions(+), 8 deletions(-)
> >>>
> >>> diff --git a/include/libcamera/internal/v4l2_videodevice.h b/include/libcamera/internal/v4l2_videodevice.h
> >>> index 7a145f608a5b..6096cd609b97 100644
> >>> --- a/include/libcamera/internal/v4l2_videodevice.h
> >>> +++ b/include/libcamera/internal/v4l2_videodevice.h
> >>> @@ -243,6 +243,7 @@ private:
> >>>  
> >>>  	V4L2Capability caps_;
> >>>  	V4L2DeviceFormat format_;
> >>> +	const PixelFormatInfo *formatInfo_;
> >>>  
> >>>  	enum v4l2_buf_type bufferType_;
> >>>  	enum v4l2_memory memoryType_;
> >>> diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> >>> index 82ddaed3656f..2d9a94c3c974 100644
> >>> --- a/src/libcamera/v4l2_videodevice.cpp
> >>> +++ b/src/libcamera/v4l2_videodevice.cpp
> >>> @@ -482,8 +482,8 @@ const std::string V4L2DeviceFormat::toString() const
> >>>   * \param[in] deviceNode The file-system path to the video device node
> >>>   */
> >>>  V4L2VideoDevice::V4L2VideoDevice(const std::string &deviceNode)
> >>> -	: V4L2Device(deviceNode), cache_(nullptr), fdBufferNotifier_(nullptr),
> >>> -	  streaming_(false)
> >>> +	: V4L2Device(deviceNode), formatInfo_(nullptr), cache_(nullptr),
> >>> +	  fdBufferNotifier_(nullptr), streaming_(false)
> >>>  {
> >>>  	/*
> >>>  	 * We default to an MMAP based CAPTURE video device, however this will
> >>> @@ -586,6 +586,7 @@ int V4L2VideoDevice::open()
> >>>  		return ret;
> >>>  	}
> >>>  
> >>> +	formatInfo_ = &PixelFormatInfo::info(format_.fourcc);
> >>
> >> I shudder a little seeing code hugging the return like that ;-)
> >>
> >> But it's purely subjective, same below of course.
> > 
> > We have various patterns indeed :-) I haven't been able to establish yet
> > what my mental process to deal with these are, sometimes it bothers me
> > too, but not always.
> > 
> >>>  	return 0;
> >>>  }
> >>>  
> >>> @@ -681,6 +682,7 @@ int V4L2VideoDevice::open(int handle, enum v4l2_buf_type type)
> >>>  		return ret;
> >>>  	}
> >>>  
> >>> +	formatInfo_ = &PixelFormatInfo::info(format_.fourcc);
> >>>  	return 0;
> >>>  }
> >>>  
> >>> @@ -695,6 +697,8 @@ void V4L2VideoDevice::close()
> >>>  	releaseBuffers();
> >>>  	delete fdBufferNotifier_;
> >>>  
> >>> +	formatInfo_ = nullptr;
> >>> +
> >>>  	V4L2Device::close();
> >>>  }
> >>>  
> >>> @@ -787,6 +791,7 @@ int V4L2VideoDevice::setFormat(V4L2DeviceFormat *format)
> >>>  		return ret;
> >>>  
> >>>  	format_ = *format;
> >>> +	formatInfo_ = &PixelFormatInfo::info(format_.fourcc);
> >>
> >> Looks like we were already using that pattern though ;-)
> >>
> >>>  	return 0;
> >>>  }
> >>>  
> >>> @@ -1325,19 +1330,17 @@ std::unique_ptr<FrameBuffer> V4L2VideoDevice::createBuffer(unsigned int index)
> >>>  		planes.push_back(std::move(plane));
> >>>  	}
> >>>  
> >>> -	const auto &info = PixelFormatInfo::info(format_.fourcc);
> >>> -	if (info.isValid() && info.numPlanes() != numPlanes) {
> >>> +	if (formatInfo_->isValid() && formatInfo_->numPlanes() != numPlanes) {
> >>>  		ASSERT(numPlanes == 1u);
> >>
> >> Not for this patch, but I guess this assert needed a
> >>   \todo Multiplanar support
> >>
> >> around it?
> > 
> > No, when formatInfo_->numPlanes() != numPlanes, it means that we have a
> > multi-planar FrameBuffer with a single-planar V4L2 buffer format (e.g.
> > V4L2_PIX_FMT_NV12 as opposed to V4L2_PIX_FMT_NV12M). numPlanes must be 1
> > in that case, otherwise it's a kernel bug.
> 
> As you're updating the conditional there anyway, could you add a short
> brief to explain that before the ASSERT please?
> 
> I don't think that extra context is clear from just the if statement.
> (Perhaps there is more context outside of the hunk maybe, but it's not
> clear from just this diff).

That's because this diff only deals with format caching :-) It's a good
point though, it's not trivial so I'll add a separate patch to document
this properly.

> >> However, this patch is just for caching the formatInfo, which seems
> >> reasonable so
> >>
> >> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> >>
> >>> -		const size_t numColorPlanes = info.numPlanes();
> >>> -		planes.resize(numColorPlanes);
> >>> +		planes.resize(formatInfo_->numPlanes());
> >>>  		const FileDescriptor &fd = planes[0].fd;
> >>>  		size_t offset = 0;
> >>> -		for (size_t i = 0; i < numColorPlanes; ++i) {
> >>> +		for (size_t i = 0; i < planes.size(); ++i) {
> >>>  			planes[i].fd = fd;
> >>>  			planes[i].offset = offset;
> >>>  
> >>>  			/* \todo Take the V4L2 stride into account */
> >>> -			planes[i].length = info.planeSize(format_.size, i);
> >>> +			planes[i].length = formatInfo_->planeSize(format_.size, i);
> >>>  			offset += planes[i].length;
> >>>  		}
> >>>  	}

Hirokazu Honda Sept. 3, 2021, 10:35 a.m. UTC | #20

Hi Laurent, thank you for the patch.

On Thu, Sep 2, 2021 at 5:49 PM Kieran Bingham
<kieran.bingham@ideasonboard.com> wrote:
>
> On 02/09/2021 05:22, Laurent Pinchart wrote:
> > Add two helpers functions to the PixelFormatInfo class to compute the
> > byte size of a given plane, taking the frame size, the stride, the
> > alignement constraints and the vertical subsampling into account.
>
> s/alignement/alignment/
>
>
> > Use the new functions through the code base to replace manual
> > implementations.
>
>
> \o/ that makes me happy.
>
> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
>
> > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > ---
> >  include/libcamera/internal/formats.h     |  4 ++
> >  src/android/mm/generic_camera_buffer.cpp | 11 +---
> >  src/android/yuv/post_processor_yuv.cpp   | 10 ++-
> >  src/libcamera/formats.cpp                | 82 ++++++++++++++++++++----
> >  src/libcamera/v4l2_videodevice.cpp       |  6 +-
> >  5 files changed, 79 insertions(+), 34 deletions(-)
> >
> > diff --git a/include/libcamera/internal/formats.h b/include/libcamera/internal/formats.h
> > index 51a8a6b8b0ae..8f314e99889b 100644
> > --- a/include/libcamera/internal/formats.h
> > +++ b/include/libcamera/internal/formats.h
> > @@ -42,6 +42,10 @@ public:
> >
> >       unsigned int stride(unsigned int width, unsigned int plane,
> >                           unsigned int align = 1) const;
> > +     unsigned int planeSize(const Size &size, unsigned int plane,
> > +                            unsigned int align = 1) const;
> > +     unsigned int planeSize(unsigned int height, unsigned int plane,
> > +                            unsigned int stride) const;
> >       unsigned int frameSize(const Size &size, unsigned int align = 1) const;
> >       unsigned int frameSize(const Size &size,
> >                              const std::array<unsigned int, 3> &strides) const;
> > diff --git a/src/android/mm/generic_camera_buffer.cpp b/src/android/mm/generic_camera_buffer.cpp
> > index 22efc4d4b13a..93aa5821e470 100644
> > --- a/src/android/mm/generic_camera_buffer.cpp
> > +++ b/src/android/mm/generic_camera_buffer.cpp
> > @@ -108,16 +108,9 @@ CameraBuffer::Private::Private([[maybe_unused]] CameraBuffer *cameraBuffer,
> >
> >       unsigned int offset = 0;
> >       for (unsigned int i = 0; i < numPlanes; ++i) {
> > -             /*
> > -              * \todo Remove if this plane size computation function is
> > -              * added to PixelFormatInfo.
> > -              */
> > -             const unsigned int vertSubSample = info.planes[i].verticalSubSampling;
> > -             const unsigned int stride = info.stride(size.width, i, 1u);
> > -             const unsigned int planeSize =
> > -                     stride * ((size.height + vertSubSample - 1) / vertSubSample);
> > +             const unsigned int planeSize = info.planeSize(size, i);
> >
> > -             planeInfo_[i].stride = stride;
> > +             planeInfo_[i].stride = info.stride(size.width, i, 1u);
> >               planeInfo_[i].offset = offset;
> >               planeInfo_[i].size = planeSize;
> >
> > diff --git a/src/android/yuv/post_processor_yuv.cpp b/src/android/yuv/post_processor_yuv.cpp
> > index 6952fc38b0ef..7b3b49609cb1 100644
> > --- a/src/android/yuv/post_processor_yuv.cpp
> > +++ b/src/android/yuv/post_processor_yuv.cpp
> > @@ -134,11 +134,9 @@ void PostProcessorYuv::calculateLengths(const StreamConfiguration &inCfg,
> >               sourceStride_[i] = inCfg.stride;
> >               destinationStride_[i] = nv12Info.stride(destinationSize_.width, i, 1);
> >
> > -             const unsigned int vertSubSample =
> > -                     nv12Info.planes[i].verticalSubSampling;
> > -             sourceLength_[i] = sourceStride_[i] *
> > -                     ((sourceSize_.height + vertSubSample - 1) / vertSubSample);
> > -             destinationLength_[i] = destinationStride_[i] *
> > -                     ((destinationSize_.height + vertSubSample - 1) / vertSubSample);
> > +             sourceLength_[i] = nv12Info.planeSize(sourceSize_.height, i,
> > +                                                   sourceStride_[i]);
> > +             destinationLength_[i] = nv12Info.planeSize(destinationSize_.height, i,
> > +                                                        destinationStride_[i]);
> >       }
> >  }
> > diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
> > index 603d88619fe0..76be93bc1c5c 100644
> > --- a/src/libcamera/formats.cpp
> > +++ b/src/libcamera/formats.cpp
> > @@ -11,6 +11,7 @@
> >  #include <errno.h>
> >
> >  #include <libcamera/base/log.h>
> > +#include <libcamera/base/utils.h>
> >
> >  #include <libcamera/formats.h>
> >
> > @@ -801,32 +802,85 @@ unsigned int PixelFormatInfo::stride(unsigned int width, unsigned int plane,
> >  }
> >
> >  /**
> > - * \brief Compute the number of bytes necessary to store a frame
> > + * \brief Compute the number of bytes necessary to store a plane of a frame
> >   * \param[in] size The size of the frame, in pixels
> > + * \param[in] plane The plane index
> >   * \param[in] align The stride alignment, in bytes (1 for default alignment)
> >   *
> > - * The frame is computed by adding the product of the line stride and the frame
> > - * height for all planes, taking subsampling and other format characteristics
> > - * into account. Additional stride alignment constraints may be specified
> > - * through the \a align parameter, and will apply to all planes. For more
> > - * complex stride constraints, use the frameSize() overloaded version that takes
> > - * an array of stride values.
> > + * The plane size is computed by multiplying the line stride and the frame
> > + * height, taking subsampling and other format characteristics into account.
> > + * Stride alignment constraints may be specified through the \a align parameter.
> >   *
> >   * \sa stride()
> >   *
> > + * \return The number of bytes necessary to store the plane, or 0 if the
> > + * PixelFormatInfo instance is not valid or the plane number isn't valid for the
> > + * format
> > + */
> > +unsigned int PixelFormatInfo::planeSize(const Size &size, unsigned int plane,
> > +                                     unsigned int align) const
> > +{
> > +     unsigned int stride = PixelFormatInfo::stride(size.width, plane, align);
> > +     if (!stride)
> > +             return 0;
> > +

The below can be replaced with
return planeSize(size.height, plane, stride);

Nice clean up!

Reviewed-by: Hirokazu Honda <hiroh@chromium.org>

-Hiro
> > +     unsigned int vertSubSample = planes[plane].verticalSubSampling;
> > +     if (!vertSubSample)
> > +             return 0;
> > +
> > +     /* stride * ceil(height / verticalSubSampling) */
> > +     return stride * ((size.height + vertSubSample - 1) / vertSubSample);
> > +}
> > +
> > +/**
> > + * \brief Compute the number of bytes necessary to store a plane of a frame
> > + * \param[in] height The height of the frame, in pixels
> > + * \param[in] plane The plane index
> > + * \param[in] stride The plane stride, in bytes
> > + *
> > + * The plane size is computed by multiplying the line stride and the frame
> > + * height, taking subsampling and other format characteristics into account.
> > + * Stride alignment constraints may be specified through the \a align parameter.
> > + *
> > + * \return The number of bytes necessary to store the plane, or 0 if the
> > + * PixelFormatInfo instance is not valid or the plane number isn't valid for the
> > + * format
> > + */
> > +unsigned int PixelFormatInfo::planeSize(unsigned int height, unsigned int plane,
> > +                                     unsigned int stride) const
> > +{
> > +     unsigned int vertSubSample = planes[plane].verticalSubSampling;
> > +     if (!vertSubSample)
> > +             return 0;
> > +
> > +     /* stride * ceil(height / verticalSubSampling) */
> > +     return stride * ((height + vertSubSample - 1) / vertSubSample);
> > +}
> > +
> > +/**
> > + * \brief Compute the number of bytes necessary to store a frame
> > + * \param[in] size The size of the frame, in pixels
> > + * \param[in] align The stride alignment, in bytes (1 for default alignment)
> > + *
> > + * The frame size is computed by adding the size of all planes, as computed by
> > + * planeSize(), using the specified alignment constraints for all planes. For
> > + * more complex stride constraints, use the frameSize() overloaded version that
> > + * takes an array of stride values.
> > + *
> > + * \sa planeSize()
> > + *
> >   * \return The number of bytes necessary to store the frame, or 0 if the
> >   * PixelFormatInfo instance is not valid
> >   */
> >  unsigned int PixelFormatInfo::frameSize(const Size &size, unsigned int align) const
> >  {
> > -     /* stride * ceil(height / verticalSubSampling) */
> >       unsigned int sum = 0;
> > -     for (unsigned int i = 0; i < 3; i++) {
> > -             unsigned int vertSubSample = planes[i].verticalSubSampling;
> > -             if (!vertSubSample)
> > -                     continue;
> > -             sum += stride(size.width, i, align)
> > -                  * ((size.height + vertSubSample - 1) / vertSubSample);
> > +
> > +     for (const auto &[i, plane] : utils::enumerate(planes)) {
> > +             if (plane.bytesPerGroup == 0)
> > +                     break;
> > +
> > +             sum += planeSize(size, i, align);
> >       }
> >
> >       return sum;
> > diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> > index 4e1c2b7cef5e..adabd4720668 100644
> > --- a/src/libcamera/v4l2_videodevice.cpp
> > +++ b/src/libcamera/v4l2_videodevice.cpp
> > @@ -1337,11 +1337,7 @@ std::unique_ptr<FrameBuffer> V4L2VideoDevice::createBuffer(unsigned int index)
> >                       planes[i].offset = offset;
> >
> >                       /* \todo Take the V4L2 stride into account */
> > -                     const unsigned int vertSubSample =
> > -                             info.planes[i].verticalSubSampling;
> > -                     planes[i].length =
> > -                             info.stride(format_.size.width, i, 1u) *
> > -                             ((format_.size.height + vertSubSample - 1) / vertSubSample);
> > +                     planes[i].length = info.planeSize(format_.size, i);
> >                       offset += planes[i].length;
> >               }
> >       }
> >

Hirokazu Honda Sept. 3, 2021, 10:53 a.m. UTC | #21

On Fri, Sep 3, 2021 at 3:36 AM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Kieran,
>
> On Thu, Sep 02, 2021 at 10:23:48AM +0100, Kieran Bingham wrote:
> > On 02/09/2021 05:22, Laurent Pinchart wrote:
> > > V4L2 describes multi-planar formats with different 4CCs depending on
> > > whether or not the planes are stored contiguously in memory. Support
> > > this when translating between PixelFormat and V4L2PixelFormat.
> > >
> > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > ---
> > >  include/libcamera/internal/formats.h |   2 +-
> > >  src/libcamera/formats.cpp            | 141 ++++++++++++++++-----------
> > >  src/libcamera/pipeline/ipu3/cio2.cpp |   2 +-
> > >  src/libcamera/v4l2_pixelformat.cpp   |  11 ++-
> > >  src/v4l2/v4l2_camera_proxy.cpp       |   6 +-
> > >  5 files changed, 99 insertions(+), 63 deletions(-)
> > >
> > > diff --git a/include/libcamera/internal/formats.h b/include/libcamera/internal/formats.h
> > > index 8f314e99889b..f9ce5f0834fa 100644
> > > --- a/include/libcamera/internal/formats.h
> > > +++ b/include/libcamera/internal/formats.h
> > > @@ -55,7 +55,7 @@ public:
> > >     /* \todo Add support for non-contiguous memory planes */
> > >     const char *name;
> > >     PixelFormat format;
> > > -   V4L2PixelFormat v4l2Format;
> > > +   std::array<V4L2PixelFormat, 2> v4l2Format;
> >
> > As discussed below, I'm curious if a struct would be clearer here and
> > help prevent bugs later...
> >
> >       struct {
> >               V4L2PixelFormat planar;
> >               V4L2PixelFormat multiplanar;
> >       } v4l2Format;
> >
> > >     unsigned int bitsPerPixel;
> > >     enum ColourEncoding colourEncoding;
> > >     bool packed;
> > > diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
> > > index 76be93bc1c5c..ce7c5a2d267e 100644
> > > --- a/src/libcamera/formats.cpp
> > > +++ b/src/libcamera/formats.cpp
> > > @@ -56,7 +56,14 @@ LOG_DEFINE_CATEGORY(Formats)
> > >   * \brief The PixelFormat described by this instance
> > >   *
> > >   * \var PixelFormatInfo::v4l2Format
> > > - * \brief The V4L2 pixel format corresponding to the PixelFormat
> > > + * \brief The V4L2 pixel formats corresponding to the PixelFormat
> > > + *
> > > + * Multiple V4L2 formats may exist for one PixelFormat when the format uses
> > > + * multiple planes, as V4L2 defines separate 4CCs for contiguous and separate
> > > + * planes formats. The two entries in the array store the contiguous and
> > > + * non-contiguous V4L2 formats respectively. If the PixelFormat isn't a
> > > + * multiplanar format, or if no corresponding non-contiguous V4L2 format
> > > + * exists, the second entry is invalid.
> > >   *
> > >   * \var PixelFormatInfo::bitsPerPixel
> > >   * \brief The average number of bits per pixel
> > > @@ -149,7 +156,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::RGB565, {
> > >             .name = "RGB565",
> > >             .format = formats::RGB565,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGB565),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGB565), },
> > >             .bitsPerPixel = 16,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > >             .packed = false,
> > > @@ -159,7 +166,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::RGB565_BE, {
> > >             .name = "RGB565_BE",
> > >             .format = formats::RGB565_BE,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGB565X),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGB565X), },
> > >             .bitsPerPixel = 16,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > >             .packed = false,
> > > @@ -169,7 +176,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::BGR888, {
> > >             .name = "BGR888",
> > >             .format = formats::BGR888,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGB24),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGB24), },
> > >             .bitsPerPixel = 24,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > >             .packed = false,
> > > @@ -179,7 +186,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::RGB888, {
> > >             .name = "RGB888",
> > >             .format = formats::RGB888,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_BGR24),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_BGR24), },
> > >             .bitsPerPixel = 24,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > >             .packed = false,
> > > @@ -189,7 +196,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::XRGB8888, {
> > >             .name = "XRGB8888",
> > >             .format = formats::XRGB8888,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_XBGR32),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_XBGR32), },
> > >             .bitsPerPixel = 32,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > >             .packed = false,
> > > @@ -199,7 +206,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::XBGR8888, {
> > >             .name = "XBGR8888",
> > >             .format = formats::XBGR8888,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGBX32),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGBX32), },
> > >             .bitsPerPixel = 32,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > >             .packed = false,
> > > @@ -209,7 +216,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::BGRX8888, {
> > >             .name = "BGRX8888",
> > >             .format = formats::BGRX8888,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_XRGB32),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_XRGB32), },
> > >             .bitsPerPixel = 32,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > >             .packed = false,
> > > @@ -219,7 +226,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::ABGR8888, {
> > >             .name = "ABGR8888",
> > >             .format = formats::ABGR8888,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGBA32),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGBA32), },
> > >             .bitsPerPixel = 32,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > >             .packed = false,
> > > @@ -229,7 +236,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::ARGB8888, {
> > >             .name = "ARGB8888",
> > >             .format = formats::ARGB8888,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_ABGR32),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_ABGR32), },
> > >             .bitsPerPixel = 32,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > >             .packed = false,
> > > @@ -239,7 +246,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::BGRA8888, {
> > >             .name = "BGRA8888",
> > >             .format = formats::BGRA8888,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_ARGB32),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_ARGB32), },
> > >             .bitsPerPixel = 32,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > >             .packed = false,
> > > @@ -249,7 +256,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::RGBA8888, {
> > >             .name = "RGBA8888",
> > >             .format = formats::RGBA8888,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_BGRA32),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_BGRA32), },
> > >             .bitsPerPixel = 32,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > >             .packed = false,
> > > @@ -261,7 +268,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::YUYV, {
> > >             .name = "YUYV",
> > >             .format = formats::YUYV,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YUYV),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_YUYV), },
> > >             .bitsPerPixel = 16,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > >             .packed = false,
> > > @@ -271,7 +278,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::YVYU, {
> > >             .name = "YVYU",
> > >             .format = formats::YVYU,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YVYU),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_YVYU), },
> > >             .bitsPerPixel = 16,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > >             .packed = false,
> > > @@ -281,7 +288,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::UYVY, {
> > >             .name = "UYVY",
> > >             .format = formats::UYVY,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_UYVY),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_UYVY), },
> > >             .bitsPerPixel = 16,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > >             .packed = false,
> > > @@ -291,7 +298,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::VYUY, {
> > >             .name = "VYUY",
> > >             .format = formats::VYUY,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_VYUY),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_VYUY), },
> > >             .bitsPerPixel = 16,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > >             .packed = false,
> > > @@ -303,7 +310,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::NV12, {
> > >             .name = "NV12",
> > >             .format = formats::NV12,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV12),
> > > +           .v4l2Format = {
> > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV12),
> > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV12M),
> > > +           },
> > >             .bitsPerPixel = 12,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > >             .packed = false,
> > > @@ -313,7 +323,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::NV21, {
> > >             .name = "NV21",
> > >             .format = formats::NV21,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV21),
> > > +           .v4l2Format = {
> > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV21),
> > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV21M),
> > > +           },
> > >             .bitsPerPixel = 12,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > >             .packed = false,
> > > @@ -323,7 +336,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::NV16, {
> > >             .name = "NV16",
> > >             .format = formats::NV16,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV16),
> > > +           .v4l2Format = {
> > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV16),
> > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV16M),
> > > +           },
> > >             .bitsPerPixel = 16,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > >             .packed = false,
> > > @@ -333,7 +349,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::NV61, {
> > >             .name = "NV61",
> > >             .format = formats::NV61,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV61),
> > > +           .v4l2Format = {
> > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV61),
> > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV61M),
> > > +           },
> > >             .bitsPerPixel = 16,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > >             .packed = false,
> > > @@ -343,7 +362,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::NV24, {
> > >             .name = "NV24",
> > >             .format = formats::NV24,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV24),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_NV24), },
> > >             .bitsPerPixel = 24,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > >             .packed = false,
> > > @@ -353,7 +372,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::NV42, {
> > >             .name = "NV42",
> > >             .format = formats::NV42,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV42),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_NV42), },
> > >             .bitsPerPixel = 24,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > >             .packed = false,
> > > @@ -363,7 +382,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::YUV420, {
> > >             .name = "YUV420",
> > >             .format = formats::YUV420,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YUV420),
> > > +           .v4l2Format = {
> > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YUV420),
> > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YUV420M),
> > > +           },
> > >             .bitsPerPixel = 12,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > >             .packed = false,
> > > @@ -373,7 +395,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::YVU420, {
> > >             .name = "YVU420",
> > >             .format = formats::YVU420,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YVU420),
> > > +           .v4l2Format = {
> > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YVU420),
> > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YVU420M),
> > > +           },
> > >             .bitsPerPixel = 12,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > >             .packed = false,
> > > @@ -383,7 +408,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::YUV422, {
> > >             .name = "YUV422",
> > >             .format = formats::YUV422,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YUV422P),
> > > +           .v4l2Format = {
> > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YUV422P),
> > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YUV422M),
> > > +           },
> > >             .bitsPerPixel = 16,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > >             .packed = false,
> > > @@ -395,7 +423,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::R8, {
> > >             .name = "R8",
> > >             .format = formats::R8,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_GREY),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_GREY), },
> > >             .bitsPerPixel = 8,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > >             .packed = false,
> > > @@ -407,7 +435,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SBGGR8, {
> > >             .name = "SBGGR8",
> > >             .format = formats::SBGGR8,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR8),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR8), },
> > >             .bitsPerPixel = 8,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = false,
> > > @@ -417,7 +445,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SGBRG8, {
> > >             .name = "SGBRG8",
> > >             .format = formats::SGBRG8,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG8),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG8), },
> > >             .bitsPerPixel = 8,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = false,
> > > @@ -427,7 +455,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SGRBG8, {
> > >             .name = "SGRBG8",
> > >             .format = formats::SGRBG8,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG8),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG8), },
> > >             .bitsPerPixel = 8,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = false,
> > > @@ -437,7 +465,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SRGGB8, {
> > >             .name = "SRGGB8",
> > >             .format = formats::SRGGB8,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB8),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB8), },
> > >             .bitsPerPixel = 8,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = false,
> > > @@ -447,7 +475,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SBGGR10, {
> > >             .name = "SBGGR10",
> > >             .format = formats::SBGGR10,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10), },
> > >             .bitsPerPixel = 10,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = false,
> > > @@ -457,7 +485,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SGBRG10, {
> > >             .name = "SGBRG10",
> > >             .format = formats::SGBRG10,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10), },
> > >             .bitsPerPixel = 10,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = false,
> > > @@ -467,7 +495,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SGRBG10, {
> > >             .name = "SGRBG10",
> > >             .format = formats::SGRBG10,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10), },
> > >             .bitsPerPixel = 10,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = false,
> > > @@ -477,7 +505,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SRGGB10, {
> > >             .name = "SRGGB10",
> > >             .format = formats::SRGGB10,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10), },
> > >             .bitsPerPixel = 10,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = false,
> > > @@ -487,7 +515,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SBGGR10_CSI2P, {
> > >             .name = "SBGGR10_CSI2P",
> > >             .format = formats::SBGGR10_CSI2P,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10P),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10P), },
> > >             .bitsPerPixel = 10,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = true,
> > > @@ -497,7 +525,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SGBRG10_CSI2P, {
> > >             .name = "SGBRG10_CSI2P",
> > >             .format = formats::SGBRG10_CSI2P,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10P),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10P), },
> > >             .bitsPerPixel = 10,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = true,
> > > @@ -507,7 +535,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SGRBG10_CSI2P, {
> > >             .name = "SGRBG10_CSI2P",
> > >             .format = formats::SGRBG10_CSI2P,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10P),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10P), },
> > >             .bitsPerPixel = 10,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = true,
> > > @@ -517,7 +545,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SRGGB10_CSI2P, {
> > >             .name = "SRGGB10_CSI2P",
> > >             .format = formats::SRGGB10_CSI2P,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10P),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10P), },
> > >             .bitsPerPixel = 10,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = true,
> > > @@ -527,7 +555,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SBGGR12, {
> > >             .name = "SBGGR12",
> > >             .format = formats::SBGGR12,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12), },
> > >             .bitsPerPixel = 12,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = false,
> > > @@ -537,7 +565,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SGBRG12, {
> > >             .name = "SGBRG12",
> > >             .format = formats::SGBRG12,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12), },
> > >             .bitsPerPixel = 12,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = false,
> > > @@ -547,7 +575,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SGRBG12, {
> > >             .name = "SGRBG12",
> > >             .format = formats::SGRBG12,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12), },
> > >             .bitsPerPixel = 12,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = false,
> > > @@ -557,7 +585,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SRGGB12, {
> > >             .name = "SRGGB12",
> > >             .format = formats::SRGGB12,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12), },
> > >             .bitsPerPixel = 12,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = false,
> > > @@ -567,7 +595,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SBGGR12_CSI2P, {
> > >             .name = "SBGGR12_CSI2P",
> > >             .format = formats::SBGGR12_CSI2P,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12P),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12P), },
> > >             .bitsPerPixel = 12,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = true,
> > > @@ -577,7 +605,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SGBRG12_CSI2P, {
> > >             .name = "SGBRG12_CSI2P",
> > >             .format = formats::SGBRG12_CSI2P,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12P),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12P), },
> > >             .bitsPerPixel = 12,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = true,
> > > @@ -587,7 +615,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SGRBG12_CSI2P, {
> > >             .name = "SGRBG12_CSI2P",
> > >             .format = formats::SGRBG12_CSI2P,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12P),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12P), },
> > >             .bitsPerPixel = 12,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = true,
> > > @@ -597,7 +625,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SRGGB12_CSI2P, {
> > >             .name = "SRGGB12_CSI2P",
> > >             .format = formats::SRGGB12_CSI2P,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12P),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12P), },
> > >             .bitsPerPixel = 12,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = true,
> > > @@ -607,7 +635,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SBGGR16, {
> > >             .name = "SBGGR16",
> > >             .format = formats::SBGGR16,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR16),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR16), },
> > >             .bitsPerPixel = 16,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = false,
> > > @@ -617,7 +645,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SGBRG16, {
> > >             .name = "SGBRG16",
> > >             .format = formats::SGBRG16,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG16),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG16), },
> > >             .bitsPerPixel = 16,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = false,
> > > @@ -627,7 +655,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SGRBG16, {
> > >             .name = "SGRBG16",
> > >             .format = formats::SGRBG16,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG16),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG16), },
> > >             .bitsPerPixel = 16,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = false,
> > > @@ -637,7 +665,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SRGGB16, {
> > >             .name = "SRGGB16",
> > >             .format = formats::SRGGB16,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB16),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB16), },
> > >             .bitsPerPixel = 16,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = false,
> > > @@ -647,7 +675,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SBGGR10_IPU3, {
> > >             .name = "SBGGR10_IPU3",
> > >             .format = formats::SBGGR10_IPU3,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SBGGR10),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SBGGR10), },
> > >             .bitsPerPixel = 10,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = true,
> > > @@ -658,7 +686,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SGBRG10_IPU3, {
> > >             .name = "SGBRG10_IPU3",
> > >             .format = formats::SGBRG10_IPU3,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGBRG10),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGBRG10), },
> > >             .bitsPerPixel = 10,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = true,
> > > @@ -668,7 +696,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SGRBG10_IPU3, {
> > >             .name = "SGRBG10_IPU3",
> > >             .format = formats::SGRBG10_IPU3,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGRBG10),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGRBG10), },
> > >             .bitsPerPixel = 10,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = true,
> > > @@ -678,7 +706,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::SRGGB10_IPU3, {
> > >             .name = "SRGGB10_IPU3",
> > >             .format = formats::SRGGB10_IPU3,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SRGGB10),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SRGGB10), },
> > >             .bitsPerPixel = 10,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > >             .packed = true,
> > > @@ -690,7 +718,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >     { formats::MJPEG, {
> > >             .name = "MJPEG",
> > >             .format = formats::MJPEG,
> > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_MJPEG),
> > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_MJPEG), },
> > >             .bitsPerPixel = 0,
> > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > >             .packed = false,
> > > @@ -736,7 +764,8 @@ const PixelFormatInfo &PixelFormatInfo::info(const V4L2PixelFormat &format)
> > >  {
> > >     const auto &info = std::find_if(pixelFormatInfo.begin(), pixelFormatInfo.end(),
> > >                                     [format](auto pair) {
> > > -                                           return pair.second.v4l2Format == format;
> > > +                                           return pair.second.v4l2Format[0] == format ||
> > > +                                                  pair.second.v4l2Format[1] == format;
> > >                                     });
> > >     if (info == pixelFormatInfo.end())
> > >             return pixelFormatInfoInvalid;
> > > diff --git a/src/libcamera/pipeline/ipu3/cio2.cpp b/src/libcamera/pipeline/ipu3/cio2.cpp
> > > index 9cedcb5b2879..5a9cffc80c8d 100644
> > > --- a/src/libcamera/pipeline/ipu3/cio2.cpp
> > > +++ b/src/libcamera/pipeline/ipu3/cio2.cpp
> > > @@ -205,7 +205,7 @@ int CIO2Device::configure(const Size &size, V4L2DeviceFormat *outputFormat)
> > >
> > >     const PixelFormatInfo &info = PixelFormatInfo::info(itInfo->second);
> > >
> > > -   outputFormat->fourcc = info.v4l2Format;
> > > +   outputFormat->fourcc = info.v4l2Format[0];

I probably miss something here. Why does selecting single planar
format always work here?
Ditto for v4l2_camera_proxy.cpp

Best Regards,
-Hiro
> > >     outputFormat->size = sensorFormat.size;
> > >     outputFormat->planesCount = 1;
> > >
> > > diff --git a/src/libcamera/v4l2_pixelformat.cpp b/src/libcamera/v4l2_pixelformat.cpp
> > > index 93fc4446cc64..93ead8928ed7 100644
> > > --- a/src/libcamera/v4l2_pixelformat.cpp
> > > +++ b/src/libcamera/v4l2_pixelformat.cpp
> > > @@ -67,12 +67,19 @@ const std::map<V4L2PixelFormat, PixelFormat> vpf2pf{
> > >
> > >     /* YUV planar formats. */
> > >     { V4L2PixelFormat(V4L2_PIX_FMT_NV16), formats::NV16 },
> > > +   { V4L2PixelFormat(V4L2_PIX_FMT_NV16M), formats::NV16 },
> > >     { V4L2PixelFormat(V4L2_PIX_FMT_NV61), formats::NV61 },
> > > +   { V4L2PixelFormat(V4L2_PIX_FMT_NV61M), formats::NV61 },
> > >     { V4L2PixelFormat(V4L2_PIX_FMT_NV12), formats::NV12 },
> > > +   { V4L2PixelFormat(V4L2_PIX_FMT_NV12M), formats::NV12 },
> > >     { V4L2PixelFormat(V4L2_PIX_FMT_NV21), formats::NV21 },
> > > +   { V4L2PixelFormat(V4L2_PIX_FMT_NV21M), formats::NV21 },
> > >     { V4L2PixelFormat(V4L2_PIX_FMT_YUV420), formats::YUV420 },
> > > +   { V4L2PixelFormat(V4L2_PIX_FMT_YUV420M), formats::YUV420 },
> > >     { V4L2PixelFormat(V4L2_PIX_FMT_YVU420), formats::YVU420 },
> > > +   { V4L2PixelFormat(V4L2_PIX_FMT_YVU420M), formats::YVU420 },
> > >     { V4L2PixelFormat(V4L2_PIX_FMT_YUV422P), formats::YUV422 },
> > > +   { V4L2PixelFormat(V4L2_PIX_FMT_YUV422M), formats::YUV422 },
> >
> > I'm worried that we're losing information here.
> > But presumably it's handled by whether we use mplane or not on the
> > device...?
>
> We don't expose to applications whether the camera requires contiguous
> buffers or can support non-contiguous buffers. We don't otherwise lose
> information, as we pick a specific V4L2 format from the PixelFormat,
> based on an explicit selection of multiplanar or singleplanar V4L2
> format by the pipeline handler.
>
> > >     /* Greyscale formats. */
> > >     { V4L2PixelFormat(V4L2_PIX_FMT_GREY), formats::R8 },
> > > @@ -202,13 +209,13 @@ PixelFormat V4L2PixelFormat::toPixelFormat() const
> > >   * \return The V4L2PixelFormat corresponding to \a pixelFormat
> > >   */
> > >  V4L2PixelFormat V4L2PixelFormat::fromPixelFormat(const PixelFormat &pixelFormat,
> > > -                                            [[maybe_unused]] bool multiplanar)
> > > +                                            bool multiplanar)
> > >  {
> > >     const PixelFormatInfo &info = PixelFormatInfo::info(pixelFormat);
> > >     if (!info.isValid())
> > >             return V4L2PixelFormat();
> > >
> > > -   return info.v4l2Format;
> > > +   return info.v4l2Format[multiplanar ? 1 : 0];
> >
> > I was going to say, a struct with named fields of 'planar', and
> > 'multiplanar' might be a better, though I like the simplicity of this bit.
>
> Would be easy to rewrite though.
>
> > >  }
> > >
> > >  } /* namespace libcamera */
> > > diff --git a/src/v4l2/v4l2_camera_proxy.cpp b/src/v4l2/v4l2_camera_proxy.cpp
> > > index 7682c4bddf90..8d8ee395954f 100644
> > > --- a/src/v4l2/v4l2_camera_proxy.cpp
> > > +++ b/src/v4l2/v4l2_camera_proxy.cpp
> > > @@ -169,7 +169,7 @@ void V4L2CameraProxy::setFmtFromConfig(const StreamConfiguration &streamConfig)
> > >
> > >     v4l2PixFormat_.width        = size.width;
> > >     v4l2PixFormat_.height       = size.height;
> > > -   v4l2PixFormat_.pixelformat  = info.v4l2Format;
> > > +   v4l2PixFormat_.pixelformat  = info.v4l2Format[0];
> > >     v4l2PixFormat_.field        = V4L2_FIELD_NONE;
> > >     v4l2PixFormat_.bytesperline = streamConfig.stride;
> > >     v4l2PixFormat_.sizeimage    = streamConfig.frameSize;
> > > @@ -276,7 +276,7 @@ int V4L2CameraProxy::vidioc_enum_fmt(V4L2CameraFile *file, struct v4l2_fmtdesc *
> > >     /* \todo Add map from format to description. */
> > >     utils::strlcpy(reinterpret_cast<char *>(arg->description),
> > >                    "Video Format Description", sizeof(arg->description));
> > > -   arg->pixelformat = PixelFormatInfo::info(format).v4l2Format;
> > > +   arg->pixelformat = PixelFormatInfo::info(format).v4l2Format[0];
> > >
> > >     memset(arg->reserved, 0, sizeof(arg->reserved));
> > >
> > > @@ -315,7 +315,7 @@ int V4L2CameraProxy::tryFormat(struct v4l2_format *arg)
> > >
> > >     arg->fmt.pix.width        = config.size.width;
> > >     arg->fmt.pix.height       = config.size.height;
> > > -   arg->fmt.pix.pixelformat  = info.v4l2Format;
> > > +   arg->fmt.pix.pixelformat  = info.v4l2Format[0];
> >
> > But for occasions like here, and in the CIO2
> >       arg->fmt.pix.pixelformat  = info.v4l2Format.planar;
> >
> > Would be far more descriptive over which one is being chosen, and might
> > help make it easier to spot issues when debugging multiplanar format
> > bugs ...
>
> I agree. I don't like the names "planar" and "multiplanar" though, as
> that's a bit ambiguous. V4L2 did a really bad job when it comes to
> naming here. I'll try to think of better names after sleeping over it,
> but please feel free to suggest alternatives :-) Bonus points of they're
> short and have the same length :-)
>
> > >     arg->fmt.pix.field        = V4L2_FIELD_NONE;
> > >     arg->fmt.pix.bytesperline = config.stride;
> > >     arg->fmt.pix.sizeimage    = config.frameSize;
> > >
>
> --
> Regards,
>
> Laurent Pinchart

Laurent Pinchart Sept. 3, 2021, 10:59 a.m. UTC | #22

Hi Hiro,

On Fri, Sep 03, 2021 at 07:53:48PM +0900, Hirokazu Honda wrote:
> On Fri, Sep 3, 2021 at 3:36 AM Laurent Pinchart wrote:
> > On Thu, Sep 02, 2021 at 10:23:48AM +0100, Kieran Bingham wrote:
> > > On 02/09/2021 05:22, Laurent Pinchart wrote:
> > > > V4L2 describes multi-planar formats with different 4CCs depending on
> > > > whether or not the planes are stored contiguously in memory. Support
> > > > this when translating between PixelFormat and V4L2PixelFormat.
> > > >
> > > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > > ---
> > > >  include/libcamera/internal/formats.h |   2 +-
> > > >  src/libcamera/formats.cpp            | 141 ++++++++++++++++-----------
> > > >  src/libcamera/pipeline/ipu3/cio2.cpp |   2 +-
> > > >  src/libcamera/v4l2_pixelformat.cpp   |  11 ++-
> > > >  src/v4l2/v4l2_camera_proxy.cpp       |   6 +-
> > > >  5 files changed, 99 insertions(+), 63 deletions(-)
> > > >
> > > > diff --git a/include/libcamera/internal/formats.h b/include/libcamera/internal/formats.h
> > > > index 8f314e99889b..f9ce5f0834fa 100644
> > > > --- a/include/libcamera/internal/formats.h
> > > > +++ b/include/libcamera/internal/formats.h
> > > > @@ -55,7 +55,7 @@ public:
> > > >     /* \todo Add support for non-contiguous memory planes */
> > > >     const char *name;
> > > >     PixelFormat format;
> > > > -   V4L2PixelFormat v4l2Format;
> > > > +   std::array<V4L2PixelFormat, 2> v4l2Format;
> > >
> > > As discussed below, I'm curious if a struct would be clearer here and
> > > help prevent bugs later...
> > >
> > >       struct {
> > >               V4L2PixelFormat planar;
> > >               V4L2PixelFormat multiplanar;
> > >       } v4l2Format;
> > >
> > > >     unsigned int bitsPerPixel;
> > > >     enum ColourEncoding colourEncoding;
> > > >     bool packed;
> > > > diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
> > > > index 76be93bc1c5c..ce7c5a2d267e 100644
> > > > --- a/src/libcamera/formats.cpp
> > > > +++ b/src/libcamera/formats.cpp
> > > > @@ -56,7 +56,14 @@ LOG_DEFINE_CATEGORY(Formats)
> > > >   * \brief The PixelFormat described by this instance
> > > >   *
> > > >   * \var PixelFormatInfo::v4l2Format
> > > > - * \brief The V4L2 pixel format corresponding to the PixelFormat
> > > > + * \brief The V4L2 pixel formats corresponding to the PixelFormat
> > > > + *
> > > > + * Multiple V4L2 formats may exist for one PixelFormat when the format uses
> > > > + * multiple planes, as V4L2 defines separate 4CCs for contiguous and separate
> > > > + * planes formats. The two entries in the array store the contiguous and
> > > > + * non-contiguous V4L2 formats respectively. If the PixelFormat isn't a
> > > > + * multiplanar format, or if no corresponding non-contiguous V4L2 format
> > > > + * exists, the second entry is invalid.
> > > >   *
> > > >   * \var PixelFormatInfo::bitsPerPixel
> > > >   * \brief The average number of bits per pixel
> > > > @@ -149,7 +156,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::RGB565, {
> > > >             .name = "RGB565",
> > > >             .format = formats::RGB565,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGB565),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGB565), },
> > > >             .bitsPerPixel = 16,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > >             .packed = false,
> > > > @@ -159,7 +166,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::RGB565_BE, {
> > > >             .name = "RGB565_BE",
> > > >             .format = formats::RGB565_BE,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGB565X),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGB565X), },
> > > >             .bitsPerPixel = 16,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > >             .packed = false,
> > > > @@ -169,7 +176,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::BGR888, {
> > > >             .name = "BGR888",
> > > >             .format = formats::BGR888,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGB24),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGB24), },
> > > >             .bitsPerPixel = 24,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > >             .packed = false,
> > > > @@ -179,7 +186,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::RGB888, {
> > > >             .name = "RGB888",
> > > >             .format = formats::RGB888,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_BGR24),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_BGR24), },
> > > >             .bitsPerPixel = 24,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > >             .packed = false,
> > > > @@ -189,7 +196,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::XRGB8888, {
> > > >             .name = "XRGB8888",
> > > >             .format = formats::XRGB8888,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_XBGR32),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_XBGR32), },
> > > >             .bitsPerPixel = 32,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > >             .packed = false,
> > > > @@ -199,7 +206,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::XBGR8888, {
> > > >             .name = "XBGR8888",
> > > >             .format = formats::XBGR8888,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGBX32),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGBX32), },
> > > >             .bitsPerPixel = 32,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > >             .packed = false,
> > > > @@ -209,7 +216,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::BGRX8888, {
> > > >             .name = "BGRX8888",
> > > >             .format = formats::BGRX8888,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_XRGB32),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_XRGB32), },
> > > >             .bitsPerPixel = 32,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > >             .packed = false,
> > > > @@ -219,7 +226,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::ABGR8888, {
> > > >             .name = "ABGR8888",
> > > >             .format = formats::ABGR8888,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGBA32),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGBA32), },
> > > >             .bitsPerPixel = 32,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > >             .packed = false,
> > > > @@ -229,7 +236,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::ARGB8888, {
> > > >             .name = "ARGB8888",
> > > >             .format = formats::ARGB8888,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_ABGR32),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_ABGR32), },
> > > >             .bitsPerPixel = 32,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > >             .packed = false,
> > > > @@ -239,7 +246,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::BGRA8888, {
> > > >             .name = "BGRA8888",
> > > >             .format = formats::BGRA8888,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_ARGB32),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_ARGB32), },
> > > >             .bitsPerPixel = 32,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > >             .packed = false,
> > > > @@ -249,7 +256,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::RGBA8888, {
> > > >             .name = "RGBA8888",
> > > >             .format = formats::RGBA8888,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_BGRA32),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_BGRA32), },
> > > >             .bitsPerPixel = 32,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > >             .packed = false,
> > > > @@ -261,7 +268,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::YUYV, {
> > > >             .name = "YUYV",
> > > >             .format = formats::YUYV,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YUYV),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_YUYV), },
> > > >             .bitsPerPixel = 16,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > >             .packed = false,
> > > > @@ -271,7 +278,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::YVYU, {
> > > >             .name = "YVYU",
> > > >             .format = formats::YVYU,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YVYU),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_YVYU), },
> > > >             .bitsPerPixel = 16,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > >             .packed = false,
> > > > @@ -281,7 +288,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::UYVY, {
> > > >             .name = "UYVY",
> > > >             .format = formats::UYVY,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_UYVY),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_UYVY), },
> > > >             .bitsPerPixel = 16,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > >             .packed = false,
> > > > @@ -291,7 +298,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::VYUY, {
> > > >             .name = "VYUY",
> > > >             .format = formats::VYUY,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_VYUY),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_VYUY), },
> > > >             .bitsPerPixel = 16,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > >             .packed = false,
> > > > @@ -303,7 +310,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::NV12, {
> > > >             .name = "NV12",
> > > >             .format = formats::NV12,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV12),
> > > > +           .v4l2Format = {
> > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV12),
> > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV12M),
> > > > +           },
> > > >             .bitsPerPixel = 12,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > >             .packed = false,
> > > > @@ -313,7 +323,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::NV21, {
> > > >             .name = "NV21",
> > > >             .format = formats::NV21,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV21),
> > > > +           .v4l2Format = {
> > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV21),
> > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV21M),
> > > > +           },
> > > >             .bitsPerPixel = 12,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > >             .packed = false,
> > > > @@ -323,7 +336,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::NV16, {
> > > >             .name = "NV16",
> > > >             .format = formats::NV16,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV16),
> > > > +           .v4l2Format = {
> > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV16),
> > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV16M),
> > > > +           },
> > > >             .bitsPerPixel = 16,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > >             .packed = false,
> > > > @@ -333,7 +349,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::NV61, {
> > > >             .name = "NV61",
> > > >             .format = formats::NV61,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV61),
> > > > +           .v4l2Format = {
> > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV61),
> > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV61M),
> > > > +           },
> > > >             .bitsPerPixel = 16,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > >             .packed = false,
> > > > @@ -343,7 +362,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::NV24, {
> > > >             .name = "NV24",
> > > >             .format = formats::NV24,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV24),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_NV24), },
> > > >             .bitsPerPixel = 24,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > >             .packed = false,
> > > > @@ -353,7 +372,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::NV42, {
> > > >             .name = "NV42",
> > > >             .format = formats::NV42,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV42),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_NV42), },
> > > >             .bitsPerPixel = 24,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > >             .packed = false,
> > > > @@ -363,7 +382,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::YUV420, {
> > > >             .name = "YUV420",
> > > >             .format = formats::YUV420,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YUV420),
> > > > +           .v4l2Format = {
> > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YUV420),
> > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YUV420M),
> > > > +           },
> > > >             .bitsPerPixel = 12,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > >             .packed = false,
> > > > @@ -373,7 +395,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::YVU420, {
> > > >             .name = "YVU420",
> > > >             .format = formats::YVU420,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YVU420),
> > > > +           .v4l2Format = {
> > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YVU420),
> > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YVU420M),
> > > > +           },
> > > >             .bitsPerPixel = 12,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > >             .packed = false,
> > > > @@ -383,7 +408,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::YUV422, {
> > > >             .name = "YUV422",
> > > >             .format = formats::YUV422,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YUV422P),
> > > > +           .v4l2Format = {
> > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YUV422P),
> > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YUV422M),
> > > > +           },
> > > >             .bitsPerPixel = 16,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > >             .packed = false,
> > > > @@ -395,7 +423,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::R8, {
> > > >             .name = "R8",
> > > >             .format = formats::R8,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_GREY),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_GREY), },
> > > >             .bitsPerPixel = 8,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > >             .packed = false,
> > > > @@ -407,7 +435,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SBGGR8, {
> > > >             .name = "SBGGR8",
> > > >             .format = formats::SBGGR8,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR8),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR8), },
> > > >             .bitsPerPixel = 8,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = false,
> > > > @@ -417,7 +445,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SGBRG8, {
> > > >             .name = "SGBRG8",
> > > >             .format = formats::SGBRG8,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG8),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG8), },
> > > >             .bitsPerPixel = 8,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = false,
> > > > @@ -427,7 +455,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SGRBG8, {
> > > >             .name = "SGRBG8",
> > > >             .format = formats::SGRBG8,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG8),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG8), },
> > > >             .bitsPerPixel = 8,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = false,
> > > > @@ -437,7 +465,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SRGGB8, {
> > > >             .name = "SRGGB8",
> > > >             .format = formats::SRGGB8,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB8),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB8), },
> > > >             .bitsPerPixel = 8,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = false,
> > > > @@ -447,7 +475,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SBGGR10, {
> > > >             .name = "SBGGR10",
> > > >             .format = formats::SBGGR10,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10), },
> > > >             .bitsPerPixel = 10,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = false,
> > > > @@ -457,7 +485,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SGBRG10, {
> > > >             .name = "SGBRG10",
> > > >             .format = formats::SGBRG10,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10), },
> > > >             .bitsPerPixel = 10,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = false,
> > > > @@ -467,7 +495,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SGRBG10, {
> > > >             .name = "SGRBG10",
> > > >             .format = formats::SGRBG10,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10), },
> > > >             .bitsPerPixel = 10,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = false,
> > > > @@ -477,7 +505,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SRGGB10, {
> > > >             .name = "SRGGB10",
> > > >             .format = formats::SRGGB10,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10), },
> > > >             .bitsPerPixel = 10,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = false,
> > > > @@ -487,7 +515,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SBGGR10_CSI2P, {
> > > >             .name = "SBGGR10_CSI2P",
> > > >             .format = formats::SBGGR10_CSI2P,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10P),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10P), },
> > > >             .bitsPerPixel = 10,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = true,
> > > > @@ -497,7 +525,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SGBRG10_CSI2P, {
> > > >             .name = "SGBRG10_CSI2P",
> > > >             .format = formats::SGBRG10_CSI2P,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10P),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10P), },
> > > >             .bitsPerPixel = 10,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = true,
> > > > @@ -507,7 +535,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SGRBG10_CSI2P, {
> > > >             .name = "SGRBG10_CSI2P",
> > > >             .format = formats::SGRBG10_CSI2P,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10P),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10P), },
> > > >             .bitsPerPixel = 10,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = true,
> > > > @@ -517,7 +545,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SRGGB10_CSI2P, {
> > > >             .name = "SRGGB10_CSI2P",
> > > >             .format = formats::SRGGB10_CSI2P,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10P),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10P), },
> > > >             .bitsPerPixel = 10,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = true,
> > > > @@ -527,7 +555,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SBGGR12, {
> > > >             .name = "SBGGR12",
> > > >             .format = formats::SBGGR12,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12), },
> > > >             .bitsPerPixel = 12,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = false,
> > > > @@ -537,7 +565,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SGBRG12, {
> > > >             .name = "SGBRG12",
> > > >             .format = formats::SGBRG12,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12), },
> > > >             .bitsPerPixel = 12,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = false,
> > > > @@ -547,7 +575,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SGRBG12, {
> > > >             .name = "SGRBG12",
> > > >             .format = formats::SGRBG12,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12), },
> > > >             .bitsPerPixel = 12,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = false,
> > > > @@ -557,7 +585,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SRGGB12, {
> > > >             .name = "SRGGB12",
> > > >             .format = formats::SRGGB12,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12), },
> > > >             .bitsPerPixel = 12,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = false,
> > > > @@ -567,7 +595,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SBGGR12_CSI2P, {
> > > >             .name = "SBGGR12_CSI2P",
> > > >             .format = formats::SBGGR12_CSI2P,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12P),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12P), },
> > > >             .bitsPerPixel = 12,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = true,
> > > > @@ -577,7 +605,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SGBRG12_CSI2P, {
> > > >             .name = "SGBRG12_CSI2P",
> > > >             .format = formats::SGBRG12_CSI2P,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12P),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12P), },
> > > >             .bitsPerPixel = 12,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = true,
> > > > @@ -587,7 +615,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SGRBG12_CSI2P, {
> > > >             .name = "SGRBG12_CSI2P",
> > > >             .format = formats::SGRBG12_CSI2P,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12P),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12P), },
> > > >             .bitsPerPixel = 12,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = true,
> > > > @@ -597,7 +625,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SRGGB12_CSI2P, {
> > > >             .name = "SRGGB12_CSI2P",
> > > >             .format = formats::SRGGB12_CSI2P,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12P),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12P), },
> > > >             .bitsPerPixel = 12,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = true,
> > > > @@ -607,7 +635,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SBGGR16, {
> > > >             .name = "SBGGR16",
> > > >             .format = formats::SBGGR16,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR16),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR16), },
> > > >             .bitsPerPixel = 16,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = false,
> > > > @@ -617,7 +645,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SGBRG16, {
> > > >             .name = "SGBRG16",
> > > >             .format = formats::SGBRG16,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG16),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG16), },
> > > >             .bitsPerPixel = 16,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = false,
> > > > @@ -627,7 +655,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SGRBG16, {
> > > >             .name = "SGRBG16",
> > > >             .format = formats::SGRBG16,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG16),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG16), },
> > > >             .bitsPerPixel = 16,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = false,
> > > > @@ -637,7 +665,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SRGGB16, {
> > > >             .name = "SRGGB16",
> > > >             .format = formats::SRGGB16,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB16),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB16), },
> > > >             .bitsPerPixel = 16,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = false,
> > > > @@ -647,7 +675,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SBGGR10_IPU3, {
> > > >             .name = "SBGGR10_IPU3",
> > > >             .format = formats::SBGGR10_IPU3,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SBGGR10),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SBGGR10), },
> > > >             .bitsPerPixel = 10,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = true,
> > > > @@ -658,7 +686,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SGBRG10_IPU3, {
> > > >             .name = "SGBRG10_IPU3",
> > > >             .format = formats::SGBRG10_IPU3,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGBRG10),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGBRG10), },
> > > >             .bitsPerPixel = 10,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = true,
> > > > @@ -668,7 +696,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SGRBG10_IPU3, {
> > > >             .name = "SGRBG10_IPU3",
> > > >             .format = formats::SGRBG10_IPU3,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGRBG10),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGRBG10), },
> > > >             .bitsPerPixel = 10,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = true,
> > > > @@ -678,7 +706,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::SRGGB10_IPU3, {
> > > >             .name = "SRGGB10_IPU3",
> > > >             .format = formats::SRGGB10_IPU3,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SRGGB10),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SRGGB10), },
> > > >             .bitsPerPixel = 10,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > >             .packed = true,
> > > > @@ -690,7 +718,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >     { formats::MJPEG, {
> > > >             .name = "MJPEG",
> > > >             .format = formats::MJPEG,
> > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_MJPEG),
> > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_MJPEG), },
> > > >             .bitsPerPixel = 0,
> > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > >             .packed = false,
> > > > @@ -736,7 +764,8 @@ const PixelFormatInfo &PixelFormatInfo::info(const V4L2PixelFormat &format)
> > > >  {
> > > >     const auto &info = std::find_if(pixelFormatInfo.begin(), pixelFormatInfo.end(),
> > > >                                     [format](auto pair) {
> > > > -                                           return pair.second.v4l2Format == format;
> > > > +                                           return pair.second.v4l2Format[0] == format ||
> > > > +                                                  pair.second.v4l2Format[1] == format;
> > > >                                     });
> > > >     if (info == pixelFormatInfo.end())
> > > >             return pixelFormatInfoInvalid;
> > > > diff --git a/src/libcamera/pipeline/ipu3/cio2.cpp b/src/libcamera/pipeline/ipu3/cio2.cpp
> > > > index 9cedcb5b2879..5a9cffc80c8d 100644
> > > > --- a/src/libcamera/pipeline/ipu3/cio2.cpp
> > > > +++ b/src/libcamera/pipeline/ipu3/cio2.cpp
> > > > @@ -205,7 +205,7 @@ int CIO2Device::configure(const Size &size, V4L2DeviceFormat *outputFormat)
> > > >
> > > >     const PixelFormatInfo &info = PixelFormatInfo::info(itInfo->second);
> > > >
> > > > -   outputFormat->fourcc = info.v4l2Format;
> > > > +   outputFormat->fourcc = info.v4l2Format[0];
> 
> I probably miss something here. Why does selecting single planar
> format always work here?

Because the CIO2 only supports Bayer formats, and those are all
single-planar.

> Ditto for v4l2_camera_proxy.cpp

Because the V4L2 compat layer doesn't support V4L2 multi-planar buffers
yet.

For the CIO2 I think that's good enough, for the V4L2 compat layer we
should add full multi-planar support in the future.

> > > >     outputFormat->size = sensorFormat.size;
> > > >     outputFormat->planesCount = 1;
> > > >
> > > > diff --git a/src/libcamera/v4l2_pixelformat.cpp b/src/libcamera/v4l2_pixelformat.cpp
> > > > index 93fc4446cc64..93ead8928ed7 100644
> > > > --- a/src/libcamera/v4l2_pixelformat.cpp
> > > > +++ b/src/libcamera/v4l2_pixelformat.cpp
> > > > @@ -67,12 +67,19 @@ const std::map<V4L2PixelFormat, PixelFormat> vpf2pf{
> > > >
> > > >     /* YUV planar formats. */
> > > >     { V4L2PixelFormat(V4L2_PIX_FMT_NV16), formats::NV16 },
> > > > +   { V4L2PixelFormat(V4L2_PIX_FMT_NV16M), formats::NV16 },
> > > >     { V4L2PixelFormat(V4L2_PIX_FMT_NV61), formats::NV61 },
> > > > +   { V4L2PixelFormat(V4L2_PIX_FMT_NV61M), formats::NV61 },
> > > >     { V4L2PixelFormat(V4L2_PIX_FMT_NV12), formats::NV12 },
> > > > +   { V4L2PixelFormat(V4L2_PIX_FMT_NV12M), formats::NV12 },
> > > >     { V4L2PixelFormat(V4L2_PIX_FMT_NV21), formats::NV21 },
> > > > +   { V4L2PixelFormat(V4L2_PIX_FMT_NV21M), formats::NV21 },
> > > >     { V4L2PixelFormat(V4L2_PIX_FMT_YUV420), formats::YUV420 },
> > > > +   { V4L2PixelFormat(V4L2_PIX_FMT_YUV420M), formats::YUV420 },
> > > >     { V4L2PixelFormat(V4L2_PIX_FMT_YVU420), formats::YVU420 },
> > > > +   { V4L2PixelFormat(V4L2_PIX_FMT_YVU420M), formats::YVU420 },
> > > >     { V4L2PixelFormat(V4L2_PIX_FMT_YUV422P), formats::YUV422 },
> > > > +   { V4L2PixelFormat(V4L2_PIX_FMT_YUV422M), formats::YUV422 },
> > >
> > > I'm worried that we're losing information here.
> > > But presumably it's handled by whether we use mplane or not on the
> > > device...?
> >
> > We don't expose to applications whether the camera requires contiguous
> > buffers or can support non-contiguous buffers. We don't otherwise lose
> > information, as we pick a specific V4L2 format from the PixelFormat,
> > based on an explicit selection of multiplanar or singleplanar V4L2
> > format by the pipeline handler.
> >
> > > >     /* Greyscale formats. */
> > > >     { V4L2PixelFormat(V4L2_PIX_FMT_GREY), formats::R8 },
> > > > @@ -202,13 +209,13 @@ PixelFormat V4L2PixelFormat::toPixelFormat() const
> > > >   * \return The V4L2PixelFormat corresponding to \a pixelFormat
> > > >   */
> > > >  V4L2PixelFormat V4L2PixelFormat::fromPixelFormat(const PixelFormat &pixelFormat,
> > > > -                                            [[maybe_unused]] bool multiplanar)
> > > > +                                            bool multiplanar)
> > > >  {
> > > >     const PixelFormatInfo &info = PixelFormatInfo::info(pixelFormat);
> > > >     if (!info.isValid())
> > > >             return V4L2PixelFormat();
> > > >
> > > > -   return info.v4l2Format;
> > > > +   return info.v4l2Format[multiplanar ? 1 : 0];
> > >
> > > I was going to say, a struct with named fields of 'planar', and
> > > 'multiplanar' might be a better, though I like the simplicity of this bit.
> >
> > Would be easy to rewrite though.
> >
> > > >  }
> > > >
> > > >  } /* namespace libcamera */
> > > > diff --git a/src/v4l2/v4l2_camera_proxy.cpp b/src/v4l2/v4l2_camera_proxy.cpp
> > > > index 7682c4bddf90..8d8ee395954f 100644
> > > > --- a/src/v4l2/v4l2_camera_proxy.cpp
> > > > +++ b/src/v4l2/v4l2_camera_proxy.cpp
> > > > @@ -169,7 +169,7 @@ void V4L2CameraProxy::setFmtFromConfig(const StreamConfiguration &streamConfig)
> > > >
> > > >     v4l2PixFormat_.width        = size.width;
> > > >     v4l2PixFormat_.height       = size.height;
> > > > -   v4l2PixFormat_.pixelformat  = info.v4l2Format;
> > > > +   v4l2PixFormat_.pixelformat  = info.v4l2Format[0];
> > > >     v4l2PixFormat_.field        = V4L2_FIELD_NONE;
> > > >     v4l2PixFormat_.bytesperline = streamConfig.stride;
> > > >     v4l2PixFormat_.sizeimage    = streamConfig.frameSize;
> > > > @@ -276,7 +276,7 @@ int V4L2CameraProxy::vidioc_enum_fmt(V4L2CameraFile *file, struct v4l2_fmtdesc *
> > > >     /* \todo Add map from format to description. */
> > > >     utils::strlcpy(reinterpret_cast<char *>(arg->description),
> > > >                    "Video Format Description", sizeof(arg->description));
> > > > -   arg->pixelformat = PixelFormatInfo::info(format).v4l2Format;
> > > > +   arg->pixelformat = PixelFormatInfo::info(format).v4l2Format[0];
> > > >
> > > >     memset(arg->reserved, 0, sizeof(arg->reserved));
> > > >
> > > > @@ -315,7 +315,7 @@ int V4L2CameraProxy::tryFormat(struct v4l2_format *arg)
> > > >
> > > >     arg->fmt.pix.width        = config.size.width;
> > > >     arg->fmt.pix.height       = config.size.height;
> > > > -   arg->fmt.pix.pixelformat  = info.v4l2Format;
> > > > +   arg->fmt.pix.pixelformat  = info.v4l2Format[0];
> > >
> > > But for occasions like here, and in the CIO2
> > >       arg->fmt.pix.pixelformat  = info.v4l2Format.planar;
> > >
> > > Would be far more descriptive over which one is being chosen, and might
> > > help make it easier to spot issues when debugging multiplanar format
> > > bugs ...
> >
> > I agree. I don't like the names "planar" and "multiplanar" though, as
> > that's a bit ambiguous. V4L2 did a really bad job when it comes to
> > naming here. I'll try to think of better names after sleeping over it,
> > but please feel free to suggest alternatives :-) Bonus points of they're
> > short and have the same length :-)
> >
> > > >     arg->fmt.pix.field        = V4L2_FIELD_NONE;
> > > >     arg->fmt.pix.bytesperline = config.stride;
> > > >     arg->fmt.pix.sizeimage    = config.frameSize;
> > > >

Hirokazu Honda Sept. 3, 2021, 11:09 a.m. UTC | #23

Hi Laurent,

On Fri, Sep 3, 2021 at 5:40 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Kieran,
>
> On Fri, Sep 03, 2021 at 09:24:02AM +0100, Kieran Bingham wrote:
> > On 02/09/2021 19:07, Laurent Pinchart wrote:
> > > On Thu, Sep 02, 2021 at 10:43:56AM +0100, Kieran Bingham wrote:
> > >> On 02/09/2021 05:22, Laurent Pinchart wrote:
> > >>> Multi-planar frame buffers can store their planes contiguously in
> > >>> memory, or split them in discontiguous memory areas. Add a private
> > >>> function to check in which of these two categories the frame buffer
> > >>> belongs. This will be used to correctly handle the differences between
> > >>> the V4L2 single and multi planar APIs.
> > >>>
> > >>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > >>> ---
> > >>>  include/libcamera/internal/framebuffer.h |  2 ++
> > >>>  src/libcamera/framebuffer.cpp            | 45 +++++++++++++++++++++++-
> > >>>  2 files changed, 46 insertions(+), 1 deletion(-)
> > >>>
> > >>> diff --git a/include/libcamera/internal/framebuffer.h b/include/libcamera/internal/framebuffer.h
> > >>> index 606aed2b4782..cd33c295466e 100644
> > >>> --- a/include/libcamera/internal/framebuffer.h
> > >>> +++ b/include/libcamera/internal/framebuffer.h
> > >>> @@ -21,9 +21,11 @@ public:
> > >>>   Private();
> > >>>
> > >>>   void setRequest(Request *request) { request_ = request; }
> > >>> + bool isContiguous() const { return isContiguous_; }
> > >>>
> > >>>  private:
> > >>>   Request *request_;
> > >>> + bool isContiguous_;
> > >>>  };
> > >>>
> > >>>  } /* namespace libcamera */
> > >>> diff --git a/src/libcamera/framebuffer.cpp b/src/libcamera/framebuffer.cpp
> > >>> index 53ef89bf458f..99265b44da43 100644
> > >>> --- a/src/libcamera/framebuffer.cpp
> > >>> +++ b/src/libcamera/framebuffer.cpp
> > >>> @@ -106,7 +106,7 @@ LOG_DEFINE_CATEGORY(Buffer)
> > >>>   */
> > >>>
> > >>>  FrameBuffer::Private::Private()
> > >>> - : request_(nullptr)
> > >>> + : request_(nullptr), isContiguous_(true)
> > >>>  {
> > >>>  }
> > >>>
> > >>> @@ -120,6 +120,17 @@ FrameBuffer::Private::Private()
> > >>>   * handlers, it is called by the pipeline handlers themselves.
> > >>>   */
> > >>>
> > >>> +/**
> > >>> + * \fn FrameBuffer::Private::isContiguous()
> > >>> + * \brief Check if the frame buffer stores planes contiguously in memory
> > >>> + *
> > >>> + * Multi-planar frame buffers can store their planes contiguously in memory, or
> > >>> + * split them in discontiguous memory areas. This function checks in which of
> > >>
> > >> 'split them into discontiguous'
> > >>
> > >>> + * these two categories the frame buffer belongs.
> > >>> + *
> > >>> + * \return True if the planes are stored contiguously in memory, false otherwise
> > >>> + */
> > >>> +
> > >>>  /**
> > >>>   * \class FrameBuffer
> > >>>   * \brief Frame buffer data and its associated dynamic metadata
> > >>> @@ -199,9 +210,41 @@ FrameBuffer::FrameBuffer(const std::vector<Plane> &planes, unsigned int cookie)
> > >>>   : Extensible(std::make_unique<Private>()), planes_(planes),
> > >>>     cookie_(cookie)
> > >>>  {
> > >>> + unsigned int offset = 0;
> > >>> +
> > >>>   /* \todo Remove the assertions after sufficient testing */
> > >>>   for (const auto &plane : planes_)
> > >>>           ASSERT(plane.offset != Plane::kInvalidOffset);

nit:
Shall we merge the two for loops?
At least, I would move offset to before isContiguous.

> > >>> +
> > >>> + bool isContiguous = true;
> > >>> + ino_t inode = 0;
> > >>> +
> > >>> + for (const auto &plane : planes_) {
> > >>> +         if (plane.offset != offset) {
> > >>> +                 isContiguous = false;
> > >>> +                 break;
> > >>> +         }
> > >>> +
> > >>> +         /*
> > >>> +          * Two different dmabuf file descriptors may still refer to the
> > >>> +          * same dmabuf instance. Check this using inodes.
> > >>
> > >> Going back to the FileDescriptor::inode() extension, if that cached the
> > >> inode (it can't change) on either first call, or construction, couldn't
> > >> this whole check be simplified to:
> > >>
> > >>            if (plane.fd.inode() != planes_[0].fd.inode()) {
> > >>                    isContiguous = false;
> > >>                    break;
> > >>            }
> > >
> > > This would call fstat() every time, while the fd comparison here is
> > > meant to skip the fstat() calls in case the numerical fd match. We could
> > > do
> >
> > Did you miss the part where I said "If we cache the inode in
> > FileDescriptor"?
>
> No I didn't :-)
>
> > My intention was that caching there would mean we can simplify
> > comparisons here, without incurring repeated fstat calls.
>
> If the numerical fds are identical, there's no need to compare the
> inodes, and we can avoid calling fstat() at all.
>
> > >             if (plane.fd != planes_[0].fd)
> > >
> > > if we introduced an operator==(), but as explained in a reply to the
> > > patch that adds inode(), I'd like to handle that later, to avoid
> > > blocking this series.
> >
> > Ok, I still don't see the harm in caching the inode within
> > FileDescriptor(), it can't change once read the first time.
>
> What we'd need to do in operator==() is to first compare the numerical
> fds, and only compare the inodes if the fds differ. Technically that's
> not an issue, but it requires deciding what the semantics of equality is
> as explaining in a separate e-mail, and that's the part that I'd like to
> defer to avoid delaying the offset fixes.
>
> > But, it can be deferred:
> >
> > Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> >
> > >>> +          */
> > >>> +         if (plane.fd.fd() != planes_[0].fd.fd()) {
> > >>> +                 if (!inode)
> > >>> +                         inode = planes_[0].fd.inode();

Shall error case (i.e plane.fd.inode() == 0) be handled?

-Hiro
> > >>> +                 if (plane.fd.inode() != inode) {
> > >>> +                         isContiguous = false;
> > >>> +                         break;
> > >>> +                 }
> > >>> +         }
> > >>> +
> > >>> +         offset += plane.length;
> > >>> + }
> > >>> +
> > >>> + LOG(Buffer, Debug)
> > >>> +         << "Buffer is " << (isContiguous ? "not " : "") << "contiguous";
> > >>> +
> > >>> + _d()->isContiguous_ = isContiguous;
> > >>>  }
> > >>>
> > >>>  /**
>
> --
> Regards,
>
> Laurent Pinchart

Hirokazu Honda Sept. 3, 2021, 11:11 a.m. UTC | #24

Hi Laurent,

On Fri, Sep 3, 2021 at 7:59 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Hiro,
>
> On Fri, Sep 03, 2021 at 07:53:48PM +0900, Hirokazu Honda wrote:
> > On Fri, Sep 3, 2021 at 3:36 AM Laurent Pinchart wrote:
> > > On Thu, Sep 02, 2021 at 10:23:48AM +0100, Kieran Bingham wrote:
> > > > On 02/09/2021 05:22, Laurent Pinchart wrote:
> > > > > V4L2 describes multi-planar formats with different 4CCs depending on
> > > > > whether or not the planes are stored contiguously in memory. Support
> > > > > this when translating between PixelFormat and V4L2PixelFormat.
> > > > >
> > > > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > > > ---
> > > > >  include/libcamera/internal/formats.h |   2 +-
> > > > >  src/libcamera/formats.cpp            | 141 ++++++++++++++++-----------
> > > > >  src/libcamera/pipeline/ipu3/cio2.cpp |   2 +-
> > > > >  src/libcamera/v4l2_pixelformat.cpp   |  11 ++-
> > > > >  src/v4l2/v4l2_camera_proxy.cpp       |   6 +-
> > > > >  5 files changed, 99 insertions(+), 63 deletions(-)
> > > > >
> > > > > diff --git a/include/libcamera/internal/formats.h b/include/libcamera/internal/formats.h
> > > > > index 8f314e99889b..f9ce5f0834fa 100644
> > > > > --- a/include/libcamera/internal/formats.h
> > > > > +++ b/include/libcamera/internal/formats.h
> > > > > @@ -55,7 +55,7 @@ public:
> > > > >     /* \todo Add support for non-contiguous memory planes */
> > > > >     const char *name;
> > > > >     PixelFormat format;
> > > > > -   V4L2PixelFormat v4l2Format;
> > > > > +   std::array<V4L2PixelFormat, 2> v4l2Format;
> > > >
> > > > As discussed below, I'm curious if a struct would be clearer here and
> > > > help prevent bugs later...
> > > >
> > > >       struct {
> > > >               V4L2PixelFormat planar;
> > > >               V4L2PixelFormat multiplanar;
> > > >       } v4l2Format;
> > > >
> > > > >     unsigned int bitsPerPixel;
> > > > >     enum ColourEncoding colourEncoding;
> > > > >     bool packed;
> > > > > diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
> > > > > index 76be93bc1c5c..ce7c5a2d267e 100644
> > > > > --- a/src/libcamera/formats.cpp
> > > > > +++ b/src/libcamera/formats.cpp
> > > > > @@ -56,7 +56,14 @@ LOG_DEFINE_CATEGORY(Formats)
> > > > >   * \brief The PixelFormat described by this instance
> > > > >   *
> > > > >   * \var PixelFormatInfo::v4l2Format
> > > > > - * \brief The V4L2 pixel format corresponding to the PixelFormat
> > > > > + * \brief The V4L2 pixel formats corresponding to the PixelFormat
> > > > > + *
> > > > > + * Multiple V4L2 formats may exist for one PixelFormat when the format uses
> > > > > + * multiple planes, as V4L2 defines separate 4CCs for contiguous and separate
> > > > > + * planes formats. The two entries in the array store the contiguous and
> > > > > + * non-contiguous V4L2 formats respectively. If the PixelFormat isn't a
> > > > > + * multiplanar format, or if no corresponding non-contiguous V4L2 format
> > > > > + * exists, the second entry is invalid.
> > > > >   *
> > > > >   * \var PixelFormatInfo::bitsPerPixel
> > > > >   * \brief The average number of bits per pixel
> > > > > @@ -149,7 +156,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::RGB565, {
> > > > >             .name = "RGB565",
> > > > >             .format = formats::RGB565,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGB565),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGB565), },
> > > > >             .bitsPerPixel = 16,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > > >             .packed = false,
> > > > > @@ -159,7 +166,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::RGB565_BE, {
> > > > >             .name = "RGB565_BE",
> > > > >             .format = formats::RGB565_BE,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGB565X),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGB565X), },
> > > > >             .bitsPerPixel = 16,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > > >             .packed = false,
> > > > > @@ -169,7 +176,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::BGR888, {
> > > > >             .name = "BGR888",
> > > > >             .format = formats::BGR888,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGB24),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGB24), },
> > > > >             .bitsPerPixel = 24,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > > >             .packed = false,
> > > > > @@ -179,7 +186,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::RGB888, {
> > > > >             .name = "RGB888",
> > > > >             .format = formats::RGB888,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_BGR24),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_BGR24), },
> > > > >             .bitsPerPixel = 24,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > > >             .packed = false,
> > > > > @@ -189,7 +196,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::XRGB8888, {
> > > > >             .name = "XRGB8888",
> > > > >             .format = formats::XRGB8888,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_XBGR32),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_XBGR32), },
> > > > >             .bitsPerPixel = 32,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > > >             .packed = false,
> > > > > @@ -199,7 +206,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::XBGR8888, {
> > > > >             .name = "XBGR8888",
> > > > >             .format = formats::XBGR8888,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGBX32),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGBX32), },
> > > > >             .bitsPerPixel = 32,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > > >             .packed = false,
> > > > > @@ -209,7 +216,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::BGRX8888, {
> > > > >             .name = "BGRX8888",
> > > > >             .format = formats::BGRX8888,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_XRGB32),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_XRGB32), },
> > > > >             .bitsPerPixel = 32,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > > >             .packed = false,
> > > > > @@ -219,7 +226,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::ABGR8888, {
> > > > >             .name = "ABGR8888",
> > > > >             .format = formats::ABGR8888,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_RGBA32),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_RGBA32), },
> > > > >             .bitsPerPixel = 32,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > > >             .packed = false,
> > > > > @@ -229,7 +236,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::ARGB8888, {
> > > > >             .name = "ARGB8888",
> > > > >             .format = formats::ARGB8888,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_ABGR32),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_ABGR32), },
> > > > >             .bitsPerPixel = 32,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > > >             .packed = false,
> > > > > @@ -239,7 +246,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::BGRA8888, {
> > > > >             .name = "BGRA8888",
> > > > >             .format = formats::BGRA8888,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_ARGB32),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_ARGB32), },
> > > > >             .bitsPerPixel = 32,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > > >             .packed = false,
> > > > > @@ -249,7 +256,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::RGBA8888, {
> > > > >             .name = "RGBA8888",
> > > > >             .format = formats::RGBA8888,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_BGRA32),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_BGRA32), },
> > > > >             .bitsPerPixel = 32,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRGB,
> > > > >             .packed = false,
> > > > > @@ -261,7 +268,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::YUYV, {
> > > > >             .name = "YUYV",
> > > > >             .format = formats::YUYV,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YUYV),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_YUYV), },
> > > > >             .bitsPerPixel = 16,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > > >             .packed = false,
> > > > > @@ -271,7 +278,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::YVYU, {
> > > > >             .name = "YVYU",
> > > > >             .format = formats::YVYU,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YVYU),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_YVYU), },
> > > > >             .bitsPerPixel = 16,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > > >             .packed = false,
> > > > > @@ -281,7 +288,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::UYVY, {
> > > > >             .name = "UYVY",
> > > > >             .format = formats::UYVY,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_UYVY),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_UYVY), },
> > > > >             .bitsPerPixel = 16,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > > >             .packed = false,
> > > > > @@ -291,7 +298,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::VYUY, {
> > > > >             .name = "VYUY",
> > > > >             .format = formats::VYUY,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_VYUY),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_VYUY), },
> > > > >             .bitsPerPixel = 16,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > > >             .packed = false,
> > > > > @@ -303,7 +310,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::NV12, {
> > > > >             .name = "NV12",
> > > > >             .format = formats::NV12,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV12),
> > > > > +           .v4l2Format = {
> > > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV12),
> > > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV12M),
> > > > > +           },
> > > > >             .bitsPerPixel = 12,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > > >             .packed = false,
> > > > > @@ -313,7 +323,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::NV21, {
> > > > >             .name = "NV21",
> > > > >             .format = formats::NV21,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV21),
> > > > > +           .v4l2Format = {
> > > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV21),
> > > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV21M),
> > > > > +           },
> > > > >             .bitsPerPixel = 12,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > > >             .packed = false,
> > > > > @@ -323,7 +336,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::NV16, {
> > > > >             .name = "NV16",
> > > > >             .format = formats::NV16,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV16),
> > > > > +           .v4l2Format = {
> > > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV16),
> > > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV16M),
> > > > > +           },
> > > > >             .bitsPerPixel = 16,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > > >             .packed = false,
> > > > > @@ -333,7 +349,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::NV61, {
> > > > >             .name = "NV61",
> > > > >             .format = formats::NV61,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV61),
> > > > > +           .v4l2Format = {
> > > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV61),
> > > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_NV61M),
> > > > > +           },
> > > > >             .bitsPerPixel = 16,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > > >             .packed = false,
> > > > > @@ -343,7 +362,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::NV24, {
> > > > >             .name = "NV24",
> > > > >             .format = formats::NV24,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV24),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_NV24), },
> > > > >             .bitsPerPixel = 24,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > > >             .packed = false,
> > > > > @@ -353,7 +372,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::NV42, {
> > > > >             .name = "NV42",
> > > > >             .format = formats::NV42,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_NV42),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_NV42), },
> > > > >             .bitsPerPixel = 24,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > > >             .packed = false,
> > > > > @@ -363,7 +382,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::YUV420, {
> > > > >             .name = "YUV420",
> > > > >             .format = formats::YUV420,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YUV420),
> > > > > +           .v4l2Format = {
> > > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YUV420),
> > > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YUV420M),
> > > > > +           },
> > > > >             .bitsPerPixel = 12,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > > >             .packed = false,
> > > > > @@ -373,7 +395,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::YVU420, {
> > > > >             .name = "YVU420",
> > > > >             .format = formats::YVU420,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YVU420),
> > > > > +           .v4l2Format = {
> > > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YVU420),
> > > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YVU420M),
> > > > > +           },
> > > > >             .bitsPerPixel = 12,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > > >             .packed = false,
> > > > > @@ -383,7 +408,10 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::YUV422, {
> > > > >             .name = "YUV422",
> > > > >             .format = formats::YUV422,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_YUV422P),
> > > > > +           .v4l2Format = {
> > > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YUV422P),
> > > > > +                   V4L2PixelFormat(V4L2_PIX_FMT_YUV422M),
> > > > > +           },
> > > > >             .bitsPerPixel = 16,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > > >             .packed = false,
> > > > > @@ -395,7 +423,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::R8, {
> > > > >             .name = "R8",
> > > > >             .format = formats::R8,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_GREY),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_GREY), },
> > > > >             .bitsPerPixel = 8,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > > >             .packed = false,
> > > > > @@ -407,7 +435,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SBGGR8, {
> > > > >             .name = "SBGGR8",
> > > > >             .format = formats::SBGGR8,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR8),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR8), },
> > > > >             .bitsPerPixel = 8,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = false,
> > > > > @@ -417,7 +445,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SGBRG8, {
> > > > >             .name = "SGBRG8",
> > > > >             .format = formats::SGBRG8,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG8),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG8), },
> > > > >             .bitsPerPixel = 8,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = false,
> > > > > @@ -427,7 +455,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SGRBG8, {
> > > > >             .name = "SGRBG8",
> > > > >             .format = formats::SGRBG8,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG8),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG8), },
> > > > >             .bitsPerPixel = 8,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = false,
> > > > > @@ -437,7 +465,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SRGGB8, {
> > > > >             .name = "SRGGB8",
> > > > >             .format = formats::SRGGB8,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB8),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB8), },
> > > > >             .bitsPerPixel = 8,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = false,
> > > > > @@ -447,7 +475,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SBGGR10, {
> > > > >             .name = "SBGGR10",
> > > > >             .format = formats::SBGGR10,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10), },
> > > > >             .bitsPerPixel = 10,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = false,
> > > > > @@ -457,7 +485,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SGBRG10, {
> > > > >             .name = "SGBRG10",
> > > > >             .format = formats::SGBRG10,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10), },
> > > > >             .bitsPerPixel = 10,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = false,
> > > > > @@ -467,7 +495,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SGRBG10, {
> > > > >             .name = "SGRBG10",
> > > > >             .format = formats::SGRBG10,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10), },
> > > > >             .bitsPerPixel = 10,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = false,
> > > > > @@ -477,7 +505,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SRGGB10, {
> > > > >             .name = "SRGGB10",
> > > > >             .format = formats::SRGGB10,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10), },
> > > > >             .bitsPerPixel = 10,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = false,
> > > > > @@ -487,7 +515,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SBGGR10_CSI2P, {
> > > > >             .name = "SBGGR10_CSI2P",
> > > > >             .format = formats::SBGGR10_CSI2P,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10P),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR10P), },
> > > > >             .bitsPerPixel = 10,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = true,
> > > > > @@ -497,7 +525,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SGBRG10_CSI2P, {
> > > > >             .name = "SGBRG10_CSI2P",
> > > > >             .format = formats::SGBRG10_CSI2P,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10P),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG10P), },
> > > > >             .bitsPerPixel = 10,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = true,
> > > > > @@ -507,7 +535,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SGRBG10_CSI2P, {
> > > > >             .name = "SGRBG10_CSI2P",
> > > > >             .format = formats::SGRBG10_CSI2P,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10P),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG10P), },
> > > > >             .bitsPerPixel = 10,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = true,
> > > > > @@ -517,7 +545,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SRGGB10_CSI2P, {
> > > > >             .name = "SRGGB10_CSI2P",
> > > > >             .format = formats::SRGGB10_CSI2P,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10P),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB10P), },
> > > > >             .bitsPerPixel = 10,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = true,
> > > > > @@ -527,7 +555,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SBGGR12, {
> > > > >             .name = "SBGGR12",
> > > > >             .format = formats::SBGGR12,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12), },
> > > > >             .bitsPerPixel = 12,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = false,
> > > > > @@ -537,7 +565,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SGBRG12, {
> > > > >             .name = "SGBRG12",
> > > > >             .format = formats::SGBRG12,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12), },
> > > > >             .bitsPerPixel = 12,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = false,
> > > > > @@ -547,7 +575,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SGRBG12, {
> > > > >             .name = "SGRBG12",
> > > > >             .format = formats::SGRBG12,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12), },
> > > > >             .bitsPerPixel = 12,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = false,
> > > > > @@ -557,7 +585,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SRGGB12, {
> > > > >             .name = "SRGGB12",
> > > > >             .format = formats::SRGGB12,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12), },
> > > > >             .bitsPerPixel = 12,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = false,
> > > > > @@ -567,7 +595,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SBGGR12_CSI2P, {
> > > > >             .name = "SBGGR12_CSI2P",
> > > > >             .format = formats::SBGGR12_CSI2P,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12P),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR12P), },
> > > > >             .bitsPerPixel = 12,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = true,
> > > > > @@ -577,7 +605,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SGBRG12_CSI2P, {
> > > > >             .name = "SGBRG12_CSI2P",
> > > > >             .format = formats::SGBRG12_CSI2P,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12P),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG12P), },
> > > > >             .bitsPerPixel = 12,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = true,
> > > > > @@ -587,7 +615,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SGRBG12_CSI2P, {
> > > > >             .name = "SGRBG12_CSI2P",
> > > > >             .format = formats::SGRBG12_CSI2P,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12P),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG12P), },
> > > > >             .bitsPerPixel = 12,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = true,
> > > > > @@ -597,7 +625,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SRGGB12_CSI2P, {
> > > > >             .name = "SRGGB12_CSI2P",
> > > > >             .format = formats::SRGGB12_CSI2P,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12P),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB12P), },
> > > > >             .bitsPerPixel = 12,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = true,
> > > > > @@ -607,7 +635,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SBGGR16, {
> > > > >             .name = "SBGGR16",
> > > > >             .format = formats::SBGGR16,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SBGGR16),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SBGGR16), },
> > > > >             .bitsPerPixel = 16,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = false,
> > > > > @@ -617,7 +645,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SGBRG16, {
> > > > >             .name = "SGBRG16",
> > > > >             .format = formats::SGBRG16,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGBRG16),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGBRG16), },
> > > > >             .bitsPerPixel = 16,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = false,
> > > > > @@ -627,7 +655,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SGRBG16, {
> > > > >             .name = "SGRBG16",
> > > > >             .format = formats::SGRBG16,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SGRBG16),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SGRBG16), },
> > > > >             .bitsPerPixel = 16,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = false,
> > > > > @@ -637,7 +665,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SRGGB16, {
> > > > >             .name = "SRGGB16",
> > > > >             .format = formats::SRGGB16,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_SRGGB16),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_SRGGB16), },
> > > > >             .bitsPerPixel = 16,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = false,
> > > > > @@ -647,7 +675,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SBGGR10_IPU3, {
> > > > >             .name = "SBGGR10_IPU3",
> > > > >             .format = formats::SBGGR10_IPU3,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SBGGR10),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SBGGR10), },
> > > > >             .bitsPerPixel = 10,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = true,
> > > > > @@ -658,7 +686,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SGBRG10_IPU3, {
> > > > >             .name = "SGBRG10_IPU3",
> > > > >             .format = formats::SGBRG10_IPU3,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGBRG10),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGBRG10), },
> > > > >             .bitsPerPixel = 10,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = true,
> > > > > @@ -668,7 +696,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SGRBG10_IPU3, {
> > > > >             .name = "SGRBG10_IPU3",
> > > > >             .format = formats::SGRBG10_IPU3,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGRBG10),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SGRBG10), },
> > > > >             .bitsPerPixel = 10,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = true,
> > > > > @@ -678,7 +706,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::SRGGB10_IPU3, {
> > > > >             .name = "SRGGB10_IPU3",
> > > > >             .format = formats::SRGGB10_IPU3,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SRGGB10),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_IPU3_SRGGB10), },
> > > > >             .bitsPerPixel = 10,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingRAW,
> > > > >             .packed = true,
> > > > > @@ -690,7 +718,7 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > > >     { formats::MJPEG, {
> > > > >             .name = "MJPEG",
> > > > >             .format = formats::MJPEG,
> > > > > -           .v4l2Format = V4L2PixelFormat(V4L2_PIX_FMT_MJPEG),
> > > > > +           .v4l2Format = { V4L2PixelFormat(V4L2_PIX_FMT_MJPEG), },
> > > > >             .bitsPerPixel = 0,
> > > > >             .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > > >             .packed = false,
> > > > > @@ -736,7 +764,8 @@ const PixelFormatInfo &PixelFormatInfo::info(const V4L2PixelFormat &format)
> > > > >  {
> > > > >     const auto &info = std::find_if(pixelFormatInfo.begin(), pixelFormatInfo.end(),
> > > > >                                     [format](auto pair) {
> > > > > -                                           return pair.second.v4l2Format == format;
> > > > > +                                           return pair.second.v4l2Format[0] == format ||
> > > > > +                                                  pair.second.v4l2Format[1] == format;
> > > > >                                     });
> > > > >     if (info == pixelFormatInfo.end())
> > > > >             return pixelFormatInfoInvalid;
> > > > > diff --git a/src/libcamera/pipeline/ipu3/cio2.cpp b/src/libcamera/pipeline/ipu3/cio2.cpp
> > > > > index 9cedcb5b2879..5a9cffc80c8d 100644
> > > > > --- a/src/libcamera/pipeline/ipu3/cio2.cpp
> > > > > +++ b/src/libcamera/pipeline/ipu3/cio2.cpp
> > > > > @@ -205,7 +205,7 @@ int CIO2Device::configure(const Size &size, V4L2DeviceFormat *outputFormat)
> > > > >
> > > > >     const PixelFormatInfo &info = PixelFormatInfo::info(itInfo->second);
> > > > >
> > > > > -   outputFormat->fourcc = info.v4l2Format;
> > > > > +   outputFormat->fourcc = info.v4l2Format[0];
> >
> > I probably miss something here. Why does selecting single planar
> > format always work here?
>
> Because the CIO2 only supports Bayer formats, and those are all
> single-planar.
>
> > Ditto for v4l2_camera_proxy.cpp
>
> Because the V4L2 compat layer doesn't support V4L2 multi-planar buffers
> yet.
>
> For the CIO2 I think that's good enough, for the V4L2 compat layer we
> should add full multi-planar support in the future.

Thanks for explaining.
Perhaps should these comments be added?

Reviewed-by: Hirokazu Honda <hiroh@chromium.org>

-Hiro
>
> > > > >     outputFormat->size = sensorFormat.size;
> > > > >     outputFormat->planesCount = 1;
> > > > >
> > > > > diff --git a/src/libcamera/v4l2_pixelformat.cpp b/src/libcamera/v4l2_pixelformat.cpp
> > > > > index 93fc4446cc64..93ead8928ed7 100644
> > > > > --- a/src/libcamera/v4l2_pixelformat.cpp
> > > > > +++ b/src/libcamera/v4l2_pixelformat.cpp
> > > > > @@ -67,12 +67,19 @@ const std::map<V4L2PixelFormat, PixelFormat> vpf2pf{
> > > > >
> > > > >     /* YUV planar formats. */
> > > > >     { V4L2PixelFormat(V4L2_PIX_FMT_NV16), formats::NV16 },
> > > > > +   { V4L2PixelFormat(V4L2_PIX_FMT_NV16M), formats::NV16 },
> > > > >     { V4L2PixelFormat(V4L2_PIX_FMT_NV61), formats::NV61 },
> > > > > +   { V4L2PixelFormat(V4L2_PIX_FMT_NV61M), formats::NV61 },
> > > > >     { V4L2PixelFormat(V4L2_PIX_FMT_NV12), formats::NV12 },
> > > > > +   { V4L2PixelFormat(V4L2_PIX_FMT_NV12M), formats::NV12 },
> > > > >     { V4L2PixelFormat(V4L2_PIX_FMT_NV21), formats::NV21 },
> > > > > +   { V4L2PixelFormat(V4L2_PIX_FMT_NV21M), formats::NV21 },
> > > > >     { V4L2PixelFormat(V4L2_PIX_FMT_YUV420), formats::YUV420 },
> > > > > +   { V4L2PixelFormat(V4L2_PIX_FMT_YUV420M), formats::YUV420 },
> > > > >     { V4L2PixelFormat(V4L2_PIX_FMT_YVU420), formats::YVU420 },
> > > > > +   { V4L2PixelFormat(V4L2_PIX_FMT_YVU420M), formats::YVU420 },
> > > > >     { V4L2PixelFormat(V4L2_PIX_FMT_YUV422P), formats::YUV422 },
> > > > > +   { V4L2PixelFormat(V4L2_PIX_FMT_YUV422M), formats::YUV422 },
> > > >
> > > > I'm worried that we're losing information here.
> > > > But presumably it's handled by whether we use mplane or not on the
> > > > device...?
> > >
> > > We don't expose to applications whether the camera requires contiguous
> > > buffers or can support non-contiguous buffers. We don't otherwise lose
> > > information, as we pick a specific V4L2 format from the PixelFormat,
> > > based on an explicit selection of multiplanar or singleplanar V4L2
> > > format by the pipeline handler.
> > >
> > > > >     /* Greyscale formats. */
> > > > >     { V4L2PixelFormat(V4L2_PIX_FMT_GREY), formats::R8 },
> > > > > @@ -202,13 +209,13 @@ PixelFormat V4L2PixelFormat::toPixelFormat() const
> > > > >   * \return The V4L2PixelFormat corresponding to \a pixelFormat
> > > > >   */
> > > > >  V4L2PixelFormat V4L2PixelFormat::fromPixelFormat(const PixelFormat &pixelFormat,
> > > > > -                                            [[maybe_unused]] bool multiplanar)
> > > > > +                                            bool multiplanar)
> > > > >  {
> > > > >     const PixelFormatInfo &info = PixelFormatInfo::info(pixelFormat);
> > > > >     if (!info.isValid())
> > > > >             return V4L2PixelFormat();
> > > > >
> > > > > -   return info.v4l2Format;
> > > > > +   return info.v4l2Format[multiplanar ? 1 : 0];
> > > >
> > > > I was going to say, a struct with named fields of 'planar', and
> > > > 'multiplanar' might be a better, though I like the simplicity of this bit.
> > >
> > > Would be easy to rewrite though.
> > >
> > > > >  }
> > > > >
> > > > >  } /* namespace libcamera */
> > > > > diff --git a/src/v4l2/v4l2_camera_proxy.cpp b/src/v4l2/v4l2_camera_proxy.cpp
> > > > > index 7682c4bddf90..8d8ee395954f 100644
> > > > > --- a/src/v4l2/v4l2_camera_proxy.cpp
> > > > > +++ b/src/v4l2/v4l2_camera_proxy.cpp
> > > > > @@ -169,7 +169,7 @@ void V4L2CameraProxy::setFmtFromConfig(const StreamConfiguration &streamConfig)
> > > > >
> > > > >     v4l2PixFormat_.width        = size.width;
> > > > >     v4l2PixFormat_.height       = size.height;
> > > > > -   v4l2PixFormat_.pixelformat  = info.v4l2Format;
> > > > > +   v4l2PixFormat_.pixelformat  = info.v4l2Format[0];
> > > > >     v4l2PixFormat_.field        = V4L2_FIELD_NONE;
> > > > >     v4l2PixFormat_.bytesperline = streamConfig.stride;
> > > > >     v4l2PixFormat_.sizeimage    = streamConfig.frameSize;
> > > > > @@ -276,7 +276,7 @@ int V4L2CameraProxy::vidioc_enum_fmt(V4L2CameraFile *file, struct v4l2_fmtdesc *
> > > > >     /* \todo Add map from format to description. */
> > > > >     utils::strlcpy(reinterpret_cast<char *>(arg->description),
> > > > >                    "Video Format Description", sizeof(arg->description));
> > > > > -   arg->pixelformat = PixelFormatInfo::info(format).v4l2Format;
> > > > > +   arg->pixelformat = PixelFormatInfo::info(format).v4l2Format[0];
> > > > >
> > > > >     memset(arg->reserved, 0, sizeof(arg->reserved));
> > > > >
> > > > > @@ -315,7 +315,7 @@ int V4L2CameraProxy::tryFormat(struct v4l2_format *arg)
> > > > >
> > > > >     arg->fmt.pix.width        = config.size.width;
> > > > >     arg->fmt.pix.height       = config.size.height;
> > > > > -   arg->fmt.pix.pixelformat  = info.v4l2Format;
> > > > > +   arg->fmt.pix.pixelformat  = info.v4l2Format[0];
> > > >
> > > > But for occasions like here, and in the CIO2
> > > >       arg->fmt.pix.pixelformat  = info.v4l2Format.planar;
> > > >
> > > > Would be far more descriptive over which one is being chosen, and might
> > > > help make it easier to spot issues when debugging multiplanar format
> > > > bugs ...
> > >
> > > I agree. I don't like the names "planar" and "multiplanar" though, as
> > > that's a bit ambiguous. V4L2 did a really bad job when it comes to
> > > naming here. I'll try to think of better names after sleeping over it,
> > > but please feel free to suggest alternatives :-) Bonus points of they're
> > > short and have the same length :-)
> > >
> > > > >     arg->fmt.pix.field        = V4L2_FIELD_NONE;
> > > > >     arg->fmt.pix.bytesperline = config.stride;
> > > > >     arg->fmt.pix.sizeimage    = config.frameSize;
> > > > >
>
> --
> Regards,
>
> Laurent Pinchart

Laurent Pinchart Sept. 3, 2021, 11:23 a.m. UTC | #25

Hi Hiro,

On Fri, Sep 03, 2021 at 08:09:07PM +0900, Hirokazu Honda wrote:
> On Fri, Sep 3, 2021 at 5:40 PM Laurent Pinchart wrote:
> > On Fri, Sep 03, 2021 at 09:24:02AM +0100, Kieran Bingham wrote:
> > > On 02/09/2021 19:07, Laurent Pinchart wrote:
> > > > On Thu, Sep 02, 2021 at 10:43:56AM +0100, Kieran Bingham wrote:
> > > >> On 02/09/2021 05:22, Laurent Pinchart wrote:
> > > >>> Multi-planar frame buffers can store their planes contiguously in
> > > >>> memory, or split them in discontiguous memory areas. Add a private
> > > >>> function to check in which of these two categories the frame buffer
> > > >>> belongs. This will be used to correctly handle the differences between
> > > >>> the V4L2 single and multi planar APIs.
> > > >>>
> > > >>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > >>> ---
> > > >>>  include/libcamera/internal/framebuffer.h |  2 ++
> > > >>>  src/libcamera/framebuffer.cpp            | 45 +++++++++++++++++++++++-
> > > >>>  2 files changed, 46 insertions(+), 1 deletion(-)
> > > >>>
> > > >>> diff --git a/include/libcamera/internal/framebuffer.h b/include/libcamera/internal/framebuffer.h
> > > >>> index 606aed2b4782..cd33c295466e 100644
> > > >>> --- a/include/libcamera/internal/framebuffer.h
> > > >>> +++ b/include/libcamera/internal/framebuffer.h
> > > >>> @@ -21,9 +21,11 @@ public:
> > > >>>   Private();
> > > >>>
> > > >>>   void setRequest(Request *request) { request_ = request; }
> > > >>> + bool isContiguous() const { return isContiguous_; }
> > > >>>
> > > >>>  private:
> > > >>>   Request *request_;
> > > >>> + bool isContiguous_;
> > > >>>  };
> > > >>>
> > > >>>  } /* namespace libcamera */
> > > >>> diff --git a/src/libcamera/framebuffer.cpp b/src/libcamera/framebuffer.cpp
> > > >>> index 53ef89bf458f..99265b44da43 100644
> > > >>> --- a/src/libcamera/framebuffer.cpp
> > > >>> +++ b/src/libcamera/framebuffer.cpp
> > > >>> @@ -106,7 +106,7 @@ LOG_DEFINE_CATEGORY(Buffer)
> > > >>>   */
> > > >>>
> > > >>>  FrameBuffer::Private::Private()
> > > >>> - : request_(nullptr)
> > > >>> + : request_(nullptr), isContiguous_(true)
> > > >>>  {
> > > >>>  }
> > > >>>
> > > >>> @@ -120,6 +120,17 @@ FrameBuffer::Private::Private()
> > > >>>   * handlers, it is called by the pipeline handlers themselves.
> > > >>>   */
> > > >>>
> > > >>> +/**
> > > >>> + * \fn FrameBuffer::Private::isContiguous()
> > > >>> + * \brief Check if the frame buffer stores planes contiguously in memory
> > > >>> + *
> > > >>> + * Multi-planar frame buffers can store their planes contiguously in memory, or
> > > >>> + * split them in discontiguous memory areas. This function checks in which of
> > > >>
> > > >> 'split them into discontiguous'
> > > >>
> > > >>> + * these two categories the frame buffer belongs.
> > > >>> + *
> > > >>> + * \return True if the planes are stored contiguously in memory, false otherwise
> > > >>> + */
> > > >>> +
> > > >>>  /**
> > > >>>   * \class FrameBuffer
> > > >>>   * \brief Frame buffer data and its associated dynamic metadata
> > > >>> @@ -199,9 +210,41 @@ FrameBuffer::FrameBuffer(const std::vector<Plane> &planes, unsigned int cookie)
> > > >>>   : Extensible(std::make_unique<Private>()), planes_(planes),
> > > >>>     cookie_(cookie)
> > > >>>  {
> > > >>> + unsigned int offset = 0;
> > > >>> +
> > > >>>   /* \todo Remove the assertions after sufficient testing */
> > > >>>   for (const auto &plane : planes_)
> > > >>>           ASSERT(plane.offset != Plane::kInvalidOffset);
> 
> nit:
> Shall we merge the two for loops?

I've kept them separate to make it easier to remove the first one (and
now I wonder why I thought it would be easier...), but as the ASSERT is
now here to stay, I'll merge them.

> At least, I would move offset to before isContiguous.

Done already :-)

> > > >>> +
> > > >>> + bool isContiguous = true;
> > > >>> + ino_t inode = 0;
> > > >>> +
> > > >>> + for (const auto &plane : planes_) {
> > > >>> +         if (plane.offset != offset) {
> > > >>> +                 isContiguous = false;
> > > >>> +                 break;
> > > >>> +         }
> > > >>> +
> > > >>> +         /*
> > > >>> +          * Two different dmabuf file descriptors may still refer to the
> > > >>> +          * same dmabuf instance. Check this using inodes.
> > > >>
> > > >> Going back to the FileDescriptor::inode() extension, if that cached the
> > > >> inode (it can't change) on either first call, or construction, couldn't
> > > >> this whole check be simplified to:
> > > >>
> > > >>            if (plane.fd.inode() != planes_[0].fd.inode()) {
> > > >>                    isContiguous = false;
> > > >>                    break;
> > > >>            }
> > > >
> > > > This would call fstat() every time, while the fd comparison here is
> > > > meant to skip the fstat() calls in case the numerical fd match. We could
> > > > do
> > >
> > > Did you miss the part where I said "If we cache the inode in
> > > FileDescriptor"?
> >
> > No I didn't :-)
> >
> > > My intention was that caching there would mean we can simplify
> > > comparisons here, without incurring repeated fstat calls.
> >
> > If the numerical fds are identical, there's no need to compare the
> > inodes, and we can avoid calling fstat() at all.
> >
> > > >             if (plane.fd != planes_[0].fd)
> > > >
> > > > if we introduced an operator==(), but as explained in a reply to the
> > > > patch that adds inode(), I'd like to handle that later, to avoid
> > > > blocking this series.
> > >
> > > Ok, I still don't see the harm in caching the inode within
> > > FileDescriptor(), it can't change once read the first time.
> >
> > What we'd need to do in operator==() is to first compare the numerical
> > fds, and only compare the inodes if the fds differ. Technically that's
> > not an issue, but it requires deciding what the semantics of equality is
> > as explaining in a separate e-mail, and that's the part that I'd like to
> > defer to avoid delaying the offset fixes.
> >
> > > But, it can be deferred:
> > >
> > > Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> > >
> > > >>> +          */
> > > >>> +         if (plane.fd.fd() != planes_[0].fd.fd()) {
> > > >>> +                 if (!inode)
> > > >>> +                         inode = planes_[0].fd.inode();
> 
> Shall error case (i.e plane.fd.inode() == 0) be handled?

This can only happen if the application gives us an incorrect fd, I'm
not sure how much we need to protect against applications doing very
very bad things.

> > > >>> +                 if (plane.fd.inode() != inode) {
> > > >>> +                         isContiguous = false;
> > > >>> +                         break;
> > > >>> +                 }
> > > >>> +         }
> > > >>> +
> > > >>> +         offset += plane.length;
> > > >>> + }
> > > >>> +
> > > >>> + LOG(Buffer, Debug)
> > > >>> +         << "Buffer is " << (isContiguous ? "not " : "") << "contiguous";
> > > >>> +
> > > >>> + _d()->isContiguous_ = isContiguous;
> > > >>>  }
> > > >>>
> > > >>>  /**

Hirokazu Honda Sept. 3, 2021, 11:29 a.m. UTC | #26

Hi Laurent,

On Fri, Sep 3, 2021 at 3:11 AM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Kieran,
>
> On Thu, Sep 02, 2021 at 10:54:19AM +0100, Kieran Bingham wrote:
> > On 02/09/2021 05:22, Laurent Pinchart wrote:
> > > The metadata planes are allocated by V4L2VideoDevice when dequeuing a
> > > buffer. This causes the metadata planes to only be allocated after a
> > > buffer gets dequeued, and doesn't provide any strong guarantee that
> > > their number matches the number of FrameBuffer planes. The lack of this
> > > invariant makes the FrameBuffer class fragile.
> > >
> > > As a first step towards fixing this, allocate the metadata planes when
> > > the FrameBuffer is constructed. The FrameMetadata API should be further
> > > improved by preventing a change in the number of planes.
> > >
> > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > ---
> > >  src/libcamera/framebuffer.cpp      |  2 ++
> > >  src/libcamera/v4l2_videodevice.cpp | 12 +++++++++---
> > >  2 files changed, 11 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/src/libcamera/framebuffer.cpp b/src/libcamera/framebuffer.cpp
> > > index 99265b44da43..c1529dfb0ad1 100644
> > > --- a/src/libcamera/framebuffer.cpp
> > > +++ b/src/libcamera/framebuffer.cpp
> > > @@ -210,6 +210,8 @@ FrameBuffer::FrameBuffer(const std::vector<Plane> &planes, unsigned int cookie)
> > >     : Extensible(std::make_unique<Private>()), planes_(planes),
> > >       cookie_(cookie)
> > >  {
> > > +   metadata_.planes.resize(planes_.size());

I would set bytesused to some default value, probably 0?
struct Plane {
  unsigned int bytesused = 0;
};

> > > +
> > >     unsigned int offset = 0;
> > >
> > >     /* \todo Remove the assertions after sufficient testing */
> > > diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> > > index adabd4720668..a51971879e75 100644
> > > --- a/src/libcamera/v4l2_videodevice.cpp
> > > +++ b/src/libcamera/v4l2_videodevice.cpp
> > > @@ -1586,12 +1586,18 @@ FrameBuffer *V4L2VideoDevice::dequeueBuffer()
> > >     buffer->metadata_.timestamp = buf.timestamp.tv_sec * 1000000000ULL
> > >                                 + buf.timestamp.tv_usec * 1000ULL;
> > >
> > > -   buffer->metadata_.planes.clear();
> > >     if (multiPlanar) {
> > > +           if (buf.length > buffer->metadata_.planes.size()) {
> > > +                   LOG(V4L2, Error)
> >
> > Are we going to 'leak' queued FrameBuffers at this point?
> > Is there a way to prevent this occurring at queue time rather than dequeue?
> >
> > In fact, Can it even happen? or should it be an ASSERT/Fatal?
>
> It's not supposed to happen, I'd say it's a kernel bug in that case. But
> it's a good point, I think we can return the buffer, and mark its status
> as erroneous. I'll do so.
>
> > > +                           << "Invalid number of planes (" << buf.length
> > > +                           << " != " << buffer->metadata_.planes.size() << ")";
> > > +                   return nullptr;
> > > +           }
> > > +
> > >             for (unsigned int nplane = 0; nplane < buf.length; nplane++)
> > > -                   buffer->metadata_.planes.push_back({ planes[nplane].bytesused });
> > > +                   buffer->metadata_.planes[nplane].bytesused = planes[nplane].bytesused;
> > >     } else {
> > > -           buffer->metadata_.planes.push_back({ buf.bytesused });
> > > +           buffer->metadata_.planes[0].bytesused = buf.bytesused;

Is metadata_.planes[1+] kept unfilled? Shall we execute
metadata_.planes.resize(1)?

Regards,
-Hiro
> > >     }
> > >
> > >     return buffer;
> > >
>
> --
> Regards,
>
> Laurent Pinchart

Hirokazu Honda Sept. 3, 2021, 11:32 a.m. UTC | #27

Hi Laurent,

On Fri, Sep 3, 2021 at 8:23 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Hiro,
>
> On Fri, Sep 03, 2021 at 08:09:07PM +0900, Hirokazu Honda wrote:
> > On Fri, Sep 3, 2021 at 5:40 PM Laurent Pinchart wrote:
> > > On Fri, Sep 03, 2021 at 09:24:02AM +0100, Kieran Bingham wrote:
> > > > On 02/09/2021 19:07, Laurent Pinchart wrote:
> > > > > On Thu, Sep 02, 2021 at 10:43:56AM +0100, Kieran Bingham wrote:
> > > > >> On 02/09/2021 05:22, Laurent Pinchart wrote:
> > > > >>> Multi-planar frame buffers can store their planes contiguously in
> > > > >>> memory, or split them in discontiguous memory areas. Add a private
> > > > >>> function to check in which of these two categories the frame buffer
> > > > >>> belongs. This will be used to correctly handle the differences between
> > > > >>> the V4L2 single and multi planar APIs.
> > > > >>>
> > > > >>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > > >>> ---
> > > > >>>  include/libcamera/internal/framebuffer.h |  2 ++
> > > > >>>  src/libcamera/framebuffer.cpp            | 45 +++++++++++++++++++++++-
> > > > >>>  2 files changed, 46 insertions(+), 1 deletion(-)
> > > > >>>
> > > > >>> diff --git a/include/libcamera/internal/framebuffer.h b/include/libcamera/internal/framebuffer.h
> > > > >>> index 606aed2b4782..cd33c295466e 100644
> > > > >>> --- a/include/libcamera/internal/framebuffer.h
> > > > >>> +++ b/include/libcamera/internal/framebuffer.h
> > > > >>> @@ -21,9 +21,11 @@ public:
> > > > >>>   Private();
> > > > >>>
> > > > >>>   void setRequest(Request *request) { request_ = request; }
> > > > >>> + bool isContiguous() const { return isContiguous_; }
> > > > >>>
> > > > >>>  private:
> > > > >>>   Request *request_;
> > > > >>> + bool isContiguous_;
> > > > >>>  };
> > > > >>>
> > > > >>>  } /* namespace libcamera */
> > > > >>> diff --git a/src/libcamera/framebuffer.cpp b/src/libcamera/framebuffer.cpp
> > > > >>> index 53ef89bf458f..99265b44da43 100644
> > > > >>> --- a/src/libcamera/framebuffer.cpp
> > > > >>> +++ b/src/libcamera/framebuffer.cpp
> > > > >>> @@ -106,7 +106,7 @@ LOG_DEFINE_CATEGORY(Buffer)
> > > > >>>   */
> > > > >>>
> > > > >>>  FrameBuffer::Private::Private()
> > > > >>> - : request_(nullptr)
> > > > >>> + : request_(nullptr), isContiguous_(true)
> > > > >>>  {
> > > > >>>  }
> > > > >>>
> > > > >>> @@ -120,6 +120,17 @@ FrameBuffer::Private::Private()
> > > > >>>   * handlers, it is called by the pipeline handlers themselves.
> > > > >>>   */
> > > > >>>
> > > > >>> +/**
> > > > >>> + * \fn FrameBuffer::Private::isContiguous()
> > > > >>> + * \brief Check if the frame buffer stores planes contiguously in memory
> > > > >>> + *
> > > > >>> + * Multi-planar frame buffers can store their planes contiguously in memory, or
> > > > >>> + * split them in discontiguous memory areas. This function checks in which of
> > > > >>
> > > > >> 'split them into discontiguous'
> > > > >>
> > > > >>> + * these two categories the frame buffer belongs.
> > > > >>> + *
> > > > >>> + * \return True if the planes are stored contiguously in memory, false otherwise
> > > > >>> + */
> > > > >>> +
> > > > >>>  /**
> > > > >>>   * \class FrameBuffer
> > > > >>>   * \brief Frame buffer data and its associated dynamic metadata
> > > > >>> @@ -199,9 +210,41 @@ FrameBuffer::FrameBuffer(const std::vector<Plane> &planes, unsigned int cookie)
> > > > >>>   : Extensible(std::make_unique<Private>()), planes_(planes),
> > > > >>>     cookie_(cookie)
> > > > >>>  {
> > > > >>> + unsigned int offset = 0;
> > > > >>> +
> > > > >>>   /* \todo Remove the assertions after sufficient testing */
> > > > >>>   for (const auto &plane : planes_)
> > > > >>>           ASSERT(plane.offset != Plane::kInvalidOffset);
> >
> > nit:
> > Shall we merge the two for loops?
>
> I've kept them separate to make it easier to remove the first one (and
> now I wonder why I thought it would be easier...), but as the ASSERT is
> now here to stay, I'll merge them.
>
> > At least, I would move offset to before isContiguous.
>
> Done already :-)
>
> > > > >>> +
> > > > >>> + bool isContiguous = true;
> > > > >>> + ino_t inode = 0;
> > > > >>> +
> > > > >>> + for (const auto &plane : planes_) {
> > > > >>> +         if (plane.offset != offset) {
> > > > >>> +                 isContiguous = false;
> > > > >>> +                 break;
> > > > >>> +         }
> > > > >>> +
> > > > >>> +         /*
> > > > >>> +          * Two different dmabuf file descriptors may still refer to the
> > > > >>> +          * same dmabuf instance. Check this using inodes.
> > > > >>
> > > > >> Going back to the FileDescriptor::inode() extension, if that cached the
> > > > >> inode (it can't change) on either first call, or construction, couldn't
> > > > >> this whole check be simplified to:
> > > > >>
> > > > >>            if (plane.fd.inode() != planes_[0].fd.inode()) {
> > > > >>                    isContiguous = false;
> > > > >>                    break;
> > > > >>            }
> > > > >
> > > > > This would call fstat() every time, while the fd comparison here is
> > > > > meant to skip the fstat() calls in case the numerical fd match. We could
> > > > > do
> > > >
> > > > Did you miss the part where I said "If we cache the inode in
> > > > FileDescriptor"?
> > >
> > > No I didn't :-)
> > >
> > > > My intention was that caching there would mean we can simplify
> > > > comparisons here, without incurring repeated fstat calls.
> > >
> > > If the numerical fds are identical, there's no need to compare the
> > > inodes, and we can avoid calling fstat() at all.
> > >
> > > > >             if (plane.fd != planes_[0].fd)
> > > > >
> > > > > if we introduced an operator==(), but as explained in a reply to the
> > > > > patch that adds inode(), I'd like to handle that later, to avoid
> > > > > blocking this series.
> > > >
> > > > Ok, I still don't see the harm in caching the inode within
> > > > FileDescriptor(), it can't change once read the first time.
> > >
> > > What we'd need to do in operator==() is to first compare the numerical
> > > fds, and only compare the inodes if the fds differ. Technically that's
> > > not an issue, but it requires deciding what the semantics of equality is
> > > as explaining in a separate e-mail, and that's the part that I'd like to
> > > defer to avoid delaying the offset fixes.
> > >
> > > > But, it can be deferred:
> > > >
> > > > Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> > > >
> > > > >>> +          */
> > > > >>> +         if (plane.fd.fd() != planes_[0].fd.fd()) {
> > > > >>> +                 if (!inode)
> > > > >>> +                         inode = planes_[0].fd.inode();
> >
> > Shall error case (i.e plane.fd.inode() == 0) be handled?
>
> This can only happen if the application gives us an incorrect fd, I'm
> not sure how much we need to protect against applications doing very
> very bad things.
>

Okay, so let's assume this is caught by other places. :-)

Reviewed-by: Hirokazu Honda <hiroh@chromium.org>

-Hiro

> > > > >>> +                 if (plane.fd.inode() != inode) {
> > > > >>> +                         isContiguous = false;
> > > > >>> +                         break;
> > > > >>> +                 }
> > > > >>> +         }
> > > > >>> +
> > > > >>> +         offset += plane.length;
> > > > >>> + }
> > > > >>> +
> > > > >>> + LOG(Buffer, Debug)
> > > > >>> +         << "Buffer is " << (isContiguous ? "not " : "") << "contiguous";
> > > > >>> +
> > > > >>> + _d()->isContiguous_ = isContiguous;
> > > > >>>  }
> > > > >>>
> > > > >>>  /**
>
> --
> Regards,
>
> Laurent Pinchart

Hirokazu Honda Sept. 3, 2021, 11:40 a.m. UTC | #28

Hi Laurent, thank you for the patch.

On Fri, Sep 3, 2021 at 5:26 PM Kieran Bingham
<kieran.bingham@ideasonboard.com> wrote:
>
> On 02/09/2021 19:20, Laurent Pinchart wrote:
> > Hi Kieran,
> >
> > On Thu, Sep 02, 2021 at 11:03:47AM +0100, Kieran Bingham wrote:
> >> On 02/09/2021 05:22, Laurent Pinchart wrote:
> >>> The number of metadata planes should always match the number of frame
> >>> buffer planes. Enforce this by making the vector private and providing
> >>> accessor functions.
> >>>

I am confused by the definition of Metadata::Planes.
It has bytesused and it is set to v4l2_buf.bytesused. It seems to
match the v4l2 format plane.
That is, it is not always the number of frame buffer planes because
planes in FrameBuffer are color format planes.

Regards,
-Hiro


> >>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> >>> ---
> >>>  include/libcamera/framebuffer.h    | 10 +++++++++-
> >>>  src/cam/camera_session.cpp         |  4 ++--
> >>>  src/cam/file_sink.cpp              |  2 +-
> >>>  src/libcamera/framebuffer.cpp      | 12 +++++++++---
> >>>  src/libcamera/v4l2_videodevice.cpp | 14 +++++++-------
> >>>  src/qcam/main_window.cpp           |  2 +-
> >>>  src/qcam/viewfinder_gl.cpp         |  2 +-
> >>>  src/qcam/viewfinder_qt.cpp         |  2 +-
> >>>  src/v4l2/v4l2_camera_proxy.cpp     |  2 +-
> >>>  9 files changed, 32 insertions(+), 18 deletions(-)
> >>>
> >>> diff --git a/include/libcamera/framebuffer.h b/include/libcamera/framebuffer.h
> >>> index fd68ed0a139d..7f2f176af691 100644
> >>> --- a/include/libcamera/framebuffer.h
> >>> +++ b/include/libcamera/framebuffer.h
> >>> @@ -13,6 +13,7 @@
> >>>  #include <vector>
> >>>
> >>>  #include <libcamera/base/class.h>
> >>> +#include <libcamera/base/span.h>
> >>>
> >>>  #include <libcamera/file_descriptor.h>
> >>>
> >>> @@ -34,7 +35,14 @@ struct FrameMetadata {
> >>>     Status status;
> >>>     unsigned int sequence;
> >>>     uint64_t timestamp;
> >>> -   std::vector<Plane> planes;
> >>> +
> >>> +   Span<Plane> planes() { return planes_; }
> >>> +   Span<const Plane> planes() const { return planes_; }
> >>
> >> Returning a span here is nice.
> >>
> >> This likely causes compile breakage for any external app.
> >>
> >> I know we're not ABI/API stable, but I wonder if we should highlight
> >> when we do cause breakage somehow, perhaps in the commit message at
> >> least, as we already have external users who we might want to notify.
> >
> > Most changes to public header files will be ABI/API breakages at this
> > point, and users will certainly notice when they get a compilation
> > failure :-) What value do you think this would bring, who would read
> > those messages ? I think it could be different for things that change
> > the ABI without breaking compilation and that would require more than a
> > recompilation to fix, there a warning seems to have more value, but I'm
> > also sure we'll forget from time to time, if not most of the time.
>
> Because I think we need to start being more aware of when we introduce
> both ABI and API breakages.
>
> At least making it explicitly noted in the commit message states that we
> were aware of the breakage at that point.
>
>
> >>> +
> >>> +private:
> >>> +   friend class FrameBuffer;
> >>> +
> >>> +   std::vector<Plane> planes_;
> >>>  };
> >>>
> >>>  class FrameBuffer final : public Extensible
> >>> diff --git a/src/cam/camera_session.cpp b/src/cam/camera_session.cpp
> >>> index 60d640f2b15c..32a373a99b72 100644
> >>> --- a/src/cam/camera_session.cpp
> >>> +++ b/src/cam/camera_session.cpp
> >>> @@ -374,9 +374,9 @@ void CameraSession::processRequest(Request *request)
> >>>                  << " bytesused: ";
> >>>
> >>>             unsigned int nplane = 0;
> >>> -           for (const FrameMetadata::Plane &plane : metadata.planes) {
> >>> +           for (const FrameMetadata::Plane &plane : metadata.planes()) {
> >>>                     info << plane.bytesused;
> >>> -                   if (++nplane < metadata.planes.size())
> >>> +                   if (++nplane < metadata.planes().size())
> >>>                             info << "/";
> >>>             }
> >>>     }
> >>> diff --git a/src/cam/file_sink.cpp b/src/cam/file_sink.cpp
> >>> index 0b529e3eb767..0fc7d621f50b 100644
> >>> --- a/src/cam/file_sink.cpp
> >>> +++ b/src/cam/file_sink.cpp
> >>> @@ -110,7 +110,7 @@ void FileSink::writeBuffer(const Stream *stream, FrameBuffer *buffer)
> >>>
> >>>     for (unsigned int i = 0; i < buffer->planes().size(); ++i) {
> >>>             const FrameBuffer::Plane &plane = buffer->planes()[i];
> >>> -           const FrameMetadata::Plane &meta = buffer->metadata().planes[i];
> >>> +           const FrameMetadata::Plane &meta = buffer->metadata().planes()[i];
> >>>
> >>>             uint8_t *data = planeData_[&plane];
> >>>             unsigned int length = std::min(meta.bytesused, plane.length);
> >>> diff --git a/src/libcamera/framebuffer.cpp b/src/libcamera/framebuffer.cpp
> >>> index c1529dfb0ad1..83cf7b83d182 100644
> >>> --- a/src/libcamera/framebuffer.cpp
> >>> +++ b/src/libcamera/framebuffer.cpp
> >>> @@ -91,8 +91,14 @@ LOG_DEFINE_CATEGORY(Buffer)
> >>>   */
> >>>
> >>>  /**
> >>> - * \var FrameMetadata::planes
> >>> - * \brief Array of per-plane metadata
> >>> + * \fn FrameMetadata::planes()
> >>> + * \copydoc FrameMetadata::planes() const
> >>> + */
> >>> +
> >>> +/**
> >>> + * \fn FrameMetadata::planes() const
> >>> + * \brief Retrieve the array of per-plane metadata
> >>> + * \return The array of per-plane metadata
> >>>   */
> >>>
> >>>  /**
> >>> @@ -210,7 +216,7 @@ FrameBuffer::FrameBuffer(const std::vector<Plane> &planes, unsigned int cookie)
> >>>     : Extensible(std::make_unique<Private>()), planes_(planes),
> >>>       cookie_(cookie)
> >>>  {
> >>> -   metadata_.planes.resize(planes_.size());
> >>> +   metadata_.planes_.resize(planes_.size());
> >>>
> >>>     unsigned int offset = 0;
> >>>
> >>> diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> >>> index a51971879e75..82ddaed3656f 100644
> >>> --- a/src/libcamera/v4l2_videodevice.cpp
> >>> +++ b/src/libcamera/v4l2_videodevice.cpp
> >>> @@ -1485,14 +1485,14 @@ int V4L2VideoDevice::queueBuffer(FrameBuffer *buffer)
> >>>
> >>>             if (multiPlanar) {
> >>>                     unsigned int nplane = 0;
> >>> -                   for (const FrameMetadata::Plane &plane : metadata.planes) {
> >>> +                   for (const FrameMetadata::Plane &plane : metadata.planes()) {
> >>>                             v4l2Planes[nplane].bytesused = plane.bytesused;
> >>>                             v4l2Planes[nplane].length = buffer->planes()[nplane].length;
> >>>                             nplane++;
> >>>                     }
> >>>             } else {
> >>> -                   if (metadata.planes.size())
> >>> -                           buf.bytesused = metadata.planes[0].bytesused;
> >>> +                   if (metadata.planes().size())
> >>> +                           buf.bytesused = metadata.planes()[0].bytesused;
> >>>             }
> >>>
> >>>             buf.sequence = metadata.sequence;
> >>> @@ -1587,17 +1587,17 @@ FrameBuffer *V4L2VideoDevice::dequeueBuffer()
> >>>                                 + buf.timestamp.tv_usec * 1000ULL;
> >>>
> >>>     if (multiPlanar) {
> >>> -           if (buf.length > buffer->metadata_.planes.size()) {
> >>> +           if (buf.length > buffer->metadata_.planes().size()) {
> >>>                     LOG(V4L2, Error)
> >>>                             << "Invalid number of planes (" << buf.length
> >>> -                           << " != " << buffer->metadata_.planes.size() << ")";
> >>> +                           << " != " << buffer->metadata_.planes().size() << ")";
> >>>                     return nullptr;
> >>>             }
> >>>
> >>>             for (unsigned int nplane = 0; nplane < buf.length; nplane++)
> >>> -                   buffer->metadata_.planes[nplane].bytesused = planes[nplane].bytesused;
> >>> +                   buffer->metadata_.planes()[nplane].bytesused = planes[nplane].bytesused;
> >>>     } else {
> >>> -           buffer->metadata_.planes[0].bytesused = buf.bytesused;
> >>> +           buffer->metadata_.planes()[0].bytesused = buf.bytesused;
> >>>     }
> >>>
> >>>     return buffer;
> >>> diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp
> >>> index 1536b2b5bd66..ac853e360aea 100644
> >>> --- a/src/qcam/main_window.cpp
> >>> +++ b/src/qcam/main_window.cpp
> >>> @@ -756,7 +756,7 @@ void MainWindow::processViewfinder(FrameBuffer *buffer)
> >>>
> >>>     qDebug().noquote()
> >>>             << QString("seq: %1").arg(metadata.sequence, 6, 10, QLatin1Char('0'))
> >>> -           << "bytesused:" << metadata.planes[0].bytesused
> >>> +           << "bytesused:" << metadata.planes()[0].bytesused
> >>>             << "timestamp:" << metadata.timestamp
> >>>             << "fps:" << Qt::fixed << qSetRealNumberPrecision(2) << fps;
> >>>
> >>> diff --git a/src/qcam/viewfinder_gl.cpp b/src/qcam/viewfinder_gl.cpp
> >>> index 40226601f9fd..d2ef036974f4 100644
> >>> --- a/src/qcam/viewfinder_gl.cpp
> >>> +++ b/src/qcam/viewfinder_gl.cpp
> >>> @@ -125,7 +125,7 @@ void ViewFinderGL::render(libcamera::FrameBuffer *buffer,
> >>>     /*
> >>>      * \todo Get the stride from the buffer instead of computing it naively
> >>>      */
> >>> -   stride_ = buffer->metadata().planes[0].bytesused / size_.height();
> >>> +   stride_ = buffer->metadata().planes()[0].bytesused / size_.height();
> >>
> >> Can this be obtained from the PixelFormatInfo now ?
> >> or do we still not expose that to applications...
> >
> > PixelFormatInfo is still internal. I'm considering exposing it, but then
> > it would be good to move the V4L2-specific part somewhere else.
> >
> >> Anyway, that's likely a change on top even if we could to solve the \todo.
> >>
> >> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> >>
> >>>     update();
> >>>     buffer_ = buffer;
> >>>  }
> >>> diff --git a/src/qcam/viewfinder_qt.cpp b/src/qcam/viewfinder_qt.cpp
> >>> index efa1d412584b..a0bf99b0b522 100644
> >>> --- a/src/qcam/viewfinder_qt.cpp
> >>> +++ b/src/qcam/viewfinder_qt.cpp
> >>> @@ -87,7 +87,7 @@ void ViewFinderQt::render(libcamera::FrameBuffer *buffer,
> >>>     }
> >>>
> >>>     unsigned char *memory = mem.data();
> >>> -   size_t size = buffer->metadata().planes[0].bytesused;
> >>> +   size_t size = buffer->metadata().planes()[0].bytesused;
> >>>
> >>>     {
> >>>             QMutexLocker locker(&mutex_);
> >>> diff --git a/src/v4l2/v4l2_camera_proxy.cpp b/src/v4l2/v4l2_camera_proxy.cpp
> >>> index 8d8ee395954f..68e47ee81834 100644
> >>> --- a/src/v4l2/v4l2_camera_proxy.cpp
> >>> +++ b/src/v4l2/v4l2_camera_proxy.cpp
> >>> @@ -212,7 +212,7 @@ void V4L2CameraProxy::updateBuffers()
> >>>
> >>>             switch (fmd.status) {
> >>>             case FrameMetadata::FrameSuccess:
> >>> -                   buf.bytesused = fmd.planes[0].bytesused;
> >>> +                   buf.bytesused = fmd.planes()[0].bytesused;
> >>>                     buf.field = V4L2_FIELD_NONE;
> >>>                     buf.timestamp.tv_sec = fmd.timestamp / 1000000000;
> >>>                     buf.timestamp.tv_usec = fmd.timestamp % 1000000;
> >

Hirokazu Honda Sept. 3, 2021, 11:54 a.m. UTC | #29

Hi Laurent, thank you for the patch.

On Fri, Sep 3, 2021 at 5:48 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Kieran,
>
> On Fri, Sep 03, 2021 at 09:29:40AM +0100, Kieran Bingham wrote:
> > On 02/09/2021 19:26, Laurent Pinchart wrote:
> > > On Thu, Sep 02, 2021 at 11:21:29AM +0100, Kieran Bingham wrote:
> > >> On 02/09/2021 05:23, Laurent Pinchart wrote:
> > >>> Cache the PixelFormatInfo instead of looking it up in every call to
> > >>> createBuffer(). This prepare for usage of the info in queueBuffer(), to
> > >>
> > >> s/prepare/prepares/
> > >>
> > >>> avoid a looking every time a buffer is queued.
> > >>>
> > >>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > >>> ---
> > >>>  include/libcamera/internal/v4l2_videodevice.h |  1 +
> > >>>  src/libcamera/v4l2_videodevice.cpp            | 19 +++++++++++--------
> > >>>  2 files changed, 12 insertions(+), 8 deletions(-)
> > >>>
> > >>> diff --git a/include/libcamera/internal/v4l2_videodevice.h b/include/libcamera/internal/v4l2_videodevice.h
> > >>> index 7a145f608a5b..6096cd609b97 100644
> > >>> --- a/include/libcamera/internal/v4l2_videodevice.h
> > >>> +++ b/include/libcamera/internal/v4l2_videodevice.h
> > >>> @@ -243,6 +243,7 @@ private:
> > >>>
> > >>>   V4L2Capability caps_;
> > >>>   V4L2DeviceFormat format_;
> > >>> + const PixelFormatInfo *formatInfo_;
> > >>>
> > >>>   enum v4l2_buf_type bufferType_;
> > >>>   enum v4l2_memory memoryType_;
> > >>> diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> > >>> index 82ddaed3656f..2d9a94c3c974 100644
> > >>> --- a/src/libcamera/v4l2_videodevice.cpp
> > >>> +++ b/src/libcamera/v4l2_videodevice.cpp
> > >>> @@ -482,8 +482,8 @@ const std::string V4L2DeviceFormat::toString() const
> > >>>   * \param[in] deviceNode The file-system path to the video device node
> > >>>   */
> > >>>  V4L2VideoDevice::V4L2VideoDevice(const std::string &deviceNode)
> > >>> - : V4L2Device(deviceNode), cache_(nullptr), fdBufferNotifier_(nullptr),
> > >>> -   streaming_(false)
> > >>> + : V4L2Device(deviceNode), formatInfo_(nullptr), cache_(nullptr),
> > >>> +   fdBufferNotifier_(nullptr), streaming_(false)
> > >>>  {
> > >>>   /*
> > >>>    * We default to an MMAP based CAPTURE video device, however this will
> > >>> @@ -586,6 +586,7 @@ int V4L2VideoDevice::open()
> > >>>           return ret;
> > >>>   }
> > >>>
> > >>> + formatInfo_ = &PixelFormatInfo::info(format_.fourcc);
> > >>
> > >> I shudder a little seeing code hugging the return like that ;-)
> > >>
> > >> But it's purely subjective, same below of course.
> > >
> > > We have various patterns indeed :-) I haven't been able to establish yet
> > > what my mental process to deal with these are, sometimes it bothers me
> > > too, but not always.
> > >
> > >>>   return 0;
> > >>>  }
> > >>>
> > >>> @@ -681,6 +682,7 @@ int V4L2VideoDevice::open(int handle, enum v4l2_buf_type type)
> > >>>           return ret;
> > >>>   }
> > >>>
> > >>> + formatInfo_ = &PixelFormatInfo::info(format_.fourcc);
> > >>>   return 0;
> > >>>  }
> > >>>
> > >>> @@ -695,6 +697,8 @@ void V4L2VideoDevice::close()
> > >>>   releaseBuffers();
> > >>>   delete fdBufferNotifier_;
> > >>>
> > >>> + formatInfo_ = nullptr;
> > >>> +
> > >>>   V4L2Device::close();
> > >>>  }
> > >>>
> > >>> @@ -787,6 +791,7 @@ int V4L2VideoDevice::setFormat(V4L2DeviceFormat *format)
> > >>>           return ret;
> > >>>
> > >>>   format_ = *format;
> > >>> + formatInfo_ = &PixelFormatInfo::info(format_.fourcc);
> > >>
> > >> Looks like we were already using that pattern though ;-)
> > >>
> > >>>   return 0;
> > >>>  }
> > >>>
> > >>> @@ -1325,19 +1330,17 @@ std::unique_ptr<FrameBuffer> V4L2VideoDevice::createBuffer(unsigned int index)
> > >>>           planes.push_back(std::move(plane));
> > >>>   }
> > >>>
> > >>> - const auto &info = PixelFormatInfo::info(format_.fourcc);
> > >>> - if (info.isValid() && info.numPlanes() != numPlanes) {
> > >>> + if (formatInfo_->isValid() && formatInfo_->numPlanes() != numPlanes) {
> > >>>           ASSERT(numPlanes == 1u);
> > >>
> > >> Not for this patch, but I guess this assert needed a
> > >>   \todo Multiplanar support
> > >>
> > >> around it?
> > >
> > > No, when formatInfo_->numPlanes() != numPlanes, it means that we have a
> > > multi-planar FrameBuffer with a single-planar V4L2 buffer format (e.g.
> > > V4L2_PIX_FMT_NV12 as opposed to V4L2_PIX_FMT_NV12M). numPlanes must be 1
> > > in that case, otherwise it's a kernel bug.
> >
> > As you're updating the conditional there anyway, could you add a short
> > brief to explain that before the ASSERT please?
> >
> > I don't think that extra context is clear from just the if statement.
> > (Perhaps there is more context outside of the hunk maybe, but it's not
> > clear from just this diff).
>
> That's because this diff only deals with format caching :-) It's a good
> point though, it's not trivial so I'll add a separate patch to document
> this properly.
>
> > >> However, this patch is just for caching the formatInfo, which seems
> > >> reasonable so
> > >>
> > >> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> > >>
> > >>> -         const size_t numColorPlanes = info.numPlanes();
> > >>> -         planes.resize(numColorPlanes);
> > >>> +         planes.resize(formatInfo_->numPlanes());

By the way, what I would not like in PixelFormatInfo(), numPlanes()
does for-loops always.
Shall we add numPlanes to PixelFormatInfo?

Definitely, it is not related to this patch.

Reviewed-by: Hirokazu Honda <hiroh@chromium.org>

-Hiro
> > >>>           const FileDescriptor &fd = planes[0].fd;
> > >>>           size_t offset = 0;
> > >>> -         for (size_t i = 0; i < numColorPlanes; ++i) {
> > >>> +         for (size_t i = 0; i < planes.size(); ++i) {
> > >>>                   planes[i].fd = fd;
> > >>>                   planes[i].offset = offset;
> > >>>
> > >>>                   /* \todo Take the V4L2 stride into account */
> > >>> -                 planes[i].length = info.planeSize(format_.size, i);
> > >>> +                 planes[i].length = formatInfo_->planeSize(format_.size, i);
> > >>>                   offset += planes[i].length;
> > >>>           }
> > >>>   }
>
> --
> Regards,
>
> Laurent Pinchart

Hirokazu Honda Sept. 3, 2021, 12:25 p.m. UTC | #30

Hi Laurent, thank you for the patch.

On Thu, Sep 2, 2021 at 8:25 PM Kieran Bingham
<kieran.bingham@ideasonboard.com> wrote:
>
> On 02/09/2021 05:23, Laurent Pinchart wrote:
> > When queueing a buffer to a V4L2VideoDevice, the number of planes in the
> > FrameBuffer may not match the number of V4L2 buffer planes if the
> > PixelFormat is multi-planar (has multiple colour planes) and the V4L2
> > format is single-planar (has a single buffer plane). In this case, we
> > need to coalesce all FrameBuffer planes into a single V4L2 buffer plane.
> > Do so, and add validity checks to reject frame buffers that can't be
> > described using a single V4L2 buffer plane.
>
>
> Yikes ;-) lets see...
>
> >
> > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > ---
> >  src/libcamera/v4l2_videodevice.cpp | 68 +++++++++++++++++++++++++-----
> >  1 file changed, 58 insertions(+), 10 deletions(-)
> >
> > diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> > index 2d9a94c3c974..79cb792117d5 100644
> > --- a/src/libcamera/v4l2_videodevice.cpp
> > +++ b/src/libcamera/v4l2_videodevice.cpp
> > @@ -22,10 +22,12 @@
> >
> >  #include <libcamera/base/event_notifier.h>
> >  #include <libcamera/base/log.h>
> > +#include <libcamera/base/utils.h>
> >
> >  #include <libcamera/file_descriptor.h>
> >
> >  #include "libcamera/internal/formats.h"
> > +#include "libcamera/internal/framebuffer.h"
> >  #include "libcamera/internal/media_device.h"
> >  #include "libcamera/internal/media_object.h"
> >
> > @@ -1468,10 +1470,21 @@ int V4L2VideoDevice::queueBuffer(FrameBuffer *buffer)
> >
> >       bool multiPlanar = V4L2_TYPE_IS_MULTIPLANAR(buf.type);
> >       const std::vector<FrameBuffer::Plane> &planes = buffer->planes();
> > +     unsigned int numV4l2Planes = format_.fourcc == formatInfo_->v4l2Format[0]
> > +                                ? 1 : planes.size();

nit const unsigned int.
I would replace formatInfo->numPlanes() with planes.size().

>
> Aha, ok so that's how we recover the planar/multiplanar information I
> was concerned we had lost earlier.
>
> I wonder if we should set/cache numV4l2Planes_ when we set formatInfo_
> as they are so closely paired .. But it's a trivial operation here.
>
>
>
> > +
> > +     /*
> > +      * If the frame buffer has multiple planes and the V4L2 format requires
> > +      * contiguous planes, ensure that's the case.
> > +      */
> > +     if (planes.size() != numV4l2Planes && !buffer->_d()->isContiguous()) {
> > +             LOG(V4L2, Error) << "Device format requires contiguous buffer";
> > +             return -EINVAL;
> > +     }
> >
> >       if (buf.memory == V4L2_MEMORY_DMABUF) {
> >               if (multiPlanar) {
> > -                     for (unsigned int p = 0; p < planes.size(); ++p)
> > +                     for (unsigned int p = 0; p < numV4l2Planes; ++p)
> >                               v4l2Planes[p].m.fd = planes[p].fd.fd();
> >               } else {
> >                       buf.m.fd = planes[0].fd.fd();
> > @@ -1479,23 +1492,58 @@ int V4L2VideoDevice::queueBuffer(FrameBuffer *buffer)
> >       }
> >
> >       if (multiPlanar) {
> > -             buf.length = planes.size();
> > +             buf.length = numV4l2Planes;
> >               buf.m.planes = v4l2Planes;
> >       }
> >
> >       if (V4L2_TYPE_IS_OUTPUT(buf.type)) {
> >               const FrameMetadata &metadata = buffer->metadata();
> >
> > -             if (multiPlanar) {
> > -                     unsigned int nplane = 0;
> > -                     for (const FrameMetadata::Plane &plane : metadata.planes()) {
> > -                             v4l2Planes[nplane].bytesused = plane.bytesused;
> > -                             v4l2Planes[nplane].length = buffer->planes()[nplane].length;
> > -                             nplane++;
> > +             if (numV4l2Planes != planes.size()) {
> > +                     /*
> > +                      * If we have a multi-planar buffer with a V4L2
> > +                      * single-planar format, coalesce all planes. The length
> > +                      * and number of bytes used may only differ in the last
> > +                      * plane as any other situation can't be represented.
> > +                      */
> > +                     unsigned int bytesused = 0;
> > +                     unsigned int length = 0;
> > +
> > +                     for (auto [i, plane] : utils::enumerate(planes)) {
> > +                             bytesused += metadata.planes()[i].bytesused;
> > +                             length += plane.length;
> > +
> > +                             if (i != planes.size() - 1 && bytesused != length) {
>
> Ah, i almost thought that was wrong, but it's
>    "If we're not the last plane, and the plane is not fully used..."
>
> So it's fine.
>
>
> > +                                     LOG(V4L2, Error)
> > +                                             << "Holes in multi-planar buffer not supported";
> > +                                     return -EINVAL;
> > +                             }
> > +                     }
> > +
> > +                     if (multiPlanar) {
> > +                             v4l2Planes[0].bytesused = bytesused;
> > +                             v4l2Planes[0].length = length;

I think multiPlanar must be false here.

v4l2 format | plane.size() | numV4L2Planes |     numV4L2Planes ==
planes.size()
multi          |         1        |      planes.size() |          true
multi          |         2+      |      planes.size() |           true
single        |         1        |       1                 |           true
single        |         2+        |       1                 |           false

So I would implement as follows.
if (multiPlanar) {
   ...
} else {
  if (numV4l2Planes != planes.size()) {
    ...
  } else {
     ...
  }
}


> > +                     } else {
> > +                             buf.bytesused = bytesused;
> > +                             buf.length = length;
> > +                     }
> > +             } else if (multiPlanar) {
> > +                     /*
> > +                      * If we use the multi-planar API, fill in the planes.
> > +                      * The number of planes in the frame buffer and in the
> > +                      * V4L2 buffer is guaranteed to be equal at this point.
> > +                      */
> > +                     for (auto [i, plane] : utils::enumerate(planes)) {
> > +                             v4l2Planes[i].bytesused = metadata.planes()[i].bytesused;
> > +                             v4l2Planes[i].length = plane.length;
> >                       }
> >               } else {
> > -                     if (metadata.planes().size())
>
> Guaranteeing the planes are there is much nicer here.
>
> Phew, this patch 'looks' a lot more complex than it actually is,
>
> So I think ... this is fine.
>
> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
>
> > -                             buf.bytesused = metadata.planes()[0].bytesused;
> > +                     /*
> > +                      * Single-planar API with a single plane in the buffer
> > +                      * is trivial to handle.
> > +                      */
> > +                     buf.bytesused = metadata.planes()[0].bytesused;
> > +                     buf.length = planes[0].length;
> >               }
> >
> >               buf.sequence = metadata.sequence;
> >

Hirokazu Honda Sept. 3, 2021, 12:36 p.m. UTC | #31

Hi Laurent, thank you for the patch.

On Fri, Sep 3, 2021 at 3:31 AM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Kieran,
>
> On Thu, Sep 02, 2021 at 12:17:31PM +0100, Kieran Bingham wrote:
> > On 02/09/2021 05:23, Laurent Pinchart wrote:
> > > When dequeueing a buffer from a V4L2VideoDevice, the number of planes in
> > > the FrameBuffer may not match the number of V4L2 buffer planes if the
> > > PixelFormat is multi-planar (has multiple colour planes) and the V4L2
> > > format is single-planar (has a single buffer plane). In this case, we
> > > need to split the single V4L2 buffer plane into FrameBuffer planes. Do
> > > so, and add checks to reject invalid V4L2 buffers in case of a driver
> > > issue.
> > >
> > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > ---
> > >  src/libcamera/v4l2_videodevice.cpp | 54 ++++++++++++++++++++++++------
> > >  1 file changed, 43 insertions(+), 11 deletions(-)
> > >
> > > diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> > > index 79cb792117d5..b80b9038a914 100644
> > > --- a/src/libcamera/v4l2_videodevice.cpp
> > > +++ b/src/libcamera/v4l2_videodevice.cpp
> > > @@ -7,6 +7,7 @@
> > >
> > >  #include "libcamera/internal/v4l2_videodevice.h"
> > >
> > > +#include <algorithm>
> > >  #include <array>
> > >  #include <fcntl.h>
> > >  #include <iomanip>
> > > @@ -1637,18 +1638,49 @@ FrameBuffer *V4L2VideoDevice::dequeueBuffer()
> > >     buffer->metadata_.timestamp = buf.timestamp.tv_sec * 1000000000ULL
> > >                                 + buf.timestamp.tv_usec * 1000ULL;
> > >
> > > -   if (multiPlanar) {
> > > -           if (buf.length > buffer->metadata_.planes().size()) {
> > > -                   LOG(V4L2, Error)
> > > -                           << "Invalid number of planes (" << buf.length
> > > -                           << " != " << buffer->metadata_.planes().size() << ")";
> > > -                   return nullptr;
> > > +   if (V4L2_TYPE_IS_CAPTURE(buf.type)) {
> >
> > Should we swap this around and return the buffer early to lower the
> > indents below?
> >
> >  if (!V4L2_TYPE_IS_CAPTURE(buf.type)) {
> >       /* Output buffer is completed */
> >       return buffer;
> >  }
>
> Yes, good point.
>
> > > +           unsigned int numV4l2Planes = multiPlanar ? buf.length : 1;
> > > +           FrameMetadata &metadata = buffer->metadata_;
> > > +
> > > +           if (numV4l2Planes != metadata.planes().size()) {

If my comment to 10/12 is correct, could you implement this like
So I would implement as follows.
if (multiPlanar) {
   ...
} else {
  if (numV4l2Planes != metadata.planes.size()) {
    ...
  } else {
     ...
}

Thanks,
-Hiro

> > > +                   /*
> > > +                    * If we have a multi-planar buffer with a V4L2
> > > +                    * single-planar format, split the V4L2 buffer across
> > > +                    * the buffer planes. Only the last plane may have less
> > > +                    * bytes used than its length.
> > > +                    */
> > > +                   if (numV4l2Planes != 1) {
> > > +                           LOG(V4L2, Error)
> > > +                                   << "Invalid number of planes (" << numV4l2Planes
> > > +                                   << " != " << metadata.planes().size() << ")";
> > > +                           return nullptr;
> > > +                   }
> > > +
> > > +                   unsigned int bytesused = multiPlanar ? planes[0].bytesused
> > > +                                          : buf.bytesused;
> > > +
> > > +                   for (auto [i, plane] : utils::enumerate(buffer->planes())) {
> > > +                           if (!bytesused) {
> > > +                                   LOG(V4L2, Error)
> > > +                                           << "Dequeued buffer is too small";
> >
> > Presumably if we got here - it's likely something (hardware) already
> > overwrote memory it likely didn't own?
>
> It likely means something bad happened, yes. It's a bit of defensive
> programming, to output a clear error message if the kernel does
> something bad, but also to possibly catch bugs in the implementation of
> this function.
>
> > > +                                   return nullptr;
> >
> > I'm still very weary of returning nullptrs on dequeue when a buffer is
> > removed from queuedBuffers_
> >
> >
> > Wouldn't it be better to return the buffer, but mark it as corrupted or
> > invalid or such ?
>
> Yes, I'll do that.
>
> > > +                           }
> > > +
> > > +                           metadata.planes()[i].bytesused =
> > > +                                   std::min(plane.length, bytesused);
> > > +                           bytesused -= metadata.planes()[i].bytesused;
> > > +                   }
> > > +           } else if (multiPlanar) {
> > > +                   /*
> > > +                    * If we use the multi-planar API, fill in the planes.
> > > +                    * The number of planes in the frame buffer and in the
> > > +                    * V4L2 buffer is guaranteed to be equal at this point.
> > > +                    */
> > > +                   for (unsigned int i = 0; i < numV4l2Planes; ++i)
> > > +                           metadata.planes()[i].bytesused = planes[i].bytesused;
> > > +           } else {
> > > +                   metadata.planes()[0].bytesused = buf.bytesused;
> > >             }
> > > -
> > > -           for (unsigned int nplane = 0; nplane < buf.length; nplane++)
> > > -                   buffer->metadata_.planes()[nplane].bytesused = planes[nplane].bytesused;
> > > -   } else {
> > > -           buffer->metadata_.planes()[0].bytesused = buf.bytesused;
> > >     }
> > >
> > >     return buffer;
> > >
>
> --
> Regards,
>
> Laurent Pinchart

Laurent Pinchart Sept. 3, 2021, 1:56 p.m. UTC | #32

Hi Hiro,

On Fri, Sep 03, 2021 at 08:54:52PM +0900, Hirokazu Honda wrote:
> On Fri, Sep 3, 2021 at 5:48 PM Laurent Pinchart wrote:
> > On Fri, Sep 03, 2021 at 09:29:40AM +0100, Kieran Bingham wrote:
> > > On 02/09/2021 19:26, Laurent Pinchart wrote:
> > > > On Thu, Sep 02, 2021 at 11:21:29AM +0100, Kieran Bingham wrote:
> > > >> On 02/09/2021 05:23, Laurent Pinchart wrote:
> > > >>> Cache the PixelFormatInfo instead of looking it up in every call to
> > > >>> createBuffer(). This prepare for usage of the info in queueBuffer(), to
> > > >>
> > > >> s/prepare/prepares/
> > > >>
> > > >>> avoid a looking every time a buffer is queued.
> > > >>>
> > > >>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > >>> ---
> > > >>>  include/libcamera/internal/v4l2_videodevice.h |  1 +
> > > >>>  src/libcamera/v4l2_videodevice.cpp            | 19 +++++++++++--------
> > > >>>  2 files changed, 12 insertions(+), 8 deletions(-)
> > > >>>
> > > >>> diff --git a/include/libcamera/internal/v4l2_videodevice.h b/include/libcamera/internal/v4l2_videodevice.h
> > > >>> index 7a145f608a5b..6096cd609b97 100644
> > > >>> --- a/include/libcamera/internal/v4l2_videodevice.h
> > > >>> +++ b/include/libcamera/internal/v4l2_videodevice.h
> > > >>> @@ -243,6 +243,7 @@ private:
> > > >>>
> > > >>>   V4L2Capability caps_;
> > > >>>   V4L2DeviceFormat format_;
> > > >>> + const PixelFormatInfo *formatInfo_;
> > > >>>
> > > >>>   enum v4l2_buf_type bufferType_;
> > > >>>   enum v4l2_memory memoryType_;
> > > >>> diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> > > >>> index 82ddaed3656f..2d9a94c3c974 100644
> > > >>> --- a/src/libcamera/v4l2_videodevice.cpp
> > > >>> +++ b/src/libcamera/v4l2_videodevice.cpp
> > > >>> @@ -482,8 +482,8 @@ const std::string V4L2DeviceFormat::toString() const
> > > >>>   * \param[in] deviceNode The file-system path to the video device node
> > > >>>   */
> > > >>>  V4L2VideoDevice::V4L2VideoDevice(const std::string &deviceNode)
> > > >>> - : V4L2Device(deviceNode), cache_(nullptr), fdBufferNotifier_(nullptr),
> > > >>> -   streaming_(false)
> > > >>> + : V4L2Device(deviceNode), formatInfo_(nullptr), cache_(nullptr),
> > > >>> +   fdBufferNotifier_(nullptr), streaming_(false)
> > > >>>  {
> > > >>>   /*
> > > >>>    * We default to an MMAP based CAPTURE video device, however this will
> > > >>> @@ -586,6 +586,7 @@ int V4L2VideoDevice::open()
> > > >>>           return ret;
> > > >>>   }
> > > >>>
> > > >>> + formatInfo_ = &PixelFormatInfo::info(format_.fourcc);
> > > >>
> > > >> I shudder a little seeing code hugging the return like that ;-)
> > > >>
> > > >> But it's purely subjective, same below of course.
> > > >
> > > > We have various patterns indeed :-) I haven't been able to establish yet
> > > > what my mental process to deal with these are, sometimes it bothers me
> > > > too, but not always.
> > > >
> > > >>>   return 0;
> > > >>>  }
> > > >>>
> > > >>> @@ -681,6 +682,7 @@ int V4L2VideoDevice::open(int handle, enum v4l2_buf_type type)
> > > >>>           return ret;
> > > >>>   }
> > > >>>
> > > >>> + formatInfo_ = &PixelFormatInfo::info(format_.fourcc);
> > > >>>   return 0;
> > > >>>  }
> > > >>>
> > > >>> @@ -695,6 +697,8 @@ void V4L2VideoDevice::close()
> > > >>>   releaseBuffers();
> > > >>>   delete fdBufferNotifier_;
> > > >>>
> > > >>> + formatInfo_ = nullptr;
> > > >>> +
> > > >>>   V4L2Device::close();
> > > >>>  }
> > > >>>
> > > >>> @@ -787,6 +791,7 @@ int V4L2VideoDevice::setFormat(V4L2DeviceFormat *format)
> > > >>>           return ret;
> > > >>>
> > > >>>   format_ = *format;
> > > >>> + formatInfo_ = &PixelFormatInfo::info(format_.fourcc);
> > > >>
> > > >> Looks like we were already using that pattern though ;-)
> > > >>
> > > >>>   return 0;
> > > >>>  }
> > > >>>
> > > >>> @@ -1325,19 +1330,17 @@ std::unique_ptr<FrameBuffer> V4L2VideoDevice::createBuffer(unsigned int index)
> > > >>>           planes.push_back(std::move(plane));
> > > >>>   }
> > > >>>
> > > >>> - const auto &info = PixelFormatInfo::info(format_.fourcc);
> > > >>> - if (info.isValid() && info.numPlanes() != numPlanes) {
> > > >>> + if (formatInfo_->isValid() && formatInfo_->numPlanes() != numPlanes) {
> > > >>>           ASSERT(numPlanes == 1u);
> > > >>
> > > >> Not for this patch, but I guess this assert needed a
> > > >>   \todo Multiplanar support
> > > >>
> > > >> around it?
> > > >
> > > > No, when formatInfo_->numPlanes() != numPlanes, it means that we have a
> > > > multi-planar FrameBuffer with a single-planar V4L2 buffer format (e.g.
> > > > V4L2_PIX_FMT_NV12 as opposed to V4L2_PIX_FMT_NV12M). numPlanes must be 1
> > > > in that case, otherwise it's a kernel bug.
> > >
> > > As you're updating the conditional there anyway, could you add a short
> > > brief to explain that before the ASSERT please?
> > >
> > > I don't think that extra context is clear from just the if statement.
> > > (Perhaps there is more context outside of the hunk maybe, but it's not
> > > clear from just this diff).
> >
> > That's because this diff only deals with format caching :-) It's a good
> > point though, it's not trivial so I'll add a separate patch to document
> > this properly.
> >
> > > >> However, this patch is just for caching the formatInfo, which seems
> > > >> reasonable so
> > > >>
> > > >> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> > > >>
> > > >>> -         const size_t numColorPlanes = info.numPlanes();
> > > >>> -         planes.resize(numColorPlanes);
> > > >>> +         planes.resize(formatInfo_->numPlanes());
> 
> By the way, what I would not like in PixelFormatInfo(), numPlanes()
> does for-loops always.
> Shall we add numPlanes to PixelFormatInfo?

The information is redundant, but I think it would make sense.

> Definitely, it is not related to this patch.
> 
> Reviewed-by: Hirokazu Honda <hiroh@chromium.org>
> 
> -Hiro
> > > >>>           const FileDescriptor &fd = planes[0].fd;
> > > >>>           size_t offset = 0;
> > > >>> -         for (size_t i = 0; i < numColorPlanes; ++i) {
> > > >>> +         for (size_t i = 0; i < planes.size(); ++i) {
> > > >>>                   planes[i].fd = fd;
> > > >>>                   planes[i].offset = offset;
> > > >>>
> > > >>>                   /* \todo Take the V4L2 stride into account */
> > > >>> -                 planes[i].length = info.planeSize(format_.size, i);
> > > >>> +                 planes[i].length = formatInfo_->planeSize(format_.size, i);
> > > >>>                   offset += planes[i].length;
> > > >>>           }
> > > >>>   }

Laurent Pinchart Sept. 3, 2021, 2:34 p.m. UTC | #33

Hi Hiro,

On Fri, Sep 03, 2021 at 09:25:17PM +0900, Hirokazu Honda wrote:
> On Thu, Sep 2, 2021 at 8:25 PM Kieran Bingham wrote:
> > On 02/09/2021 05:23, Laurent Pinchart wrote:
> > > When queueing a buffer to a V4L2VideoDevice, the number of planes in the
> > > FrameBuffer may not match the number of V4L2 buffer planes if the
> > > PixelFormat is multi-planar (has multiple colour planes) and the V4L2
> > > format is single-planar (has a single buffer plane). In this case, we
> > > need to coalesce all FrameBuffer planes into a single V4L2 buffer plane.
> > > Do so, and add validity checks to reject frame buffers that can't be
> > > described using a single V4L2 buffer plane.
> >
> > Yikes ;-) lets see...
> >
> > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > ---
> > >  src/libcamera/v4l2_videodevice.cpp | 68 +++++++++++++++++++++++++-----
> > >  1 file changed, 58 insertions(+), 10 deletions(-)
> > >
> > > diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> > > index 2d9a94c3c974..79cb792117d5 100644
> > > --- a/src/libcamera/v4l2_videodevice.cpp
> > > +++ b/src/libcamera/v4l2_videodevice.cpp
> > > @@ -22,10 +22,12 @@
> > >
> > >  #include <libcamera/base/event_notifier.h>
> > >  #include <libcamera/base/log.h>
> > > +#include <libcamera/base/utils.h>
> > >
> > >  #include <libcamera/file_descriptor.h>
> > >
> > >  #include "libcamera/internal/formats.h"
> > > +#include "libcamera/internal/framebuffer.h"
> > >  #include "libcamera/internal/media_device.h"
> > >  #include "libcamera/internal/media_object.h"
> > >
> > > @@ -1468,10 +1470,21 @@ int V4L2VideoDevice::queueBuffer(FrameBuffer *buffer)
> > >
> > >       bool multiPlanar = V4L2_TYPE_IS_MULTIPLANAR(buf.type);
> > >       const std::vector<FrameBuffer::Plane> &planes = buffer->planes();
> > > +     unsigned int numV4l2Planes = format_.fourcc == formatInfo_->v4l2Format[0]
> > > +                                ? 1 : planes.size();
> 
> nit const unsigned int.

Agreed.

> I would replace formatInfo->numPlanes() with planes.size().

Where ?

> > Aha, ok so that's how we recover the planar/multiplanar information I
> > was concerned we had lost earlier.
> >
> > I wonder if we should set/cache numV4l2Planes_ when we set formatInfo_
> > as they are so closely paired .. But it's a trivial operation here.
> >
> > > +
> > > +     /*
> > > +      * If the frame buffer has multiple planes and the V4L2 format requires
> > > +      * contiguous planes, ensure that's the case.
> > > +      */
> > > +     if (planes.size() != numV4l2Planes && !buffer->_d()->isContiguous()) {
> > > +             LOG(V4L2, Error) << "Device format requires contiguous buffer";
> > > +             return -EINVAL;
> > > +     }
> > >
> > >       if (buf.memory == V4L2_MEMORY_DMABUF) {
> > >               if (multiPlanar) {
> > > -                     for (unsigned int p = 0; p < planes.size(); ++p)
> > > +                     for (unsigned int p = 0; p < numV4l2Planes; ++p)
> > >                               v4l2Planes[p].m.fd = planes[p].fd.fd();
> > >               } else {
> > >                       buf.m.fd = planes[0].fd.fd();
> > > @@ -1479,23 +1492,58 @@ int V4L2VideoDevice::queueBuffer(FrameBuffer *buffer)
> > >       }
> > >
> > >       if (multiPlanar) {
> > > -             buf.length = planes.size();
> > > +             buf.length = numV4l2Planes;
> > >               buf.m.planes = v4l2Planes;
> > >       }
> > >
> > >       if (V4L2_TYPE_IS_OUTPUT(buf.type)) {
> > >               const FrameMetadata &metadata = buffer->metadata();
> > >
> > > -             if (multiPlanar) {
> > > -                     unsigned int nplane = 0;
> > > -                     for (const FrameMetadata::Plane &plane : metadata.planes()) {
> > > -                             v4l2Planes[nplane].bytesused = plane.bytesused;
> > > -                             v4l2Planes[nplane].length = buffer->planes()[nplane].length;
> > > -                             nplane++;
> > > +             if (numV4l2Planes != planes.size()) {
> > > +                     /*
> > > +                      * If we have a multi-planar buffer with a V4L2
> > > +                      * single-planar format, coalesce all planes. The length
> > > +                      * and number of bytes used may only differ in the last
> > > +                      * plane as any other situation can't be represented.
> > > +                      */
> > > +                     unsigned int bytesused = 0;
> > > +                     unsigned int length = 0;
> > > +
> > > +                     for (auto [i, plane] : utils::enumerate(planes)) {
> > > +                             bytesused += metadata.planes()[i].bytesused;
> > > +                             length += plane.length;
> > > +
> > > +                             if (i != planes.size() - 1 && bytesused != length) {
> >
> > Ah, i almost thought that was wrong, but it's
> >    "If we're not the last plane, and the plane is not fully used..."
> >
> > So it's fine.
> >
> >
> > > +                                     LOG(V4L2, Error)
> > > +                                             << "Holes in multi-planar buffer not supported";
> > > +                                     return -EINVAL;
> > > +                             }
> > > +                     }
> > > +
> > > +                     if (multiPlanar) {
> > > +                             v4l2Planes[0].bytesused = bytesused;
> > > +                             v4l2Planes[0].length = length;
> 
> I think multiPlanar must be false here.
> 
> v4l2 format | plane.size() | numV4L2Planes | numV4L2Planes == planes.size()
> multi       | 1            | planes.size() | true
> multi       | 2+           | planes.size() | true
> single      | 1            | 1             | true
> single      | 2+           | 1             | false

multiPlanar indicates whether or not the driver uses the V4L2
multi-planar API, that is, if it stores buffer data in buf.length and
buf.bytesused, or in buf.m.planes[].length and buf.m.planes[].bytesused.
Formats that use multiple V4L2 buffer planes (such as
V4L2_PIX_FMT_NV12M) require the multi-planar API, but formats that use a
single V4L2 buffer plane (such as V4L2_PIX_FMT_NV12) can use either.

We are here under the context of the condition

	if (numV4l2Planes != planes.size())

numV4l2Planes stores the number of V4L2 buffer planes, so for
V4L2_PIX_FMT_NV12, it will be 1. planes.size(), for V4L2_PIX_FMT_NV12,
will be 2. We can thus reach this point with V4L2_PIX_FMT_NV12, and that
format can be used with both the single-planar API and the multi-planar
API.

> So I would implement as follows.
> if (multiPlanar) {
>    ...
> } else {
>   if (numV4l2Planes != planes.size()) {
>     ...
>   } else {
>      ...
>   }
> }
> 
> > > +                     } else {
> > > +                             buf.bytesused = bytesused;
> > > +                             buf.length = length;
> > > +                     }
> > > +             } else if (multiPlanar) {
> > > +                     /*
> > > +                      * If we use the multi-planar API, fill in the planes.
> > > +                      * The number of planes in the frame buffer and in the
> > > +                      * V4L2 buffer is guaranteed to be equal at this point.
> > > +                      */
> > > +                     for (auto [i, plane] : utils::enumerate(planes)) {
> > > +                             v4l2Planes[i].bytesused = metadata.planes()[i].bytesused;
> > > +                             v4l2Planes[i].length = plane.length;
> > >                       }
> > >               } else {
> > > -                     if (metadata.planes().size())
> >
> > Guaranteeing the planes are there is much nicer here.
> >
> > Phew, this patch 'looks' a lot more complex than it actually is,
> >
> > So I think ... this is fine.
> >
> > Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> >
> > > -                             buf.bytesused = metadata.planes()[0].bytesused;
> > > +                     /*
> > > +                      * Single-planar API with a single plane in the buffer
> > > +                      * is trivial to handle.
> > > +                      */
> > > +                     buf.bytesused = metadata.planes()[0].bytesused;
> > > +                     buf.length = planes[0].length;
> > >               }
> > >
> > >               buf.sequence = metadata.sequence;

Hirokazu Honda Sept. 3, 2021, 2:46 p.m. UTC | #34

Hi Laurent,

On Fri, Sep 3, 2021 at 11:34 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Hiro,
>
> On Fri, Sep 03, 2021 at 09:25:17PM +0900, Hirokazu Honda wrote:
> > On Thu, Sep 2, 2021 at 8:25 PM Kieran Bingham wrote:
> > > On 02/09/2021 05:23, Laurent Pinchart wrote:
> > > > When queueing a buffer to a V4L2VideoDevice, the number of planes in the
> > > > FrameBuffer may not match the number of V4L2 buffer planes if the
> > > > PixelFormat is multi-planar (has multiple colour planes) and the V4L2
> > > > format is single-planar (has a single buffer plane). In this case, we
> > > > need to coalesce all FrameBuffer planes into a single V4L2 buffer plane.
> > > > Do so, and add validity checks to reject frame buffers that can't be
> > > > described using a single V4L2 buffer plane.
> > >
> > > Yikes ;-) lets see...
> > >
> > > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > > ---
> > > >  src/libcamera/v4l2_videodevice.cpp | 68 +++++++++++++++++++++++++-----
> > > >  1 file changed, 58 insertions(+), 10 deletions(-)
> > > >
> > > > diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> > > > index 2d9a94c3c974..79cb792117d5 100644
> > > > --- a/src/libcamera/v4l2_videodevice.cpp
> > > > +++ b/src/libcamera/v4l2_videodevice.cpp
> > > > @@ -22,10 +22,12 @@
> > > >
> > > >  #include <libcamera/base/event_notifier.h>
> > > >  #include <libcamera/base/log.h>
> > > > +#include <libcamera/base/utils.h>
> > > >
> > > >  #include <libcamera/file_descriptor.h>
> > > >
> > > >  #include "libcamera/internal/formats.h"
> > > > +#include "libcamera/internal/framebuffer.h"
> > > >  #include "libcamera/internal/media_device.h"
> > > >  #include "libcamera/internal/media_object.h"
> > > >
> > > > @@ -1468,10 +1470,21 @@ int V4L2VideoDevice::queueBuffer(FrameBuffer *buffer)
> > > >
> > > >       bool multiPlanar = V4L2_TYPE_IS_MULTIPLANAR(buf.type);
> > > >       const std::vector<FrameBuffer::Plane> &planes = buffer->planes();
> > > > +     unsigned int numV4l2Planes = format_.fourcc == formatInfo_->v4l2Format[0]
> > > > +                                ? 1 : planes.size();
> >
> > nit const unsigned int.
>
> Agreed.
>
> > I would replace formatInfo->numPlanes() with planes.size().
>

This is the opposite. replace planes.size() with formatinfo_->numPlanes().

> Where ?
>
> > > Aha, ok so that's how we recover the planar/multiplanar information I
> > > was concerned we had lost earlier.
> > >
> > > I wonder if we should set/cache numV4l2Planes_ when we set formatInfo_
> > > as they are so closely paired .. But it's a trivial operation here.
> > >
> > > > +
> > > > +     /*
> > > > +      * If the frame buffer has multiple planes and the V4L2 format requires
> > > > +      * contiguous planes, ensure that's the case.
> > > > +      */
> > > > +     if (planes.size() != numV4l2Planes && !buffer->_d()->isContiguous()) {
> > > > +             LOG(V4L2, Error) << "Device format requires contiguous buffer";
> > > > +             return -EINVAL;
> > > > +     }
> > > >
> > > >       if (buf.memory == V4L2_MEMORY_DMABUF) {
> > > >               if (multiPlanar) {
> > > > -                     for (unsigned int p = 0; p < planes.size(); ++p)
> > > > +                     for (unsigned int p = 0; p < numV4l2Planes; ++p)
> > > >                               v4l2Planes[p].m.fd = planes[p].fd.fd();
> > > >               } else {
> > > >                       buf.m.fd = planes[0].fd.fd();
> > > > @@ -1479,23 +1492,58 @@ int V4L2VideoDevice::queueBuffer(FrameBuffer *buffer)
> > > >       }
> > > >
> > > >       if (multiPlanar) {
> > > > -             buf.length = planes.size();
> > > > +             buf.length = numV4l2Planes;
> > > >               buf.m.planes = v4l2Planes;
> > > >       }
> > > >
> > > >       if (V4L2_TYPE_IS_OUTPUT(buf.type)) {
> > > >               const FrameMetadata &metadata = buffer->metadata();
> > > >
> > > > -             if (multiPlanar) {
> > > > -                     unsigned int nplane = 0;
> > > > -                     for (const FrameMetadata::Plane &plane : metadata.planes()) {
> > > > -                             v4l2Planes[nplane].bytesused = plane.bytesused;
> > > > -                             v4l2Planes[nplane].length = buffer->planes()[nplane].length;
> > > > -                             nplane++;
> > > > +             if (numV4l2Planes != planes.size()) {
> > > > +                     /*
> > > > +                      * If we have a multi-planar buffer with a V4L2
> > > > +                      * single-planar format, coalesce all planes. The length
> > > > +                      * and number of bytes used may only differ in the last
> > > > +                      * plane as any other situation can't be represented.
> > > > +                      */
> > > > +                     unsigned int bytesused = 0;
> > > > +                     unsigned int length = 0;
> > > > +
> > > > +                     for (auto [i, plane] : utils::enumerate(planes)) {
> > > > +                             bytesused += metadata.planes()[i].bytesused;
> > > > +                             length += plane.length;
> > > > +
> > > > +                             if (i != planes.size() - 1 && bytesused != length) {
> > >
> > > Ah, i almost thought that was wrong, but it's
> > >    "If we're not the last plane, and the plane is not fully used..."
> > >
> > > So it's fine.
> > >
> > >
> > > > +                                     LOG(V4L2, Error)
> > > > +                                             << "Holes in multi-planar buffer not supported";
> > > > +                                     return -EINVAL;
> > > > +                             }
> > > > +                     }
> > > > +
> > > > +                     if (multiPlanar) {
> > > > +                             v4l2Planes[0].bytesused = bytesused;
> > > > +                             v4l2Planes[0].length = length;
> >
> > I think multiPlanar must be false here.
> >
> > v4l2 format | plane.size() | numV4L2Planes | numV4L2Planes == planes.size()
> > multi       | 1            | planes.size() | true
> > multi       | 2+           | planes.size() | true
> > single      | 1            | 1             | true
> > single      | 2+           | 1             | false
>
> multiPlanar indicates whether or not the driver uses the V4L2
> multi-planar API, that is, if it stores buffer data in buf.length and
> buf.bytesused, or in buf.m.planes[].length and buf.m.planes[].bytesused.
> Formats that use multiple V4L2 buffer planes (such as
> V4L2_PIX_FMT_NV12M) require the multi-planar API, but formats that use a
> single V4L2 buffer plane (such as V4L2_PIX_FMT_NV12) can use either.
>
> We are here under the context of the condition
>
>         if (numV4l2Planes != planes.size())
>
> numV4l2Planes stores the number of V4L2 buffer planes, so for
> V4L2_PIX_FMT_NV12, it will be 1. planes.size(), for V4L2_PIX_FMT_NV12,
> will be 2. We can thus reach this point with V4L2_PIX_FMT_NV12, and that
> format can be used with both the single-planar API and the multi-planar
> API.

Ah, I got it.
Thanks

Reviewed-by: Hirokazu Honda <hiroh@chromium.org>

-Hiro
>
> > So I would implement as follows.
> > if (multiPlanar) {
> >    ...
> > } else {
> >   if (numV4l2Planes != planes.size()) {
> >     ...
> >   } else {
> >      ...
> >   }
> > }
> >
> > > > +                     } else {
> > > > +                             buf.bytesused = bytesused;
> > > > +                             buf.length = length;
> > > > +                     }
> > > > +             } else if (multiPlanar) {
> > > > +                     /*
> > > > +                      * If we use the multi-planar API, fill in the planes.
> > > > +                      * The number of planes in the frame buffer and in the
> > > > +                      * V4L2 buffer is guaranteed to be equal at this point.
> > > > +                      */
> > > > +                     for (auto [i, plane] : utils::enumerate(planes)) {
> > > > +                             v4l2Planes[i].bytesused = metadata.planes()[i].bytesused;
> > > > +                             v4l2Planes[i].length = plane.length;
> > > >                       }
> > > >               } else {
> > > > -                     if (metadata.planes().size())
> > >
> > > Guaranteeing the planes are there is much nicer here.
> > >
> > > Phew, this patch 'looks' a lot more complex than it actually is,
> > >
> > > So I think ... this is fine.
> > >
> > > Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> > >
> > > > -                             buf.bytesused = metadata.planes()[0].bytesused;
> > > > +                     /*
> > > > +                      * Single-planar API with a single plane in the buffer
> > > > +                      * is trivial to handle.
> > > > +                      */
> > > > +                     buf.bytesused = metadata.planes()[0].bytesused;
> > > > +                     buf.length = planes[0].length;
> > > >               }
> > > >
> > > >               buf.sequence = metadata.sequence;
>
> --
> Regards,
>
> Laurent Pinchart

Laurent Pinchart Sept. 3, 2021, 4:23 p.m. UTC | #35

Hi Hiro,

On Fri, Sep 03, 2021 at 08:29:19PM +0900, Hirokazu Honda wrote:
> On Fri, Sep 3, 2021 at 3:11 AM Laurent Pinchart wrote:
> > On Thu, Sep 02, 2021 at 10:54:19AM +0100, Kieran Bingham wrote:
> > > On 02/09/2021 05:22, Laurent Pinchart wrote:
> > > > The metadata planes are allocated by V4L2VideoDevice when dequeuing a
> > > > buffer. This causes the metadata planes to only be allocated after a
> > > > buffer gets dequeued, and doesn't provide any strong guarantee that
> > > > their number matches the number of FrameBuffer planes. The lack of this
> > > > invariant makes the FrameBuffer class fragile.
> > > >
> > > > As a first step towards fixing this, allocate the metadata planes when
> > > > the FrameBuffer is constructed. The FrameMetadata API should be further
> > > > improved by preventing a change in the number of planes.
> > > >
> > > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > > ---
> > > >  src/libcamera/framebuffer.cpp      |  2 ++
> > > >  src/libcamera/v4l2_videodevice.cpp | 12 +++++++++---
> > > >  2 files changed, 11 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/src/libcamera/framebuffer.cpp b/src/libcamera/framebuffer.cpp
> > > > index 99265b44da43..c1529dfb0ad1 100644
> > > > --- a/src/libcamera/framebuffer.cpp
> > > > +++ b/src/libcamera/framebuffer.cpp
> > > > @@ -210,6 +210,8 @@ FrameBuffer::FrameBuffer(const std::vector<Plane> &planes, unsigned int cookie)
> > > >     : Extensible(std::make_unique<Private>()), planes_(planes),
> > > >       cookie_(cookie)
> > > >  {
> > > > +   metadata_.planes.resize(planes_.size());
> 
> I would set bytesused to some default value, probably 0?
> struct Plane {
>   unsigned int bytesused = 0;
> };

That should go in another patch, which should then also initialize the
other members of the metadata.

> > > > +
> > > >     unsigned int offset = 0;
> > > >
> > > >     /* \todo Remove the assertions after sufficient testing */
> > > > diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> > > > index adabd4720668..a51971879e75 100644
> > > > --- a/src/libcamera/v4l2_videodevice.cpp
> > > > +++ b/src/libcamera/v4l2_videodevice.cpp
> > > > @@ -1586,12 +1586,18 @@ FrameBuffer *V4L2VideoDevice::dequeueBuffer()
> > > >     buffer->metadata_.timestamp = buf.timestamp.tv_sec * 1000000000ULL
> > > >                                 + buf.timestamp.tv_usec * 1000ULL;
> > > >
> > > > -   buffer->metadata_.planes.clear();
> > > >     if (multiPlanar) {
> > > > +           if (buf.length > buffer->metadata_.planes.size()) {
> > > > +                   LOG(V4L2, Error)
> > >
> > > Are we going to 'leak' queued FrameBuffers at this point?
> > > Is there a way to prevent this occurring at queue time rather than dequeue?
> > >
> > > In fact, Can it even happen? or should it be an ASSERT/Fatal?
> >
> > It's not supposed to happen, I'd say it's a kernel bug in that case. But
> > it's a good point, I think we can return the buffer, and mark its status
> > as erroneous. I'll do so.
> >
> > > > +                           << "Invalid number of planes (" << buf.length
> > > > +                           << " != " << buffer->metadata_.planes.size() << ")";
> > > > +                   return nullptr;
> > > > +           }
> > > > +
> > > >             for (unsigned int nplane = 0; nplane < buf.length; nplane++)
> > > > -                   buffer->metadata_.planes.push_back({ planes[nplane].bytesused });
> > > > +                   buffer->metadata_.planes[nplane].bytesused = planes[nplane].bytesused;
> > > >     } else {
> > > > -           buffer->metadata_.planes.push_back({ buf.bytesused });
> > > > +           buffer->metadata_.planes[0].bytesused = buf.bytesused;
> 
> Is metadata_.planes[1+] kept unfilled? Shall we execute
> metadata_.planes.resize(1)?

The whole point of this patch is to make sure that FrameBuffer::planes_
will always have the same size as FrameMetadata::planes_. As the former
never changes after the construction of the FrameBuffer, the latter
should never change either. In this patch we only update bytesused in
the first plane indeed, but later in the series that separate issue is
addressed.

> > > >     }
> > > >
> > > >     return buffer;
> > > >

Laurent Pinchart Sept. 3, 2021, 4:30 p.m. UTC | #36

Hi Hiro,

On Fri, Sep 03, 2021 at 08:40:16PM +0900, Hirokazu Honda wrote:
> On Fri, Sep 3, 2021 at 5:26 PM Kieran Bingham wrote:
> > On 02/09/2021 19:20, Laurent Pinchart wrote:
> > > On Thu, Sep 02, 2021 at 11:03:47AM +0100, Kieran Bingham wrote:
> > >> On 02/09/2021 05:22, Laurent Pinchart wrote:
> > >>> The number of metadata planes should always match the number of frame
> > >>> buffer planes. Enforce this by making the vector private and providing
> > >>> accessor functions.
> 
> I am confused by the definition of Metadata::Planes.
> It has bytesused and it is set to v4l2_buf.bytesused. It seems to
> match the v4l2 format plane.
> That is, it is not always the number of frame buffer planes because
> planes in FrameBuffer are color format planes.

Yes, and that's the problem that this series tries to address. It's
something that was missing from your series, that I thought we could
live with for the time being, but it turns out to cause issues.

I'll continue working to fix the fallout of the offset addition over the
weekend, but if I run out of time, I'll revert it all before Monday.

> > >>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > >>> ---
> > >>>  include/libcamera/framebuffer.h    | 10 +++++++++-
> > >>>  src/cam/camera_session.cpp         |  4 ++--
> > >>>  src/cam/file_sink.cpp              |  2 +-
> > >>>  src/libcamera/framebuffer.cpp      | 12 +++++++++---
> > >>>  src/libcamera/v4l2_videodevice.cpp | 14 +++++++-------
> > >>>  src/qcam/main_window.cpp           |  2 +-
> > >>>  src/qcam/viewfinder_gl.cpp         |  2 +-
> > >>>  src/qcam/viewfinder_qt.cpp         |  2 +-
> > >>>  src/v4l2/v4l2_camera_proxy.cpp     |  2 +-
> > >>>  9 files changed, 32 insertions(+), 18 deletions(-)
> > >>>
> > >>> diff --git a/include/libcamera/framebuffer.h b/include/libcamera/framebuffer.h
> > >>> index fd68ed0a139d..7f2f176af691 100644
> > >>> --- a/include/libcamera/framebuffer.h
> > >>> +++ b/include/libcamera/framebuffer.h
> > >>> @@ -13,6 +13,7 @@
> > >>>  #include <vector>
> > >>>
> > >>>  #include <libcamera/base/class.h>
> > >>> +#include <libcamera/base/span.h>
> > >>>
> > >>>  #include <libcamera/file_descriptor.h>
> > >>>
> > >>> @@ -34,7 +35,14 @@ struct FrameMetadata {
> > >>>     Status status;
> > >>>     unsigned int sequence;
> > >>>     uint64_t timestamp;
> > >>> -   std::vector<Plane> planes;
> > >>> +
> > >>> +   Span<Plane> planes() { return planes_; }
> > >>> +   Span<const Plane> planes() const { return planes_; }
> > >>
> > >> Returning a span here is nice.
> > >>
> > >> This likely causes compile breakage for any external app.
> > >>
> > >> I know we're not ABI/API stable, but I wonder if we should highlight
> > >> when we do cause breakage somehow, perhaps in the commit message at
> > >> least, as we already have external users who we might want to notify.
> > >
> > > Most changes to public header files will be ABI/API breakages at this
> > > point, and users will certainly notice when they get a compilation
> > > failure :-) What value do you think this would bring, who would read
> > > those messages ? I think it could be different for things that change
> > > the ABI without breaking compilation and that would require more than a
> > > recompilation to fix, there a warning seems to have more value, but I'm
> > > also sure we'll forget from time to time, if not most of the time.
> >
> > Because I think we need to start being more aware of when we introduce
> > both ABI and API breakages.
> >
> > At least making it explicitly noted in the commit message states that we
> > were aware of the breakage at that point.
> >
> > >>> +
> > >>> +private:
> > >>> +   friend class FrameBuffer;
> > >>> +
> > >>> +   std::vector<Plane> planes_;
> > >>>  };
> > >>>
> > >>>  class FrameBuffer final : public Extensible
> > >>> diff --git a/src/cam/camera_session.cpp b/src/cam/camera_session.cpp
> > >>> index 60d640f2b15c..32a373a99b72 100644
> > >>> --- a/src/cam/camera_session.cpp
> > >>> +++ b/src/cam/camera_session.cpp
> > >>> @@ -374,9 +374,9 @@ void CameraSession::processRequest(Request *request)
> > >>>                  << " bytesused: ";
> > >>>
> > >>>             unsigned int nplane = 0;
> > >>> -           for (const FrameMetadata::Plane &plane : metadata.planes) {
> > >>> +           for (const FrameMetadata::Plane &plane : metadata.planes()) {
> > >>>                     info << plane.bytesused;
> > >>> -                   if (++nplane < metadata.planes.size())
> > >>> +                   if (++nplane < metadata.planes().size())
> > >>>                             info << "/";
> > >>>             }
> > >>>     }
> > >>> diff --git a/src/cam/file_sink.cpp b/src/cam/file_sink.cpp
> > >>> index 0b529e3eb767..0fc7d621f50b 100644
> > >>> --- a/src/cam/file_sink.cpp
> > >>> +++ b/src/cam/file_sink.cpp
> > >>> @@ -110,7 +110,7 @@ void FileSink::writeBuffer(const Stream *stream, FrameBuffer *buffer)
> > >>>
> > >>>     for (unsigned int i = 0; i < buffer->planes().size(); ++i) {
> > >>>             const FrameBuffer::Plane &plane = buffer->planes()[i];
> > >>> -           const FrameMetadata::Plane &meta = buffer->metadata().planes[i];
> > >>> +           const FrameMetadata::Plane &meta = buffer->metadata().planes()[i];
> > >>>
> > >>>             uint8_t *data = planeData_[&plane];
> > >>>             unsigned int length = std::min(meta.bytesused, plane.length);
> > >>> diff --git a/src/libcamera/framebuffer.cpp b/src/libcamera/framebuffer.cpp
> > >>> index c1529dfb0ad1..83cf7b83d182 100644
> > >>> --- a/src/libcamera/framebuffer.cpp
> > >>> +++ b/src/libcamera/framebuffer.cpp
> > >>> @@ -91,8 +91,14 @@ LOG_DEFINE_CATEGORY(Buffer)
> > >>>   */
> > >>>
> > >>>  /**
> > >>> - * \var FrameMetadata::planes
> > >>> - * \brief Array of per-plane metadata
> > >>> + * \fn FrameMetadata::planes()
> > >>> + * \copydoc FrameMetadata::planes() const
> > >>> + */
> > >>> +
> > >>> +/**
> > >>> + * \fn FrameMetadata::planes() const
> > >>> + * \brief Retrieve the array of per-plane metadata
> > >>> + * \return The array of per-plane metadata
> > >>>   */
> > >>>
> > >>>  /**
> > >>> @@ -210,7 +216,7 @@ FrameBuffer::FrameBuffer(const std::vector<Plane> &planes, unsigned int cookie)
> > >>>     : Extensible(std::make_unique<Private>()), planes_(planes),
> > >>>       cookie_(cookie)
> > >>>  {
> > >>> -   metadata_.planes.resize(planes_.size());
> > >>> +   metadata_.planes_.resize(planes_.size());
> > >>>
> > >>>     unsigned int offset = 0;
> > >>>
> > >>> diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp
> > >>> index a51971879e75..82ddaed3656f 100644
> > >>> --- a/src/libcamera/v4l2_videodevice.cpp
> > >>> +++ b/src/libcamera/v4l2_videodevice.cpp
> > >>> @@ -1485,14 +1485,14 @@ int V4L2VideoDevice::queueBuffer(FrameBuffer *buffer)
> > >>>
> > >>>             if (multiPlanar) {
> > >>>                     unsigned int nplane = 0;
> > >>> -                   for (const FrameMetadata::Plane &plane : metadata.planes) {
> > >>> +                   for (const FrameMetadata::Plane &plane : metadata.planes()) {
> > >>>                             v4l2Planes[nplane].bytesused = plane.bytesused;
> > >>>                             v4l2Planes[nplane].length = buffer->planes()[nplane].length;
> > >>>                             nplane++;
> > >>>                     }
> > >>>             } else {
> > >>> -                   if (metadata.planes.size())
> > >>> -                           buf.bytesused = metadata.planes[0].bytesused;
> > >>> +                   if (metadata.planes().size())
> > >>> +                           buf.bytesused = metadata.planes()[0].bytesused;
> > >>>             }
> > >>>
> > >>>             buf.sequence = metadata.sequence;
> > >>> @@ -1587,17 +1587,17 @@ FrameBuffer *V4L2VideoDevice::dequeueBuffer()
> > >>>                                 + buf.timestamp.tv_usec * 1000ULL;
> > >>>
> > >>>     if (multiPlanar) {
> > >>> -           if (buf.length > buffer->metadata_.planes.size()) {
> > >>> +           if (buf.length > buffer->metadata_.planes().size()) {
> > >>>                     LOG(V4L2, Error)
> > >>>                             << "Invalid number of planes (" << buf.length
> > >>> -                           << " != " << buffer->metadata_.planes.size() << ")";
> > >>> +                           << " != " << buffer->metadata_.planes().size() << ")";
> > >>>                     return nullptr;
> > >>>             }
> > >>>
> > >>>             for (unsigned int nplane = 0; nplane < buf.length; nplane++)
> > >>> -                   buffer->metadata_.planes[nplane].bytesused = planes[nplane].bytesused;
> > >>> +                   buffer->metadata_.planes()[nplane].bytesused = planes[nplane].bytesused;
> > >>>     } else {
> > >>> -           buffer->metadata_.planes[0].bytesused = buf.bytesused;
> > >>> +           buffer->metadata_.planes()[0].bytesused = buf.bytesused;
> > >>>     }
> > >>>
> > >>>     return buffer;
> > >>> diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp
> > >>> index 1536b2b5bd66..ac853e360aea 100644
> > >>> --- a/src/qcam/main_window.cpp
> > >>> +++ b/src/qcam/main_window.cpp
> > >>> @@ -756,7 +756,7 @@ void MainWindow::processViewfinder(FrameBuffer *buffer)
> > >>>
> > >>>     qDebug().noquote()
> > >>>             << QString("seq: %1").arg(metadata.sequence, 6, 10, QLatin1Char('0'))
> > >>> -           << "bytesused:" << metadata.planes[0].bytesused
> > >>> +           << "bytesused:" << metadata.planes()[0].bytesused
> > >>>             << "timestamp:" << metadata.timestamp
> > >>>             << "fps:" << Qt::fixed << qSetRealNumberPrecision(2) << fps;
> > >>>
> > >>> diff --git a/src/qcam/viewfinder_gl.cpp b/src/qcam/viewfinder_gl.cpp
> > >>> index 40226601f9fd..d2ef036974f4 100644
> > >>> --- a/src/qcam/viewfinder_gl.cpp
> > >>> +++ b/src/qcam/viewfinder_gl.cpp
> > >>> @@ -125,7 +125,7 @@ void ViewFinderGL::render(libcamera::FrameBuffer *buffer,
> > >>>     /*
> > >>>      * \todo Get the stride from the buffer instead of computing it naively
> > >>>      */
> > >>> -   stride_ = buffer->metadata().planes[0].bytesused / size_.height();
> > >>> +   stride_ = buffer->metadata().planes()[0].bytesused / size_.height();
> > >>
> > >> Can this be obtained from the PixelFormatInfo now ?
> > >> or do we still not expose that to applications...
> > >
> > > PixelFormatInfo is still internal. I'm considering exposing it, but then
> > > it would be good to move the V4L2-specific part somewhere else.
> > >
> > >> Anyway, that's likely a change on top even if we could to solve the \todo.
> > >>
> > >> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> > >>
> > >>>     update();
> > >>>     buffer_ = buffer;
> > >>>  }
> > >>> diff --git a/src/qcam/viewfinder_qt.cpp b/src/qcam/viewfinder_qt.cpp
> > >>> index efa1d412584b..a0bf99b0b522 100644
> > >>> --- a/src/qcam/viewfinder_qt.cpp
> > >>> +++ b/src/qcam/viewfinder_qt.cpp
> > >>> @@ -87,7 +87,7 @@ void ViewFinderQt::render(libcamera::FrameBuffer *buffer,
> > >>>     }
> > >>>
> > >>>     unsigned char *memory = mem.data();
> > >>> -   size_t size = buffer->metadata().planes[0].bytesused;
> > >>> +   size_t size = buffer->metadata().planes()[0].bytesused;
> > >>>
> > >>>     {
> > >>>             QMutexLocker locker(&mutex_);
> > >>> diff --git a/src/v4l2/v4l2_camera_proxy.cpp b/src/v4l2/v4l2_camera_proxy.cpp
> > >>> index 8d8ee395954f..68e47ee81834 100644
> > >>> --- a/src/v4l2/v4l2_camera_proxy.cpp
> > >>> +++ b/src/v4l2/v4l2_camera_proxy.cpp
> > >>> @@ -212,7 +212,7 @@ void V4L2CameraProxy::updateBuffers()
> > >>>
> > >>>             switch (fmd.status) {
> > >>>             case FrameMetadata::FrameSuccess:
> > >>> -                   buf.bytesused = fmd.planes[0].bytesused;
> > >>> +                   buf.bytesused = fmd.planes()[0].bytesused;
> > >>>                     buf.field = V4L2_FIELD_NONE;
> > >>>                     buf.timestamp.tv_sec = fmd.timestamp / 1000000000;
> > >>>                     buf.timestamp.tv_usec = fmd.timestamp % 1000000;

[libcamera-devel,RFC,v1,00/12] libcamera: Handle fallout of FrameBuffer offset support
mbox series

Message

Comments

[libcamera-devel,RFC,v1,00/12] libcamera: Handle fallout of FrameBuffer offset support 13596 mbox series

Message

Comments

[libcamera-devel,RFC,v1,00/12] libcamera: Handle fallout of FrameBuffer offset support
mbox series