[libcamera-devel,v4,02/21] libcamera: formats: Add fields to info to ease calculating stride

8671 diff mbox series

Packed formats make it difficult to calculate stride as well as
frame size with the fields that PixelFormatInfo currently has.
bitsPerPixel is defined as the average number of bits per pixel, and
only counts effective bits, so it is not useful for calculating
stride and frame size.

To fix this, we introduce a concept of a "pixel group". The size of this
group is defined as the minimum number of pixels (including padding)
necessary in a row when the image has only one column of effective
pixels. The pixel group has one more attribute, that is the "bytes per
group". This determines how many bytes one pixel group consumes. These
are the fields pixelsPerGroup and bytesPerGroup that are defined in this
patch. Defining these two values makes it really simple to calculate
bytes-per-line, as ceil(width / pixelsPerGroup) * bytesPerGroup, where
width is measured in number of pixels. The ceiling accounts for padding.

The pixel group has another contraint, which is that the pixel group
(bytesPerGroup and pixelsPerGroup) is the smallest repeatable unit. What
this means is that, for example, in the IPU3 formats, if there is only
one column of effective pixels, it looks like it could be fit in 5 bytes
with 3 padding pixels (for a total of 4 pixels over 5 bytes). However,
this unit is not repeatable, as at the 7th group in the same row, the
pattern is broken. Therefore, the pixel group for IPU3 formats must be
25 pixels over 32 bytes.

Clearly, pixelsPerGroup must be constant for all planes in the format.
The bytesPerGroup then, must be a per-plane attribute. There is one more
field, verticalSubSampling, that is per-plane. This is simply a divider,
to divide the number of rows of pixels by the sub-sampling value, to
obtain the number of rows of pixels for the subsampled plane.

For example, for something simple like BGR888, it is self-explanatory:
the pixel group size is 1, and the bytes necessary is 3, and there is
only one plane with no (= 1) vertical subsampling. For YUYV, the
CbCr pair is shared between two pixels, so even if you have only one
pixel, you would still need a padded second Y, therefore the pixel
group size is 2, and bytes necessary is 4 (as opposed to 1 and 2). YUYV
also has no vertical subsampling. NV12 has a pixel group size of 2
pixels, due to the CbCr plane. The bytes per group then, for both
planes, is 2. The first plane has no vertical subsampling, but the
second plane is subsampled by a factor of 2.
The IPU3 formats are also self-explanatory, as they are single-planar,
and have a pixel group size of 25, consuming 32 bytes. Although a
comment in the driver suggests that it should be 50 and 64,
respectively, this is an attribute of the driver, and not the format, so
this shall be set by the ipu3 pipeline handler.

Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

---
Changes in v4:
- expanded documentation and definition of pixel group

Changes in v3:
- add planes
- redefine the parameters for the formats
  - pixelsPerGroup is for whole format
  - add verticalSubSampling per plane

Changes in v2:
- add documentation for bytesPerGroup pixelsPerGroup
- fix wording in commit message
  - bytes-per-line -> stride
  - buffer size -> frame size
- changed MJPEG todo to allowing pipeline handlers to set parameters of
  format infos
---
 include/libcamera/internal/formats.h |  10 ++
 src/libcamera/formats.cpp            | 148 +++++++++++++++++++++++++++
 2 files changed, 158 insertions(+)

Hi Paul,
  there a still a few minor points from my previous comments that went
un-adressed (the fact a single plane has no sub-sampling in example)
but I like this version more than the previous one.

I would have included the RGB10 example I gave, as it's more
informative than RGB888, being it not byte aligned.

But I won't bother any more, you already did a great job putting this
down in writing, it's not easy stuff to grasp

Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>

Thanks for you effort on this
   j

On Wed, Jul 08, 2020 at 10:43:58PM +0900, Paul Elder wrote:
> Packed formats make it difficult to calculate stride as well as
> frame size with the fields that PixelFormatInfo currently has.
> bitsPerPixel is defined as the average number of bits per pixel, and
> only counts effective bits, so it is not useful for calculating
> stride and frame size.
>
> To fix this, we introduce a concept of a "pixel group". The size of this
> group is defined as the minimum number of pixels (including padding)
> necessary in a row when the image has only one column of effective
> pixels. The pixel group has one more attribute, that is the "bytes per
> group". This determines how many bytes one pixel group consumes. These
> are the fields pixelsPerGroup and bytesPerGroup that are defined in this
> patch. Defining these two values makes it really simple to calculate
> bytes-per-line, as ceil(width / pixelsPerGroup) * bytesPerGroup, where
> width is measured in number of pixels. The ceiling accounts for padding.
>
> The pixel group has another contraint, which is that the pixel group
> (bytesPerGroup and pixelsPerGroup) is the smallest repeatable unit. What
> this means is that, for example, in the IPU3 formats, if there is only
> one column of effective pixels, it looks like it could be fit in 5 bytes
> with 3 padding pixels (for a total of 4 pixels over 5 bytes). However,
> this unit is not repeatable, as at the 7th group in the same row, the
> pattern is broken. Therefore, the pixel group for IPU3 formats must be
> 25 pixels over 32 bytes.
>
> Clearly, pixelsPerGroup must be constant for all planes in the format.
> The bytesPerGroup then, must be a per-plane attribute. There is one more
> field, verticalSubSampling, that is per-plane. This is simply a divider,
> to divide the number of rows of pixels by the sub-sampling value, to
> obtain the number of rows of pixels for the subsampled plane.
>
> For example, for something simple like BGR888, it is self-explanatory:
> the pixel group size is 1, and the bytes necessary is 3, and there is
> only one plane with no (= 1) vertical subsampling. For YUYV, the
> CbCr pair is shared between two pixels, so even if you have only one
> pixel, you would still need a padded second Y, therefore the pixel
> group size is 2, and bytes necessary is 4 (as opposed to 1 and 2). YUYV
> also has no vertical subsampling. NV12 has a pixel group size of 2
> pixels, due to the CbCr plane. The bytes per group then, for both
> planes, is 2. The first plane has no vertical subsampling, but the
> second plane is subsampled by a factor of 2.
> The IPU3 formats are also self-explanatory, as they are single-planar,
> and have a pixel group size of 25, consuming 32 bytes. Although a
> comment in the driver suggests that it should be 50 and 64,
> respectively, this is an attribute of the driver, and not the format, so
> this shall be set by the ipu3 pipeline handler.
>
> Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>
> ---
> Changes in v4:
> - expanded documentation and definition of pixel group
>
> Changes in v3:
> - add planes
> - redefine the parameters for the formats
>   - pixelsPerGroup is for whole format
>   - add verticalSubSampling per plane
>
> Changes in v2:
> - add documentation for bytesPerGroup pixelsPerGroup
> - fix wording in commit message
>   - bytes-per-line -> stride
>   - buffer size -> frame size
> - changed MJPEG todo to allowing pipeline handlers to set parameters of
>   format infos
> ---
>  include/libcamera/internal/formats.h |  10 ++
>  src/libcamera/formats.cpp            | 148 +++++++++++++++++++++++++++
>  2 files changed, 158 insertions(+)
>
> diff --git a/include/libcamera/internal/formats.h b/include/libcamera/internal/formats.h
> index f59ac8f..61e5ff5 100644
> --- a/include/libcamera/internal/formats.h
> +++ b/include/libcamera/internal/formats.h
> @@ -32,6 +32,12 @@ private:
>  	std::map<unsigned int, std::vector<SizeRange>> data_;
>  };
>
> +struct PixelFormatPlaneInfo
> +{
> +	unsigned int bytesPerGroup;
> +	unsigned int verticalSubSampling;
> +};
> +
>  class PixelFormatInfo
>  {
>  public:
> @@ -52,6 +58,10 @@ public:
>  	unsigned int bitsPerPixel;
>  	enum ColourEncoding colourEncoding;
>  	bool packed;
> +
> +	unsigned int pixelsPerGroup;
> +
> +	std::array<PixelFormatPlaneInfo, 3> planes;
>  };
>
>  } /* namespace libcamera */
> diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
> index d3b722c..e6d3d3b 100644
> --- a/src/libcamera/formats.cpp
> +++ b/src/libcamera/formats.cpp
> @@ -110,6 +110,22 @@ const std::map<unsigned int, std::vector<SizeRange>> &ImageFormats::data() const
>  	return data_;
>  }
>
> +/**
> + * \class PixelFormatPlaneInfo
> + * \brief Information about a single plane of a pixel format
> + *
> + * \var PixelFormatPlaneInfo::bytesPerGroup
> + * \brief The number of bytes that a pixel group consumes
> + *
> + * \sa PixelFormatInfo::pixelsPerGroup
> + *
> + * \var PixelFormatPlaneInfo::verticalSubSampling
> + * \brief Vertical subsampling multiplier
> + *
> + * This value is the ratio between the number of rows of pixels in the frame
> + * to the number of rows of pixels in the plane.
> + */
> +
>  /**
>   * \class PixelFormatInfo
>   * \brief Information about pixel formats
> @@ -152,6 +168,49 @@ const std::map<unsigned int, std::vector<SizeRange>> &ImageFormats::data() const
>   * bytes. For instance, 12-bit Bayer data with two pixels stored in three bytes
>   * is packed, while the same data stored with 4 bits of padding in two bytes
>   * per pixel is not packed.
> + *
> + * \var PixelFormatInfo::pixelsPerGroup
> + * \brief The number of pixels in a pixel group
> + *
> + * The minimum number of pixels (including padding) necessary in a row
> + * when the frame has only one column of effective pixels.
> + *
> + * A pixel group is defined as the minimum number of pixels (including padding)
> + * necessary in a row when the image has only one column of effective pixels.
> + * pixelsPerGroup refers to this value. PixelFormatPlaneInfo::bytesPerGroup,
> + * then, refers to the number of bytes that a pixel group consumes. This
> + * definition of a pixel group allows simple calculation of stride, as
> + * ceil(width / pixelsPerGroup) * bytesPerGroup. These values are determined
> + * only in terms of a row. The ceiling accounts for padding.
> + *
> + * A pixel group has a second constraint, such that the pixel group
> + * (bytesPerGroup and pixelsPerGroup) is the smallest repeatable unit.
> + * What this means is that, for example, in the IPU3 formats, if there is only
> + * one column of effective pixels, it looks like it could be fit in 5 bytes
> + * with 3 padding pixels (for a total of 4 pixels over 5 bytes). However, this
> + * unit is not repeatable, as at the 7th group in the same row, the pattern
> + * is broken. Therefore, the pixel group for IPU3 formats must be 25 pixels
> + * over 32 bytes.
> + *
> + * For example, for something simple like BGR888, it is self-explanatory:
> + * the pixel group size is 1, and the bytes necessary is 3, and there is
> + * only one plane with no (= 1) vertical subsampling. For YUYV, the
> + * CbCr pair is shared between two pixels, so even if you have only one
> + * pixel, you would still need a padded second Y sample, therefore the pixel
> + * group size is 2, and bytes necessary is 4. YUYV also has no vertical
> + * subsampling. NV12 has a pixel group size of 2 pixels, due to the CbCr plane.
> + * The bytes per group then, for both planes, is 2. The first plane has no
> + * vertical subsampling, but the second plane is subsampled by a factor of 2.
> + * The IPU3 formats are also self-explanatory, as they are single-planar,
> + * and have a pixel group size of 25, consuming 32 bytes. Although a
> + * comment in the driver suggests that it should be 50 and 64,
> + * respectively, this is an attribute of the driver, and not the format, so
> + * this shall be set by the ipu3 pipeline handler.
> + *
> + * \var PixelFormatInfo::planes
> + * \brief Information about pixels for each plane
> + *
> + * \sa PixelFormatPlaneInfo
>   */
>
>  /**
> @@ -179,6 +238,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 24,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 3, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::RGB888, {
>  		.name = "RGB888",
> @@ -187,6 +248,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 24,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 3, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::ABGR8888, {
>  		.name = "ABGR8888",
> @@ -195,6 +258,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 32,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::ARGB8888, {
>  		.name = "ARGB8888",
> @@ -203,6 +268,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 32,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::BGRA8888, {
>  		.name = "BGRA8888",
> @@ -211,6 +278,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 32,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::RGBA8888, {
>  		.name = "RGBA8888",
> @@ -219,6 +288,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 32,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>
>  	/* YUV packed formats. */
> @@ -229,6 +300,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::YVYU, {
>  		.name = "YVYU",
> @@ -237,6 +310,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::UYVY, {
>  		.name = "UYVY",
> @@ -245,6 +320,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::VYUY, {
>  		.name = "VYUY",
> @@ -253,6 +330,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>
>  	/* YUV planar formats. */
> @@ -263,6 +342,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 2, 2 }, { 0, 0 } }},
>  	} },
>  	{ formats::NV21, {
>  		.name = "NV21",
> @@ -271,6 +352,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 2, 2 }, { 0, 0 } }},
>  	} },
>  	{ formats::NV16, {
>  		.name = "NV16",
> @@ -279,6 +362,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 2, 1 }, { 0, 0 } }},
>  	} },
>  	{ formats::NV61, {
>  		.name = "NV61",
> @@ -287,6 +372,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 2, 1 }, { 0, 0 } }},
>  	} },
>  	{ formats::NV24, {
>  		.name = "NV24",
> @@ -295,6 +382,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 24,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 1, 1 }, { 2, 1 }, { 0, 0 } }},
>  	} },
>  	{ formats::NV42, {
>  		.name = "NV42",
> @@ -303,6 +392,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 24,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 1, 1 }, { 2, 1 }, { 0, 0 } }},
>  	} },
>  	{ formats::YUV420, {
>  		.name = "YUV420",
> @@ -311,6 +402,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 1, 2 }, { 1, 2 } }},
>  	} },
>  	{ formats::YUV422, {
>  		.name = "YUV422",
> @@ -319,6 +412,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 1, 1 }, { 1, 1 } }},
>  	} },
>
>  	/* Greyscale formats. */
> @@ -329,6 +424,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 8,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 1, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>
>  	/* Bayer formats. */
> @@ -339,6 +436,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 8,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGBRG8, {
>  		.name = "SGBRG8",
> @@ -347,6 +446,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 8,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGRBG8, {
>  		.name = "SGRBG8",
> @@ -355,6 +456,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 8,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SRGGB8, {
>  		.name = "SRGGB8",
> @@ -363,6 +466,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 8,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SBGGR10, {
>  		.name = "SBGGR10",
> @@ -371,6 +476,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGBRG10, {
>  		.name = "SGBRG10",
> @@ -379,6 +486,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGRBG10, {
>  		.name = "SGRBG10",
> @@ -387,6 +496,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SRGGB10, {
>  		.name = "SRGGB10",
> @@ -395,6 +506,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SBGGR10_CSI2P, {
>  		.name = "SBGGR10_CSI2P",
> @@ -403,6 +516,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 4,
> +		.planes = {{ { 5, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGBRG10_CSI2P, {
>  		.name = "SGBRG10_CSI2P",
> @@ -411,6 +526,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 4,
> +		.planes = {{ { 5, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGRBG10_CSI2P, {
>  		.name = "SGRBG10_CSI2P",
> @@ -419,6 +536,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 4,
> +		.planes = {{ { 5, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SRGGB10_CSI2P, {
>  		.name = "SRGGB10_CSI2P",
> @@ -427,6 +546,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 4,
> +		.planes = {{ { 5, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SBGGR12, {
>  		.name = "SBGGR12",
> @@ -435,6 +556,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGBRG12, {
>  		.name = "SGBRG12",
> @@ -443,6 +566,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGRBG12, {
>  		.name = "SGRBG12",
> @@ -451,6 +576,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SRGGB12, {
>  		.name = "SRGGB12",
> @@ -459,6 +586,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SBGGR12_CSI2P, {
>  		.name = "SBGGR12_CSI2P",
> @@ -467,6 +596,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 3, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGBRG12_CSI2P, {
>  		.name = "SGBRG12_CSI2P",
> @@ -475,6 +606,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 3, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGRBG12_CSI2P, {
>  		.name = "SGRBG12_CSI2P",
> @@ -483,6 +616,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 3, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SRGGB12_CSI2P, {
>  		.name = "SRGGB12_CSI2P",
> @@ -491,6 +626,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 3, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SBGGR10_IPU3, {
>  		.name = "SBGGR10_IPU3",
> @@ -499,6 +636,9 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		/* \todo remember to double this in the ipu3 pipeline handler */
> +		.pixelsPerGroup = 25,
> +		.planes = {{ { 32, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGBRG10_IPU3, {
>  		.name = "SGBRG10_IPU3",
> @@ -507,6 +647,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 25,
> +		.planes = {{ { 32, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGRBG10_IPU3, {
>  		.name = "SGRBG10_IPU3",
> @@ -515,6 +657,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 25,
> +		.planes = {{ { 32, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SRGGB10_IPU3, {
>  		.name = "SRGGB10_IPU3",
> @@ -523,6 +667,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 25,
> +		.planes = {{ { 32, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>
>  	/* Compressed formats. */
> @@ -533,6 +679,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 0,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 1, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  };
>
> --
> 2.27.0
>
> _______________________________________________
> libcamera-devel mailing list
> libcamera-devel@lists.libcamera.org
> https://lists.libcamera.org/listinfo/libcamera-devel

Hi Paul,

On 08/07/2020 14:43, Paul Elder wrote:
> Packed formats make it difficult to calculate stride as well as
> frame size with the fields that PixelFormatInfo currently has.
> bitsPerPixel is defined as the average number of bits per pixel, and
> only counts effective bits, so it is not useful for calculating
> stride and frame size.
> 
> To fix this, we introduce a concept of a "pixel group". The size of this
> group is defined as the minimum number of pixels (including padding)
> necessary in a row when the image has only one column of effective
> pixels. The pixel group has one more attribute, that is the "bytes per
> group". This determines how many bytes one pixel group consumes. These
> are the fields pixelsPerGroup and bytesPerGroup that are defined in this
> patch. Defining these two values makes it really simple to calculate
> bytes-per-line, as ceil(width / pixelsPerGroup) * bytesPerGroup, where
> width is measured in number of pixels. The ceiling accounts for padding.
> 
> The pixel group has another contraint, which is that the pixel group
> (bytesPerGroup and pixelsPerGroup) is the smallest repeatable unit. What
> this means is that, for example, in the IPU3 formats, if there is only
> one column of effective pixels, it looks like it could be fit in 5 bytes
> with 3 padding pixels (for a total of 4 pixels over 5 bytes). However,
> this unit is not repeatable, as at the 7th group in the same row, the
> pattern is broken. Therefore, the pixel group for IPU3 formats must be
> 25 pixels over 32 bytes.
> 
> Clearly, pixelsPerGroup must be constant for all planes in the format.
> The bytesPerGroup then, must be a per-plane attribute. There is one more
> field, verticalSubSampling, that is per-plane. This is simply a divider,
> to divide the number of rows of pixels by the sub-sampling value, to
> obtain the number of rows of pixels for the subsampled plane.
> 
> For example, for something simple like BGR888, it is self-explanatory:
> the pixel group size is 1, and the bytes necessary is 3, and there is
> only one plane with no (= 1) vertical subsampling. For YUYV, the
> CbCr pair is shared between two pixels, so even if you have only one
> pixel, you would still need a padded second Y, therefore the pixel
> group size is 2, and bytes necessary is 4 (as opposed to 1 and 2). YUYV
> also has no vertical subsampling. NV12 has a pixel group size of 2
> pixels, due to the CbCr plane. The bytes per group then, for both
> planes, is 2. The first plane has no vertical subsampling, but the
> second plane is subsampled by a factor of 2.
> The IPU3 formats are also self-explanatory, as they are single-planar,
> and have a pixel group size of 25, consuming 32 bytes. Although a
> comment in the driver suggests that it should be 50 and 64,
> respectively, this is an attribute of the driver, and not the format, so
> this shall be set by the ipu3 pipeline handler.
> 
> Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> ---
> Changes in v4:
> - expanded documentation and definition of pixel group
> 
> Changes in v3:
> - add planes
> - redefine the parameters for the formats
>   - pixelsPerGroup is for whole format
>   - add verticalSubSampling per plane
> 
> Changes in v2:
> - add documentation for bytesPerGroup pixelsPerGroup
> - fix wording in commit message
>   - bytes-per-line -> stride
>   - buffer size -> frame size
> - changed MJPEG todo to allowing pipeline handlers to set parameters of
>   format infos
> ---
>  include/libcamera/internal/formats.h |  10 ++
>  src/libcamera/formats.cpp            | 148 +++++++++++++++++++++++++++
>  2 files changed, 158 insertions(+)
> 
> diff --git a/include/libcamera/internal/formats.h b/include/libcamera/internal/formats.h
> index f59ac8f..61e5ff5 100644
> --- a/include/libcamera/internal/formats.h
> +++ b/include/libcamera/internal/formats.h
> @@ -32,6 +32,12 @@ private:
>  	std::map<unsigned int, std::vector<SizeRange>> data_;
>  };
>  
> +struct PixelFormatPlaneInfo
> +{
> +	unsigned int bytesPerGroup;
> +	unsigned int verticalSubSampling;
> +};
> +
>  class PixelFormatInfo
>  {
>  public:
> @@ -52,6 +58,10 @@ public:
>  	unsigned int bitsPerPixel;
>  	enum ColourEncoding colourEncoding;
>  	bool packed;
> +
> +	unsigned int pixelsPerGroup;
> +
> +	std::array<PixelFormatPlaneInfo, 3> planes;


Building this on Chrome gets quite unhappy here:


../../../../../../../../mnt/host/source/src/third_party/libcamera/include/libcamera/internal/formats.h:71:38:
error: implicit instantiation of undefined template
'std::__1::array<libcamera::PixelFormatPlaneInfo, 3>'
        std::array<PixelFormatPlaneInfo, 3> planes;
                                            ^
/usr/bin/../include/c++/v1/__tuple:219:64: note: template is declared here
template <class _Tp, size_t _Size> struct _LIBCPP_TEMPLATE_VIS array;
                                                               ^


Adding #include <array> in this formats.h resolves it.

With that,

Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>



>  };
>  
>  } /* namespace libcamera */
> diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
> index d3b722c..e6d3d3b 100644
> --- a/src/libcamera/formats.cpp
> +++ b/src/libcamera/formats.cpp
> @@ -110,6 +110,22 @@ const std::map<unsigned int, std::vector<SizeRange>> &ImageFormats::data() const
>  	return data_;
>  }
>  
> +/**
> + * \class PixelFormatPlaneInfo
> + * \brief Information about a single plane of a pixel format
> + *
> + * \var PixelFormatPlaneInfo::bytesPerGroup
> + * \brief The number of bytes that a pixel group consumes
> + *
> + * \sa PixelFormatInfo::pixelsPerGroup
> + *
> + * \var PixelFormatPlaneInfo::verticalSubSampling
> + * \brief Vertical subsampling multiplier
> + *
> + * This value is the ratio between the number of rows of pixels in the frame
> + * to the number of rows of pixels in the plane.
> + */
> +
>  /**
>   * \class PixelFormatInfo
>   * \brief Information about pixel formats
> @@ -152,6 +168,49 @@ const std::map<unsigned int, std::vector<SizeRange>> &ImageFormats::data() const
>   * bytes. For instance, 12-bit Bayer data with two pixels stored in three bytes
>   * is packed, while the same data stored with 4 bits of padding in two bytes
>   * per pixel is not packed.
> + *
> + * \var PixelFormatInfo::pixelsPerGroup
> + * \brief The number of pixels in a pixel group
> + *
> + * The minimum number of pixels (including padding) necessary in a row
> + * when the frame has only one column of effective pixels.
> + *
> + * A pixel group is defined as the minimum number of pixels (including padding)
> + * necessary in a row when the image has only one column of effective pixels.
> + * pixelsPerGroup refers to this value. PixelFormatPlaneInfo::bytesPerGroup,
> + * then, refers to the number of bytes that a pixel group consumes. This
> + * definition of a pixel group allows simple calculation of stride, as
> + * ceil(width / pixelsPerGroup) * bytesPerGroup. These values are determined
> + * only in terms of a row. The ceiling accounts for padding.
> + *
> + * A pixel group has a second constraint, such that the pixel group
> + * (bytesPerGroup and pixelsPerGroup) is the smallest repeatable unit.
> + * What this means is that, for example, in the IPU3 formats, if there is only
> + * one column of effective pixels, it looks like it could be fit in 5 bytes
> + * with 3 padding pixels (for a total of 4 pixels over 5 bytes). However, this
> + * unit is not repeatable, as at the 7th group in the same row, the pattern
> + * is broken. Therefore, the pixel group for IPU3 formats must be 25 pixels
> + * over 32 bytes.
> + *
> + * For example, for something simple like BGR888, it is self-explanatory:
> + * the pixel group size is 1, and the bytes necessary is 3, and there is
> + * only one plane with no (= 1) vertical subsampling. For YUYV, the
> + * CbCr pair is shared between two pixels, so even if you have only one
> + * pixel, you would still need a padded second Y sample, therefore the pixel
> + * group size is 2, and bytes necessary is 4. YUYV also has no vertical
> + * subsampling. NV12 has a pixel group size of 2 pixels, due to the CbCr plane.
> + * The bytes per group then, for both planes, is 2. The first plane has no
> + * vertical subsampling, but the second plane is subsampled by a factor of 2.
> + * The IPU3 formats are also self-explanatory, as they are single-planar,
> + * and have a pixel group size of 25, consuming 32 bytes. Although a
> + * comment in the driver suggests that it should be 50 and 64,
> + * respectively, this is an attribute of the driver, and not the format, so
> + * this shall be set by the ipu3 pipeline handler.
> + *
> + * \var PixelFormatInfo::planes
> + * \brief Information about pixels for each plane
> + *
> + * \sa PixelFormatPlaneInfo
>   */
>  
>  /**
> @@ -179,6 +238,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 24,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 3, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::RGB888, {
>  		.name = "RGB888",
> @@ -187,6 +248,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 24,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 3, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::ABGR8888, {
>  		.name = "ABGR8888",
> @@ -195,6 +258,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 32,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::ARGB8888, {
>  		.name = "ARGB8888",
> @@ -203,6 +268,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 32,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::BGRA8888, {
>  		.name = "BGRA8888",
> @@ -211,6 +278,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 32,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::RGBA8888, {
>  		.name = "RGBA8888",
> @@ -219,6 +288,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 32,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRGB,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  
>  	/* YUV packed formats. */
> @@ -229,6 +300,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::YVYU, {
>  		.name = "YVYU",
> @@ -237,6 +310,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::UYVY, {
>  		.name = "UYVY",
> @@ -245,6 +320,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::VYUY, {
>  		.name = "VYUY",
> @@ -253,6 +330,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  
>  	/* YUV planar formats. */
> @@ -263,6 +342,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 2, 2 }, { 0, 0 } }},
>  	} },
>  	{ formats::NV21, {
>  		.name = "NV21",
> @@ -271,6 +352,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 2, 2 }, { 0, 0 } }},
>  	} },
>  	{ formats::NV16, {
>  		.name = "NV16",
> @@ -279,6 +362,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 2, 1 }, { 0, 0 } }},
>  	} },
>  	{ formats::NV61, {
>  		.name = "NV61",
> @@ -287,6 +372,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 2, 1 }, { 0, 0 } }},
>  	} },
>  	{ formats::NV24, {
>  		.name = "NV24",
> @@ -295,6 +382,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 24,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 1, 1 }, { 2, 1 }, { 0, 0 } }},
>  	} },
>  	{ formats::NV42, {
>  		.name = "NV42",
> @@ -303,6 +392,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 24,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 1, 1 }, { 2, 1 }, { 0, 0 } }},
>  	} },
>  	{ formats::YUV420, {
>  		.name = "YUV420",
> @@ -311,6 +402,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 1, 2 }, { 1, 2 } }},
>  	} },
>  	{ formats::YUV422, {
>  		.name = "YUV422",
> @@ -319,6 +412,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 16,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 1, 1 }, { 1, 1 } }},
>  	} },
>  
>  	/* Greyscale formats. */
> @@ -329,6 +424,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 8,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 1, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  
>  	/* Bayer formats. */
> @@ -339,6 +436,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 8,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGBRG8, {
>  		.name = "SGBRG8",
> @@ -347,6 +446,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 8,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGRBG8, {
>  		.name = "SGRBG8",
> @@ -355,6 +456,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 8,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SRGGB8, {
>  		.name = "SRGGB8",
> @@ -363,6 +466,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 8,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 2, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SBGGR10, {
>  		.name = "SBGGR10",
> @@ -371,6 +476,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGBRG10, {
>  		.name = "SGBRG10",
> @@ -379,6 +486,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGRBG10, {
>  		.name = "SGRBG10",
> @@ -387,6 +496,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SRGGB10, {
>  		.name = "SRGGB10",
> @@ -395,6 +506,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SBGGR10_CSI2P, {
>  		.name = "SBGGR10_CSI2P",
> @@ -403,6 +516,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 4,
> +		.planes = {{ { 5, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGBRG10_CSI2P, {
>  		.name = "SGBRG10_CSI2P",
> @@ -411,6 +526,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 4,
> +		.planes = {{ { 5, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGRBG10_CSI2P, {
>  		.name = "SGRBG10_CSI2P",
> @@ -419,6 +536,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 4,
> +		.planes = {{ { 5, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SRGGB10_CSI2P, {
>  		.name = "SRGGB10_CSI2P",
> @@ -427,6 +546,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 4,
> +		.planes = {{ { 5, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SBGGR12, {
>  		.name = "SBGGR12",
> @@ -435,6 +556,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGBRG12, {
>  		.name = "SGBRG12",
> @@ -443,6 +566,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGRBG12, {
>  		.name = "SGRBG12",
> @@ -451,6 +576,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SRGGB12, {
>  		.name = "SRGGB12",
> @@ -459,6 +586,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = false,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 4, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SBGGR12_CSI2P, {
>  		.name = "SBGGR12_CSI2P",
> @@ -467,6 +596,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 3, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGBRG12_CSI2P, {
>  		.name = "SGBRG12_CSI2P",
> @@ -475,6 +606,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 3, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGRBG12_CSI2P, {
>  		.name = "SGRBG12_CSI2P",
> @@ -483,6 +616,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 3, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SRGGB12_CSI2P, {
>  		.name = "SRGGB12_CSI2P",
> @@ -491,6 +626,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 12,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 2,
> +		.planes = {{ { 3, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SBGGR10_IPU3, {
>  		.name = "SBGGR10_IPU3",
> @@ -499,6 +636,9 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		/* \todo remember to double this in the ipu3 pipeline handler */
> +		.pixelsPerGroup = 25,
> +		.planes = {{ { 32, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGBRG10_IPU3, {
>  		.name = "SGBRG10_IPU3",
> @@ -507,6 +647,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 25,
> +		.planes = {{ { 32, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SGRBG10_IPU3, {
>  		.name = "SGRBG10_IPU3",
> @@ -515,6 +657,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 25,
> +		.planes = {{ { 32, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  	{ formats::SRGGB10_IPU3, {
>  		.name = "SRGGB10_IPU3",
> @@ -523,6 +667,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 10,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingRAW,
>  		.packed = true,
> +		.pixelsPerGroup = 25,
> +		.planes = {{ { 32, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  
>  	/* Compressed formats. */
> @@ -533,6 +679,8 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  		.bitsPerPixel = 0,
>  		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
>  		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 1, 1 }, { 0, 0 }, { 0, 0 } }},
>  	} },
>  };
>  
>