[libcamera-devel] JPEG support for libcamera

16535 diff mbox series

JPEG is very similar to MJPEG format (AFAICT, it may contain extra
EXIF headers). Add support for it. Tested with PinePhone and command
line cam utility.

Signed-off-by: Pavel Machek <pavel@ucw.cz>

Hi Pavel,
Thanks! It worked! There was one more place to add a "JPEG". Here is an 
updated patch:
https://github.com/maemo-leste-upstream-forks/libcamera/commit/f6fabd191f6bc8f66cfddd8caf8c49759fd55a87

diff --git a/include/linux/drm_fourcc.h b/include/linux/drm_fourcc.h
index ea11dcb4..d20fd1ae 100644
--- a/include/linux/drm_fourcc.h
+++ b/include/linux/drm_fourcc.h
@@ -352,6 +352,7 @@ extern "C" {

  /* Compressed formats */
  #define DRM_FORMAT_MJPEG       fourcc_code('M', 'J', 'P', 'G') /* 
Motion-JPEG */
+#define DRM_FORMAT_JPEG        fourcc_code('J', 'P', 'E', 'G') /* JFIF 
JPEG */

  /*
   * Bayer formats
diff --git a/src/cam/sdl_sink.cpp b/src/cam/sdl_sink.cpp
index f8e3e95d..0c328e88 100644
--- a/src/cam/sdl_sink.cpp
+++ b/src/cam/sdl_sink.cpp
@@ -64,6 +64,7 @@ int SDLSink::configure(const 
libcamera::CameraConfiguration &config)
         switch (cfg.pixelFormat) {
  #ifdef HAVE_SDL_IMAGE
         case libcamera::formats::MJPEG:
+       case libcamera::formats::JPEG:
                 texture_ = std::make_unique<SDLTextureMJPG>(rect_);
                 break;
  #endif
diff --git a/src/gstreamer/gstlibcamera-utils.cpp 
b/src/gstreamer/gstlibcamera-utils.cpp
index 3f242286..dd6836c1 100644
--- a/src/gstreamer/gstlibcamera-utils.cpp
+++ b/src/gstreamer/gstlibcamera-utils.cpp
@@ -18,6 +18,7 @@ static struct {
  } format_map[] = {
         /* Compressed */
         { GST_VIDEO_FORMAT_ENCODED, formats::MJPEG },
+       { GST_VIDEO_FORMAT_ENCODED, formats::JPEG },

         /* RGB */
         { GST_VIDEO_FORMAT_RGB, formats::BGR888 },
@@ -81,6 +82,7 @@ bare_structure_from_format(const PixelFormat &format)

         switch (format) {
         case formats::MJPEG:
+       case formats::JPEG:
                 return gst_structure_new_empty("image/jpeg");
         default:
                 return nullptr;
diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
index 283ecb3d..c5e97198 100644
--- a/src/libcamera/formats.cpp
+++ b/src/libcamera/formats.cpp
@@ -944,6 +944,19 @@ const std::map<PixelFormat, PixelFormatInfo> 
pixelFormatInfo{
         } },

         /* Compressed formats. */
+       { formats::JPEG, {
+               .name = "JPEG",
+               .format = formats::JPEG,
+               .v4l2Formats = {
+                       .single = V4L2PixelFormat(V4L2_PIX_FMT_JPEG),
+                       .multi = V4L2PixelFormat(),
+               },
+               .bitsPerPixel = 0,
+               .colourEncoding = PixelFormatInfo::ColourEncodingYUV,
+               .packed = false,
+               .pixelsPerGroup = 1,
+               .planes = {{ { 1, 1 }, { 0, 0 }, { 0, 0 } }},
+       } },
         { formats::MJPEG, {
                 .name = "MJPEG",
                 .format = formats::MJPEG,
diff --git a/src/libcamera/formats.yaml b/src/libcamera/formats.yaml
index 7dda0132..6b931c34 100644
--- a/src/libcamera/formats.yaml
+++ b/src/libcamera/formats.yaml
@@ -76,6 +76,8 @@ formats:
    - YVU444:
        fourcc: DRM_FORMAT_YVU444

+  - JPEG:
+      fourcc: DRM_FORMAT_JPEG
    - MJPEG:
        fourcc: DRM_FORMAT_MJPEG

diff --git a/src/libcamera/v4l2_pixelformat.cpp 
b/src/libcamera/v4l2_pixelformat.cpp
index 58fc4e9d..e5737323 100644
--- a/src/libcamera/v4l2_pixelformat.cpp
+++ b/src/libcamera/v4l2_pixelformat.cpp
@@ -181,6 +181,8 @@ const std::map<V4L2PixelFormat, 
V4L2PixelFormat::Info> vpf2pf{
                 { formats::SRGGB16, "16-bit Bayer RGRG/GBGB" } },

         /* Compressed formats. */
+       { V4L2PixelFormat(V4L2_PIX_FMT_JPEG),
+               { formats::JPEG, "JPEG" } },
         { V4L2PixelFormat(V4L2_PIX_FMT_MJPEG),
                 { formats::MJPEG, "Motion-JPEG" } },
  };



Cheers,
Rafael

On 7/4/22 15:07, Pavel Machek wrote:
> JPEG is very similar to MJPEG format (AFAICT, it may contain extra
> EXIF headers). Add support for it. Tested with PinePhone and command
> line cam utility.
> 
> Signed-off-by: Pavel Machek <pavel@ucw.cz>
> 
> diff --git a/include/linux/drm_fourcc.h b/include/linux/drm_fourcc.h
> index ea11dcb4..b30a705d 100644
> --- a/include/linux/drm_fourcc.h
> +++ b/include/linux/drm_fourcc.h
> @@ -352,6 +352,7 @@ extern "C" {
>   
>   /* Compressed formats */
>   #define DRM_FORMAT_MJPEG	fourcc_code('M', 'J', 'P', 'G') /* Motion-JPEG */
> +#define DRM_FORMAT_JPEG	fourcc_code('J', 'P', 'E', 'G') /* JFIF JPEG */
>   
>   /*
>    * Bayer formats
> diff --git a/src/cam/sdl_sink.cpp b/src/cam/sdl_sink.cpp
> index f8e3e95d..673bd642 100644
> --- a/src/cam/sdl_sink.cpp
> +++ b/src/cam/sdl_sink.cpp
> @@ -63,6 +63,7 @@ int SDLSink::configure(const libcamera::CameraConfiguration &config)
>   
>   	switch (cfg.pixelFormat) {
>   #ifdef HAVE_SDL_IMAGE
> +	case libcamera::formats::JPEG:
>   	case libcamera::formats::MJPEG:
>   		texture_ = std::make_unique<SDLTextureMJPG>(rect_);
>   		break;
> diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-utils.cpp
> index 3f242286..fd5689d9 100644
> --- a/src/gstreamer/gstlibcamera-utils.cpp
> +++ b/src/gstreamer/gstlibcamera-utils.cpp
> @@ -80,6 +80,7 @@ bare_structure_from_format(const PixelFormat &format)
>   					 gst_video_format_to_string(gst_format), nullptr);
>   
>   	switch (format) {
> +	case formats::JPEG:
>   	case formats::MJPEG:
>   		return gst_structure_new_empty("image/jpeg");
>   	default:
> diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
> index 283ecb3d..c5e97198 100644
> --- a/src/libcamera/formats.cpp
> +++ b/src/libcamera/formats.cpp
> @@ -944,6 +944,19 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>   	} },
>   
>   	/* Compressed formats. */
> +	{ formats::JPEG, {
> +		.name = "JPEG",
> +		.format = formats::JPEG,
> +		.v4l2Formats = {
> +			.single = V4L2PixelFormat(V4L2_PIX_FMT_JPEG),
> +			.multi = V4L2PixelFormat(),
> +		},
> +		.bitsPerPixel = 0,
> +		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> +		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 1, 1 }, { 0, 0 }, { 0, 0 } }},
> +	} },
>   	{ formats::MJPEG, {
>   		.name = "MJPEG",
>   		.format = formats::MJPEG,
> diff --git a/src/libcamera/formats.yaml b/src/libcamera/formats.yaml
> index 7dda0132..6b931c34 100644
> --- a/src/libcamera/formats.yaml
> +++ b/src/libcamera/formats.yaml
> @@ -76,6 +76,8 @@ formats:
>     - YVU444:
>         fourcc: DRM_FORMAT_YVU444
>   
> +  - JPEG:
> +      fourcc: DRM_FORMAT_JPEG
>     - MJPEG:
>         fourcc: DRM_FORMAT_MJPEG
>   
> diff --git a/src/libcamera/v4l2_pixelformat.cpp b/src/libcamera/v4l2_pixelformat.cpp
> index 58fc4e9d..82dfbd1e 100644
> --- a/src/libcamera/v4l2_pixelformat.cpp
> +++ b/src/libcamera/v4l2_pixelformat.cpp
> @@ -183,6 +183,8 @@ const std::map<V4L2PixelFormat, V4L2PixelFormat::Info> vpf2pf{
>   	/* Compressed formats. */
>   	{ V4L2PixelFormat(V4L2_PIX_FMT_MJPEG),
>   		{ formats::MJPEG, "Motion-JPEG" } },
> +	{ V4L2PixelFormat(V4L2_PIX_FMT_JPEG),
> +		{ formats::JPEG, "JPEG" } },
>   };
>   
>   } /* namespace */
>

Hi Pavel,
   I discussed it with Rafael (see message id
<20220701075644.mcjbcwthzsofmssx@uno.localdomain>) and my take was
slightly different: the driver should be fixed to report MJPEG:

Let me re-paste it here, mostly to validate my understanding as I
might have got this very wrong.

-------------------------------------------------------------------------------

Libcamera only supports:
#define V4L2_PIX_FMT_MJPEG    v4l2_fourcc('M', 'J', 'P', 'G') /* Motion-JPEG   */

I admit I know nothing about JPEG related standards but looking at the
media drivers, it seems FMT_JPEG is used more often than MJPEG.

However libcamera only seems to support MJPEG. You might have noticed
libcamera uses the DRM pixelformat identifiers, and it seems that in
the drm_fourcc.h header we ship there's no JPEG format at all, but
only MJPEG. If my understanding is correct this makes sense, as the
MJPEG standard defines a video container format for streaming
JPEG-compressed frames while JFIF JPEG defines a file-encoding scheme
for exchanging single images. Being DRM about video output devices
only MJPEG applies there, so there's no "JIF/JFIF JPEG" format in
there.

Now, I would not be surprised if the multiple users of FMT_JPEG in
drivers/media/ are actually sending MJPEG but driver developers pick
FMT_JPEG because of some form of cargo-cult.

Laurent do you know more about this maybe ?

When it comes to libcamera, for the reasons explained above, as a
streaming format it seems only MJPEG would make sense, while JFIF JPEG
could rather be used as the container when saving still images,
something for which we use DNG in qcam, in example.

To remove the above error I would change your driver to report
FMT_MJPEG for the time being until the above doesn't get clarified.
-------------------------------------------------------------------------------

What do you think ?

On Mon, Jul 04, 2022 at 08:07:20PM +0200, Pavel Machek via libcamera-devel wrote:
> JPEG is very similar to MJPEG format (AFAICT, it may contain extra
> EXIF headers). Add support for it. Tested with PinePhone and command
> line cam utility.
>
> Signed-off-by: Pavel Machek <pavel@ucw.cz>
>
> diff --git a/include/linux/drm_fourcc.h b/include/linux/drm_fourcc.h
> index ea11dcb4..b30a705d 100644
> --- a/include/linux/drm_fourcc.h
> +++ b/include/linux/drm_fourcc.h
> @@ -352,6 +352,7 @@ extern "C" {
>
>  /* Compressed formats */
>  #define DRM_FORMAT_MJPEG	fourcc_code('M', 'J', 'P', 'G') /* Motion-JPEG */
> +#define DRM_FORMAT_JPEG	fourcc_code('J', 'P', 'E', 'G') /* JFIF JPEG */
>
>  /*
>   * Bayer formats
> diff --git a/src/cam/sdl_sink.cpp b/src/cam/sdl_sink.cpp
> index f8e3e95d..673bd642 100644
> --- a/src/cam/sdl_sink.cpp
> +++ b/src/cam/sdl_sink.cpp
> @@ -63,6 +63,7 @@ int SDLSink::configure(const libcamera::CameraConfiguration &config)
>
>  	switch (cfg.pixelFormat) {
>  #ifdef HAVE_SDL_IMAGE
> +	case libcamera::formats::JPEG:
>  	case libcamera::formats::MJPEG:
>  		texture_ = std::make_unique<SDLTextureMJPG>(rect_);
>  		break;
> diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-utils.cpp
> index 3f242286..fd5689d9 100644
> --- a/src/gstreamer/gstlibcamera-utils.cpp
> +++ b/src/gstreamer/gstlibcamera-utils.cpp
> @@ -80,6 +80,7 @@ bare_structure_from_format(const PixelFormat &format)
>  					 gst_video_format_to_string(gst_format), nullptr);
>
>  	switch (format) {
> +	case formats::JPEG:
>  	case formats::MJPEG:
>  		return gst_structure_new_empty("image/jpeg");
>  	default:
> diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
> index 283ecb3d..c5e97198 100644
> --- a/src/libcamera/formats.cpp
> +++ b/src/libcamera/formats.cpp
> @@ -944,6 +944,19 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	} },
>
>  	/* Compressed formats. */
> +	{ formats::JPEG, {
> +		.name = "JPEG",
> +		.format = formats::JPEG,
> +		.v4l2Formats = {
> +			.single = V4L2PixelFormat(V4L2_PIX_FMT_JPEG),
> +			.multi = V4L2PixelFormat(),
> +		},
> +		.bitsPerPixel = 0,
> +		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> +		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 1, 1 }, { 0, 0 }, { 0, 0 } }},
> +	} },
>  	{ formats::MJPEG, {
>  		.name = "MJPEG",
>  		.format = formats::MJPEG,
> diff --git a/src/libcamera/formats.yaml b/src/libcamera/formats.yaml
> index 7dda0132..6b931c34 100644
> --- a/src/libcamera/formats.yaml
> +++ b/src/libcamera/formats.yaml
> @@ -76,6 +76,8 @@ formats:
>    - YVU444:
>        fourcc: DRM_FORMAT_YVU444
>
> +  - JPEG:
> +      fourcc: DRM_FORMAT_JPEG
>    - MJPEG:
>        fourcc: DRM_FORMAT_MJPEG
>
> diff --git a/src/libcamera/v4l2_pixelformat.cpp b/src/libcamera/v4l2_pixelformat.cpp
> index 58fc4e9d..82dfbd1e 100644
> --- a/src/libcamera/v4l2_pixelformat.cpp
> +++ b/src/libcamera/v4l2_pixelformat.cpp
> @@ -183,6 +183,8 @@ const std::map<V4L2PixelFormat, V4L2PixelFormat::Info> vpf2pf{
>  	/* Compressed formats. */
>  	{ V4L2PixelFormat(V4L2_PIX_FMT_MJPEG),
>  		{ formats::MJPEG, "Motion-JPEG" } },
> +	{ V4L2PixelFormat(V4L2_PIX_FMT_JPEG),
> +		{ formats::JPEG, "JPEG" } },
>  };
>
>  } /* namespace */
>
> --
> People of Russia, stop Putin before his war on Ukraine escalates.

Hi!

> Hi Pavel,
>    I discussed it with Rafael (see message id
> <20220701075644.mcjbcwthzsofmssx@uno.localdomain>) and my take was
> slightly different: the driver should be fixed to report MJPEG:
> 
> Let me re-paste it here, mostly to validate my understanding as I
> might have got this very wrong.
> 
> -------------------------------------------------------------------------------
> 
> Libcamera only supports:
> #define V4L2_PIX_FMT_MJPEG    v4l2_fourcc('M', 'J', 'P', 'G') /* Motion-JPEG   */
> 
> I admit I know nothing about JPEG related standards but looking at the
> media drivers, it seems FMT_JPEG is used more often than MJPEG.
> 
> However libcamera only seems to support MJPEG. You might have noticed
> libcamera uses the DRM pixelformat identifiers, and it seems that in
> the drm_fourcc.h header we ship there's no JPEG format at all, but
> only MJPEG. If my understanding is correct this makes sense, as the
> MJPEG standard defines a video container format for streaming
> JPEG-compressed frames while JFIF JPEG defines a file-encoding scheme
> for exchanging single images. Being DRM about video output devices
> only MJPEG applies there, so there's no "JIF/JFIF JPEG" format in
> there.
> 
> Now, I would not be surprised if the multiple users of FMT_JPEG in
> drivers/media/ are actually sending MJPEG but driver developers pick
> FMT_JPEG because of some form of cargo-cult.

Best explanation I found so far was that JPEG = EXIF + MJPEG. Kernel
interface exposes JPEG and MJPEG as separate types, so IMO it makes
sense for libcamera to support both.

> To remove the above error I would change your driver to report
> FMT_MJPEG for the time being until the above doesn't get clarified.

I'm not really able to touch the kernel at the moment. Plus it is not
"my" driver, it is driver in upstream kernel...

If one of the formats is redundant and should be dropped, I guess we
should have that discussion on the v4l2/kernel mailing lists...

Best regards,
								Pavel

On Tue, 5 Jul 2022 at 15:48, Pavel Machek via libcamera-devel
<libcamera-devel@lists.libcamera.org> wrote:
>
> Hi!
>
> > Hi Pavel,
> >    I discussed it with Rafael (see message id
> > <20220701075644.mcjbcwthzsofmssx@uno.localdomain>) and my take was
> > slightly different: the driver should be fixed to report MJPEG:
> >
> > Let me re-paste it here, mostly to validate my understanding as I
> > might have got this very wrong.
> >
> > -------------------------------------------------------------------------------
> >
> > Libcamera only supports:
> > #define V4L2_PIX_FMT_MJPEG    v4l2_fourcc('M', 'J', 'P', 'G') /* Motion-JPEG   */
> >
> > I admit I know nothing about JPEG related standards but looking at the
> > media drivers, it seems FMT_JPEG is used more often than MJPEG.
> >
> > However libcamera only seems to support MJPEG. You might have noticed
> > libcamera uses the DRM pixelformat identifiers, and it seems that in
> > the drm_fourcc.h header we ship there's no JPEG format at all, but
> > only MJPEG. If my understanding is correct this makes sense, as the
> > MJPEG standard defines a video container format for streaming
> > JPEG-compressed frames while JFIF JPEG defines a file-encoding scheme
> > for exchanging single images. Being DRM about video output devices
> > only MJPEG applies there, so there's no "JIF/JFIF JPEG" format in
> > there.
> >
> > Now, I would not be surprised if the multiple users of FMT_JPEG in
> > drivers/media/ are actually sending MJPEG but driver developers pick
> > FMT_JPEG because of some form of cargo-cult.
>
> Best explanation I found so far was that JPEG = EXIF + MJPEG. Kernel
> interface exposes JPEG and MJPEG as separate types, so IMO it makes
> sense for libcamera to support both.

TBH JPEG at least has a specification. MJPEG largely doesn't - it's
just a concatenation of JPEG encoded frames, and there seems to be no
real definition of which JPEG blocks are required and in which order.
Some MJPEG encoders drop the Huffman tables and rely on the decoder
assuming defaults.
There are then also MJPEG-A and MJPEG-B variants defined by Apple as
part of the Quicktime spec [1], with MJPEG-B not being compatible with
JPEG decoders due to changing headers.

Which flavour V4L2_PIX_FMT_MJPEG relates to seems to be undefined.

I've had issues in the past with what conditions VLC judges to be JPEG
vs MJPEG, and never found a definitive answer [2]

So how Libcamera should handle JPEG vs MJPEG probably doesn't have an
obvious answer. There's little point in encoding EXIF in a hardware
encoder as you have the overhead of passing all the parameters in.
For mapping into V4L2, just treat both of them the same.

  Dave

[1] https://developer.apple.com/standards/qtff-2001.pdf page 95.
[2] https://github.com/raspberrypi/userland/issues/345

> > To remove the above error I would change your driver to report
> > FMT_MJPEG for the time being until the above doesn't get clarified.
>
> I'm not really able to touch the kernel at the moment. Plus it is not
> "my" driver, it is driver in upstream kernel...
>
> If one of the formats is redundant and should be dropped, I guess we
> should have that discussion on the v4l2/kernel mailing lists...
>
> Best regards,
>                                                                 Pavel
> --
> People of Russia, stop Putin before his war on Ukraine escalates.

Le mardi 05 juillet 2022 à 16:33 +0100, Dave Stevenson via libcamera-devel a
écrit :
> On Tue, 5 Jul 2022 at 15:48, Pavel Machek via libcamera-devel
> <libcamera-devel@lists.libcamera.org> wrote:
> > 
> > Hi!
> > 
> > > Hi Pavel,
> > >    I discussed it with Rafael (see message id
> > > <20220701075644.mcjbcwthzsofmssx@uno.localdomain>) and my take was
> > > slightly different: the driver should be fixed to report MJPEG:
> > > 
> > > Let me re-paste it here, mostly to validate my understanding as I
> > > might have got this very wrong.
> > > 
> > > -------------------------------------------------------------------------------
> > > 
> > > Libcamera only supports:
> > > #define V4L2_PIX_FMT_MJPEG    v4l2_fourcc('M', 'J', 'P', 'G') /* Motion-JPEG   */
> > > 
> > > I admit I know nothing about JPEG related standards but looking at the
> > > media drivers, it seems FMT_JPEG is used more often than MJPEG.
> > > 
> > > However libcamera only seems to support MJPEG. You might have noticed
> > > libcamera uses the DRM pixelformat identifiers, and it seems that in
> > > the drm_fourcc.h header we ship there's no JPEG format at all, but
> > > only MJPEG. If my understanding is correct this makes sense, as the
> > > MJPEG standard defines a video container format for streaming
> > > JPEG-compressed frames while JFIF JPEG defines a file-encoding scheme
> > > for exchanging single images. Being DRM about video output devices
> > > only MJPEG applies there, so there's no "JIF/JFIF JPEG" format in
> > > there.
> > > 
> > > Now, I would not be surprised if the multiple users of FMT_JPEG in
> > > drivers/media/ are actually sending MJPEG but driver developers pick
> > > FMT_JPEG because of some form of cargo-cult.
> > 
> > Best explanation I found so far was that JPEG = EXIF + MJPEG. Kernel
> > interface exposes JPEG and MJPEG as separate types, so IMO it makes
> > sense for libcamera to support both.
> 
> TBH JPEG at least has a specification. MJPEG largely doesn't - it's
> just a concatenation of JPEG encoded frames, and there seems to be no
> real definition of which JPEG blocks are required and in which order.
> Some MJPEG encoders drop the Huffman tables and rely on the decoder
> assuming defaults.
> There are then also MJPEG-A and MJPEG-B variants defined by Apple as
> part of the Quicktime spec [1], with MJPEG-B not being compatible with
> JPEG decoders due to changing headers.
> 
> Which flavour V4L2_PIX_FMT_MJPEG relates to seems to be undefined.
> 
> I've had issues in the past with what conditions VLC judges to be JPEG
> vs MJPEG, and never found a definitive answer [2]
> 
> So how Libcamera should handle JPEG vs MJPEG probably doesn't have an
> obvious answer. There's little point in encoding EXIF in a hardware
> encoder as you have the overhead of passing all the parameters in.
> For mapping into V4L2, just treat both of them the same.

Older C920 could be setup to produce jpeg, which embeds H.264 into the EXIF
header. You'll find support for that in GStreamer [0]. Probably not relevant to
the discussion. In GStreamer v4l2src [1], all form of jpeg is assumed to be the
same. Which made me notice that V4L2_PIX_FMT_PJPG not mentionned here. I have no
idea what this refer too, this special case [1] dates from before me. I think
kernel driver encoders today will in software append the missing table/headers
as needed.

[0] https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-plugins-bad/sys/uvch264
[1] https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-good/sys/v4l2/gstv4l2object.c#L965
[3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/media/platform/chips-media/coda-jpeg.c#n215


> 
>   Dave
> 
> [1] https://developer.apple.com/standards/qtff-2001.pdf page 95.
> [2] https://github.com/raspberrypi/userland/issues/345
> 
> > > To remove the above error I would change your driver to report
> > > FMT_MJPEG for the time being until the above doesn't get clarified.
> > 
> > I'm not really able to touch the kernel at the moment. Plus it is not
> > "my" driver, it is driver in upstream kernel...
> > 
> > If one of the formats is redundant and should be dropped, I guess we
> > should have that discussion on the v4l2/kernel mailing lists...
> > 
> > Best regards,
> >                                                                 Pavel
> > --
> > People of Russia, stop Putin before his war on Ukraine escalates.

Hi Pavel, Dave, Nicolas,

   thank you all for the clarification. I'll review Pavel's patch with
the intent of mapping both JPEG and MJPEG to a single
application visible DRM_FORMAT_MJPEG

On Tue, Jul 05, 2022 at 01:39:02PM -0400, Nicolas Dufresne via libcamera-devel wrote:
> Le mardi 05 juillet 2022 à 16:33 +0100, Dave Stevenson via libcamera-devel a
> écrit :
> > On Tue, 5 Jul 2022 at 15:48, Pavel Machek via libcamera-devel
> > <libcamera-devel@lists.libcamera.org> wrote:
> > >
> > > Hi!
> > >
> > > > Hi Pavel,
> > > >    I discussed it with Rafael (see message id
> > > > <20220701075644.mcjbcwthzsofmssx@uno.localdomain>) and my take was
> > > > slightly different: the driver should be fixed to report MJPEG:
> > > >
> > > > Let me re-paste it here, mostly to validate my understanding as I
> > > > might have got this very wrong.
> > > >
> > > > -------------------------------------------------------------------------------
> > > >
> > > > Libcamera only supports:
> > > > #define V4L2_PIX_FMT_MJPEG    v4l2_fourcc('M', 'J', 'P', 'G') /* Motion-JPEG   */
> > > >
> > > > I admit I know nothing about JPEG related standards but looking at the
> > > > media drivers, it seems FMT_JPEG is used more often than MJPEG.
> > > >
> > > > However libcamera only seems to support MJPEG. You might have noticed
> > > > libcamera uses the DRM pixelformat identifiers, and it seems that in
> > > > the drm_fourcc.h header we ship there's no JPEG format at all, but
> > > > only MJPEG. If my understanding is correct this makes sense, as the
> > > > MJPEG standard defines a video container format for streaming
> > > > JPEG-compressed frames while JFIF JPEG defines a file-encoding scheme
> > > > for exchanging single images. Being DRM about video output devices
> > > > only MJPEG applies there, so there's no "JIF/JFIF JPEG" format in
> > > > there.
> > > > Now, I would not be surprised if the multiple users of FMT_JPEG in
> > > > drivers/media/ are actually sending MJPEG but driver developers pick
> > > > FMT_JPEG because of some form of cargo-cult.
> > >
> > > Best explanation I found so far was that JPEG = EXIF + MJPEG. Kernel
> > > interface exposes JPEG and MJPEG as separate types, so IMO it makes
> > > sense for libcamera to support both.
> >
> > TBH JPEG at least has a specification. MJPEG largely doesn't - it's
> > just a concatenation of JPEG encoded frames, and there seems to be no
> > real definition of which JPEG blocks are required and in which order.
> > Some MJPEG encoders drop the Huffman tables and rely on the decoder
> > assuming defaults.
> > There are then also MJPEG-A and MJPEG-B variants defined by Apple as
> > part of the Quicktime spec [1], with MJPEG-B not being compatible with
> > JPEG decoders due to changing headers.
> >
> > Which flavour V4L2_PIX_FMT_MJPEG relates to seems to be undefined.
> >
> > I've had issues in the past with what conditions VLC judges to be JPEG
> > vs MJPEG, and never found a definitive answer [2]
> >
> > So how Libcamera should handle JPEG vs MJPEG probably doesn't have an
> > obvious answer. There's little point in encoding EXIF in a hardware
> > encoder as you have the overhead of passing all the parameters in.
> > For mapping into V4L2, just treat both of them the same.
>
> Older C920 could be setup to produce jpeg, which embeds H.264 into the EXIF
> header. You'll find support for that in GStreamer [0]. Probably not relevant to
> the discussion. In GStreamer v4l2src [1], all form of jpeg is assumed to be the
> same. Which made me notice that V4L2_PIX_FMT_PJPG not mentionned here. I have no
> idea what this refer too, this special case [1] dates from before me. I think
> kernel driver encoders today will in software append the missing table/headers
> as needed.
>
> [0] https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-plugins-bad/sys/uvch264
> [1] https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-good/sys/v4l2/gstv4l2object.c#L965
> [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/media/platform/chips-media/coda-jpeg.c#n215
>
>
> >
> >   Dave
> >
> > [1] https://developer.apple.com/standards/qtff-2001.pdf page 95.
> > [2] https://github.com/raspberrypi/userland/issues/345
> >
> > > > To remove the above error I would change your driver to report
> > > > FMT_MJPEG for the time being until the above doesn't get clarified.
> > >
> > > I'm not really able to touch the kernel at the moment. Plus it is not
> > > "my" driver, it is driver in upstream kernel...
> > >
> > > If one of the formats is redundant and should be dropped, I guess we
> > > should have that discussion on the v4l2/kernel mailing lists...
> > >
> > > Best regards,
> > >                                                                 Pavel
> > > --
> > > People of Russia, stop Putin before his war on Ukraine escalates.
>

On Tue, Jul 05, 2022 at 08:49:53PM +0200, Jacopo Mondi via libcamera-devel wrote:
> Hi Pavel, Dave, Nicolas,
> 
>    thank you all for the clarification. I'll review Pavel's patch with
> the intent of mapping both JPEG and MJPEG to a single
> application visible DRM_FORMAT_MJPEG

Sounds good to me. This mess should eventually be fixed in V4L2, until
then, keeping a single libcamera pixel format is best to avoid
propagating the problem to applications.

> On Tue, Jul 05, 2022 at 01:39:02PM -0400, Nicolas Dufresne via libcamera-devel wrote:
> > Le mardi 05 juillet 2022 à 16:33 +0100, Dave Stevenson via libcamera-devel a écrit :
> > > On Tue, 5 Jul 2022 at 15:48, Pavel Machek via libcamera-devel wrote:
> > > >
> > > > Hi!
> > > >
> > > > > Hi Pavel,
> > > > >    I discussed it with Rafael (see message id
> > > > > <20220701075644.mcjbcwthzsofmssx@uno.localdomain>) and my take was
> > > > > slightly different: the driver should be fixed to report MJPEG:
> > > > >
> > > > > Let me re-paste it here, mostly to validate my understanding as I
> > > > > might have got this very wrong.
> > > > >
> > > > > -------------------------------------------------------------------------------
> > > > >
> > > > > Libcamera only supports:
> > > > > #define V4L2_PIX_FMT_MJPEG    v4l2_fourcc('M', 'J', 'P', 'G') /* Motion-JPEG   */
> > > > >
> > > > > I admit I know nothing about JPEG related standards but looking at the
> > > > > media drivers, it seems FMT_JPEG is used more often than MJPEG.
> > > > >
> > > > > However libcamera only seems to support MJPEG. You might have noticed
> > > > > libcamera uses the DRM pixelformat identifiers, and it seems that in
> > > > > the drm_fourcc.h header we ship there's no JPEG format at all, but
> > > > > only MJPEG. If my understanding is correct this makes sense, as the
> > > > > MJPEG standard defines a video container format for streaming
> > > > > JPEG-compressed frames while JFIF JPEG defines a file-encoding scheme
> > > > > for exchanging single images. Being DRM about video output devices
> > > > > only MJPEG applies there, so there's no "JIF/JFIF JPEG" format in
> > > > > there.
> > > > > Now, I would not be surprised if the multiple users of FMT_JPEG in
> > > > > drivers/media/ are actually sending MJPEG but driver developers pick
> > > > > FMT_JPEG because of some form of cargo-cult.
> > > >
> > > > Best explanation I found so far was that JPEG = EXIF + MJPEG. Kernel
> > > > interface exposes JPEG and MJPEG as separate types, so IMO it makes
> > > > sense for libcamera to support both.
> > >
> > > TBH JPEG at least has a specification. MJPEG largely doesn't - it's
> > > just a concatenation of JPEG encoded frames, and there seems to be no
> > > real definition of which JPEG blocks are required and in which order.
> > > Some MJPEG encoders drop the Huffman tables and rely on the decoder
> > > assuming defaults.
> > > There are then also MJPEG-A and MJPEG-B variants defined by Apple as
> > > part of the Quicktime spec [1], with MJPEG-B not being compatible with
> > > JPEG decoders due to changing headers.
> > >
> > > Which flavour V4L2_PIX_FMT_MJPEG relates to seems to be undefined.
> > >
> > > I've had issues in the past with what conditions VLC judges to be JPEG
> > > vs MJPEG, and never found a definitive answer [2]
> > >
> > > So how Libcamera should handle JPEG vs MJPEG probably doesn't have an
> > > obvious answer. There's little point in encoding EXIF in a hardware
> > > encoder as you have the overhead of passing all the parameters in.
> > > For mapping into V4L2, just treat both of them the same.
> >
> > Older C920 could be setup to produce jpeg, which embeds H.264 into the EXIF
> > header. You'll find support for that in GStreamer [0]. Probably not relevant to
> > the discussion. In GStreamer v4l2src [1], all form of jpeg is assumed to be the
> > same. Which made me notice that V4L2_PIX_FMT_PJPG not mentionned here. I have no
> > idea what this refer too, this special case [1] dates from before me. I think
> > kernel driver encoders today will in software append the missing table/headers
> > as needed.
> >
> > [0] https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-plugins-bad/sys/uvch264
> > [1] https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-good/sys/v4l2/gstv4l2object.c#L965
> > [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/media/platform/chips-media/coda-jpeg.c#n215
> >
> > > [1] https://developer.apple.com/standards/qtff-2001.pdf page 95.
> > > [2] https://github.com/raspberrypi/userland/issues/345
> > >
> > > > > To remove the above error I would change your driver to report
> > > > > FMT_MJPEG for the time being until the above doesn't get clarified.
> > > >
> > > > I'm not really able to touch the kernel at the moment. Plus it is not
> > > > "my" driver, it is driver in upstream kernel...
> > > >
> > > > If one of the formats is redundant and should be dropped, I guess we
> > > > should have that discussion on the v4l2/kernel mailing lists...

This patch looks good to me.

In a related area I was thinking of removing SDL2_image (while keeping
SDL2 main library as a dependancy) and replacing it with libjpeg to
keep our dependency count low. We were only really using SDL2_image as
a wrapper for libjpeg (and using no other SDL2_image functionality).
We already have libjpeg as an optional dependency in some of the
Android code, do like this instead basically (it's what SDL2_image is
doing under the covers anyway):

https://github.com/ericcurtin/twincam/blob/main/src/sdl_texture_mjpg.cpp

SDL2_image isn't as readily available either, it's not in CentOS
Stream 9 for example.

We don't lose any functionality this way.

We could re-use this decompressor in kms_sink potentially also,
although we'd still have to do some pixel format conversion after that
to have decent MJPEG and JPEG support in kms_sink.

Is mise le meas/Regards,

Eric Curtin



On Tue, 5 Jul 2022 at 21:34, Laurent Pinchart via libcamera-devel
<libcamera-devel@lists.libcamera.org> wrote:
>
> On Tue, Jul 05, 2022 at 08:49:53PM +0200, Jacopo Mondi via libcamera-devel wrote:
> > Hi Pavel, Dave, Nicolas,
> >
> >    thank you all for the clarification. I'll review Pavel's patch with
> > the intent of mapping both JPEG and MJPEG to a single
> > application visible DRM_FORMAT_MJPEG
>
> Sounds good to me. This mess should eventually be fixed in V4L2, until
> then, keeping a single libcamera pixel format is best to avoid
> propagating the problem to applications.
>
> > On Tue, Jul 05, 2022 at 01:39:02PM -0400, Nicolas Dufresne via libcamera-devel wrote:
> > > Le mardi 05 juillet 2022 à 16:33 +0100, Dave Stevenson via libcamera-devel a écrit :
> > > > On Tue, 5 Jul 2022 at 15:48, Pavel Machek via libcamera-devel wrote:
> > > > >
> > > > > Hi!
> > > > >
> > > > > > Hi Pavel,
> > > > > >    I discussed it with Rafael (see message id
> > > > > > <20220701075644.mcjbcwthzsofmssx@uno.localdomain>) and my take was
> > > > > > slightly different: the driver should be fixed to report MJPEG:
> > > > > >
> > > > > > Let me re-paste it here, mostly to validate my understanding as I
> > > > > > might have got this very wrong.
> > > > > >
> > > > > > -------------------------------------------------------------------------------
> > > > > >
> > > > > > Libcamera only supports:
> > > > > > #define V4L2_PIX_FMT_MJPEG    v4l2_fourcc('M', 'J', 'P', 'G') /* Motion-JPEG   */
> > > > > >
> > > > > > I admit I know nothing about JPEG related standards but looking at the
> > > > > > media drivers, it seems FMT_JPEG is used more often than MJPEG.
> > > > > >
> > > > > > However libcamera only seems to support MJPEG. You might have noticed
> > > > > > libcamera uses the DRM pixelformat identifiers, and it seems that in
> > > > > > the drm_fourcc.h header we ship there's no JPEG format at all, but
> > > > > > only MJPEG. If my understanding is correct this makes sense, as the
> > > > > > MJPEG standard defines a video container format for streaming
> > > > > > JPEG-compressed frames while JFIF JPEG defines a file-encoding scheme
> > > > > > for exchanging single images. Being DRM about video output devices
> > > > > > only MJPEG applies there, so there's no "JIF/JFIF JPEG" format in
> > > > > > there.
> > > > > > Now, I would not be surprised if the multiple users of FMT_JPEG in
> > > > > > drivers/media/ are actually sending MJPEG but driver developers pick
> > > > > > FMT_JPEG because of some form of cargo-cult.
> > > > >
> > > > > Best explanation I found so far was that JPEG = EXIF + MJPEG. Kernel
> > > > > interface exposes JPEG and MJPEG as separate types, so IMO it makes
> > > > > sense for libcamera to support both.
> > > >
> > > > TBH JPEG at least has a specification. MJPEG largely doesn't - it's
> > > > just a concatenation of JPEG encoded frames, and there seems to be no
> > > > real definition of which JPEG blocks are required and in which order.
> > > > Some MJPEG encoders drop the Huffman tables and rely on the decoder
> > > > assuming defaults.
> > > > There are then also MJPEG-A and MJPEG-B variants defined by Apple as
> > > > part of the Quicktime spec [1], with MJPEG-B not being compatible with
> > > > JPEG decoders due to changing headers.
> > > >
> > > > Which flavour V4L2_PIX_FMT_MJPEG relates to seems to be undefined.
> > > >
> > > > I've had issues in the past with what conditions VLC judges to be JPEG
> > > > vs MJPEG, and never found a definitive answer [2]
> > > >
> > > > So how Libcamera should handle JPEG vs MJPEG probably doesn't have an
> > > > obvious answer. There's little point in encoding EXIF in a hardware
> > > > encoder as you have the overhead of passing all the parameters in.
> > > > For mapping into V4L2, just treat both of them the same.
> > >
> > > Older C920 could be setup to produce jpeg, which embeds H.264 into the EXIF
> > > header. You'll find support for that in GStreamer [0]. Probably not relevant to
> > > the discussion. In GStreamer v4l2src [1], all form of jpeg is assumed to be the
> > > same. Which made me notice that V4L2_PIX_FMT_PJPG not mentionned here. I have no
> > > idea what this refer too, this special case [1] dates from before me. I think
> > > kernel driver encoders today will in software append the missing table/headers
> > > as needed.
> > >
> > > [0] https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-plugins-bad/sys/uvch264
> > > [1] https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-good/sys/v4l2/gstv4l2object.c#L965
> > > [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/media/platform/chips-media/coda-jpeg.c#n215
> > >
> > > > [1] https://developer.apple.com/standards/qtff-2001.pdf page 95.
> > > > [2] https://github.com/raspberrypi/userland/issues/345
> > > >
> > > > > > To remove the above error I would change your driver to report
> > > > > > FMT_MJPEG for the time being until the above doesn't get clarified.
> > > > >
> > > > > I'm not really able to touch the kernel at the moment. Plus it is not
> > > > > "my" driver, it is driver in upstream kernel...
> > > > >
> > > > > If one of the formats is redundant and should be dropped, I guess we
> > > > > should have that discussion on the v4l2/kernel mailing lists...
>
> --
> Regards,
>
> Laurent Pinchart
>

Hi Eric,

On Wed, Jul 06, 2022 at 10:21:26PM +0100, Eric Curtin wrote:
> This patch looks good to me.
> 
> In a related area I was thinking of removing SDL2_image (while keeping
> SDL2 main library as a dependancy) and replacing it with libjpeg to
> keep our dependency count low. We were only really using SDL2_image as
> a wrapper for libjpeg (and using no other SDL2_image functionality).
> We already have libjpeg as an optional dependency in some of the
> Android code, do like this instead basically (it's what SDL2_image is
> doing under the covers anyway):
> 
> https://github.com/ericcurtin/twincam/blob/main/src/sdl_texture_mjpg.cpp
> 
> SDL2_image isn't as readily available either, it's not in CentOS
> Stream 9 for example.
> 
> We don't lose any functionality this way.

No objection in principle. Just be careful that the libjpeg API isn't
the most straightforward to use, it can be quite cumbersome, especially
when it comes to error handling.

> We could re-use this decompressor in kms_sink potentially also,
> although we'd still have to do some pixel format conversion after that
> to have decent MJPEG and JPEG support in kms_sink.
> 
> Is mise le meas/Regards,
> 
> Eric Curtin
> 
> 
> 
> On Tue, 5 Jul 2022 at 21:34, Laurent Pinchart via libcamera-devel
> <libcamera-devel@lists.libcamera.org> wrote:
> >
> > On Tue, Jul 05, 2022 at 08:49:53PM +0200, Jacopo Mondi via libcamera-devel wrote:
> > > Hi Pavel, Dave, Nicolas,
> > >
> > >    thank you all for the clarification. I'll review Pavel's patch with
> > > the intent of mapping both JPEG and MJPEG to a single
> > > application visible DRM_FORMAT_MJPEG
> >
> > Sounds good to me. This mess should eventually be fixed in V4L2, until
> > then, keeping a single libcamera pixel format is best to avoid
> > propagating the problem to applications.
> >
> > > On Tue, Jul 05, 2022 at 01:39:02PM -0400, Nicolas Dufresne via libcamera-devel wrote:
> > > > Le mardi 05 juillet 2022 à 16:33 +0100, Dave Stevenson via libcamera-devel a écrit :
> > > > > On Tue, 5 Jul 2022 at 15:48, Pavel Machek via libcamera-devel wrote:
> > > > > >
> > > > > > Hi!
> > > > > >
> > > > > > > Hi Pavel,
> > > > > > >    I discussed it with Rafael (see message id
> > > > > > > <20220701075644.mcjbcwthzsofmssx@uno.localdomain>) and my take was
> > > > > > > slightly different: the driver should be fixed to report MJPEG:
> > > > > > >
> > > > > > > Let me re-paste it here, mostly to validate my understanding as I
> > > > > > > might have got this very wrong.
> > > > > > >
> > > > > > > -------------------------------------------------------------------------------
> > > > > > >
> > > > > > > Libcamera only supports:
> > > > > > > #define V4L2_PIX_FMT_MJPEG    v4l2_fourcc('M', 'J', 'P', 'G') /* Motion-JPEG   */
> > > > > > >
> > > > > > > I admit I know nothing about JPEG related standards but looking at the
> > > > > > > media drivers, it seems FMT_JPEG is used more often than MJPEG.
> > > > > > >
> > > > > > > However libcamera only seems to support MJPEG. You might have noticed
> > > > > > > libcamera uses the DRM pixelformat identifiers, and it seems that in
> > > > > > > the drm_fourcc.h header we ship there's no JPEG format at all, but
> > > > > > > only MJPEG. If my understanding is correct this makes sense, as the
> > > > > > > MJPEG standard defines a video container format for streaming
> > > > > > > JPEG-compressed frames while JFIF JPEG defines a file-encoding scheme
> > > > > > > for exchanging single images. Being DRM about video output devices
> > > > > > > only MJPEG applies there, so there's no "JIF/JFIF JPEG" format in
> > > > > > > there.
> > > > > > > Now, I would not be surprised if the multiple users of FMT_JPEG in
> > > > > > > drivers/media/ are actually sending MJPEG but driver developers pick
> > > > > > > FMT_JPEG because of some form of cargo-cult.
> > > > > >
> > > > > > Best explanation I found so far was that JPEG = EXIF + MJPEG. Kernel
> > > > > > interface exposes JPEG and MJPEG as separate types, so IMO it makes
> > > > > > sense for libcamera to support both.
> > > > >
> > > > > TBH JPEG at least has a specification. MJPEG largely doesn't - it's
> > > > > just a concatenation of JPEG encoded frames, and there seems to be no
> > > > > real definition of which JPEG blocks are required and in which order.
> > > > > Some MJPEG encoders drop the Huffman tables and rely on the decoder
> > > > > assuming defaults.
> > > > > There are then also MJPEG-A and MJPEG-B variants defined by Apple as
> > > > > part of the Quicktime spec [1], with MJPEG-B not being compatible with
> > > > > JPEG decoders due to changing headers.
> > > > >
> > > > > Which flavour V4L2_PIX_FMT_MJPEG relates to seems to be undefined.
> > > > >
> > > > > I've had issues in the past with what conditions VLC judges to be JPEG
> > > > > vs MJPEG, and never found a definitive answer [2]
> > > > >
> > > > > So how Libcamera should handle JPEG vs MJPEG probably doesn't have an
> > > > > obvious answer. There's little point in encoding EXIF in a hardware
> > > > > encoder as you have the overhead of passing all the parameters in.
> > > > > For mapping into V4L2, just treat both of them the same.
> > > >
> > > > Older C920 could be setup to produce jpeg, which embeds H.264 into the EXIF
> > > > header. You'll find support for that in GStreamer [0]. Probably not relevant to
> > > > the discussion. In GStreamer v4l2src [1], all form of jpeg is assumed to be the
> > > > same. Which made me notice that V4L2_PIX_FMT_PJPG not mentionned here. I have no
> > > > idea what this refer too, this special case [1] dates from before me. I think
> > > > kernel driver encoders today will in software append the missing table/headers
> > > > as needed.
> > > >
> > > > [0] https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-plugins-bad/sys/uvch264
> > > > [1] https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-good/sys/v4l2/gstv4l2object.c#L965
> > > > [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/media/platform/chips-media/coda-jpeg.c#n215
> > > >
> > > > > [1] https://developer.apple.com/standards/qtff-2001.pdf page 95.
> > > > > [2] https://github.com/raspberrypi/userland/issues/345
> > > > >
> > > > > > > To remove the above error I would change your driver to report
> > > > > > > FMT_MJPEG for the time being until the above doesn't get clarified.
> > > > > >
> > > > > > I'm not really able to touch the kernel at the moment. Plus it is not
> > > > > > "my" driver, it is driver in upstream kernel...
> > > > > >
> > > > > > If one of the formats is redundant and should be dropped, I guess we
> > > > > > should have that discussion on the v4l2/kernel mailing lists...

Hi Pavel,

On Mon, Jul 04, 2022 at 08:07:20PM +0200, Pavel Machek via libcamera-devel wrote:
> JPEG is very similar to MJPEG format (AFAICT, it may contain extra
> EXIF headers). Add support for it. Tested with PinePhone and command
> line cam utility.
>
> Signed-off-by: Pavel Machek <pavel@ucw.cz>
>
> diff --git a/include/linux/drm_fourcc.h b/include/linux/drm_fourcc.h
> index ea11dcb4..b30a705d 100644
> --- a/include/linux/drm_fourcc.h
> +++ b/include/linux/drm_fourcc.h
> @@ -352,6 +352,7 @@ extern "C" {
>
>  /* Compressed formats */
>  #define DRM_FORMAT_MJPEG	fourcc_code('M', 'J', 'P', 'G') /* Motion-JPEG */
> +#define DRM_FORMAT_JPEG	fourcc_code('J', 'P', 'E', 'G') /* JFIF JPEG */

If something like this is required it should be done in upstream DRM.

However I'm not sure that's what we want, see below..

>
>  /*
>   * Bayer formats
> diff --git a/src/cam/sdl_sink.cpp b/src/cam/sdl_sink.cpp
> index f8e3e95d..673bd642 100644
> --- a/src/cam/sdl_sink.cpp
> +++ b/src/cam/sdl_sink.cpp
> @@ -63,6 +63,7 @@ int SDLSink::configure(const libcamera::CameraConfiguration &config)
>
>  	switch (cfg.pixelFormat) {
>  #ifdef HAVE_SDL_IMAGE
> +	case libcamera::formats::JPEG:
>  	case libcamera::formats::MJPEG:
>  		texture_ = std::make_unique<SDLTextureMJPG>(rect_);
>  		break;
> diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-utils.cpp
> index 3f242286..fd5689d9 100644
> --- a/src/gstreamer/gstlibcamera-utils.cpp
> +++ b/src/gstreamer/gstlibcamera-utils.cpp
> @@ -80,6 +80,7 @@ bare_structure_from_format(const PixelFormat &format)
>  					 gst_video_format_to_string(gst_format), nullptr);
>
>  	switch (format) {
> +	case formats::JPEG:
>  	case formats::MJPEG:
>  		return gst_structure_new_empty("image/jpeg");
>  	default:
> diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
> index 283ecb3d..c5e97198 100644
> --- a/src/libcamera/formats.cpp
> +++ b/src/libcamera/formats.cpp
> @@ -944,6 +944,19 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	} },
>
>  	/* Compressed formats. */
> +	{ formats::JPEG, {
> +		.name = "JPEG",
> +		.format = formats::JPEG,
> +		.v4l2Formats = {
> +			.single = V4L2PixelFormat(V4L2_PIX_FMT_JPEG),
> +			.multi = V4L2PixelFormat(),
> +		},
> +		.bitsPerPixel = 0,
> +		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> +		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 1, 1 }, { 0, 0 }, { 0, 0 } }},
> +	} },

I would rather work with the idea to associate multiple v4l2Formats to
the single formats::MJPEG identifier, by making
PixelFormatInfo::v4l2Formats an array.

However I'm not sure how we could rework this:

V4L2PixelFormat V4L2PixelFormat::fromPixelFormat(const PixelFormat &pixelFormat,
						 bool multiplanar)
{
	const PixelFormatInfo &info = PixelFormatInfo::info(pixelFormat);
	if (!info.isValid())
		return V4L2PixelFormat();

	return multiplanar ? info.v4l2Formats.multi : info.v4l2Formats.single;
}

So that it picks the right V4L2 pixelformat if not by poking the video
device and checking what it supports.

I think we already have a very similar problem in the
mapping of contiguous/non-contiguous YUV planar formats, where we
abuse the V4L2 planar/multiplanar API definition to discern which
format to map to

	{ formats::NV16, {
		.name = "NV16",
		.format = formats::NV16,
		.v4l2Formats = {
			.single = V4L2PixelFormat(V4L2_PIX_FMT_NV16),
			.multi = V4L2PixelFormat(V4L2_PIX_FMT_NV16M),
		},

Which is actually problematic as there are drivers that (rightfully)
expose V4L2_PIX_FMT_NV16 through the multiplanar API.

I would go one step further and remove the single/multiplanar formats
and just place a list there to match against the list of formats
supported by the video device which will have to be passed to
V4L2PixelFormat::fromPixelFormat().

Opinions ?

Pavel: I'm not asking you to go full circle and change such internal
machinery to have your format supported, I can look into doing that,
but in the meantime you should live with your patch out-of-tree. Is
this ok ?


>  	{ formats::MJPEG, {
>  		.name = "MJPEG",
>  		.format = formats::MJPEG,
> diff --git a/src/libcamera/formats.yaml b/src/libcamera/formats.yaml
> index 7dda0132..6b931c34 100644
> --- a/src/libcamera/formats.yaml
> +++ b/src/libcamera/formats.yaml
> @@ -76,6 +76,8 @@ formats:
>    - YVU444:
>        fourcc: DRM_FORMAT_YVU444
>
> +  - JPEG:
> +      fourcc: DRM_FORMAT_JPEG
>    - MJPEG:
>        fourcc: DRM_FORMAT_MJPEG
>
> diff --git a/src/libcamera/v4l2_pixelformat.cpp b/src/libcamera/v4l2_pixelformat.cpp
> index 58fc4e9d..82dfbd1e 100644
> --- a/src/libcamera/v4l2_pixelformat.cpp
> +++ b/src/libcamera/v4l2_pixelformat.cpp
> @@ -183,6 +183,8 @@ const std::map<V4L2PixelFormat, V4L2PixelFormat::Info> vpf2pf{
>  	/* Compressed formats. */
>  	{ V4L2PixelFormat(V4L2_PIX_FMT_MJPEG),
>  		{ formats::MJPEG, "Motion-JPEG" } },
> +	{ V4L2PixelFormat(V4L2_PIX_FMT_JPEG),
> +		{ formats::JPEG, "JPEG" } },
>  };
>
>  } /* namespace */
>
> --
> People of Russia, stop Putin before his war on Ukraine escalates.

Hi Jacopo,

On Thu, Jul 07, 2022 at 09:33:42AM +0200, Jacopo Mondi via libcamera-devel wrote:
> On Mon, Jul 04, 2022 at 08:07:20PM +0200, Pavel Machek via libcamera-devel wrote:
> > JPEG is very similar to MJPEG format (AFAICT, it may contain extra
> > EXIF headers). Add support for it. Tested with PinePhone and command
> > line cam utility.
> >
> > Signed-off-by: Pavel Machek <pavel@ucw.cz>
> >
> > diff --git a/include/linux/drm_fourcc.h b/include/linux/drm_fourcc.h
> > index ea11dcb4..b30a705d 100644
> > --- a/include/linux/drm_fourcc.h
> > +++ b/include/linux/drm_fourcc.h
> > @@ -352,6 +352,7 @@ extern "C" {
> >
> >  /* Compressed formats */
> >  #define DRM_FORMAT_MJPEG	fourcc_code('M', 'J', 'P', 'G') /* Motion-JPEG */
> > +#define DRM_FORMAT_JPEG	fourcc_code('J', 'P', 'E', 'G') /* JFIF JPEG */
> 
> If something like this is required it should be done in upstream DRM.
> 
> However I'm not sure that's what we want, see below..
> 
> >  /*
> >   * Bayer formats
> > diff --git a/src/cam/sdl_sink.cpp b/src/cam/sdl_sink.cpp
> > index f8e3e95d..673bd642 100644
> > --- a/src/cam/sdl_sink.cpp
> > +++ b/src/cam/sdl_sink.cpp
> > @@ -63,6 +63,7 @@ int SDLSink::configure(const libcamera::CameraConfiguration &config)
> >
> >  	switch (cfg.pixelFormat) {
> >  #ifdef HAVE_SDL_IMAGE
> > +	case libcamera::formats::JPEG:
> >  	case libcamera::formats::MJPEG:
> >  		texture_ = std::make_unique<SDLTextureMJPG>(rect_);
> >  		break;
> > diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-utils.cpp
> > index 3f242286..fd5689d9 100644
> > --- a/src/gstreamer/gstlibcamera-utils.cpp
> > +++ b/src/gstreamer/gstlibcamera-utils.cpp
> > @@ -80,6 +80,7 @@ bare_structure_from_format(const PixelFormat &format)
> >  					 gst_video_format_to_string(gst_format), nullptr);
> >
> >  	switch (format) {
> > +	case formats::JPEG:
> >  	case formats::MJPEG:
> >  		return gst_structure_new_empty("image/jpeg");
> >  	default:
> > diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
> > index 283ecb3d..c5e97198 100644
> > --- a/src/libcamera/formats.cpp
> > +++ b/src/libcamera/formats.cpp
> > @@ -944,6 +944,19 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> >  	} },
> >
> >  	/* Compressed formats. */
> > +	{ formats::JPEG, {
> > +		.name = "JPEG",
> > +		.format = formats::JPEG,
> > +		.v4l2Formats = {
> > +			.single = V4L2PixelFormat(V4L2_PIX_FMT_JPEG),
> > +			.multi = V4L2PixelFormat(),
> > +		},
> > +		.bitsPerPixel = 0,
> > +		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > +		.packed = false,
> > +		.pixelsPerGroup = 1,
> > +		.planes = {{ { 1, 1 }, { 0, 0 }, { 0, 0 } }},
> > +	} },
> 
> I would rather work with the idea to associate multiple v4l2Formats to
> the single formats::MJPEG identifier, by making
> PixelFormatInfo::v4l2Formats an array.
> 
> However I'm not sure how we could rework this:
> 
> V4L2PixelFormat V4L2PixelFormat::fromPixelFormat(const PixelFormat &pixelFormat,
> 						 bool multiplanar)
> {
> 	const PixelFormatInfo &info = PixelFormatInfo::info(pixelFormat);
> 	if (!info.isValid())
> 		return V4L2PixelFormat();
> 
> 	return multiplanar ? info.v4l2Formats.multi : info.v4l2Formats.single;
> }
> 
> So that it picks the right V4L2 pixelformat if not by poking the video
> device and checking what it supports.
> 
> I think we already have a very similar problem in the
> mapping of contiguous/non-contiguous YUV planar formats, where we
> abuse the V4L2 planar/multiplanar API definition to discern which
> format to map to
> 
> 	{ formats::NV16, {
> 		.name = "NV16",
> 		.format = formats::NV16,
> 		.v4l2Formats = {
> 			.single = V4L2PixelFormat(V4L2_PIX_FMT_NV16),
> 			.multi = V4L2PixelFormat(V4L2_PIX_FMT_NV16M),
> 		},
> 
> Which is actually problematic as there are drivers that (rightfully)
> expose V4L2_PIX_FMT_NV16 through the multiplanar API.

I don't think that's really a problem, the multiplanar argument to
fromPixelFormat() refers to multiplanar formats, not the multiplanar
API. What is, however, a problem is that no caller sets that argument to
true :-)

> I would go one step further and remove the single/multiplanar formats
> and just place a list there to match against the list of formats
> supported by the video device which will have to be passed to
> V4L2PixelFormat::fromPixelFormat().

I don't like the idea of passing the list to the function if all callers
have to do so manually. Not only will it complicate the callers, but
some won't even have a list to pass (see
V4L2CameraProxy::setFmtFromConfig() for instance).

Looking at commit 395d43d6d75b ("libcamera: v4l2_videodevice: Drop
toV4L2PixelFormat()"), I think we should resurect that function, and use
it to replace most V4L2PixelFormat::fromPixelFormat() calls. It can
internally call a V4L2PixelFormat::fromPixelFormat() overload with a
list of formats, while a different V4L2PixelFormat::fromPixelFormat()
would exist for usage in the V4L2 compat layer. I'm interested in
knowing if all usages of V4L2PixelFormat::fromPixelFormat() from
pipeline handlers can be converted to
V4L2VideoDevice::toV4L2PixelFormat(), or if some would be problematic.

> Opinions ?
> 
> Pavel: I'm not asking you to go full circle and change such internal
> machinery to have your format supported, I can look into doing that,
> but in the meantime you should live with your patch out-of-tree. Is
> this ok ?
> 
> >  	{ formats::MJPEG, {
> >  		.name = "MJPEG",
> >  		.format = formats::MJPEG,
> > diff --git a/src/libcamera/formats.yaml b/src/libcamera/formats.yaml
> > index 7dda0132..6b931c34 100644
> > --- a/src/libcamera/formats.yaml
> > +++ b/src/libcamera/formats.yaml
> > @@ -76,6 +76,8 @@ formats:
> >    - YVU444:
> >        fourcc: DRM_FORMAT_YVU444
> >
> > +  - JPEG:
> > +      fourcc: DRM_FORMAT_JPEG
> >    - MJPEG:
> >        fourcc: DRM_FORMAT_MJPEG
> >
> > diff --git a/src/libcamera/v4l2_pixelformat.cpp b/src/libcamera/v4l2_pixelformat.cpp
> > index 58fc4e9d..82dfbd1e 100644
> > --- a/src/libcamera/v4l2_pixelformat.cpp
> > +++ b/src/libcamera/v4l2_pixelformat.cpp
> > @@ -183,6 +183,8 @@ const std::map<V4L2PixelFormat, V4L2PixelFormat::Info> vpf2pf{
> >  	/* Compressed formats. */
> >  	{ V4L2PixelFormat(V4L2_PIX_FMT_MJPEG),
> >  		{ formats::MJPEG, "Motion-JPEG" } },
> > +	{ V4L2PixelFormat(V4L2_PIX_FMT_JPEG),
> > +		{ formats::JPEG, "JPEG" } },
> >  };
> >
> >  } /* namespace */

Hi!

> Opinions ?
> 
> Pavel: I'm not asking you to go full circle and change such internal
> machinery to have your format supported, I can look into doing that,
> but in the meantime you should live with your patch out-of-tree. Is
> this ok ?

Thank you. Any solution is okay to me, as long as it gets solved in
reasonable timeframe.

Best regards,
								Pavel

Hi Laurent,

On Thu, Jul 07, 2022 at 11:10:10AM +0300, Laurent Pinchart wrote:
> Hi Jacopo,
>
> On Thu, Jul 07, 2022 at 09:33:42AM +0200, Jacopo Mondi via libcamera-devel wrote:
> > On Mon, Jul 04, 2022 at 08:07:20PM +0200, Pavel Machek via libcamera-devel wrote:
> > > JPEG is very similar to MJPEG format (AFAICT, it may contain extra
> > > EXIF headers). Add support for it. Tested with PinePhone and command
> > > line cam utility.
> > >
> > > Signed-off-by: Pavel Machek <pavel@ucw.cz>
> > >
> > > diff --git a/include/linux/drm_fourcc.h b/include/linux/drm_fourcc.h
> > > index ea11dcb4..b30a705d 100644
> > > --- a/include/linux/drm_fourcc.h
> > > +++ b/include/linux/drm_fourcc.h
> > > @@ -352,6 +352,7 @@ extern "C" {
> > >
> > >  /* Compressed formats */
> > >  #define DRM_FORMAT_MJPEG	fourcc_code('M', 'J', 'P', 'G') /* Motion-JPEG */
> > > +#define DRM_FORMAT_JPEG	fourcc_code('J', 'P', 'E', 'G') /* JFIF JPEG */
> >
> > If something like this is required it should be done in upstream DRM.
> >
> > However I'm not sure that's what we want, see below..
> >
> > >  /*
> > >   * Bayer formats
> > > diff --git a/src/cam/sdl_sink.cpp b/src/cam/sdl_sink.cpp
> > > index f8e3e95d..673bd642 100644
> > > --- a/src/cam/sdl_sink.cpp
> > > +++ b/src/cam/sdl_sink.cpp
> > > @@ -63,6 +63,7 @@ int SDLSink::configure(const libcamera::CameraConfiguration &config)
> > >
> > >  	switch (cfg.pixelFormat) {
> > >  #ifdef HAVE_SDL_IMAGE
> > > +	case libcamera::formats::JPEG:
> > >  	case libcamera::formats::MJPEG:
> > >  		texture_ = std::make_unique<SDLTextureMJPG>(rect_);
> > >  		break;
> > > diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-utils.cpp
> > > index 3f242286..fd5689d9 100644
> > > --- a/src/gstreamer/gstlibcamera-utils.cpp
> > > +++ b/src/gstreamer/gstlibcamera-utils.cpp
> > > @@ -80,6 +80,7 @@ bare_structure_from_format(const PixelFormat &format)
> > >  					 gst_video_format_to_string(gst_format), nullptr);
> > >
> > >  	switch (format) {
> > > +	case formats::JPEG:
> > >  	case formats::MJPEG:
> > >  		return gst_structure_new_empty("image/jpeg");
> > >  	default:
> > > diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
> > > index 283ecb3d..c5e97198 100644
> > > --- a/src/libcamera/formats.cpp
> > > +++ b/src/libcamera/formats.cpp
> > > @@ -944,6 +944,19 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > >  	} },
> > >
> > >  	/* Compressed formats. */
> > > +	{ formats::JPEG, {
> > > +		.name = "JPEG",
> > > +		.format = formats::JPEG,
> > > +		.v4l2Formats = {
> > > +			.single = V4L2PixelFormat(V4L2_PIX_FMT_JPEG),
> > > +			.multi = V4L2PixelFormat(),
> > > +		},
> > > +		.bitsPerPixel = 0,
> > > +		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > +		.packed = false,
> > > +		.pixelsPerGroup = 1,
> > > +		.planes = {{ { 1, 1 }, { 0, 0 }, { 0, 0 } }},
> > > +	} },
> >
> > I would rather work with the idea to associate multiple v4l2Formats to
> > the single formats::MJPEG identifier, by making
> > PixelFormatInfo::v4l2Formats an array.
> >
> > However I'm not sure how we could rework this:
> >
> > V4L2PixelFormat V4L2PixelFormat::fromPixelFormat(const PixelFormat &pixelFormat,
> > 						 bool multiplanar)
> > {
> > 	const PixelFormatInfo &info = PixelFormatInfo::info(pixelFormat);
> > 	if (!info.isValid())
> > 		return V4L2PixelFormat();
> >
> > 	return multiplanar ? info.v4l2Formats.multi : info.v4l2Formats.single;
> > }
> >
> > So that it picks the right V4L2 pixelformat if not by poking the video
> > device and checking what it supports.
> >
> > I think we already have a very similar problem in the
> > mapping of contiguous/non-contiguous YUV planar formats, where we
> > abuse the V4L2 planar/multiplanar API definition to discern which
> > format to map to
> >
> > 	{ formats::NV16, {
> > 		.name = "NV16",
> > 		.format = formats::NV16,
> > 		.v4l2Formats = {
> > 			.single = V4L2PixelFormat(V4L2_PIX_FMT_NV16),
> > 			.multi = V4L2PixelFormat(V4L2_PIX_FMT_NV16M),
> > 		},
> >
> > Which is actually problematic as there are drivers that (rightfully)
> > expose V4L2_PIX_FMT_NV16 through the multiplanar API.
>
> I don't think that's really a problem, the multiplanar argument to
> fromPixelFormat() refers to multiplanar formats, not the multiplanar
> API. What is, however, a problem is that no caller sets that argument to
> true :-)
>

Oh, it's even documented

 * planes formats. The two entries in the array store the contiguous and
 * non-contiguous V4L2 formats respectively. If the PixelFormat isn't a

In facts planar/multiplanar API do not really play a role in format
selection, but I thought we used that to indeitify the two

> > I would go one step further and remove the single/multiplanar formats
> > and just place a list there to match against the list of formats
> > supported by the video device which will have to be passed to
> > V4L2PixelFormat::fromPixelFormat().
>
> I don't like the idea of passing the list to the function if all callers
> have to do so manually. Not only will it complicate the callers, but

I would rather pass the video device, but the concept is the same, if
not that we would need an overload

> some won't even have a list to pass (see
> V4L2CameraProxy::setFmtFromConfig() for instance).
>
> Looking at commit 395d43d6d75b ("libcamera: v4l2_videodevice: Drop
> toV4L2PixelFormat()"), I think we should resurect that function, and use
> it to replace most V4L2PixelFormat::fromPixelFormat() calls. It can
> internally call a V4L2PixelFormat::fromPixelFormat() overload with a
> list of formats, while a different V4L2PixelFormat::fromPixelFormat()
> would exist for usage in the V4L2 compat layer. I'm interested in

I wonder if an overload with no formats list won't provide an easy
way out for lazy developers not to get the video device format list..

> knowing if all usages of V4L2PixelFormat::fromPixelFormat() from
> pipeline handlers can be converted to
> V4L2VideoDevice::toV4L2PixelFormat(), or if some would be problematic.

Let's find it out then...

----------------------------------------------------------------------------
src/libcamera/pipeline/ipu3/cio2.cpp:

- should be ok as the class has access to the output video device and
  can easily retrieve the list of supported formats. Please not this
  shouldn't even be necessary, as we know what formats the CIO2 videod
  device supports

        outputFormat->fourcc = V4L2PixelFormat::fromPixelFormat(itInfo->second);
        ...
        ret = output_->setFormat(outputFormat);


----------------------------------------------------------------------------
src/libcamera/pipeline/ipu3/imgu.cpp:

- Should equally be safe as the function that calls
  V4L2PixelFormat::fromPixelFormat() receives the video device as
  function parameter.

----------------------------------------------------------------------------

src/libcamera/pipeline/raspberrypi/raspberrypi.cpp:

- The toV4L2DeviceFormat() static function can be gives the video
  device (it's always Unicam::Image) the format should be converted for

- In RPiCameraConfiguration::validate() the functions is used twice

		if (i == maxIndex)
			dev = data_->isp_[Isp::Output0].dev();
		else
			dev = data_->isp_[Isp::Output1].dev();

		V4L2VideoDevice::Formats fmts = dev->formats();

		if (fmts.find(V4L2PixelFormat::fromPixelFormat(cfgPixFmt)) == fmts.end()) {
			/* If we cannot find a native format, use a default one. */
			cfgPixFmt = formats::NV12;
			status = Adjusted;
		}

		V4L2DeviceFormat format;
		format.fourcc = V4L2PixelFormat::fromPixelFormat(cfg.pixelFormat);
		format.size = cfg.size;
		format.colorSpace = cfg.colorSpace;

   Let alone the fact this looks like a bug (format.fourcc should be
   assigned with cfgPixFmt?) the function can use 'dev'

- In  PipelineHandlerRPi::configure()

		V4L2PixelFormat fourcc = V4L2PixelFormat::fromPixelFormat(cfg.pixelFormat);
		format.size = cfg.size;
		format.fourcc = fourcc;
		format.colorSpace = cfg.colorSpace;

		LOG(RPI, Debug) << "Setting " << stream->name() << " to "
				<< format;

		ret = stream->dev()->setFormat(&format);

  stream->dev() can be used as well as in the following usages where
  formats are translated knowing on which video devices they're going
  to be applied on


		format.fourcc = V4L2PixelFormat::fromPixelFormat(formats::YUV420);
                ..
		ret = data->isp_[Isp::Output0].dev()->setFormat(&format);

----------------------------------------------------------------------------
src/libcamera/pipeline/rkisp1/rkisp1_path.cpp

- In RkISP1Path::validate()

        format.fourcc = V4L2PixelFormat::fromPixelFormat(cfg->pixelFormat);
        ..
        int ret = video_->tryFormat(&format);

  We have access to video_

- In RkISP1Path::configure()

        outputFormat.fourcc = V4L2PixelFormat::fromPixelFormat(config.pixelFormat);
        ...
        ret = video_->setFormat(&outputFormat);

  again video_ is available

----------------------------------------------------------------------------

src/libcamera/pipeline/simple/converter.cpp

- SimpleConverter::Stream::configure() applies formats on m2m_->capture()
  and m2m_->output()

        V4L2PixelFormat videoFormat =
                v4L2PixelFormat::fromPixelFormat(inputCfg.pixelFormat);
        ..
        int ret = m2m_->output()->setFormat(&format);

- SimpleConverter::format() equally has access to m2m_->output()

- SimpleConverter::strideAndFrameSize() has  m2m_->capture()

- SimpleCameraConfiguration::validate() knows on what video device the
  format has to be applied

        format.fourcc = V4L2PixelFormat::fromPixelFormat(cfg.pixelFormat);
        ...
        int ret = data_->video_->tryFormat(&format);

- SimplePipelineHandler::configure() has as well access to the video
  device

        V4L2PixelFormat videoFormat =
                V4L2PixelFormat::fromPixelFormat(pipeConfig->captureFormat);

        ..
        ret = video->setFormat(&captureFormat);

----------------------------------------------------------------------------

src/libcamera/pipeline/vimc/vimc.cpp
src/libcamera/pipeline/uvcvideo/uvcvideo.cpp

- Have very similar patter where the format is converted just before
  being applied to the video device, and so the formats list can be
  accessed.

----------------------------------------------------------------------------

>
> > Opinions ?
> >
> > Pavel: I'm not asking you to go full circle and change such internal
> > machinery to have your format supported, I can look into doing that,
> > but in the meantime you should live with your patch out-of-tree. Is
> > this ok ?
> >
> > >  	{ formats::MJPEG, {
> > >  		.name = "MJPEG",
> > >  		.format = formats::MJPEG,
> > > diff --git a/src/libcamera/formats.yaml b/src/libcamera/formats.yaml
> > > index 7dda0132..6b931c34 100644
> > > --- a/src/libcamera/formats.yaml
> > > +++ b/src/libcamera/formats.yaml
> > > @@ -76,6 +76,8 @@ formats:
> > >    - YVU444:
> > >        fourcc: DRM_FORMAT_YVU444
> > >
> > > +  - JPEG:
> > > +      fourcc: DRM_FORMAT_JPEG
> > >    - MJPEG:
> > >        fourcc: DRM_FORMAT_MJPEG
> > >
> > > diff --git a/src/libcamera/v4l2_pixelformat.cpp b/src/libcamera/v4l2_pixelformat.cpp
> > > index 58fc4e9d..82dfbd1e 100644
> > > --- a/src/libcamera/v4l2_pixelformat.cpp
> > > +++ b/src/libcamera/v4l2_pixelformat.cpp
> > > @@ -183,6 +183,8 @@ const std::map<V4L2PixelFormat, V4L2PixelFormat::Info> vpf2pf{
> > >  	/* Compressed formats. */
> > >  	{ V4L2PixelFormat(V4L2_PIX_FMT_MJPEG),
> > >  		{ formats::MJPEG, "Motion-JPEG" } },
> > > +	{ V4L2PixelFormat(V4L2_PIX_FMT_JPEG),
> > > +		{ formats::JPEG, "JPEG" } },
> > >  };
> > >
> > >  } /* namespace */
>
> --
> Regards,
>
> Laurent Pinchart

Hi Jacopo,

On Thu, Jul 07, 2022 at 05:42:38PM +0200, Jacopo Mondi wrote:
> On Thu, Jul 07, 2022 at 11:10:10AM +0300, Laurent Pinchart wrote:
> > On Thu, Jul 07, 2022 at 09:33:42AM +0200, Jacopo Mondi via libcamera-devel wrote:
> > > On Mon, Jul 04, 2022 at 08:07:20PM +0200, Pavel Machek via libcamera-devel wrote:
> > > > JPEG is very similar to MJPEG format (AFAICT, it may contain extra
> > > > EXIF headers). Add support for it. Tested with PinePhone and command
> > > > line cam utility.
> > > >
> > > > Signed-off-by: Pavel Machek <pavel@ucw.cz>
> > > >
> > > > diff --git a/include/linux/drm_fourcc.h b/include/linux/drm_fourcc.h
> > > > index ea11dcb4..b30a705d 100644
> > > > --- a/include/linux/drm_fourcc.h
> > > > +++ b/include/linux/drm_fourcc.h
> > > > @@ -352,6 +352,7 @@ extern "C" {
> > > >
> > > >  /* Compressed formats */
> > > >  #define DRM_FORMAT_MJPEG	fourcc_code('M', 'J', 'P', 'G') /* Motion-JPEG */
> > > > +#define DRM_FORMAT_JPEG	fourcc_code('J', 'P', 'E', 'G') /* JFIF JPEG */
> > >
> > > If something like this is required it should be done in upstream DRM.
> > >
> > > However I'm not sure that's what we want, see below..
> > >
> > > >  /*
> > > >   * Bayer formats
> > > > diff --git a/src/cam/sdl_sink.cpp b/src/cam/sdl_sink.cpp
> > > > index f8e3e95d..673bd642 100644
> > > > --- a/src/cam/sdl_sink.cpp
> > > > +++ b/src/cam/sdl_sink.cpp
> > > > @@ -63,6 +63,7 @@ int SDLSink::configure(const libcamera::CameraConfiguration &config)
> > > >
> > > >  	switch (cfg.pixelFormat) {
> > > >  #ifdef HAVE_SDL_IMAGE
> > > > +	case libcamera::formats::JPEG:
> > > >  	case libcamera::formats::MJPEG:
> > > >  		texture_ = std::make_unique<SDLTextureMJPG>(rect_);
> > > >  		break;
> > > > diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-utils.cpp
> > > > index 3f242286..fd5689d9 100644
> > > > --- a/src/gstreamer/gstlibcamera-utils.cpp
> > > > +++ b/src/gstreamer/gstlibcamera-utils.cpp
> > > > @@ -80,6 +80,7 @@ bare_structure_from_format(const PixelFormat &format)
> > > >  					 gst_video_format_to_string(gst_format), nullptr);
> > > >
> > > >  	switch (format) {
> > > > +	case formats::JPEG:
> > > >  	case formats::MJPEG:
> > > >  		return gst_structure_new_empty("image/jpeg");
> > > >  	default:
> > > > diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
> > > > index 283ecb3d..c5e97198 100644
> > > > --- a/src/libcamera/formats.cpp
> > > > +++ b/src/libcamera/formats.cpp
> > > > @@ -944,6 +944,19 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
> > > >  	} },
> > > >
> > > >  	/* Compressed formats. */
> > > > +	{ formats::JPEG, {
> > > > +		.name = "JPEG",
> > > > +		.format = formats::JPEG,
> > > > +		.v4l2Formats = {
> > > > +			.single = V4L2PixelFormat(V4L2_PIX_FMT_JPEG),
> > > > +			.multi = V4L2PixelFormat(),
> > > > +		},
> > > > +		.bitsPerPixel = 0,
> > > > +		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> > > > +		.packed = false,
> > > > +		.pixelsPerGroup = 1,
> > > > +		.planes = {{ { 1, 1 }, { 0, 0 }, { 0, 0 } }},
> > > > +	} },
> > >
> > > I would rather work with the idea to associate multiple v4l2Formats to
> > > the single formats::MJPEG identifier, by making
> > > PixelFormatInfo::v4l2Formats an array.
> > >
> > > However I'm not sure how we could rework this:
> > >
> > > V4L2PixelFormat V4L2PixelFormat::fromPixelFormat(const PixelFormat &pixelFormat,
> > > 						 bool multiplanar)
> > > {
> > > 	const PixelFormatInfo &info = PixelFormatInfo::info(pixelFormat);
> > > 	if (!info.isValid())
> > > 		return V4L2PixelFormat();
> > >
> > > 	return multiplanar ? info.v4l2Formats.multi : info.v4l2Formats.single;
> > > }
> > >
> > > So that it picks the right V4L2 pixelformat if not by poking the video
> > > device and checking what it supports.
> > >
> > > I think we already have a very similar problem in the
> > > mapping of contiguous/non-contiguous YUV planar formats, where we
> > > abuse the V4L2 planar/multiplanar API definition to discern which
> > > format to map to
> > >
> > > 	{ formats::NV16, {
> > > 		.name = "NV16",
> > > 		.format = formats::NV16,
> > > 		.v4l2Formats = {
> > > 			.single = V4L2PixelFormat(V4L2_PIX_FMT_NV16),
> > > 			.multi = V4L2PixelFormat(V4L2_PIX_FMT_NV16M),
> > > 		},
> > >
> > > Which is actually problematic as there are drivers that (rightfully)
> > > expose V4L2_PIX_FMT_NV16 through the multiplanar API.
> >
> > I don't think that's really a problem, the multiplanar argument to
> > fromPixelFormat() refers to multiplanar formats, not the multiplanar
> > API. What is, however, a problem is that no caller sets that argument to
> > true :-)
> 
> Oh, it's even documented
> 
>  * planes formats. The two entries in the array store the contiguous and
>  * non-contiguous V4L2 formats respectively. If the PixelFormat isn't a
> 
> In facts planar/multiplanar API do not really play a role in format
> selection, but I thought we used that to indeitify the two
> 
> > > I would go one step further and remove the single/multiplanar formats
> > > and just place a list there to match against the list of formats
> > > supported by the video device which will have to be passed to
> > > V4L2PixelFormat::fromPixelFormat().
> >
> > I don't like the idea of passing the list to the function if all callers
> > have to do so manually. Not only will it complicate the callers, but
> 
> I would rather pass the video device, but the concept is the same, if
> not that we would need an overload

Passing the video device is a good idea, I haven't thought about it.

> > some won't even have a list to pass (see
> > V4L2CameraProxy::setFmtFromConfig() for instance).
> >
> > Looking at commit 395d43d6d75b ("libcamera: v4l2_videodevice: Drop
> > toV4L2PixelFormat()"), I think we should resurect that function, and use
> > it to replace most V4L2PixelFormat::fromPixelFormat() calls. It can
> > internally call a V4L2PixelFormat::fromPixelFormat() overload with a
> > list of formats, while a different V4L2PixelFormat::fromPixelFormat()
> > would exist for usage in the V4L2 compat layer. I'm interested in
> 
> I wonder if an overload with no formats list won't provide an easy
> way out for lazy developers not to get the video device format list..

Possibly, yes, but if we require pipeline handlers to always use
V4L2VideoDevice::toV4L2PixelFormat(), this could be caught during review
(and even flagged by checkstyle.py).

> > knowing if all usages of V4L2PixelFormat::fromPixelFormat() from
> > pipeline handlers can be converted to
> > V4L2VideoDevice::toV4L2PixelFormat(), or if some would be problematic.
> 
> Let's find it out then...
> 
> ----------------------------------------------------------------------------
> src/libcamera/pipeline/ipu3/cio2.cpp:
> 
> - should be ok as the class has access to the output video device and
>   can easily retrieve the list of supported formats. Please not this
>   shouldn't even be necessary, as we know what formats the CIO2 videod
>   device supports
> 
>         outputFormat->fourcc = V4L2PixelFormat::fromPixelFormat(itInfo->second);
>         ...
>         ret = output_->setFormat(outputFormat);
> 
> 
> ----------------------------------------------------------------------------
> src/libcamera/pipeline/ipu3/imgu.cpp:
> 
> - Should equally be safe as the function that calls
>   V4L2PixelFormat::fromPixelFormat() receives the video device as
>   function parameter.
> 
> ----------------------------------------------------------------------------
> 
> src/libcamera/pipeline/raspberrypi/raspberrypi.cpp:
> 
> - The toV4L2DeviceFormat() static function can be gives the video
>   device (it's always Unicam::Image) the format should be converted for
> 
> - In RPiCameraConfiguration::validate() the functions is used twice
> 
> 		if (i == maxIndex)
> 			dev = data_->isp_[Isp::Output0].dev();
> 		else
> 			dev = data_->isp_[Isp::Output1].dev();
> 
> 		V4L2VideoDevice::Formats fmts = dev->formats();
> 
> 		if (fmts.find(V4L2PixelFormat::fromPixelFormat(cfgPixFmt)) == fmts.end()) {
> 			/* If we cannot find a native format, use a default one. */
> 			cfgPixFmt = formats::NV12;
> 			status = Adjusted;
> 		}
> 
> 		V4L2DeviceFormat format;
> 		format.fourcc = V4L2PixelFormat::fromPixelFormat(cfg.pixelFormat);
> 		format.size = cfg.size;
> 		format.colorSpace = cfg.colorSpace;
> 
>    Let alone the fact this looks like a bug (format.fourcc should be
>    assigned with cfgPixFmt?) the function can use 'dev'

Yes it looks like a bug, good point. Can I let you send a patch ?

> - In  PipelineHandlerRPi::configure()
> 
> 		V4L2PixelFormat fourcc = V4L2PixelFormat::fromPixelFormat(cfg.pixelFormat);
> 		format.size = cfg.size;
> 		format.fourcc = fourcc;
> 		format.colorSpace = cfg.colorSpace;
> 
> 		LOG(RPI, Debug) << "Setting " << stream->name() << " to "
> 				<< format;
> 
> 		ret = stream->dev()->setFormat(&format);
> 
>   stream->dev() can be used as well as in the following usages where
>   formats are translated knowing on which video devices they're going
>   to be applied on
> 
> 
> 		format.fourcc = V4L2PixelFormat::fromPixelFormat(formats::YUV420);
>                 ..
> 		ret = data->isp_[Isp::Output0].dev()->setFormat(&format);
> 
> ----------------------------------------------------------------------------
> src/libcamera/pipeline/rkisp1/rkisp1_path.cpp
> 
> - In RkISP1Path::validate()
> 
>         format.fourcc = V4L2PixelFormat::fromPixelFormat(cfg->pixelFormat);
>         ..
>         int ret = video_->tryFormat(&format);
> 
>   We have access to video_
> 
> - In RkISP1Path::configure()
> 
>         outputFormat.fourcc = V4L2PixelFormat::fromPixelFormat(config.pixelFormat);
>         ...
>         ret = video_->setFormat(&outputFormat);
> 
>   again video_ is available
> 
> ----------------------------------------------------------------------------
> 
> src/libcamera/pipeline/simple/converter.cpp
> 
> - SimpleConverter::Stream::configure() applies formats on m2m_->capture()
>   and m2m_->output()
> 
>         V4L2PixelFormat videoFormat =
>                 v4L2PixelFormat::fromPixelFormat(inputCfg.pixelFormat);
>         ..
>         int ret = m2m_->output()->setFormat(&format);
> 
> - SimpleConverter::format() equally has access to m2m_->output()
> 
> - SimpleConverter::strideAndFrameSize() has  m2m_->capture()
> 
> - SimpleCameraConfiguration::validate() knows on what video device the
>   format has to be applied
> 
>         format.fourcc = V4L2PixelFormat::fromPixelFormat(cfg.pixelFormat);
>         ...
>         int ret = data_->video_->tryFormat(&format);
> 
> - SimplePipelineHandler::configure() has as well access to the video
>   device
> 
>         V4L2PixelFormat videoFormat =
>                 V4L2PixelFormat::fromPixelFormat(pipeConfig->captureFormat);
> 
>         ..
>         ret = video->setFormat(&captureFormat);
> 
> ----------------------------------------------------------------------------
> 
> src/libcamera/pipeline/vimc/vimc.cpp
> src/libcamera/pipeline/uvcvideo/uvcvideo.cpp
> 
> - Have very similar patter where the format is converted just before
>   being applied to the video device, and so the formats list can be
>   accessed.
> 
> ----------------------------------------------------------------------------

Sounds good so far :-) I'm looking forward to reviewing the patches.

> > > Opinions ?
> > >
> > > Pavel: I'm not asking you to go full circle and change such internal
> > > machinery to have your format supported, I can look into doing that,
> > > but in the meantime you should live with your patch out-of-tree. Is
> > > this ok ?
> > >
> > > >  	{ formats::MJPEG, {
> > > >  		.name = "MJPEG",
> > > >  		.format = formats::MJPEG,
> > > > diff --git a/src/libcamera/formats.yaml b/src/libcamera/formats.yaml
> > > > index 7dda0132..6b931c34 100644
> > > > --- a/src/libcamera/formats.yaml
> > > > +++ b/src/libcamera/formats.yaml
> > > > @@ -76,6 +76,8 @@ formats:
> > > >    - YVU444:
> > > >        fourcc: DRM_FORMAT_YVU444
> > > >
> > > > +  - JPEG:
> > > > +      fourcc: DRM_FORMAT_JPEG
> > > >    - MJPEG:
> > > >        fourcc: DRM_FORMAT_MJPEG
> > > >
> > > > diff --git a/src/libcamera/v4l2_pixelformat.cpp b/src/libcamera/v4l2_pixelformat.cpp
> > > > index 58fc4e9d..82dfbd1e 100644
> > > > --- a/src/libcamera/v4l2_pixelformat.cpp
> > > > +++ b/src/libcamera/v4l2_pixelformat.cpp
> > > > @@ -183,6 +183,8 @@ const std::map<V4L2PixelFormat, V4L2PixelFormat::Info> vpf2pf{
> > > >  	/* Compressed formats. */
> > > >  	{ V4L2PixelFormat(V4L2_PIX_FMT_MJPEG),
> > > >  		{ formats::MJPEG, "Motion-JPEG" } },
> > > > +	{ V4L2PixelFormat(V4L2_PIX_FMT_JPEG),
> > > > +		{ formats::JPEG, "JPEG" } },
> > > >  };
> > > >
> > > >  } /* namespace */

Le lundi 04 juillet 2022 à 20:07 +0200, Pavel Machek via libcamera-devel a
écrit :
> JPEG is very similar to MJPEG format (AFAICT, it may contain extra
> EXIF headers). Add support for it. Tested with PinePhone and command
> line cam utility.
> 
> Signed-off-by: Pavel Machek <pavel@ucw.cz>
> 
> diff --git a/include/linux/drm_fourcc.h b/include/linux/drm_fourcc.h
> index ea11dcb4..b30a705d 100644
> --- a/include/linux/drm_fourcc.h
> +++ b/include/linux/drm_fourcc.h
> @@ -352,6 +352,7 @@ extern "C" {
>  
>  /* Compressed formats */
>  #define DRM_FORMAT_MJPEG	fourcc_code('M', 'J', 'P', 'G') /* Motion-JPEG */
> +#define DRM_FORMAT_JPEG	fourcc_code('J', 'P', 'E', 'G') /* JFIF JPEG */

I personally don't see the point of adding a second JPEG format in the public
API. JPEG being duplicated is an absurdity because there is no decoders that do
differentiate any of these. I'd rather hide deeply this strange duplication that
originates from V4L2. Remember that there is a third JPEG format in V4L2 which
is also not going to be differentiated by the decoders.

regards,
Nicolas

>  
>  /*
>   * Bayer formats
> diff --git a/src/cam/sdl_sink.cpp b/src/cam/sdl_sink.cpp
> index f8e3e95d..673bd642 100644
> --- a/src/cam/sdl_sink.cpp
> +++ b/src/cam/sdl_sink.cpp
> @@ -63,6 +63,7 @@ int SDLSink::configure(const libcamera::CameraConfiguration &config)
>  
>  	switch (cfg.pixelFormat) {
>  #ifdef HAVE_SDL_IMAGE
> +	case libcamera::formats::JPEG:
>  	case libcamera::formats::MJPEG:
>  		texture_ = std::make_unique<SDLTextureMJPG>(rect_);
>  		break;
> diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-utils.cpp
> index 3f242286..fd5689d9 100644
> --- a/src/gstreamer/gstlibcamera-utils.cpp
> +++ b/src/gstreamer/gstlibcamera-utils.cpp
> @@ -80,6 +80,7 @@ bare_structure_from_format(const PixelFormat &format)
>  					 gst_video_format_to_string(gst_format), nullptr);
>  
>  	switch (format) {
> +	case formats::JPEG:
>  	case formats::MJPEG:
>  		return gst_structure_new_empty("image/jpeg");
>  	default:
> diff --git a/src/libcamera/formats.cpp b/src/libcamera/formats.cpp
> index 283ecb3d..c5e97198 100644
> --- a/src/libcamera/formats.cpp
> +++ b/src/libcamera/formats.cpp
> @@ -944,6 +944,19 @@ const std::map<PixelFormat, PixelFormatInfo> pixelFormatInfo{
>  	} },
>  
>  	/* Compressed formats. */
> +	{ formats::JPEG, {
> +		.name = "JPEG",
> +		.format = formats::JPEG,
> +		.v4l2Formats = {
> +			.single = V4L2PixelFormat(V4L2_PIX_FMT_JPEG),
> +			.multi = V4L2PixelFormat(),
> +		},
> +		.bitsPerPixel = 0,
> +		.colourEncoding = PixelFormatInfo::ColourEncodingYUV,
> +		.packed = false,
> +		.pixelsPerGroup = 1,
> +		.planes = {{ { 1, 1 }, { 0, 0 }, { 0, 0 } }},
> +	} },
>  	{ formats::MJPEG, {
>  		.name = "MJPEG",
>  		.format = formats::MJPEG,
> diff --git a/src/libcamera/formats.yaml b/src/libcamera/formats.yaml
> index 7dda0132..6b931c34 100644
> --- a/src/libcamera/formats.yaml
> +++ b/src/libcamera/formats.yaml
> @@ -76,6 +76,8 @@ formats:
>    - YVU444:
>        fourcc: DRM_FORMAT_YVU444
>  
> +  - JPEG:
> +      fourcc: DRM_FORMAT_JPEG
>    - MJPEG:
>        fourcc: DRM_FORMAT_MJPEG
>  
> diff --git a/src/libcamera/v4l2_pixelformat.cpp b/src/libcamera/v4l2_pixelformat.cpp
> index 58fc4e9d..82dfbd1e 100644
> --- a/src/libcamera/v4l2_pixelformat.cpp
> +++ b/src/libcamera/v4l2_pixelformat.cpp
> @@ -183,6 +183,8 @@ const std::map<V4L2PixelFormat, V4L2PixelFormat::Info> vpf2pf{
>  	/* Compressed formats. */
>  	{ V4L2PixelFormat(V4L2_PIX_FMT_MJPEG),
>  		{ formats::MJPEG, "Motion-JPEG" } },
> +	{ V4L2PixelFormat(V4L2_PIX_FMT_JPEG),
> +		{ formats::JPEG, "JPEG" } },
>  };
>  
>  } /* namespace */
>

Hi!

> > JPEG is very similar to MJPEG format (AFAICT, it may contain extra
> > EXIF headers). Add support for it. Tested with PinePhone and command
> > line cam utility.
> > 
> > Signed-off-by: Pavel Machek <pavel@ucw.cz>
> > 
> > diff --git a/include/linux/drm_fourcc.h b/include/linux/drm_fourcc.h
> > index ea11dcb4..b30a705d 100644
> > --- a/include/linux/drm_fourcc.h
> > +++ b/include/linux/drm_fourcc.h
> > @@ -352,6 +352,7 @@ extern "C" {
> >  
> >  /* Compressed formats */
> >  #define DRM_FORMAT_MJPEG	fourcc_code('M', 'J', 'P', 'G') /* Motion-JPEG */
> > +#define DRM_FORMAT_JPEG	fourcc_code('J', 'P', 'E', 'G') /* JFIF JPEG */
> 
> I personally don't see the point of adding a second JPEG format in the public
> API. JPEG being duplicated is an absurdity because there is no decoders that do
> differentiate any of these. I'd rather hide deeply this strange duplication that
> originates from V4L2. Remember that there is a third JPEG format in V4L2 which
> is also not going to be differentiated by the decoders.

As long as JPEGs work for me, I'm quite happy. Can we make that
happen? :-).
								Pavel

Hi!

> > > Best explanation I found so far was that JPEG = EXIF + MJPEG. Kernel
> > > interface exposes JPEG and MJPEG as separate types, so IMO it makes
> > > sense for libcamera to support both.
> > 
> > TBH JPEG at least has a specification. MJPEG largely doesn't - it's
> > just a concatenation of JPEG encoded frames, and there seems to be no
> > real definition of which JPEG blocks are required and in which order.
> > Some MJPEG encoders drop the Huffman tables and rely on the decoder
> > assuming defaults.
> > There are then also MJPEG-A and MJPEG-B variants defined by Apple as
> > part of the Quicktime spec [1], with MJPEG-B not being compatible with
> > JPEG decoders due to changing headers.
> > 
> > Which flavour V4L2_PIX_FMT_MJPEG relates to seems to be undefined.
> > 
> > I've had issues in the past with what conditions VLC judges to be JPEG
> > vs MJPEG, and never found a definitive answer [2]
> > 
> > So how Libcamera should handle JPEG vs MJPEG probably doesn't have an
> > obvious answer. There's little point in encoding EXIF in a hardware
> > encoder as you have the overhead of passing all the parameters in.
> > For mapping into V4L2, just treat both of them the same.
> 
> Older C920 could be setup to produce jpeg, which embeds H.264 into the EXIF
> header. You'll find support for that in GStreamer [0]. Probably not relevant to
> the discussion. In GStreamer v4l2src [1], all form of jpeg is assumed to be the
> same. Which made me notice that V4L2_PIX_FMT_PJPG not mentionned here. I have no
> idea what this refer too, this special case [1] dates from before me. I think
> kernel driver encoders today will in software append the missing table/headers
> as needed.

PJPG is "progressive jpeg", AFAICT, see here:
https://cloudinary.com/blog/progressive_jpegs_and_green_martians . I
actually wishpinephone would produce that, as it would make previews
faster.

Best regards,
							Pavel