| Message ID | 20240226153854.99471-1-jacopo.mondi@ideasonboard.com |
|---|---|
| State | Accepted |
| Headers | show |
| Series |
|
| Related | show |
On Mon, 26 Feb 2024 at 16:39, Jacopo Mondi <jacopo.mondi@ideasonboard.com> wrote: > > Add modifiers for the Raspberry Pi PiSP compressed formats. > > The compressed formats are documented at: > Documentation/userspace-api/media/v4l/pixfmt-pisp-comp-rggb.rst > > and in the PiSP datasheet: > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com> > --- > > Background: > ----------- > > The Raspberry Pi PiSP camera subsystem is on its way to upstream through the > Video4Linux2 subsystem: > https://patchwork.linuxtv.org/project/linux-media/list/?series=12310 > > The PiSP camera system is composed by a "Front End" and a "Back End". > The FrontEnd part is a MIPI CSI-2 receiver that store frames to memory and > produce statistics, and the BackEnd is a memory-to-memory ISP that converts > images in a format usable by application. > > The "FrontEnd" is capable of encoding RAW Bayer images as received by the > image sensor in a 'compressed' format defined by Raspberry Pi and fully > documented in the PiSP manual: > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > The compression scheme is documented in the in-review patch series for the BE > support at: > https://patchwork.linuxtv.org/project/linux-media/patch/20240223163012.300763-7-jacopo.mondi@ideasonboard.com/ > > The "BackEnd" is capable of consuming images in the compressed format and > optionally user application might want to inspect those images for debugging > purposes. > > Why a DRM modifier > ------------------ > > The PiSP support is entirely implemented in libcamera, with the support of an > hw-specific library called 'libpisp'. > > libcamera uses the fourcc codes defined by DRM to define its formats: > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/formats.yaml > > And to define a new libcamera format for the Raspberry Pi compressed ones we > need to associate the above proposed modifiers with a RAW Bayer format > identifier. > > In example: > > - RGGB16_PISP_COMP1: > fourcc: DRM_FORMAT_SRGGB16 > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > - GRBG16_PISP_COMP1: > fourcc: DRM_FORMAT_SGRBG16 > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > - GBRG16_PISP_COMP1: > fourcc: DRM_FORMAT_SGBRG16 > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > - BGGR16_PISP_COMP1: > fourcc: DRM_FORMAT_SBGGR16 > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > - MONO_PISP_COMP1: > fourcc: DRM_FORMAT_R16 > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > See > https://patchwork.libcamera.org/patch/19503/ > > Would if be acceptable for DRM to include the above proposed modifiers for the > purpose of defining the above presented libcamera formats ? There will be no > graphic format associated with these modifiers as their purpose it not > displaying images but rather exchange them between the components of the > camera subsystem (and possibly be inspected by specialized test applications). Yeah I think libcamera using drm-fourcc formats and modifiers is absolutely ok, and has my ack in principle. And for these users we're ok with merging modifiers that the kernel doesn't use. I think it would be really good to formalize this by adding libcamera to the officially listed users in the "Open Source User Waiver" section in the drm_fourcc.h docs: https://dri.freedesktop.org/docs/drm/gpu/drm-kms.html#open-source-user-waiver You might want to convert that into a list, it could get a bit confusing. Then we can get that patch properly acked (by kernel and libcamera folks) to record the community consensus. For the rpi modifiers themselves: They need to be properly documented, least to exclude a screw-up like with the rpi modifiers we already have, which unfortunately encode the buffer height (instead of just the rounding algorithim to align the height to the right tile size) in the modifiers, which breaks assumptions everywhere. For details see https://gitlab.freedesktop.org/wlroots/wlroots/-/merge_requests/4529#note_2262057 Cheers, Sima > > --- > include/uapi/drm/drm_fourcc.h | 5 +++++ > 1 file changed, 5 insertions(+) > > diff --git a/include/uapi/drm/drm_fourcc.h b/include/uapi/drm/drm_fourcc.h > index 00db00083175..09b182a959ad 100644 > --- a/include/uapi/drm/drm_fourcc.h > +++ b/include/uapi/drm/drm_fourcc.h > @@ -425,6 +425,7 @@ extern "C" { > #define DRM_FORMAT_MOD_VENDOR_ARM 0x08 > #define DRM_FORMAT_MOD_VENDOR_ALLWINNER 0x09 > #define DRM_FORMAT_MOD_VENDOR_AMLOGIC 0x0a > +#define DRM_FORMAT_MOD_VENDOR_RPI 0x0b > > /* add more to the end as needed */ > > @@ -1568,6 +1569,10 @@ drm_fourcc_canonicalize_nvidia_format_mod(__u64 modifier) > #define AMD_FMT_MOD_CLEAR(field) \ > (~((__u64)AMD_FMT_MOD_##field##_MASK << AMD_FMT_MOD_##field##_SHIFT)) > > +/* RPI (Raspberry Pi) modifiers */ > +#define PISP_FORMAT_MOD_COMPRESS_MODE1 fourcc_mod_code(RPI, 1) > +#define PISP_FORMAT_MOD_COMPRESS_MODE2 fourcc_mod_code(RPI, 2) > + > #if defined(__cplusplus) > } > #endif > -- > 2.43.0 >
Hi Sima On Mon, Feb 26, 2024 at 04:46:24PM +0100, Daniel Vetter wrote: > On Mon, 26 Feb 2024 at 16:39, Jacopo Mondi > <jacopo.mondi@ideasonboard.com> wrote: > > > > Add modifiers for the Raspberry Pi PiSP compressed formats. > > > > The compressed formats are documented at: > > Documentation/userspace-api/media/v4l/pixfmt-pisp-comp-rggb.rst > > > > and in the PiSP datasheet: > > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > > > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com> > > --- > > > > Background: > > ----------- > > > > The Raspberry Pi PiSP camera subsystem is on its way to upstream through the > > Video4Linux2 subsystem: > > https://patchwork.linuxtv.org/project/linux-media/list/?series=12310 > > > > The PiSP camera system is composed by a "Front End" and a "Back End". > > The FrontEnd part is a MIPI CSI-2 receiver that store frames to memory and > > produce statistics, and the BackEnd is a memory-to-memory ISP that converts > > images in a format usable by application. > > > > The "FrontEnd" is capable of encoding RAW Bayer images as received by the > > image sensor in a 'compressed' format defined by Raspberry Pi and fully > > documented in the PiSP manual: > > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > > > The compression scheme is documented in the in-review patch series for the BE > > support at: > > https://patchwork.linuxtv.org/project/linux-media/patch/20240223163012.300763-7-jacopo.mondi@ideasonboard.com/ > > > > The "BackEnd" is capable of consuming images in the compressed format and > > optionally user application might want to inspect those images for debugging > > purposes. > > > > Why a DRM modifier > > ------------------ > > > > The PiSP support is entirely implemented in libcamera, with the support of an > > hw-specific library called 'libpisp'. > > > > libcamera uses the fourcc codes defined by DRM to define its formats: > > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/formats.yaml > > > > And to define a new libcamera format for the Raspberry Pi compressed ones we > > need to associate the above proposed modifiers with a RAW Bayer format > > identifier. > > > > In example: > > > > - RGGB16_PISP_COMP1: > > fourcc: DRM_FORMAT_SRGGB16 > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > - GRBG16_PISP_COMP1: > > fourcc: DRM_FORMAT_SGRBG16 > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > - GBRG16_PISP_COMP1: > > fourcc: DRM_FORMAT_SGBRG16 > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > - BGGR16_PISP_COMP1: > > fourcc: DRM_FORMAT_SBGGR16 > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > - MONO_PISP_COMP1: > > fourcc: DRM_FORMAT_R16 > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > See > > https://patchwork.libcamera.org/patch/19503/ > > > > Would if be acceptable for DRM to include the above proposed modifiers for the > > purpose of defining the above presented libcamera formats ? There will be no > > graphic format associated with these modifiers as their purpose it not > > displaying images but rather exchange them between the components of the > > camera subsystem (and possibly be inspected by specialized test applications). > > Yeah I think libcamera using drm-fourcc formats and modifiers is > absolutely ok, and has my ack in principle. And for these users we're > ok with merging modifiers that the kernel doesn't use. > Great! > I think it would be really good to formalize this by adding libcamera > to the officially listed users in the "Open Source User Waiver" > section in the drm_fourcc.h docs: > > https://dri.freedesktop.org/docs/drm/gpu/drm-kms.html#open-source-user-waiver > > You might want to convert that into a list, it could get a bit > confusing. Then we can get that patch properly acked (by kernel and > libcamera folks) to record the community consensus. > I see your point, but I wonder if libcamera actually need to be part of this list; we want in-kernel upstream driver and open-source userspace components. We allow binary 3A modules to be loaded by the library, but I'm not sure they will ever need to modify the DRM 4cc list. Anyway, with other libcamera people ack, I can certainly do so! > For the rpi modifiers themselves: They need to be properly documented, > least to exclude a screw-up like with the rpi modifiers we already > have, which unfortunately encode the buffer height (instead of just > the rounding algorithim to align the height to the right tile size) in > the modifiers, which breaks assumptions everywhere. For details see > https://gitlab.freedesktop.org/wlroots/wlroots/-/merge_requests/4529#note_2262057 I see. The formats are fully documented in the above linked datasheet, and I've provided (with the help of RPi engineers, as I don't understand the compression algorithm part :) a shorter description as part of the V4L2 patch series that upstreams the BE driver https://patchwork.linuxtv.org/project/linux-media/patch/20240223163012.300763-7-jacopo.mondi@ideasonboard.com/ The only indication about the buffer's layout I see is Each scanline is padded to a multiple of 8 pixels wide, and each block of 8 horizontally-contiguous pixels is coded using 8 bytes. While the rest of the text describes the compression algorithm which (afaiu) doesn't affect the buffer geometry. Would you be ok with me replicating the above description (or maybe just reference the V4L2 documentation ?) > > Cheers, Sima > > > > > --- > > include/uapi/drm/drm_fourcc.h | 5 +++++ > > 1 file changed, 5 insertions(+) > > > > diff --git a/include/uapi/drm/drm_fourcc.h b/include/uapi/drm/drm_fourcc.h > > index 00db00083175..09b182a959ad 100644 > > --- a/include/uapi/drm/drm_fourcc.h > > +++ b/include/uapi/drm/drm_fourcc.h > > @@ -425,6 +425,7 @@ extern "C" { > > #define DRM_FORMAT_MOD_VENDOR_ARM 0x08 > > #define DRM_FORMAT_MOD_VENDOR_ALLWINNER 0x09 > > #define DRM_FORMAT_MOD_VENDOR_AMLOGIC 0x0a > > +#define DRM_FORMAT_MOD_VENDOR_RPI 0x0b > > > > /* add more to the end as needed */ > > > > @@ -1568,6 +1569,10 @@ drm_fourcc_canonicalize_nvidia_format_mod(__u64 modifier) > > #define AMD_FMT_MOD_CLEAR(field) \ > > (~((__u64)AMD_FMT_MOD_##field##_MASK << AMD_FMT_MOD_##field##_SHIFT)) > > > > +/* RPI (Raspberry Pi) modifiers */ > > +#define PISP_FORMAT_MOD_COMPRESS_MODE1 fourcc_mod_code(RPI, 1) > > +#define PISP_FORMAT_MOD_COMPRESS_MODE2 fourcc_mod_code(RPI, 2) > > + > > #if defined(__cplusplus) > > } > > #endif > > -- > > 2.43.0 > > > > > -- > Daniel Vetter > Software Engineer, Intel Corporation > http://blog.ffwll.ch
On Mon, Feb 26, 2024 at 04:46:24PM +0100, Daniel Vetter wrote: > On Mon, 26 Feb 2024 at 16:39, Jacopo Mondi wrote: > > > > Add modifiers for the Raspberry Pi PiSP compressed formats. > > > > The compressed formats are documented at: > > Documentation/userspace-api/media/v4l/pixfmt-pisp-comp-rggb.rst > > > > and in the PiSP datasheet: > > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > > > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com> > > --- > > > > Background: > > ----------- > > > > The Raspberry Pi PiSP camera subsystem is on its way to upstream through the > > Video4Linux2 subsystem: > > https://patchwork.linuxtv.org/project/linux-media/list/?series=12310 > > > > The PiSP camera system is composed by a "Front End" and a "Back End". > > The FrontEnd part is a MIPI CSI-2 receiver that store frames to memory and > > produce statistics, and the BackEnd is a memory-to-memory ISP that converts > > images in a format usable by application. > > > > The "FrontEnd" is capable of encoding RAW Bayer images as received by the > > image sensor in a 'compressed' format defined by Raspberry Pi and fully > > documented in the PiSP manual: > > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > > > The compression scheme is documented in the in-review patch series for the BE > > support at: > > https://patchwork.linuxtv.org/project/linux-media/patch/20240223163012.300763-7-jacopo.mondi@ideasonboard.com/ > > > > The "BackEnd" is capable of consuming images in the compressed format and > > optionally user application might want to inspect those images for debugging > > purposes. > > > > Why a DRM modifier > > ------------------ > > > > The PiSP support is entirely implemented in libcamera, with the support of an > > hw-specific library called 'libpisp'. > > > > libcamera uses the fourcc codes defined by DRM to define its formats: > > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/formats.yaml > > > > And to define a new libcamera format for the Raspberry Pi compressed ones we > > need to associate the above proposed modifiers with a RAW Bayer format > > identifier. > > > > In example: > > > > - RGGB16_PISP_COMP1: > > fourcc: DRM_FORMAT_SRGGB16 An "interesting" issue here is that these formats currently live in libcamera only, we haven't merged them in DRM "yet". This may be a prerequisite ? > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > - GRBG16_PISP_COMP1: > > fourcc: DRM_FORMAT_SGRBG16 > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > - GBRG16_PISP_COMP1: > > fourcc: DRM_FORMAT_SGBRG16 > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > - BGGR16_PISP_COMP1: > > fourcc: DRM_FORMAT_SBGGR16 > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > - MONO_PISP_COMP1: > > fourcc: DRM_FORMAT_R16 > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > See > > https://patchwork.libcamera.org/patch/19503/ > > > > Would if be acceptable for DRM to include the above proposed modifiers for the > > purpose of defining the above presented libcamera formats ? There will be no > > graphic format associated with these modifiers as their purpose it not > > displaying images but rather exchange them between the components of the > > camera subsystem (and possibly be inspected by specialized test applications). > > Yeah I think libcamera using drm-fourcc formats and modifiers is > absolutely ok, and has my ack in principle. And for these users we're > ok with merging modifiers that the kernel doesn't use. > > I think it would be really good to formalize this by adding libcamera > to the officially listed users in the "Open Source User Waiver" > section in the drm_fourcc.h docs: > > https://dri.freedesktop.org/docs/drm/gpu/drm-kms.html#open-source-user-waiver The waiver states "the usual requirement for an upstream in-kernel or open source userspace user does not apply". Strictly speaking, there is an open-source userspace user with libcamera. It's only on the kernel side that the new modifier my not get used. This being said, I'm fine clarifying the documentation, clarity is always good. > You might want to convert that into a list, it could get a bit > confusing. Then we can get that patch properly acked (by kernel and > libcamera folks) to record the community consensus. > > For the rpi modifiers themselves: They need to be properly documented, > least to exclude a screw-up like with the rpi modifiers we already > have, which unfortunately encode the buffer height (instead of just > the rounding algorithim to align the height to the right tile size) in > the modifiers, which breaks assumptions everywhere. For details see > https://gitlab.freedesktop.org/wlroots/wlroots/-/merge_requests/4529#note_2262057 > > > --- > > include/uapi/drm/drm_fourcc.h | 5 +++++ > > 1 file changed, 5 insertions(+) > > > > diff --git a/include/uapi/drm/drm_fourcc.h b/include/uapi/drm/drm_fourcc.h > > index 00db00083175..09b182a959ad 100644 > > --- a/include/uapi/drm/drm_fourcc.h > > +++ b/include/uapi/drm/drm_fourcc.h > > @@ -425,6 +425,7 @@ extern "C" { > > #define DRM_FORMAT_MOD_VENDOR_ARM 0x08 > > #define DRM_FORMAT_MOD_VENDOR_ALLWINNER 0x09 > > #define DRM_FORMAT_MOD_VENDOR_AMLOGIC 0x0a > > +#define DRM_FORMAT_MOD_VENDOR_RPI 0x0b > > > > /* add more to the end as needed */ > > > > @@ -1568,6 +1569,10 @@ drm_fourcc_canonicalize_nvidia_format_mod(__u64 modifier) > > #define AMD_FMT_MOD_CLEAR(field) \ > > (~((__u64)AMD_FMT_MOD_##field##_MASK << AMD_FMT_MOD_##field##_SHIFT)) > > > > +/* RPI (Raspberry Pi) modifiers */ > > +#define PISP_FORMAT_MOD_COMPRESS_MODE1 fourcc_mod_code(RPI, 1) > > +#define PISP_FORMAT_MOD_COMPRESS_MODE2 fourcc_mod_code(RPI, 2) > > + > > #if defined(__cplusplus) > > } > > #endif
On Mon, Feb 26, 2024 at 05:24:41PM +0100, Jacopo Mondi wrote: > On Mon, Feb 26, 2024 at 04:46:24PM +0100, Daniel Vetter wrote: > > On Mon, 26 Feb 2024 at 16:39, Jacopo Mondi wrote: > > > > > > Add modifiers for the Raspberry Pi PiSP compressed formats. > > > > > > The compressed formats are documented at: > > > Documentation/userspace-api/media/v4l/pixfmt-pisp-comp-rggb.rst > > > > > > and in the PiSP datasheet: > > > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > > > > > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com> > > > --- > > > > > > Background: > > > ----------- > > > > > > The Raspberry Pi PiSP camera subsystem is on its way to upstream through the > > > Video4Linux2 subsystem: > > > https://patchwork.linuxtv.org/project/linux-media/list/?series=12310 > > > > > > The PiSP camera system is composed by a "Front End" and a "Back End". > > > The FrontEnd part is a MIPI CSI-2 receiver that store frames to memory and > > > produce statistics, and the BackEnd is a memory-to-memory ISP that converts > > > images in a format usable by application. > > > > > > The "FrontEnd" is capable of encoding RAW Bayer images as received by the > > > image sensor in a 'compressed' format defined by Raspberry Pi and fully > > > documented in the PiSP manual: > > > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > > > > > The compression scheme is documented in the in-review patch series for the BE > > > support at: > > > https://patchwork.linuxtv.org/project/linux-media/patch/20240223163012.300763-7-jacopo.mondi@ideasonboard.com/ > > > > > > The "BackEnd" is capable of consuming images in the compressed format and > > > optionally user application might want to inspect those images for debugging > > > purposes. > > > > > > Why a DRM modifier > > > ------------------ > > > > > > The PiSP support is entirely implemented in libcamera, with the support of an > > > hw-specific library called 'libpisp'. > > > > > > libcamera uses the fourcc codes defined by DRM to define its formats: > > > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/formats.yaml > > > > > > And to define a new libcamera format for the Raspberry Pi compressed ones we > > > need to associate the above proposed modifiers with a RAW Bayer format > > > identifier. > > > > > > In example: > > > > > > - RGGB16_PISP_COMP1: > > > fourcc: DRM_FORMAT_SRGGB16 > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > - GRBG16_PISP_COMP1: > > > fourcc: DRM_FORMAT_SGRBG16 > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > - GBRG16_PISP_COMP1: > > > fourcc: DRM_FORMAT_SGBRG16 > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > - BGGR16_PISP_COMP1: > > > fourcc: DRM_FORMAT_SBGGR16 > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > - MONO_PISP_COMP1: > > > fourcc: DRM_FORMAT_R16 > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > > > See > > > https://patchwork.libcamera.org/patch/19503/ > > > > > > Would if be acceptable for DRM to include the above proposed modifiers for the > > > purpose of defining the above presented libcamera formats ? There will be no > > > graphic format associated with these modifiers as their purpose it not > > > displaying images but rather exchange them between the components of the > > > camera subsystem (and possibly be inspected by specialized test applications). > > > > Yeah I think libcamera using drm-fourcc formats and modifiers is > > absolutely ok, and has my ack in principle. And for these users we're > > ok with merging modifiers that the kernel doesn't use. > > Great! > > > I think it would be really good to formalize this by adding libcamera > > to the officially listed users in the "Open Source User Waiver" > > section in the drm_fourcc.h docs: > > > > https://dri.freedesktop.org/docs/drm/gpu/drm-kms.html#open-source-user-waiver > > > > You might want to convert that into a list, it could get a bit > > confusing. Then we can get that patch properly acked (by kernel and > > libcamera folks) to record the community consensus. > > I see your point, but I wonder if libcamera actually need to be part > of this list; we want in-kernel upstream driver and open-source For the kernel side we'll use V4L2, so the DRM modifier won't have an in-kernel user. On the userspace side, yes, there will be an open-source user with libcamera. > userspace components. We allow binary 3A modules to be loaded by the > library, but I'm not sure they will ever need to modify the DRM 4cc list. > > Anyway, with other libcamera people ack, I can certainly do so! > > > For the rpi modifiers themselves: They need to be properly documented, > > least to exclude a screw-up like with the rpi modifiers we already > > have, which unfortunately encode the buffer height (instead of just > > the rounding algorithim to align the height to the right tile size) in > > the modifiers, which breaks assumptions everywhere. For details see > > https://gitlab.freedesktop.org/wlroots/wlroots/-/merge_requests/4529#note_2262057 > > I see. The formats are fully documented in the above linked datasheet, > and I've provided (with the help of RPi engineers, as I don't > understand the compression algorithm part :) a shorter description as > part of the V4L2 patch series that upstreams the BE driver > > https://patchwork.linuxtv.org/project/linux-media/patch/20240223163012.300763-7-jacopo.mondi@ideasonboard.com/ > > The only indication about the buffer's layout I see is > > Each scanline is padded to a multiple of 8 pixels wide, and each block of 8 > horizontally-contiguous pixels is coded using 8 bytes. > > While the rest of the text describes the compression algorithm which > (afaiu) doesn't affect the buffer geometry. > > Would you be ok with me replicating the above description (or maybe > just reference the V4L2 documentation ?) > > > > --- > > > include/uapi/drm/drm_fourcc.h | 5 +++++ > > > 1 file changed, 5 insertions(+) > > > > > > diff --git a/include/uapi/drm/drm_fourcc.h b/include/uapi/drm/drm_fourcc.h > > > index 00db00083175..09b182a959ad 100644 > > > --- a/include/uapi/drm/drm_fourcc.h > > > +++ b/include/uapi/drm/drm_fourcc.h > > > @@ -425,6 +425,7 @@ extern "C" { > > > #define DRM_FORMAT_MOD_VENDOR_ARM 0x08 > > > #define DRM_FORMAT_MOD_VENDOR_ALLWINNER 0x09 > > > #define DRM_FORMAT_MOD_VENDOR_AMLOGIC 0x0a > > > +#define DRM_FORMAT_MOD_VENDOR_RPI 0x0b > > > > > > /* add more to the end as needed */ > > > > > > @@ -1568,6 +1569,10 @@ drm_fourcc_canonicalize_nvidia_format_mod(__u64 modifier) > > > #define AMD_FMT_MOD_CLEAR(field) \ > > > (~((__u64)AMD_FMT_MOD_##field##_MASK << AMD_FMT_MOD_##field##_SHIFT)) > > > > > > +/* RPI (Raspberry Pi) modifiers */ > > > +#define PISP_FORMAT_MOD_COMPRESS_MODE1 fourcc_mod_code(RPI, 1) > > > +#define PISP_FORMAT_MOD_COMPRESS_MODE2 fourcc_mod_code(RPI, 2) > > > + > > > #if defined(__cplusplus) > > > } > > > #endif
Hi Laurent On Tue, Feb 27, 2024 at 03:08:27PM +0200, Laurent Pinchart wrote: > On Mon, Feb 26, 2024 at 04:46:24PM +0100, Daniel Vetter wrote: > > On Mon, 26 Feb 2024 at 16:39, Jacopo Mondi wrote: > > > > > > Add modifiers for the Raspberry Pi PiSP compressed formats. > > > > > > The compressed formats are documented at: > > > Documentation/userspace-api/media/v4l/pixfmt-pisp-comp-rggb.rst > > > > > > and in the PiSP datasheet: > > > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > > > > > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com> > > > --- > > > > > > Background: > > > ----------- > > > > > > The Raspberry Pi PiSP camera subsystem is on its way to upstream through the > > > Video4Linux2 subsystem: > > > https://patchwork.linuxtv.org/project/linux-media/list/?series=12310 > > > > > > The PiSP camera system is composed by a "Front End" and a "Back End". > > > The FrontEnd part is a MIPI CSI-2 receiver that store frames to memory and > > > produce statistics, and the BackEnd is a memory-to-memory ISP that converts > > > images in a format usable by application. > > > > > > The "FrontEnd" is capable of encoding RAW Bayer images as received by the > > > image sensor in a 'compressed' format defined by Raspberry Pi and fully > > > documented in the PiSP manual: > > > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > > > > > The compression scheme is documented in the in-review patch series for the BE > > > support at: > > > https://patchwork.linuxtv.org/project/linux-media/patch/20240223163012.300763-7-jacopo.mondi@ideasonboard.com/ > > > > > > The "BackEnd" is capable of consuming images in the compressed format and > > > optionally user application might want to inspect those images for debugging > > > purposes. > > > > > > Why a DRM modifier > > > ------------------ > > > > > > The PiSP support is entirely implemented in libcamera, with the support of an > > > hw-specific library called 'libpisp'. > > > > > > libcamera uses the fourcc codes defined by DRM to define its formats: > > > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/formats.yaml > > > > > > And to define a new libcamera format for the Raspberry Pi compressed ones we > > > need to associate the above proposed modifiers with a RAW Bayer format > > > identifier. > > > > > > In example: > > > > > > - RGGB16_PISP_COMP1: > > > fourcc: DRM_FORMAT_SRGGB16 > > An "interesting" issue here is that these formats currently live in > libcamera only, we haven't merged them in DRM "yet". This may be a > prerequisite ? > Ah right! I didn't notice! I think there are two issues at play here, one to be clarified by the DRM maintainers, the other more technically involved with the definition of the Bayer formats themselves. - Does DRM want RAW Bayer formats to be listed here, as these are not typically 'graphic' formats. What's the DRM maintainers opinion here ? - The RAW Bayer pattern ordering: we have realized that the idea of defining RAW Bayer formats by encoding the components ordering in V4L2 was, in retrospective, a bad idea. The reason is that any change in the Bayer order along the capture pipeline (usually due to flip) requires reconfiguration of the formats along the whole pipeline, requiring a start/stop of the capture operations. For this reason, we have multiple times discussed the idea of only conveying the bit depth in the format definition and to communicate the components ordering with other out-of-band mechanism, that will likely be used to negotiate between the image sensor and the CSI-2 receiver. With DRM the ideal would be something like DRM_FORMAT_RAW8 | FORMAT_MODIFIER_SBGGR DRM_FORMAT_RAW8 | FORMAT_MODIFIER_SGBRG ... DRM_FORMAT_RAW10 | FORMAT_MODIFIER_SBGGR DRM_FORMAT_RAW10 | FORMAT_MODIFIER_SGBRG ... This however prevents to use additional modifiers to convey, in example, packaging (ie MIPI CSI-2) or compressions (like done here for PISP_COMP1) as, as far as I understand it, multiple modifiers cannot be applied to a format. However, as I understand the argument of not repeating the V4L2 mistakes here, the DRM formats are aimed to be used mostly to represent the formats of images as they get stored in application buffers (at least from a libcamera perspective). The drawback of having to reconfigure the capture pipeline due to a flip does not really apply here, actually, knowing the components ordering in the final image is very relevant for applications to know how to interpret the image data. TL;DR I'm not too concerned about having to propagate the components ordering along the pipeline, as the DRM formats are used only to describe the image data as presented to the applications and not along the pipeline as it happens for the V4L2 formats. Opinions anyone ? Thanks j > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > - GRBG16_PISP_COMP1: > > > fourcc: DRM_FORMAT_SGRBG16 > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > - GBRG16_PISP_COMP1: > > > fourcc: DRM_FORMAT_SGBRG16 > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > - BGGR16_PISP_COMP1: > > > fourcc: DRM_FORMAT_SBGGR16 > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > - MONO_PISP_COMP1: > > > fourcc: DRM_FORMAT_R16 > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > > > See > > > https://patchwork.libcamera.org/patch/19503/ > > > > > > Would if be acceptable for DRM to include the above proposed modifiers for the > > > purpose of defining the above presented libcamera formats ? There will be no > > > graphic format associated with these modifiers as their purpose it not > > > displaying images but rather exchange them between the components of the > > > camera subsystem (and possibly be inspected by specialized test applications). > > > > Yeah I think libcamera using drm-fourcc formats and modifiers is > > absolutely ok, and has my ack in principle. And for these users we're > > ok with merging modifiers that the kernel doesn't use. > > > > I think it would be really good to formalize this by adding libcamera > > to the officially listed users in the "Open Source User Waiver" > > section in the drm_fourcc.h docs: > > > > https://dri.freedesktop.org/docs/drm/gpu/drm-kms.html#open-source-user-waiver > > The waiver states "the usual requirement for an upstream in-kernel or > open source userspace user does not apply". Strictly speaking, there is > an open-source userspace user with libcamera. It's only on the kernel > side that the new modifier my not get used. > > This being said, I'm fine clarifying the documentation, clarity is > always good. > > > You might want to convert that into a list, it could get a bit > > confusing. Then we can get that patch properly acked (by kernel and > > libcamera folks) to record the community consensus. > > > > For the rpi modifiers themselves: They need to be properly documented, > > least to exclude a screw-up like with the rpi modifiers we already > > have, which unfortunately encode the buffer height (instead of just > > the rounding algorithim to align the height to the right tile size) in > > the modifiers, which breaks assumptions everywhere. For details see > > https://gitlab.freedesktop.org/wlroots/wlroots/-/merge_requests/4529#note_2262057 > > > > > --- > > > include/uapi/drm/drm_fourcc.h | 5 +++++ > > > 1 file changed, 5 insertions(+) > > > > > > diff --git a/include/uapi/drm/drm_fourcc.h b/include/uapi/drm/drm_fourcc.h > > > index 00db00083175..09b182a959ad 100644 > > > --- a/include/uapi/drm/drm_fourcc.h > > > +++ b/include/uapi/drm/drm_fourcc.h > > > @@ -425,6 +425,7 @@ extern "C" { > > > #define DRM_FORMAT_MOD_VENDOR_ARM 0x08 > > > #define DRM_FORMAT_MOD_VENDOR_ALLWINNER 0x09 > > > #define DRM_FORMAT_MOD_VENDOR_AMLOGIC 0x0a > > > +#define DRM_FORMAT_MOD_VENDOR_RPI 0x0b > > > > > > /* add more to the end as needed */ > > > > > > @@ -1568,6 +1569,10 @@ drm_fourcc_canonicalize_nvidia_format_mod(__u64 modifier) > > > #define AMD_FMT_MOD_CLEAR(field) \ > > > (~((__u64)AMD_FMT_MOD_##field##_MASK << AMD_FMT_MOD_##field##_SHIFT)) > > > > > > +/* RPI (Raspberry Pi) modifiers */ > > > +#define PISP_FORMAT_MOD_COMPRESS_MODE1 fourcc_mod_code(RPI, 1) > > > +#define PISP_FORMAT_MOD_COMPRESS_MODE2 fourcc_mod_code(RPI, 2) > > > + > > > #if defined(__cplusplus) > > > } > > > #endif > > -- > Regards, > > Laurent Pinchart
On Wed, Feb 28, 2024 at 11:41:57AM +0100, Jacopo Mondi wrote: > On Tue, Feb 27, 2024 at 03:08:27PM +0200, Laurent Pinchart wrote: > > On Mon, Feb 26, 2024 at 04:46:24PM +0100, Daniel Vetter wrote: > > > On Mon, 26 Feb 2024 at 16:39, Jacopo Mondi wrote: > > > > > > > > Add modifiers for the Raspberry Pi PiSP compressed formats. > > > > > > > > The compressed formats are documented at: > > > > Documentation/userspace-api/media/v4l/pixfmt-pisp-comp-rggb.rst > > > > > > > > and in the PiSP datasheet: > > > > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > > > > > > > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com> > > > > --- > > > > > > > > Background: > > > > ----------- > > > > > > > > The Raspberry Pi PiSP camera subsystem is on its way to upstream through the > > > > Video4Linux2 subsystem: > > > > https://patchwork.linuxtv.org/project/linux-media/list/?series=12310 > > > > > > > > The PiSP camera system is composed by a "Front End" and a "Back End". > > > > The FrontEnd part is a MIPI CSI-2 receiver that store frames to memory and > > > > produce statistics, and the BackEnd is a memory-to-memory ISP that converts > > > > images in a format usable by application. > > > > > > > > The "FrontEnd" is capable of encoding RAW Bayer images as received by the > > > > image sensor in a 'compressed' format defined by Raspberry Pi and fully > > > > documented in the PiSP manual: > > > > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > > > > > > > The compression scheme is documented in the in-review patch series for the BE > > > > support at: > > > > https://patchwork.linuxtv.org/project/linux-media/patch/20240223163012.300763-7-jacopo.mondi@ideasonboard.com/ > > > > > > > > The "BackEnd" is capable of consuming images in the compressed format and > > > > optionally user application might want to inspect those images for debugging > > > > purposes. > > > > > > > > Why a DRM modifier > > > > ------------------ > > > > > > > > The PiSP support is entirely implemented in libcamera, with the support of an > > > > hw-specific library called 'libpisp'. > > > > > > > > libcamera uses the fourcc codes defined by DRM to define its formats: > > > > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/formats.yaml > > > > > > > > And to define a new libcamera format for the Raspberry Pi compressed ones we > > > > need to associate the above proposed modifiers with a RAW Bayer format > > > > identifier. > > > > > > > > In example: > > > > > > > > - RGGB16_PISP_COMP1: > > > > fourcc: DRM_FORMAT_SRGGB16 > > > > An "interesting" issue here is that these formats currently live in > > libcamera only, we haven't merged them in DRM "yet". This may be a > > prerequisite ? > > > > Ah right! I didn't notice! > > I think there are two issues at play here, one to be clarified by the > DRM maintainers, the other more technically involved with the > definition of the Bayer formats themselves. > > - Does DRM want RAW Bayer formats to be listed here, as these are not > typically 'graphic' formats. What's the DRM maintainers opinion here ? To give some context, the "historical mistake" I keep referring to regarding V4L2 is the decision to combine the bit depth of raw formats with the colour filter array (a.k.a. Bayer) pattern into a fourcc. I think we should have defined raw pixel formats that only encode a bit depth, and conveyed the CFA pattern out-of-band. This is especially the case for media bus codes (formats on the buses between hardware devices). The reasoning here is that most devices only care about the bit depth and not the CFA pattern. Adding a new CFA pattern (for instance for RGB+Ir) for a camera sensor would currently require adding a new media bus code (easy), using it in the sensor driver (obvious), and patching *every* CSI-2 receiver driver to pass it through. Userspace would similarly need to grow support for the new format, even for userspace code that only cares about capturing the raw data without processing the colour components. Having raw pixel formats that don't convey the CFA pattern would simplify all this (and it's something I'm considering adding to the media bus codes). part is lots of churn > - The RAW Bayer pattern ordering: we have realized that the idea of > defining RAW Bayer formats by encoding the components ordering in > V4L2 was, in retrospective, a bad idea. The reason is that any > change in the Bayer order along the capture pipeline (usually due to > flip) requires reconfiguration of the formats along the whole > pipeline, requiring a start/stop of the capture operations. That's also a reason, yes. > For this reason, we have multiple times discussed the idea of only > conveying the bit depth in the format definition and to communicate > the components ordering with other out-of-band mechanism, that will > likely be used to negotiate between the image sensor and the CSI-2 > receiver. > > With DRM the ideal would be something like > > DRM_FORMAT_RAW8 | FORMAT_MODIFIER_SBGGR > DRM_FORMAT_RAW8 | FORMAT_MODIFIER_SGBRG > ... > > DRM_FORMAT_RAW10 | FORMAT_MODIFIER_SBGGR > DRM_FORMAT_RAW10 | FORMAT_MODIFIER_SGBRG > ... When I said out-of-band, I meant really out of band from a format point of view, so neither in the 4CC nor in the modifiers. > This however prevents to use additional modifiers to convey, in > example, packaging (ie MIPI CSI-2) or compressions (like done here > for PISP_COMP1) as, as far as I understand it, multiple modifiers > cannot be applied to a format. > > However, as I understand the argument of not repeating the V4L2 > mistakes here, the DRM formats are aimed to be used mostly to > represent the formats of images as they get stored in application > buffers (at least from a libcamera perspective). The drawback of > having to reconfigure the capture pipeline due to a flip does not > really apply here, actually, knowing the components ordering in the > final image is very relevant for applications to know how to > interpret the image data. > > TL;DR I'm not too concerned about having to propagate the components > ordering along the pipeline, as the DRM formats are used only to > describe the image data as presented to the applications and not > along the pipeline as it happens for the V4L2 formats. > > Opinions anyone ? > > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > - GRBG16_PISP_COMP1: > > > > fourcc: DRM_FORMAT_SGRBG16 > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > - GBRG16_PISP_COMP1: > > > > fourcc: DRM_FORMAT_SGBRG16 > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > - BGGR16_PISP_COMP1: > > > > fourcc: DRM_FORMAT_SBGGR16 > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > - MONO_PISP_COMP1: > > > > fourcc: DRM_FORMAT_R16 > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > > > > > See > > > > https://patchwork.libcamera.org/patch/19503/ > > > > > > > > Would if be acceptable for DRM to include the above proposed modifiers for the > > > > purpose of defining the above presented libcamera formats ? There will be no > > > > graphic format associated with these modifiers as their purpose it not > > > > displaying images but rather exchange them between the components of the > > > > camera subsystem (and possibly be inspected by specialized test applications). > > > > > > Yeah I think libcamera using drm-fourcc formats and modifiers is > > > absolutely ok, and has my ack in principle. And for these users we're > > > ok with merging modifiers that the kernel doesn't use. > > > > > > I think it would be really good to formalize this by adding libcamera > > > to the officially listed users in the "Open Source User Waiver" > > > section in the drm_fourcc.h docs: > > > > > > https://dri.freedesktop.org/docs/drm/gpu/drm-kms.html#open-source-user-waiver > > > > The waiver states "the usual requirement for an upstream in-kernel or > > open source userspace user does not apply". Strictly speaking, there is > > an open-source userspace user with libcamera. It's only on the kernel > > side that the new modifier my not get used. > > > > This being said, I'm fine clarifying the documentation, clarity is > > always good. > > > > > You might want to convert that into a list, it could get a bit > > > confusing. Then we can get that patch properly acked (by kernel and > > > libcamera folks) to record the community consensus. > > > > > > For the rpi modifiers themselves: They need to be properly documented, > > > least to exclude a screw-up like with the rpi modifiers we already > > > have, which unfortunately encode the buffer height (instead of just > > > the rounding algorithim to align the height to the right tile size) in > > > the modifiers, which breaks assumptions everywhere. For details see > > > https://gitlab.freedesktop.org/wlroots/wlroots/-/merge_requests/4529#note_2262057 > > > > > > > --- > > > > include/uapi/drm/drm_fourcc.h | 5 +++++ > > > > 1 file changed, 5 insertions(+) > > > > > > > > diff --git a/include/uapi/drm/drm_fourcc.h b/include/uapi/drm/drm_fourcc.h > > > > index 00db00083175..09b182a959ad 100644 > > > > --- a/include/uapi/drm/drm_fourcc.h > > > > +++ b/include/uapi/drm/drm_fourcc.h > > > > @@ -425,6 +425,7 @@ extern "C" { > > > > #define DRM_FORMAT_MOD_VENDOR_ARM 0x08 > > > > #define DRM_FORMAT_MOD_VENDOR_ALLWINNER 0x09 > > > > #define DRM_FORMAT_MOD_VENDOR_AMLOGIC 0x0a > > > > +#define DRM_FORMAT_MOD_VENDOR_RPI 0x0b > > > > > > > > /* add more to the end as needed */ > > > > > > > > @@ -1568,6 +1569,10 @@ drm_fourcc_canonicalize_nvidia_format_mod(__u64 modifier) > > > > #define AMD_FMT_MOD_CLEAR(field) \ > > > > (~((__u64)AMD_FMT_MOD_##field##_MASK << AMD_FMT_MOD_##field##_SHIFT)) > > > > > > > > +/* RPI (Raspberry Pi) modifiers */ > > > > +#define PISP_FORMAT_MOD_COMPRESS_MODE1 fourcc_mod_code(RPI, 1) > > > > +#define PISP_FORMAT_MOD_COMPRESS_MODE2 fourcc_mod_code(RPI, 2) > > > > + > > > > #if defined(__cplusplus) > > > > } > > > > #endif
On Tue, Feb 27, 2024 at 03:10:06PM +0200, Laurent Pinchart wrote: > On Mon, Feb 26, 2024 at 05:24:41PM +0100, Jacopo Mondi wrote: > > On Mon, Feb 26, 2024 at 04:46:24PM +0100, Daniel Vetter wrote: > > > On Mon, 26 Feb 2024 at 16:39, Jacopo Mondi wrote: > > > > > > > > Add modifiers for the Raspberry Pi PiSP compressed formats. > > > > > > > > The compressed formats are documented at: > > > > Documentation/userspace-api/media/v4l/pixfmt-pisp-comp-rggb.rst > > > > > > > > and in the PiSP datasheet: > > > > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > > > > > > > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com> > > > > --- > > > > > > > > Background: > > > > ----------- > > > > > > > > The Raspberry Pi PiSP camera subsystem is on its way to upstream through the > > > > Video4Linux2 subsystem: > > > > https://patchwork.linuxtv.org/project/linux-media/list/?series=12310 > > > > > > > > The PiSP camera system is composed by a "Front End" and a "Back End". > > > > The FrontEnd part is a MIPI CSI-2 receiver that store frames to memory and > > > > produce statistics, and the BackEnd is a memory-to-memory ISP that converts > > > > images in a format usable by application. > > > > > > > > The "FrontEnd" is capable of encoding RAW Bayer images as received by the > > > > image sensor in a 'compressed' format defined by Raspberry Pi and fully > > > > documented in the PiSP manual: > > > > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > > > > > > > The compression scheme is documented in the in-review patch series for the BE > > > > support at: > > > > https://patchwork.linuxtv.org/project/linux-media/patch/20240223163012.300763-7-jacopo.mondi@ideasonboard.com/ > > > > > > > > The "BackEnd" is capable of consuming images in the compressed format and > > > > optionally user application might want to inspect those images for debugging > > > > purposes. > > > > > > > > Why a DRM modifier > > > > ------------------ > > > > > > > > The PiSP support is entirely implemented in libcamera, with the support of an > > > > hw-specific library called 'libpisp'. > > > > > > > > libcamera uses the fourcc codes defined by DRM to define its formats: > > > > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/formats.yaml > > > > > > > > And to define a new libcamera format for the Raspberry Pi compressed ones we > > > > need to associate the above proposed modifiers with a RAW Bayer format > > > > identifier. > > > > > > > > In example: > > > > > > > > - RGGB16_PISP_COMP1: > > > > fourcc: DRM_FORMAT_SRGGB16 > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > - GRBG16_PISP_COMP1: > > > > fourcc: DRM_FORMAT_SGRBG16 > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > - GBRG16_PISP_COMP1: > > > > fourcc: DRM_FORMAT_SGBRG16 > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > - BGGR16_PISP_COMP1: > > > > fourcc: DRM_FORMAT_SBGGR16 > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > - MONO_PISP_COMP1: > > > > fourcc: DRM_FORMAT_R16 > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > > > > > See > > > > https://patchwork.libcamera.org/patch/19503/ > > > > > > > > Would if be acceptable for DRM to include the above proposed modifiers for the > > > > purpose of defining the above presented libcamera formats ? There will be no > > > > graphic format associated with these modifiers as their purpose it not > > > > displaying images but rather exchange them between the components of the > > > > camera subsystem (and possibly be inspected by specialized test applications). > > > > > > Yeah I think libcamera using drm-fourcc formats and modifiers is > > > absolutely ok, and has my ack in principle. And for these users we're > > > ok with merging modifiers that the kernel doesn't use. > > > > Great! > > > > > I think it would be really good to formalize this by adding libcamera > > > to the officially listed users in the "Open Source User Waiver" > > > section in the drm_fourcc.h docs: > > > > > > https://dri.freedesktop.org/docs/drm/gpu/drm-kms.html#open-source-user-waiver > > > > > > You might want to convert that into a list, it could get a bit > > > confusing. Then we can get that patch properly acked (by kernel and > > > libcamera folks) to record the community consensus. > > > > I see your point, but I wonder if libcamera actually need to be part > > of this list; we want in-kernel upstream driver and open-source > > For the kernel side we'll use V4L2, so the DRM modifier won't have an > in-kernel user. Also if you e.g. use a gpu or accel thing for post-processing, which would be a drm driver, then you still don't have an in-kernel user for these because the userspace driver handles that all directly and just sends command buffers to the kernel. So I think adding libcamera explicitly would be really good. Also if you convert the list to a proper list it might be good to link to the relevant gl/vk extensions directly instead of just vaguely hinting at them. > On the userspace side, yes, there will be an open-source user with > libcamera. > > > userspace components. We allow binary 3A modules to be loaded by the > > library, but I'm not sure they will ever need to modify the DRM 4cc list. > > > > Anyway, with other libcamera people ack, I can certainly do so! > > > > > For the rpi modifiers themselves: They need to be properly documented, > > > least to exclude a screw-up like with the rpi modifiers we already > > > have, which unfortunately encode the buffer height (instead of just > > > the rounding algorithim to align the height to the right tile size) in > > > the modifiers, which breaks assumptions everywhere. For details see > > > https://gitlab.freedesktop.org/wlroots/wlroots/-/merge_requests/4529#note_2262057 > > > > I see. The formats are fully documented in the above linked datasheet, > > and I've provided (with the help of RPi engineers, as I don't > > understand the compression algorithm part :) a shorter description as > > part of the V4L2 patch series that upstreams the BE driver I think linking to the datasheet in the comment above the modifiers is ok too, if we don't have anything better. Just not entirely undocumented magic values, because those result in giantic confusion all around. Also for the namespace, I'm assuming this format is actually rpi IP, and not something they've lincensed from someone else? Because the point here is interop, so if it's a licensed format we need to put the modifiers under the name of the original IP vendor, or people will create duplicated modifiers and the entire interop dream is just a dream. -Sima > > > > https://patchwork.linuxtv.org/project/linux-media/patch/20240223163012.300763-7-jacopo.mondi@ideasonboard.com/ > > > > The only indication about the buffer's layout I see is > > > > Each scanline is padded to a multiple of 8 pixels wide, and each block of 8 > > horizontally-contiguous pixels is coded using 8 bytes. > > > > While the rest of the text describes the compression algorithm which > > (afaiu) doesn't affect the buffer geometry. > > > > Would you be ok with me replicating the above description (or maybe > > just reference the V4L2 documentation ?) > > > > > > --- > > > > include/uapi/drm/drm_fourcc.h | 5 +++++ > > > > 1 file changed, 5 insertions(+) > > > > > > > > diff --git a/include/uapi/drm/drm_fourcc.h b/include/uapi/drm/drm_fourcc.h > > > > index 00db00083175..09b182a959ad 100644 > > > > --- a/include/uapi/drm/drm_fourcc.h > > > > +++ b/include/uapi/drm/drm_fourcc.h > > > > @@ -425,6 +425,7 @@ extern "C" { > > > > #define DRM_FORMAT_MOD_VENDOR_ARM 0x08 > > > > #define DRM_FORMAT_MOD_VENDOR_ALLWINNER 0x09 > > > > #define DRM_FORMAT_MOD_VENDOR_AMLOGIC 0x0a > > > > +#define DRM_FORMAT_MOD_VENDOR_RPI 0x0b > > > > > > > > /* add more to the end as needed */ > > > > > > > > @@ -1568,6 +1569,10 @@ drm_fourcc_canonicalize_nvidia_format_mod(__u64 modifier) > > > > #define AMD_FMT_MOD_CLEAR(field) \ > > > > (~((__u64)AMD_FMT_MOD_##field##_MASK << AMD_FMT_MOD_##field##_SHIFT)) > > > > > > > > +/* RPI (Raspberry Pi) modifiers */ > > > > +#define PISP_FORMAT_MOD_COMPRESS_MODE1 fourcc_mod_code(RPI, 1) > > > > +#define PISP_FORMAT_MOD_COMPRESS_MODE2 fourcc_mod_code(RPI, 2) > > > > + > > > > #if defined(__cplusplus) > > > > } > > > > #endif > > -- > Regards, > > Laurent Pinchart
On Wed, Feb 28, 2024 at 01:13:45PM +0200, Laurent Pinchart wrote: > On Wed, Feb 28, 2024 at 11:41:57AM +0100, Jacopo Mondi wrote: > > On Tue, Feb 27, 2024 at 03:08:27PM +0200, Laurent Pinchart wrote: > > > On Mon, Feb 26, 2024 at 04:46:24PM +0100, Daniel Vetter wrote: > > > > On Mon, 26 Feb 2024 at 16:39, Jacopo Mondi wrote: > > > > > > > > > > Add modifiers for the Raspberry Pi PiSP compressed formats. > > > > > > > > > > The compressed formats are documented at: > > > > > Documentation/userspace-api/media/v4l/pixfmt-pisp-comp-rggb.rst > > > > > > > > > > and in the PiSP datasheet: > > > > > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > > > > > > > > > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com> > > > > > --- > > > > > > > > > > Background: > > > > > ----------- > > > > > > > > > > The Raspberry Pi PiSP camera subsystem is on its way to upstream through the > > > > > Video4Linux2 subsystem: > > > > > https://patchwork.linuxtv.org/project/linux-media/list/?series=12310 > > > > > > > > > > The PiSP camera system is composed by a "Front End" and a "Back End". > > > > > The FrontEnd part is a MIPI CSI-2 receiver that store frames to memory and > > > > > produce statistics, and the BackEnd is a memory-to-memory ISP that converts > > > > > images in a format usable by application. > > > > > > > > > > The "FrontEnd" is capable of encoding RAW Bayer images as received by the > > > > > image sensor in a 'compressed' format defined by Raspberry Pi and fully > > > > > documented in the PiSP manual: > > > > > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > > > > > > > > > The compression scheme is documented in the in-review patch series for the BE > > > > > support at: > > > > > https://patchwork.linuxtv.org/project/linux-media/patch/20240223163012.300763-7-jacopo.mondi@ideasonboard.com/ > > > > > > > > > > The "BackEnd" is capable of consuming images in the compressed format and > > > > > optionally user application might want to inspect those images for debugging > > > > > purposes. > > > > > > > > > > Why a DRM modifier > > > > > ------------------ > > > > > > > > > > The PiSP support is entirely implemented in libcamera, with the support of an > > > > > hw-specific library called 'libpisp'. > > > > > > > > > > libcamera uses the fourcc codes defined by DRM to define its formats: > > > > > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/formats.yaml > > > > > > > > > > And to define a new libcamera format for the Raspberry Pi compressed ones we > > > > > need to associate the above proposed modifiers with a RAW Bayer format > > > > > identifier. > > > > > > > > > > In example: > > > > > > > > > > - RGGB16_PISP_COMP1: > > > > > fourcc: DRM_FORMAT_SRGGB16 > > > > > > An "interesting" issue here is that these formats currently live in > > > libcamera only, we haven't merged them in DRM "yet". This may be a > > > prerequisite ? > > > > > > > Ah right! I didn't notice! > > > > I think there are two issues at play here, one to be clarified by the > > DRM maintainers, the other more technically involved with the > > definition of the Bayer formats themselves. > > > > - Does DRM want RAW Bayer formats to be listed here, as these are not > > typically 'graphic' formats. What's the DRM maintainers opinion here ? > > To give some context, the "historical mistake" I keep referring to > regarding V4L2 is the decision to combine the bit depth of raw formats > with the colour filter array (a.k.a. Bayer) pattern into a fourcc. I > think we should have defined raw pixel formats that only encode a bit > depth, and conveyed the CFA pattern out-of-band. This is especially the > case for media bus codes (formats on the buses between hardware > devices). The reasoning here is that most devices only care about the > bit depth and not the CFA pattern. Adding a new CFA pattern (for > instance for RGB+Ir) for a camera sensor would currently require adding > a new media bus code (easy), using it in the sensor driver (obvious), > and patching *every* CSI-2 receiver driver to pass it through. Userspace > would similarly need to grow support for the new format, even for > userspace code that only cares about capturing the raw data without > processing the colour components. Having raw pixel formats that don't > convey the CFA pattern would simplify all this (and it's something I'm > considering adding to the media bus codes). > part is lots of churn So both drm fourcc and modifer formats are meant to be entirely opaque and complete. Which means if you take them from one driver (whether userspace or kernel driver shouldn't matter) and pass it to another, together with the height, width (in pixels) and the pitch (in linearized bytes, people got confused and wanted to do tile row pitch here and create and absolute mess) must _fully_ describe the buffer for the other side. Which would mean the bayer pattern would need to be in the modifier. Same way we have BGRA and RGBA and all these swizzles of the same stuff. But also if these are purely internal to libcamera I guess we can make exceptions, after all we do have _R8 and _RG8 formats too, and those have nothing to do with red/green colors - it's simply historical naming for the first/second color channel. > > - The RAW Bayer pattern ordering: we have realized that the idea of > > defining RAW Bayer formats by encoding the components ordering in > > V4L2 was, in retrospective, a bad idea. The reason is that any > > change in the Bayer order along the capture pipeline (usually due to > > flip) requires reconfiguration of the formats along the whole > > pipeline, requiring a start/stop of the capture operations. > > That's also a reason, yes. > > > For this reason, we have multiple times discussed the idea of only > > conveying the bit depth in the format definition and to communicate > > the components ordering with other out-of-band mechanism, that will > > likely be used to negotiate between the image sensor and the CSI-2 > > receiver. > > > > With DRM the ideal would be something like > > > > DRM_FORMAT_RAW8 | FORMAT_MODIFIER_SBGGR > > DRM_FORMAT_RAW8 | FORMAT_MODIFIER_SGBRG > > ... > > > > DRM_FORMAT_RAW10 | FORMAT_MODIFIER_SBGGR > > DRM_FORMAT_RAW10 | FORMAT_MODIFIER_SGBRG > > ... > > When I said out-of-band, I meant really out of band from a format point > of view, so neither in the 4CC nor in the modifiers. See above, but drm fourcc don't have flags. Yes we have the big/little-endian flag, and it's absolute lolz and mostly doesn't work. Also _RAW8 exist already, it's R8, again see above for the historical reasons why gl/vk call them like that. > > This however prevents to use additional modifiers to convey, in > > example, packaging (ie MIPI CSI-2) or compressions (like done here > > for PISP_COMP1) as, as far as I understand it, multiple modifiers > > cannot be applied to a format. > > > > However, as I understand the argument of not repeating the V4L2 > > mistakes here, the DRM formats are aimed to be used mostly to > > represent the formats of images as they get stored in application > > buffers (at least from a libcamera perspective). The drawback of > > having to reconfigure the capture pipeline due to a flip does not > > really apply here, actually, knowing the components ordering in the > > final image is very relevant for applications to know how to > > interpret the image data. > > > > TL;DR I'm not too concerned about having to propagate the components > > ordering along the pipeline, as the DRM formats are used only to > > describe the image data as presented to the applications and not > > along the pipeline as it happens for the V4L2 formats. > > > > Opinions anyone ? > > > > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > > - GRBG16_PISP_COMP1: > > > > > fourcc: DRM_FORMAT_SGRBG16 > > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > > - GBRG16_PISP_COMP1: > > > > > fourcc: DRM_FORMAT_SGBRG16 > > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > > - BGGR16_PISP_COMP1: > > > > > fourcc: DRM_FORMAT_SBGGR16 > > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > > - MONO_PISP_COMP1: > > > > > fourcc: DRM_FORMAT_R16 > > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > > > > > > > See > > > > > https://patchwork.libcamera.org/patch/19503/ > > > > > > > > > > Would if be acceptable for DRM to include the above proposed modifiers for the > > > > > purpose of defining the above presented libcamera formats ? There will be no > > > > > graphic format associated with these modifiers as their purpose it not > > > > > displaying images but rather exchange them between the components of the > > > > > camera subsystem (and possibly be inspected by specialized test applications). > > > > > > > > Yeah I think libcamera using drm-fourcc formats and modifiers is > > > > absolutely ok, and has my ack in principle. And for these users we're > > > > ok with merging modifiers that the kernel doesn't use. > > > > > > > > I think it would be really good to formalize this by adding libcamera > > > > to the officially listed users in the "Open Source User Waiver" > > > > section in the drm_fourcc.h docs: > > > > > > > > https://dri.freedesktop.org/docs/drm/gpu/drm-kms.html#open-source-user-waiver > > > > > > The waiver states "the usual requirement for an upstream in-kernel or > > > open source userspace user does not apply". Strictly speaking, there is > > > an open-source userspace user with libcamera. It's only on the kernel > > > side that the new modifier my not get used. So the usual requirement here means for merging a drm driver into the upstream drm subsystem. We might want to link to that section explicitly: https://dri.freedesktop.org/docs/drm/gpu/drm-uapi.html#open-source-userspace-requirements And yes this exception is for proprietary gl/vk implementations, which would not apply to upstream libcamera. But my understanding is that libcamera does allow for proprierty extensions (out-of-tree), so personally I fell like adding libcamera userspace to this exception list makes sense. I think allowing this would help to push libcamera towards an actual industry standard that hopefully everyone can and will use. But that's just my take, I'm happy to go with whatever libcamera people want here. > > > This being said, I'm fine clarifying the documentation, clarity is > > > always good. Absolutely :-) -Sima > > > > > > > You might want to convert that into a list, it could get a bit > > > > confusing. Then we can get that patch properly acked (by kernel and > > > > libcamera folks) to record the community consensus. > > > > > > > > For the rpi modifiers themselves: They need to be properly documented, > > > > least to exclude a screw-up like with the rpi modifiers we already > > > > have, which unfortunately encode the buffer height (instead of just > > > > the rounding algorithim to align the height to the right tile size) in > > > > the modifiers, which breaks assumptions everywhere. For details see > > > > https://gitlab.freedesktop.org/wlroots/wlroots/-/merge_requests/4529#note_2262057 > > > > > > > > > --- > > > > > include/uapi/drm/drm_fourcc.h | 5 +++++ > > > > > 1 file changed, 5 insertions(+) > > > > > > > > > > diff --git a/include/uapi/drm/drm_fourcc.h b/include/uapi/drm/drm_fourcc.h > > > > > index 00db00083175..09b182a959ad 100644 > > > > > --- a/include/uapi/drm/drm_fourcc.h > > > > > +++ b/include/uapi/drm/drm_fourcc.h > > > > > @@ -425,6 +425,7 @@ extern "C" { > > > > > #define DRM_FORMAT_MOD_VENDOR_ARM 0x08 > > > > > #define DRM_FORMAT_MOD_VENDOR_ALLWINNER 0x09 > > > > > #define DRM_FORMAT_MOD_VENDOR_AMLOGIC 0x0a > > > > > +#define DRM_FORMAT_MOD_VENDOR_RPI 0x0b > > > > > > > > > > /* add more to the end as needed */ > > > > > > > > > > @@ -1568,6 +1569,10 @@ drm_fourcc_canonicalize_nvidia_format_mod(__u64 modifier) > > > > > #define AMD_FMT_MOD_CLEAR(field) \ > > > > > (~((__u64)AMD_FMT_MOD_##field##_MASK << AMD_FMT_MOD_##field##_SHIFT)) > > > > > > > > > > +/* RPI (Raspberry Pi) modifiers */ > > > > > +#define PISP_FORMAT_MOD_COMPRESS_MODE1 fourcc_mod_code(RPI, 1) > > > > > +#define PISP_FORMAT_MOD_COMPRESS_MODE2 fourcc_mod_code(RPI, 2) > > > > > + > > > > > #if defined(__cplusplus) > > > > > } > > > > > #endif > > -- > Regards, > > Laurent Pinchart
On Thu, Feb 29, 2024 at 11:39:24AM +0100, Daniel Vetter wrote: > On Wed, Feb 28, 2024 at 01:13:45PM +0200, Laurent Pinchart wrote: > > On Wed, Feb 28, 2024 at 11:41:57AM +0100, Jacopo Mondi wrote: > > > On Tue, Feb 27, 2024 at 03:08:27PM +0200, Laurent Pinchart wrote: > > > > On Mon, Feb 26, 2024 at 04:46:24PM +0100, Daniel Vetter wrote: > > > > > On Mon, 26 Feb 2024 at 16:39, Jacopo Mondi wrote: > > > > > > > > > > > > Add modifiers for the Raspberry Pi PiSP compressed formats. > > > > > > > > > > > > The compressed formats are documented at: > > > > > > Documentation/userspace-api/media/v4l/pixfmt-pisp-comp-rggb.rst > > > > > > > > > > > > and in the PiSP datasheet: > > > > > > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > > > > > > > > > > > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com> > > > > > > --- > > > > > > > > > > > > Background: > > > > > > ----------- > > > > > > > > > > > > The Raspberry Pi PiSP camera subsystem is on its way to upstream through the > > > > > > Video4Linux2 subsystem: > > > > > > https://patchwork.linuxtv.org/project/linux-media/list/?series=12310 > > > > > > > > > > > > The PiSP camera system is composed by a "Front End" and a "Back End". > > > > > > The FrontEnd part is a MIPI CSI-2 receiver that store frames to memory and > > > > > > produce statistics, and the BackEnd is a memory-to-memory ISP that converts > > > > > > images in a format usable by application. > > > > > > > > > > > > The "FrontEnd" is capable of encoding RAW Bayer images as received by the > > > > > > image sensor in a 'compressed' format defined by Raspberry Pi and fully > > > > > > documented in the PiSP manual: > > > > > > https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf > > > > > > > > > > > > The compression scheme is documented in the in-review patch series for the BE > > > > > > support at: > > > > > > https://patchwork.linuxtv.org/project/linux-media/patch/20240223163012.300763-7-jacopo.mondi@ideasonboard.com/ > > > > > > > > > > > > The "BackEnd" is capable of consuming images in the compressed format and > > > > > > optionally user application might want to inspect those images for debugging > > > > > > purposes. > > > > > > > > > > > > Why a DRM modifier > > > > > > ------------------ > > > > > > > > > > > > The PiSP support is entirely implemented in libcamera, with the support of an > > > > > > hw-specific library called 'libpisp'. > > > > > > > > > > > > libcamera uses the fourcc codes defined by DRM to define its formats: > > > > > > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/formats.yaml > > > > > > > > > > > > And to define a new libcamera format for the Raspberry Pi compressed ones we > > > > > > need to associate the above proposed modifiers with a RAW Bayer format > > > > > > identifier. > > > > > > > > > > > > In example: > > > > > > > > > > > > - RGGB16_PISP_COMP1: > > > > > > fourcc: DRM_FORMAT_SRGGB16 > > > > > > > > An "interesting" issue here is that these formats currently live in > > > > libcamera only, we haven't merged them in DRM "yet". This may be a > > > > prerequisite ? > > > > > > > > > > Ah right! I didn't notice! > > > > > > I think there are two issues at play here, one to be clarified by the > > > DRM maintainers, the other more technically involved with the > > > definition of the Bayer formats themselves. > > > > > > - Does DRM want RAW Bayer formats to be listed here, as these are not > > > typically 'graphic' formats. What's the DRM maintainers opinion here ? > > > > To give some context, the "historical mistake" I keep referring to > > regarding V4L2 is the decision to combine the bit depth of raw formats > > with the colour filter array (a.k.a. Bayer) pattern into a fourcc. I > > think we should have defined raw pixel formats that only encode a bit > > depth, and conveyed the CFA pattern out-of-band. This is especially the > > case for media bus codes (formats on the buses between hardware > > devices). The reasoning here is that most devices only care about the > > bit depth and not the CFA pattern. Adding a new CFA pattern (for > > instance for RGB+Ir) for a camera sensor would currently require adding > > a new media bus code (easy), using it in the sensor driver (obvious), > > and patching *every* CSI-2 receiver driver to pass it through. Userspace > > would similarly need to grow support for the new format, even for > > userspace code that only cares about capturing the raw data without > > processing the colour components. Having raw pixel formats that don't > > convey the CFA pattern would simplify all this (and it's something I'm > > considering adding to the media bus codes). > > part is lots of churn > > So both drm fourcc and modifer formats are meant to be entirely opaque and > complete. Which means if you take them from one driver (whether userspace > or kernel driver shouldn't matter) and pass it to another, together with > the height, width (in pixels) and the pitch (in linearized bytes, people > got confused and wanted to do tile row pitch here and create and absolute > mess) must _fully_ describe the buffer for the other side. How full does fully mean ? :-) For instance, one thing that is not conveyed by the format today is colourspace information. It is communicated out-of-band, and if the source and sink have different configurations, you'll get sub-optimal results. Same thing for monochrome formats, R8 could store visible light data, infrared data or depth data. Granted, when it comes to the colour filter array pattern the "suboptimal" results will be quite significantly more suboptimal than what you'll get from mismatches in colourspaces. > Which would mean the bayer pattern would need to be in the modifier. Same > way we have BGRA and RGBA and all these swizzles of the same stuff. > > But also if these are purely internal to libcamera I guess we can make > exceptions, after all we do have _R8 and _RG8 formats too, and those have > nothing to do with red/green colors - it's simply historical naming for > the first/second color channel. > > > > - The RAW Bayer pattern ordering: we have realized that the idea of > > > defining RAW Bayer formats by encoding the components ordering in > > > V4L2 was, in retrospective, a bad idea. The reason is that any > > > change in the Bayer order along the capture pipeline (usually due to > > > flip) requires reconfiguration of the formats along the whole > > > pipeline, requiring a start/stop of the capture operations. > > > > That's also a reason, yes. > > > > > For this reason, we have multiple times discussed the idea of only > > > conveying the bit depth in the format definition and to communicate > > > the components ordering with other out-of-band mechanism, that will > > > likely be used to negotiate between the image sensor and the CSI-2 > > > receiver. > > > > > > With DRM the ideal would be something like > > > > > > DRM_FORMAT_RAW8 | FORMAT_MODIFIER_SBGGR > > > DRM_FORMAT_RAW8 | FORMAT_MODIFIER_SGBRG > > > ... > > > > > > DRM_FORMAT_RAW10 | FORMAT_MODIFIER_SBGGR > > > DRM_FORMAT_RAW10 | FORMAT_MODIFIER_SGBRG > > > ... > > > > When I said out-of-band, I meant really out of band from a format point > > of view, so neither in the 4CC nor in the modifiers. > > See above, but drm fourcc don't have flags. Yes we have the > big/little-endian flag, and it's absolute lolz and mostly doesn't work. > > Also _RAW8 exist already, it's R8, again see above for the historical > reasons why gl/vk call them like that. > > > > This however prevents to use additional modifiers to convey, in > > > example, packaging (ie MIPI CSI-2) or compressions (like done here > > > for PISP_COMP1) as, as far as I understand it, multiple modifiers > > > cannot be applied to a format. > > > > > > However, as I understand the argument of not repeating the V4L2 > > > mistakes here, the DRM formats are aimed to be used mostly to > > > represent the formats of images as they get stored in application > > > buffers (at least from a libcamera perspective). The drawback of > > > having to reconfigure the capture pipeline due to a flip does not > > > really apply here, actually, knowing the components ordering in the > > > final image is very relevant for applications to know how to > > > interpret the image data. > > > > > > TL;DR I'm not too concerned about having to propagate the components > > > ordering along the pipeline, as the DRM formats are used only to > > > describe the image data as presented to the applications and not > > > along the pipeline as it happens for the V4L2 formats. > > > > > > Opinions anyone ? > > > > > > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > > > - GRBG16_PISP_COMP1: > > > > > > fourcc: DRM_FORMAT_SGRBG16 > > > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > > > - GBRG16_PISP_COMP1: > > > > > > fourcc: DRM_FORMAT_SGBRG16 > > > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > > > - BGGR16_PISP_COMP1: > > > > > > fourcc: DRM_FORMAT_SBGGR16 > > > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > > > - MONO_PISP_COMP1: > > > > > > fourcc: DRM_FORMAT_R16 > > > > > > mod: PISP_FORMAT_MOD_COMPRESS_MODE1 > > > > > > > > > > > > See > > > > > > https://patchwork.libcamera.org/patch/19503/ > > > > > > > > > > > > Would if be acceptable for DRM to include the above proposed modifiers for the > > > > > > purpose of defining the above presented libcamera formats ? There will be no > > > > > > graphic format associated with these modifiers as their purpose it not > > > > > > displaying images but rather exchange them between the components of the > > > > > > camera subsystem (and possibly be inspected by specialized test applications). > > > > > > > > > > Yeah I think libcamera using drm-fourcc formats and modifiers is > > > > > absolutely ok, and has my ack in principle. And for these users we're > > > > > ok with merging modifiers that the kernel doesn't use. > > > > > > > > > > I think it would be really good to formalize this by adding libcamera > > > > > to the officially listed users in the "Open Source User Waiver" > > > > > section in the drm_fourcc.h docs: > > > > > > > > > > https://dri.freedesktop.org/docs/drm/gpu/drm-kms.html#open-source-user-waiver > > > > > > > > The waiver states "the usual requirement for an upstream in-kernel or > > > > open source userspace user does not apply". Strictly speaking, there is > > > > an open-source userspace user with libcamera. It's only on the kernel > > > > side that the new modifier my not get used. > > So the usual requirement here means for merging a drm driver into the > upstream drm subsystem. We might want to link to that section explicitly: > > https://dri.freedesktop.org/docs/drm/gpu/drm-uapi.html#open-source-userspace-requirements > > And yes this exception is for proprietary gl/vk implementations, which > would not apply to upstream libcamera. > > But my understanding is that libcamera does allow for proprierty > extensions (out-of-tree), For the sake of completeness, those extensions are not allowed to define new pixel formats at the moment. I don't envision changing that, but I can't fully rule it out either. > so personally I fell like adding libcamera > userspace to this exception list makes sense. I think allowing this would > help to push libcamera towards an actual industry standard that hopefully > everyone can and will use. > > But that's just my take, I'm happy to go with whatever libcamera people > want here. > > > > > This being said, I'm fine clarifying the documentation, clarity is > > > > always good. > > Absolutely :-) > > > > > > > > > > You might want to convert that into a list, it could get a bit > > > > > confusing. Then we can get that patch properly acked (by kernel and > > > > > libcamera folks) to record the community consensus. > > > > > > > > > > For the rpi modifiers themselves: They need to be properly documented, > > > > > least to exclude a screw-up like with the rpi modifiers we already > > > > > have, which unfortunately encode the buffer height (instead of just > > > > > the rounding algorithim to align the height to the right tile size) in > > > > > the modifiers, which breaks assumptions everywhere. For details see > > > > > https://gitlab.freedesktop.org/wlroots/wlroots/-/merge_requests/4529#note_2262057 > > > > > > > > > > > --- > > > > > > include/uapi/drm/drm_fourcc.h | 5 +++++ > > > > > > 1 file changed, 5 insertions(+) > > > > > > > > > > > > diff --git a/include/uapi/drm/drm_fourcc.h b/include/uapi/drm/drm_fourcc.h > > > > > > index 00db00083175..09b182a959ad 100644 > > > > > > --- a/include/uapi/drm/drm_fourcc.h > > > > > > +++ b/include/uapi/drm/drm_fourcc.h > > > > > > @@ -425,6 +425,7 @@ extern "C" { > > > > > > #define DRM_FORMAT_MOD_VENDOR_ARM 0x08 > > > > > > #define DRM_FORMAT_MOD_VENDOR_ALLWINNER 0x09 > > > > > > #define DRM_FORMAT_MOD_VENDOR_AMLOGIC 0x0a > > > > > > +#define DRM_FORMAT_MOD_VENDOR_RPI 0x0b > > > > > > > > > > > > /* add more to the end as needed */ > > > > > > > > > > > > @@ -1568,6 +1569,10 @@ drm_fourcc_canonicalize_nvidia_format_mod(__u64 modifier) > > > > > > #define AMD_FMT_MOD_CLEAR(field) \ > > > > > > (~((__u64)AMD_FMT_MOD_##field##_MASK << AMD_FMT_MOD_##field##_SHIFT)) > > > > > > > > > > > > +/* RPI (Raspberry Pi) modifiers */ > > > > > > +#define PISP_FORMAT_MOD_COMPRESS_MODE1 fourcc_mod_code(RPI, 1) > > > > > > +#define PISP_FORMAT_MOD_COMPRESS_MODE2 fourcc_mod_code(RPI, 2) > > > > > > + > > > > > > #if defined(__cplusplus) > > > > > > } > > > > > > #endif
diff --git a/include/uapi/drm/drm_fourcc.h b/include/uapi/drm/drm_fourcc.h index 00db00083175..09b182a959ad 100644 --- a/include/uapi/drm/drm_fourcc.h +++ b/include/uapi/drm/drm_fourcc.h @@ -425,6 +425,7 @@ extern "C" { #define DRM_FORMAT_MOD_VENDOR_ARM 0x08 #define DRM_FORMAT_MOD_VENDOR_ALLWINNER 0x09 #define DRM_FORMAT_MOD_VENDOR_AMLOGIC 0x0a +#define DRM_FORMAT_MOD_VENDOR_RPI 0x0b /* add more to the end as needed */ @@ -1568,6 +1569,10 @@ drm_fourcc_canonicalize_nvidia_format_mod(__u64 modifier) #define AMD_FMT_MOD_CLEAR(field) \ (~((__u64)AMD_FMT_MOD_##field##_MASK << AMD_FMT_MOD_##field##_SHIFT)) +/* RPI (Raspberry Pi) modifiers */ +#define PISP_FORMAT_MOD_COMPRESS_MODE1 fourcc_mod_code(RPI, 1) +#define PISP_FORMAT_MOD_COMPRESS_MODE2 fourcc_mod_code(RPI, 2) + #if defined(__cplusplus) } #endif
Add modifiers for the Raspberry Pi PiSP compressed formats. The compressed formats are documented at: Documentation/userspace-api/media/v4l/pixfmt-pisp-comp-rggb.rst and in the PiSP datasheet: https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com> --- Background: ----------- The Raspberry Pi PiSP camera subsystem is on its way to upstream through the Video4Linux2 subsystem: https://patchwork.linuxtv.org/project/linux-media/list/?series=12310 The PiSP camera system is composed by a "Front End" and a "Back End". The FrontEnd part is a MIPI CSI-2 receiver that store frames to memory and produce statistics, and the BackEnd is a memory-to-memory ISP that converts images in a format usable by application. The "FrontEnd" is capable of encoding RAW Bayer images as received by the image sensor in a 'compressed' format defined by Raspberry Pi and fully documented in the PiSP manual: https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf The compression scheme is documented in the in-review patch series for the BE support at: https://patchwork.linuxtv.org/project/linux-media/patch/20240223163012.300763-7-jacopo.mondi@ideasonboard.com/ The "BackEnd" is capable of consuming images in the compressed format and optionally user application might want to inspect those images for debugging purposes. Why a DRM modifier ------------------ The PiSP support is entirely implemented in libcamera, with the support of an hw-specific library called 'libpisp'. libcamera uses the fourcc codes defined by DRM to define its formats: https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/formats.yaml And to define a new libcamera format for the Raspberry Pi compressed ones we need to associate the above proposed modifiers with a RAW Bayer format identifier. In example: - RGGB16_PISP_COMP1: fourcc: DRM_FORMAT_SRGGB16 mod: PISP_FORMAT_MOD_COMPRESS_MODE1 - GRBG16_PISP_COMP1: fourcc: DRM_FORMAT_SGRBG16 mod: PISP_FORMAT_MOD_COMPRESS_MODE1 - GBRG16_PISP_COMP1: fourcc: DRM_FORMAT_SGBRG16 mod: PISP_FORMAT_MOD_COMPRESS_MODE1 - BGGR16_PISP_COMP1: fourcc: DRM_FORMAT_SBGGR16 mod: PISP_FORMAT_MOD_COMPRESS_MODE1 - MONO_PISP_COMP1: fourcc: DRM_FORMAT_R16 mod: PISP_FORMAT_MOD_COMPRESS_MODE1 See https://patchwork.libcamera.org/patch/19503/ Would if be acceptable for DRM to include the above proposed modifiers for the purpose of defining the above presented libcamera formats ? There will be no graphic format associated with these modifiers as their purpose it not displaying images but rather exchange them between the components of the camera subsystem (and possibly be inspected by specialized test applications). --- include/uapi/drm/drm_fourcc.h | 5 +++++ 1 file changed, 5 insertions(+) -- 2.43.0