[WIP,RFC,00/26] Support the new common raw sensor model

28570 mbox series

NOTE: This is a WIP series that doesn't compile for all platforms :-)

Hi all,

This series introduces support for the V4L2 Common Raw Sensor Model [1].
I'm posting this as an RFC/WIP for early feedback.

PATCH 1-6: Baseline, adding headers for the new model, fixing minor issues
PATCH 7-11: Updates to CameraSensorRaw and enabling the capability
	    bit for the new model
PATCH 12-17: Split out ColorFilterArray from BayerFormat, drop
	     bayer-specific PixelFormats, switch to generic ones
PATCH 18-26: Migrate three pipelines (vc4, pisp, mali-c55) and
	     an application (dng_writer) to the new architecture

There are of course many more pipeline handlers and applications that
can be converted, but each requires carefully going through it to do it
cleanly, so I've stopped at two for now.

I've also completely dropped the legacy bayer-order specific
PixelFormats, which would break all applications. This is only
temporary. Once the migration is complete, the plan is to add them back
as deprecated formats, so applications have some notice to switch to the
newer model.

This series depends on my earlier series for migrating the Raspberry Pi
4 pipeline and IPA to work with mainline kernel [2].

I have also prepared a kernel branch [3] and libcamera branch [4] you
can use for testing this.

The kernel branch has support for the raw sensor model, including
3 sensors (IMX678, IMX708, IMX219) that fully support the raw sensor
model, and updates to the raspberry pi drivers to work with the new
formats.

I've only tested this on Raspberry Pi 5. The Mali-C55 and dng writer
changes are only compile-tested.

Lastly, there are quite a few features of the new raw sensor model that
are not yet supported, like binning, skipping and new frame rate
controls. I'm still working on those :-)

[1]: https://lore.kernel.org/all/20260409201501.975242-1-sakari.ailus@linux.intel.com/
[2]: https://patchwork.libcamera.org/cover/28300/
[3]: https://gitlab.freedesktop.org/linux-media/users/jluthra/-/commits/mainline-metadata
[4]: https://gitlab.freedesktop.org/jluthra/libcamera/-/commits/mainline-metadata

Signed-off-by: Jai Luthra <jai.luthra@ideasonboard.com>
---
Jai Luthra (26):
      include: linux: Update uAPI headers to v7.3 + Common Raw Sensor Model
      libcamera: v4l2_subdevice: Add mediaBusInfo for generic raw formats
      libcamera: v4l2_pixelformat: Update to latest version of generic META formats
      pipeline: rpi: vc4: Map subdev metadata formats to V4L2 fourcc
      include: uapi: videodev2: Add formats for unpacked RAW
      include: uapi: videodev2: Add RAW device-specific bayer formats
      libcamera: Drop MediaBusFormatInfo::Type::EmbeddedData
      libcamera: camera_sensor_raw: Read the new CFA_PATTERN control
      libcamera: camera_sensor_raw: Enumerate formats using sink pads
      libcamera: camera_sensor_raw: Check for CONFIG_MODEL
      libcamera: Support raw sensor model
      libcamera: bayer_format: Introduce a ColorFilterArray class
      include: uapi: drm_fourcc: Add 14-bit R format
      libcamera: formats: Switch to generic layout-agnostic formats
      HACK: libcamera: formats: Add a modifier for RAW colorspace
      libcamera: v4l2_pixelformat: Add toNonGeneric() helper
      libcamera: bayer_format: Strip out ColorFilterArray from BayerFormat
      libcamera: stream: Add bayerOrder for raw formats
      libcamera: camera_sensor: Add helpers for isRaw() and isMono()
      pipeline: rpi: Use CameraSensor::isMono() helper
      pipeline: rpi: Split bayerOrder from PixelFormat
      pipeline: rpi: Prefer generic raw formats
      pipeline: rpi: pisp: Migrate to RAW formats
      pipeline: rpi: vc4: Migrate to generic RAW formats
      pipeline: mali-c55: Use generic RAW formats
      apps: dng_writer: Split cfa pattern handling

 include/libcamera/color_filter_array.h             |  58 +++
 include/libcamera/internal/bayer_format.h          |  18 +-
 include/libcamera/internal/camera_sensor.h         |   6 +-
 .../libcamera/internal/software_isp/swstats_cpu.h  |   2 +-
 include/libcamera/internal/v4l2_pixelformat.h      |   3 +
 include/libcamera/internal/v4l2_subdevice.h        |   2 +-
 include/libcamera/meson.build                      |   1 +
 include/libcamera/stream.h                         |  17 +
 include/linux/drm_fourcc.h                         |   7 +
 include/linux/media-bus-format.h                   |  12 +-
 include/linux/v4l2-controls.h                      |  58 ++-
 include/linux/v4l2-subdev.h                        |  15 +-
 include/linux/videodev2.h                          |  35 +-
 src/apps/common/dng_writer.cpp                     | 189 ++-------
 src/ipa/mali-c55/algorithms/agc.cpp                |  14 +-
 src/ipa/mali-c55/algorithms/agc.h                  |   5 +-
 src/ipa/mali-c55/ipa_context.h                     |   5 +-
 src/ipa/mali-c55/mali-c55.cpp                      |   4 +-
 src/libcamera/bayer_format.cpp                     | 303 ++++----------
 src/libcamera/color_filter_array.cpp               | 334 ++++++++++++++++
 src/libcamera/formats.cpp                          | 440 +++++----------------
 src/libcamera/formats.yaml                         | 149 ++-----
 src/libcamera/meson.build                          |   1 +
 src/libcamera/pipeline/mali-c55/mali-c55.cpp       | 165 ++++----
 .../pipeline/rpi/common/pipeline_base.cpp          |  38 +-
 src/libcamera/pipeline/rpi/pisp/pisp.cpp           |  86 ++--
 src/libcamera/pipeline/rpi/vc4/vc4.cpp             |  56 +--
 src/libcamera/pipeline/simple/simple.cpp           |   3 +-
 src/libcamera/sensor/camera_sensor_legacy.cpp      |  45 ++-
 src/libcamera/sensor/camera_sensor_raw.cpp         | 124 +++---
 src/libcamera/software_isp/debayer.cpp             |   8 +-
 src/libcamera/software_isp/debayer.h               |   2 +-
 src/libcamera/software_isp/debayer_cpu.cpp         |  58 +--
 src/libcamera/software_isp/debayer_cpu.h           |   2 +-
 src/libcamera/software_isp/debayer_egl.cpp         |   6 +-
 src/libcamera/software_isp/swstats_cpu.cpp         |  26 +-
 src/libcamera/v4l2_pixelformat.cpp                 | 221 +++++++++--
 src/libcamera/v4l2_subdevice.cpp                   | 203 ++++++++--
 test/bayer-format.cpp                              |  40 +-
 39 files changed, 1495 insertions(+), 1266 deletions(-)
---
base-commit: 01772f1c0c8a48a77dc99a47f5f50afa51c58f1f
change-id: 20261023-b4-raw-model-3ee1409b15c2

Best regards,