From patchwork Fri Mar 1 21:20:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Pinchart X-Patchwork-Id: 19591 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 26BE0BD160 for ; Fri, 1 Mar 2024 21:21:22 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5F7506286F; Fri, 1 Mar 2024 22:21:21 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="hR6cdFwC"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 564536285F for ; Fri, 1 Mar 2024 22:21:19 +0100 (CET) Received: from pendragon.ideasonboard.com (89-27-53-110.bb.dnainternet.fi [89.27.53.110]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id ADC2C673; Fri, 1 Mar 2024 22:21:04 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1709328064; bh=csbfQ56eQ4d4ma8G6b14BNrZXULzSzSc15AvxwypVCo=; h=From:To:Cc:Subject:Date:From; b=hR6cdFwCpL8w30nLzNvC3XhYfcaOGRvXNqyQuIlwdGiLFtLL6o29HgikV7W+o7akM FwoqUNS68qBqGfEvLWWtL6yMoSoP9d8rYPBc1gKQKOSTI2f6xh3E4lSnwWfpCJlXD/ Ezww4uNZE2Bu1mIXRUT91grFnb4k+09ZNi0VhX90= From: Laurent Pinchart To: libcamera-devel@lists.libcamera.org Subject: [PATCH/RFC 00/32] libcamera: Support the upstream Unicam driver Date: Fri, 1 Mar 2024 23:20:49 +0200 Message-ID: <20240301212121.9072-1-laurent.pinchart@ideasonboard.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sakari Ailus Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Hello everybody, This large(-ish) patch series is the first complete implementation of support for the Unicam upstream kernel driver for libcamera. Or, it would be, if the Unicam driver was upstream. While I have also completed a version of the Unicam driver for the upstream kernel, there's a bit of a chicken-and-egg issue here as I would like to cross-reference the cover letters. To break the loop, I'm posting this series first, and will reply with a link to the Unicam patches ([1]). Support for Raspberry Pi 4 in libcamera currently relies on downstream drivers that live in the Raspberry Pi kernel tree, for Unicam (the CSI-2 receiver) and the ISP. The Unicam driver, in particular, includes a few workarounds to implement support for sensor embedded data, to compensate for features that are or were until recently missing in the V4L2 in-kernel and userspace APIs. The new Unicam submission for the kernel reworks the driver extensively to use the recently merged V4L2 streams API, as well as the under development V4L2 generic metadata and internal pads APIs ([2]). The series starts with nine patches that clean up, rework and improve the V4L2Subdevice class (01/32 to 09/32). They have been posted to the list before, and mostly reviewed. Patches 10/32 is where the fun starts, as it pulls the APIs from [2] in the kernel headers. The next three patches, 11/32 to 13/32, then update the V4L2Subdevice and V4L2VideoDevice classes to support those APIs. The next ten patches rework the CameraSensor class to support sensor drivers that use the new APIs. Patches 14/32 to to 21/32 contain various reworks, including moving the camera sensor support to the new src/libcamera/sensor/ directory, and turning CameraSensor into an abstract base class with multiple implementations, using a factory pattern to pick the right match. Patches 22/32 introduces a new CameraSensor subclass for the new APIs, and patch 23/32 then extends the CameraSensor interface to support embedded data. With all the building blocks in place, the next seven patches focus on the Raspberry Pi VC4 pipeline handler to use the new APIs, when interacting with the sensor (24/32), or with the mainline Unicam driver (27/32 to 30/32). Once done, patch 31/32 drops the V4L2_META_FMT_SENSOR_DATA format from the videodev2.h kernel header, as the format is not used anymore and has never been present in the upstream kernel. The last patch, 32/32, is a hack that unconditionally enables embedded data for the IMX219 sensor, as that is the platform I've been using to develop and test this series. Worthy of a note, this series breaks compatibility with the downstream Raspberry Pi kernel. I have decided not to preserve backward compatibiliy to ease development, and to make the new code easier to review. We still need to discuss how to integrate the various components (libcamera and kernel space) in a way that will not generate lots of anger among users. This may lead to some temporary changes to the code to help with the transition. For convenience, the patches can be found at [3]. [1] TBD due to lack of a time machine [2] https://lore.kernel.org/linux-media/20231106122539.1268265-1-sakari.ailus@linux.intel.com [3] https://git.libcamera.org/libcamera/pinchartl/libcamera.git/log/?h=rpi/streams/next Jacopo Mondi (1): libcamera: camera_sensor: Introduce CameraSensorFactory Laurent Pinchart (31): libcamera: v4l2_subdevice: Rename V4L2SubdeviceFormatInfo libcamera: v4l2_subdevice: Add code member to MediaBusFormatInfo libcamera: v4l2_subdevice: Expose media bus format info as internal API libcamera: v4l2_subdevice: Extend MediaBusFormatInfo with metadata formats libcamera: v4l2_subdevice: Drop V4L2SubdeviceFormat::bitsPerPixel() libcamera: v4l2_subdevice: Rename V4L2SubdeviceFormat::mbus_code to code libcamera: v4l2_subdevice: Add stream support to get/set functions libcamera: v4l2_subdevice: Replace Routing::toString() with operator<<() libcamera: v4l2_subdevice: Add V4L2Subdevice::Route structure [DNI] include: linux: Update kernel headers to metadata API libcamera: v4l2_subdevice: Update to the new kernel routing API libcamera: v4l2_subdevice: Add new metadata formats libcamera: v4l2_videodevice: Update to the new kernel metadata API libcamera: camera_sensor: Move related classes to subdirectory libcamera: camera_sensor: Drop updateControlInfo() function libcamera: camera_sensor: Reorder functions libcamera: camera_sensor: Test for read-only HBLANK with READ_ONLY flag libcamera: camera_sensor: Expose the Bayer order libcamera: camera_sensor: Create abstract base class libcamera: camera_sensor: Sort factories by priority libcamera: Add CameraSensor implementation for raw V4L2 sensors libcamera: camera_sensor: Add support for embedded data pipeline: raspberrypi: common: Configure sensor embedded data pipeline: raspberrypi: vc4: Use operator<<(V4L2VideoFormat) pipeline: raspberrypi: vc4: Reorganize platformConfigure() pipeline: raspberrypi: vc4: Use the CameraSensor embedded data API pipeline: raspberrypi: vc4: Unconditionally create embedded data stream pipeline: raspberrypi: vc4: Configure format on Unicam subdev pipeline: raspberrypi: vc4: Fix configuration of the embedded data node include: linux: Drop V4L2_META_FMT_SENSOR_DATA [HACK]: ipa: rpi: cam_helper_imx219: Enable embedded data Documentation/Doxyfile.in | 2 + include/libcamera/internal/camera_sensor.h | 162 +-- include/libcamera/internal/v4l2_subdevice.h | 103 +- include/linux/README | 2 +- include/linux/media-bus-format.h | 13 + include/linux/media.h | 1 + include/linux/v4l2-controls.h | 16 +- include/linux/v4l2-mediabus.h | 18 +- include/linux/v4l2-subdev.h | 23 +- include/linux/videodev2.h | 30 +- src/ipa/rpi/cam_helper/cam_helper_imx219.cpp | 2 +- src/libcamera/meson.build | 3 +- src/libcamera/pipeline/imx8-isi/imx8-isi.cpp | 19 +- src/libcamera/pipeline/ipu3/cio2.cpp | 15 +- src/libcamera/pipeline/ipu3/imgu.cpp | 4 +- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 9 +- src/libcamera/pipeline/rkisp1/rkisp1_path.cpp | 2 +- .../pipeline/rpi/common/pipeline_base.cpp | 85 +- .../pipeline/rpi/common/pipeline_base.h | 6 +- src/libcamera/pipeline/rpi/vc4/vc4.cpp | 174 ++- src/libcamera/pipeline/simple/simple.cpp | 29 +- src/libcamera/pipeline/vimc/vimc.cpp | 11 +- src/libcamera/sensor/camera_sensor.cpp | 557 ++++++++ .../camera_sensor_legacy.cpp} | 717 ++++------ .../{ => sensor}/camera_sensor_properties.cpp | 0 src/libcamera/sensor/camera_sensor_raw.cpp | 1135 +++++++++++++++ src/libcamera/sensor/meson.build | 8 + src/libcamera/v4l2_subdevice.cpp | 1239 ++++++++++++++--- src/libcamera/v4l2_videodevice.cpp | 31 +- test/camera-sensor.cpp | 9 +- .../v4l2_videodevice_test.cpp | 7 +- test/v4l2_videodevice/v4l2_videodevice_test.h | 2 +- 32 files changed, 3471 insertions(+), 963 deletions(-) create mode 100644 src/libcamera/sensor/camera_sensor.cpp rename src/libcamera/{camera_sensor.cpp => sensor/camera_sensor_legacy.cpp} (60%) rename src/libcamera/{ => sensor}/camera_sensor_properties.cpp (100%) create mode 100644 src/libcamera/sensor/camera_sensor_raw.cpp create mode 100644 src/libcamera/sensor/meson.build base-commit: c64446c226d4e629884d2f5b148a01969e8ee84a