| Message ID | 20261119-multicontext-prep-v1-0-f68ea7ef960f@ideasonboard.com |
|---|---|
| Headers | show
Return-Path: <libcamera-devel-bounces@lists.libcamera.org> X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D4BE9BDE17 for <parsemail@patchwork.libcamera.org>; Thu, 19 Nov 2026 16:39:57 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id EF3826A934; Thu, 19 Nov 2026 17:39:55 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="RykVzVMV"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B5D0F6A861 for <libcamera-devel@lists.libcamera.org>; Thu, 19 Nov 2026 17:39:53 +0100 (CET) Received: from [192.168.1.7] (unknown [93.65.100.155]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 3345F13D3; Thu, 19 Nov 2026 17:37:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1795106244; bh=m0XdFnCFkzauD04ML0lhs3NDLucyWOPkHn0XDJ+nXXY=; h=From:Subject:Date:To:Cc:From; b=RykVzVMVAA5Vks5Gg03I1XF9hKz1ttucdZ0MDHB9PLtGJ4Vi6eKM3EDOQT61ypQ0y Ww/ua3mMcBP7hsAsSaI9QVkxbXQmnpNe9G6Bt0vURBfsQrRq6g5dxG+qrozWSL2JRB gJaXesyyM3EvYHB8AQ42B4g7aVZLsGfezMZDrXsY= From: Jacopo Mondi <jacopo.mondi@ideasonboard.com> Subject: [PATCH 0/6] libcamera: Prepare to add MediaContext Date: Thu, 19 Nov 2026 17:39:42 +0100 Message-Id: <20261119-multicontext-prep-v1-0-f68ea7ef960f@ideasonboard.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit X-B4-Tracking: v=1; b=H4sIAAAAAAAC/yXM0QpAMBTG8VfRubbaphGvIhfMwRGjbaS0dzdc/ r76/jc4tIQOquQGiyc52kyESBPQU2tGZNRHg+QyF0KUbD0WT3ozHi/Pdos741mhpC6wV5xD/MV xoOtr1s1vd3Qzav+GIIQHGlJBs3UAAAA= X-Change-ID: 20261119-multicontext-prep-03752c7ed500 To: libcamera-devel@lists.libcamera.org Cc: Jacopo Mondi <jacopo.mondi@ideasonboard.com> X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=13990; i=jacopo.mondi@ideasonboard.com; h=from:subject:message-id; bh=m0XdFnCFkzauD04ML0lhs3NDLucyWOPkHn0XDJ+nXXY=; b=owEBbQKS/ZANAwAKAXI0Bo8WoVY8AcsmYgBq/yZXPh0Ln7Z/w4PWTpfBZP/jkFdTwepCNqnZy pL/qzmyW+OJAjMEAAEKAB0WIQS1xD1IgJogio9YOMByNAaPFqFWPAUCav8mVwAKCRByNAaPFqFW PIXQD/wK3hLsS7qIhHtsI20QFPkJzND7HDoE/7dj370TSqA0t0rRe2sVYAksI1sAjnNU/cGbJcz 7fnnNHIisQaEXzMpAwF8nelCylP0P26dt0+Zvkd4m0e3zvl9pEQ1KkIZNhOiV2Aj3kUkPKMuu9M SkCvRYtY7uWm+tdcmKa64OCzzZp0e4tuLJDMbFh1w0FfgDkRfgVXS8zsTRZ1V3cHUYgl+dzMlbg +jsm3N7SRXG7c//2z3WqdOGkxBw8G74LP99HSD9jHvEiOTBYPdxoVPKvYE05sSVXKWSTtowYE7U GOM7wr8dwZXoSJp34MBTT1la7B3egCAPCWnhnlhGd6oZ9aP3lX3ppWaRhkOqvQ86BzsBU5jve/a MK4CiWE6ir7Ufjm4nuXC1D2nKJrvxPMOf5VHHnBcgD3zh0NMx+U8xcr2migsE8y/R/9oaCi9gbe 0q5tokOH8U5gc9yijTQE/mDGl8RyTvzOYF7XwbT8DKKNxc7J89o3oj/V3L9eS2oTaraS++z2T3r bK3m0saBH8Smn4cvv1jj8Gd6865cIKWjKmJFga7CJWPOHqkZPxGuH1x6KZpHPrg/yarUcncj134 GXuNmC9EYDLTXEyu9WMEeCD9gp/h4x/VsJmv1dCY5riTGOFKQWon3uAihVa+AUm0l/5QtxbwDQT Dr9dfyYM7Y0S+lw== X-Developer-Key: i=jacopo.mondi@ideasonboard.com; a=openpgp; fpr=72392EDC88144A65C701EA9BA5826A2587AD026B X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: <libcamera-devel.lists.libcamera.org> List-Unsubscribe: <https://lists.libcamera.org/options/libcamera-devel>, <mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe> List-Archive: <https://lists.libcamera.org/pipermail/libcamera-devel/> List-Post: <mailto:libcamera-devel@lists.libcamera.org> List-Help: <mailto:libcamera-devel-request@lists.libcamera.org?subject=help> List-Subscribe: <https://lists.libcamera.org/listinfo/libcamera-devel>, <mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe> Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" <libcamera-devel-bounces@lists.libcamera.org> |
| Series |
|
| Related |
show
|
This series prepares the code base for the introduction of a new type name MediaContext that will help with the support of multi-context time-multiplexed ISP devices. This first part addresses one fundamental design goal: de-coupling the MediaDevice and its users from the MediaObject class hierarchy. This serves the purpose of moving MediaObjects to the MediaContext class, so that MediaObjects are only accessible to PipelineHandlers after they have acquired a Context. If this confuses you, don't worry. I'll paste here below the design document I'm trying to keep up-to-date that should explain a bit of the rationale behind all of this (at the very end). Tested on uvcvideo, vimc, pisp and rppx1. CI pipeline: https://gitlab.freedesktop.org/jmondi/libcamera/-/pipelines/1798086 ------------------------------------------------------------------------------- This section provides an introduction on the current (upstream) design and usage of the MediaDevice class. The media device class serves mostly at two purposes: - Enumerate the entities, pads and link in a media graph - Enable/disable links The users of MediaDevice are: - DeviceEnumerator: - enumerate entities to perform a `match()` - Pipeline Handlers: - to discover specific entities (CAMERA_SENSOR) and create cameras - to enable/disable links MediaDevices are created by DeviceEnumerator and populated at creation time. A MediaDevice is: - "acquired" by a pipeline handler during a `match()` using `PipelineHandler::acquireMediaDevice` - Which calls `MediaDevice::acquire()` - causes the MediaDevice to be `open()` - "locked" through an flock when a Camera is acquired using `PipelineHandler::acquire()` - Which calls `MediaDevice::lock()` - causes the MediaDevice to be flock-ed Acquiring a media device at match() time marks it as acquired and prevent further instances of the PipelineHandler class to match() once a media device has been acquired Locking a MediaDevice with a file lock prevents other applications in the system to use the same media graph The topology of a MediaDevice is stored as a list of MediaObject instances. One of the design goals of the MediaObject class hierarchy (MediaEntity, MediaPad and MediaLnks) is the ability to access pads from entities, and links from pads without going through a MediaDevice. MediaLink enablement can be performed without knowing what MediaDevice a link belongs to ``` class MediaLink : public MediaObject { public: ... int setEnabled(bool enable); ... ``` To do so, every MediaObject retains a reference to the MediaDevice that created it. MediaLink are the only users of that referece, in the implementation of `setEnabled()` : ``` int MediaLink::setEnabled(bool enable) { unsigned int flags = (flags_ & ~MEDIA_LNK_FL_ENABLED) | (enable ? MEDIA_LNK_FL_ENABLED : 0); int ret = dev_->setupLink(this, flags); ``` The MediaLink class delegates to the MediaDevice class the task of actually performing the `ioctl` on the fd_ ``` int MediaDevice::setupLink(const MediaLink *link, unsigned int flags) { ... int ret = ioctl(fd_.get(), MEDIA_IOC_SETUP_LINK, &linkDesc); ... } ``` The userspace API for multi-context are quite simple: - A media device context is created everytime a media device is open - Video devices and sub-devices are bound to a media device context with `VIDIOC_BIND_CONTEXT/VIDIOC_SUBDEV_BIND_CONTEXT` using the media device file descriptor as the context identifier ``` int media_fd = open(/dev/media0, ...); int video_fd = open(/dev/video0, ...); int sd_fd = open(/dev/v4l-subdev0, ...) struct v4l2_context video_context = { .context_fd = media_fd, }; struct v4l2_subdev_context sd_context = { .context_fd = media_fd, }; ioctl(video_fd, VIDIOC_BIND_CONTEXT, &media_context); ioctl(sd_fd, VIDIOC_SUBDEV_BIND_CONTEXT, &sd_context); ``` The "binding" isolates all the operations on the video device/sub-device in a media device context. All ioctls performed on `video_fd` and `sd_fd` are isolated from all other operations performed on different file descriptors obtained by re-opening the video device/sub-device. The current implementation of the multi-context framework in Linux abstracts the ioctl operations for format configuration and buffer handling, but we want more: - Link enablement status should be per-context - v4l2-controls should be per-context - Support for per-context Requests The MediaDevice class and the MediaObject class hierarchy should be re-structured to allow: - The creation of multiple context by allowing opening a media device multiple times - Acquiring a shared media graph can be performed multiple times - Each time `MediaDevice::acquire()` is performed during the pipeline handler `match()`, the media device is opened again, creating a new context - flock for a shared media device doesn't make much sense - The "context" on which libcamera operates will be isolated from the ones created by other applications in the system It is required to redirect the ioctls performed on a media device (only `MEDIA_IOC_SETUP_LINK` and `MEDIA_IOC_REQUEST_ALLOC` at the moment) on the `context` the pipeline handler is operating on. Currently the media device is only opened once, and all the ioctls are directed to the file descriptor created at `aquire()` time. Once we'll allow to open a media device at every new `acquire()` each pipeline handler will operate on a different context, with its own file descriptor. A new class is introduced: `MediaContext` It will represent a "mutable" `MediaDevice` which will allow pipeline handlers to: - Bind video device/sub-devices to it - Iterate the list of entities and MediaObjects like the MediaDevice class did - Perform ioctls (LINK_SETUP and REQUEST_ALLOC) on the open file descriptor In order to allow pipeline handlers to operate on new contexts, the prototype of the `PipelineHandler::acquire()` function needs to change: ``` 43 ⋮ │- std::shared_ptr<MediaDevice> 44 ⋮ │- acquireMediaContext(DeviceEnumerator *enumerator, 45 ⋮ │- const DeviceMatch &dm); ⋮ 43 │+ MediaContext *acquireMediaDevice(DeviceEnumerator *enumerator, ⋮ 44 │+ const DeviceMatch &dm); ``` The `MediaDevice::acquire()` operation will now create a new MediaContext and this will be given to pipeline handlers to operate on. This design applies to legacy (non-multicontext) MediaDevices as well. They will only be able to create one MediaContext. In order to implement proper support for both context and non-context media devices the MediaDevice class performs access arbitration: - Sharabled (== multicontext) media devices are identified at creation time through the `MEDIA_DEVICE_FL_CONTEXT` flag. - If a MediaDevice is sharable, then it gets opened once and a MediaContext is created with the newly obtained file descriptor - If a MediaDevice is not sharable, a single MediaContext can be created A similar reasoning goes for locking. At `MediaDevice::lock()` time, if `MEDIA_DEVICE_FL_CONTEXT` is _not_ set then a file lock is created, otherwise it is skipped `MediaDevice` instances are created in the DeviceEnumertor and held as `shared_ptr<>` in a class member vector. `std::vector<std::shared_ptr<MediaDevice>> devices_;` All the other users of the class receive a new `shared_ptr<>` to guarantee the MediaDevice stays alive until there are users owning a reference to it. Pipeline handlers are a good example of this pattern: ``` std::shared_ptr<MediaDevice> PipelineHandler::acquireMediaDevice(DeviceEnumerator *enumerator, const DeviceMatch &dm) { std::shared_ptr<MediaDevice> media = enumerator->search(dm); if (!media) return nullptr; if (!media->acquire()) return nullptr; mediaDevices_.push_back(media); return media; } ``` The newly introduced `MediaContext` is created from a `MediaDevice` and requires that the `MediaDevice` lifetime is tied to its own (iow _the MediaDevice lives as long as it has Contexts referencing it_). For this reason the `MediaContext` will retain a `shared_ptr<>` reference to the MediaDevice that created it. The ownership of the `MediaContext` is passed to the PipelineHandler base class, which will maintain it alive as long as they need. To implement this 1) Make MediaDevice inherit `enable_shared_from_this` ``` +class MediaDevice : public std::enable_shared_from_this<MediaDevice>, ``` To make sure this is safe the MediaDevice constructor will be made private and a new ``` static std::shared_ptr<MediaDevice> MediaDevice::create() ``` function needs to be added 2) Pass to the newly created `MediaContext` a reference to the `MediaDevice` by increasing the `shared_ptr<>` usage count using `shared_from_this()` ``` MediaContext c(shared_from_this(), std::move(fd)) ``` 3) Create a MediaContext as a `unique_ptr<>` to pass its ownership to pipeline handlers ``` ⋮ 745│+std::unique_ptr<MediaContext> MediaDevice::acquire() ⋮ 746│+{ ... ⋮ 749│+ ⋮ 750│+ UniqueFD fd = open(); ⋮ 751│+ if (!fd.isValid()) ⋮ 752│+ return false; ⋮ 753│+ ⋮ 754│+ acquired_ = true; ⋮ 755│+ return std::make_unique<MediaContext>(shared_from_this(), std::move(fd)); ⋮ 756│+} ``` This ties the MediaDevice lifetime to the one of its contexts and gives to pipelines ownership of the `MediaContext` they acquired. The newly created MediaContext is stored in the PipelineHandler base class and given to PipelineHandlers as a raw pointer (or reference). In the PipelineHandler base class we can now replace mediaDevices_ with mediaContexts_. So far, it's all good. `MediaContext` implements the same enumeration capabilities (entities(), getEntityByName(), link() etc) of the existing `MediaDevice` class. The `MediaDevice` class now solely collectes the static information of the media device (`driver`, `hwVersion` etc) and does not open the media devnode anymore after the initialization. Simply `s/MediaDevice/MediaContext` in all pipeline handlers, bind() where you want to support multi-context and you're done. There are two main problems still: - Entities and MediaObject enumeration Both `DeviceEnumerator` and `PipelineHandlers` need to access entities and the MediaObject list. - `DeviceEnumerator` operates on a MediaDevice before any context is created - `PipelineHandler` operates on a MediaContext after the MediaDevice has been acquired The media topology needs to be available both from `MediaDevice` and from `MediaContext` - MediaLink handling As said above, all MediaObjects are created with a reference to the media device which created them. The action of creating the list of MediaObjects in a MediaDevice is named `populate()` and is performed by the `DeviceEnumerator` when the MediaDevice is first created. The topology need to be immediately available for `DeviceEnumerator` to perform a match With this design, all MediaObject will retain a reference to a MediaDevice, and this is a problem for MediaLink enablement as the `MediaDevice` class now doesn't have access to an open file descriptor anymore. As long as we don't have media link state in a context on the kernel side this is possibily fine, but once support for media link state handling is supported in the context framework, we'll need to associate a MediaLink to a MediaContext AND not to a MediaDevice. The only solution that might help is to re-enumerate all entities at `MediaContext` creation time and inject the `MediaContext` as the device on which the link enablement is performed. To solve both problems, the MediaObjects creation need to be delayed and be performed by `MediaContext`. This requires to remove all usages of `MediaEntities` before a `MediaContext` is acquired. To do so the DeviceEnumerator match() logic and the devnode path injection need to be reworked to remove dependencies on `MediaEntities`. ------------------------------------------------------------------------------- Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com> --- Jacopo Mondi (6): libcamera: Move DeviceMatch::match() to MediaDevice libcamera: media_device: Align the two loops in match() libcamera: media_device: Store the media v2 topology libcamera: media_device: Store entities ids by devnum libcamera: device_enumerator: Use devnum to identify entities libcamera: media_device: Delay topology creation at acquire() include/libcamera/internal/device_enumerator.h | 10 +- .../libcamera/internal/device_enumerator_udev.h | 3 +- include/libcamera/internal/media_device.h | 74 ++++++++ src/libcamera/device_enumerator.cpp | 89 +++------ src/libcamera/device_enumerator_sysfs.cpp | 15 +- src/libcamera/device_enumerator_udev.cpp | 29 ++- src/libcamera/media_device.cpp | 199 +++++++++++++++++---- test/camera-sensor.cpp | 8 + test/delayed_controls.cpp | 12 ++ test/libtest/buffer_source.cpp | 6 + test/v4l2_subdevice/v4l2_subdevice_test.cpp | 7 + test/v4l2_videodevice/v4l2_m2mdevice.cpp | 8 + 12 files changed, 328 insertions(+), 132 deletions(-) --- base-commit: 5efdc682e5b54f7b9f793b1fbf99884d522fd9f9 change-id: 20261119-multicontext-prep-03752c7ed500 Best regards, -- Jacopo Mondi <jacopo.mondi@ideasonboard.com>