[0/6] libcamera: Prepare to add MediaContext
mbox series

Message ID 20261119-multicontext-prep-v1-0-f68ea7ef960f@ideasonboard.com
Headers show
Series
  • libcamera: Prepare to add MediaContext
Related show

Message

Jacopo Mondi Nov. 19, 2026, 4:39 p.m. UTC
This series prepares the code base for the introduction of a new type
name MediaContext that will help with the support of multi-context
time-multiplexed ISP devices.

This first part addresses one fundamental design goal: de-coupling the
MediaDevice and its users from the MediaObject class hierarchy. This
serves the purpose of moving MediaObjects to the MediaContext class,
so that MediaObjects are only accessible to PipelineHandlers after they
have acquired a Context.

If this confuses you, don't worry. I'll paste here below the design
document I'm trying to keep up-to-date that should explain a bit of the
rationale behind all of this (at the very end).

Tested on uvcvideo, vimc, pisp and rppx1.

CI pipeline:
https://gitlab.freedesktop.org/jmondi/libcamera/-/pipelines/1798086

-------------------------------------------------------------------------------

This section provides an introduction on the current (upstream) design
and usage of the MediaDevice class.

The media device class serves mostly at two purposes:

- Enumerate the entities, pads and link in a media graph
- Enable/disable links

The users of MediaDevice are:

- DeviceEnumerator:
  - enumerate entities to perform a `match()`

- Pipeline Handlers:
  - to discover specific entities (CAMERA_SENSOR) and create cameras
  - to enable/disable links

MediaDevices are created by DeviceEnumerator and populated at creation
time.

A MediaDevice is:

- "acquired" by a pipeline handler during a `match()` using
  `PipelineHandler::acquireMediaDevice`
  - Which calls `MediaDevice::acquire()`
    - causes the MediaDevice to be `open()`

- "locked" through an flock when a Camera is acquired using
  `PipelineHandler::acquire()`
  - Which calls `MediaDevice::lock()`
    - causes the MediaDevice to be flock-ed

Acquiring a media device at match() time marks it as acquired and
prevent further instances of the PipelineHandler class to match() once a
media device has been acquired

Locking a MediaDevice with a file lock prevents other applications in
the system to use the same media graph

The topology of a MediaDevice is stored as a list of MediaObject
instances. One of the design goals of the MediaObject class hierarchy
(MediaEntity, MediaPad and MediaLnks) is the ability to access pads from
entities, and links from pads without going through a MediaDevice.

MediaLink enablement can be performed without knowing what MediaDevice a
link belongs to

```
class MediaLink : public MediaObject
{
 public:
	...
	int setEnabled(bool enable);
	...

```

To do so, every MediaObject retains a reference to the MediaDevice that
created it. MediaLink are the only users of that referece, in the
implementation of `setEnabled()` :

```
int MediaLink::setEnabled(bool enable)
{
	 unsigned int flags = (flags_ & ~MEDIA_LNK_FL_ENABLED) | (enable ? MEDIA_LNK_FL_ENABLED : 0);

	int ret = dev_->setupLink(this, flags);
 ```

The MediaLink class delegates to the MediaDevice class the task of
actually performing the `ioctl` on the fd_

```
int MediaDevice::setupLink(const MediaLink *link, unsigned int	flags) {

	 ...

	int ret = ioctl(fd_.get(), MEDIA_IOC_SETUP_LINK, &linkDesc);
	...
}
```

The userspace API for multi-context are quite simple:

- A media device context is created everytime a media device is open
- Video devices and sub-devices are bound to a media device context with
  `VIDIOC_BIND_CONTEXT/VIDIOC_SUBDEV_BIND_CONTEXT` using the media
device file descriptor as the context identifier

```

  int media_fd = open(/dev/media0, ...);
  int video_fd = open(/dev/video0, ...);
  int sd_fd = open(/dev/v4l-subdev0, ...)

  struct v4l2_context video_context = { .context_fd = media_fd, };

  struct v4l2_subdev_context sd_context = { .context_fd = media_fd, };

  ioctl(video_fd, VIDIOC_BIND_CONTEXT, &media_context);
  ioctl(sd_fd, VIDIOC_SUBDEV_BIND_CONTEXT, &sd_context);
```

The "binding" isolates all the operations on the video device/sub-device
in a media device context. All ioctls performed on `video_fd` and
`sd_fd` are isolated from all other operations performed on different
file descriptors obtained by re-opening the video device/sub-device.

The current implementation of the multi-context framework in Linux
abstracts the ioctl operations for format configuration and buffer
handling, but we want more:

- Link enablement status should be per-context
- v4l2-controls should be per-context
- Support for per-context Requests

The MediaDevice class and the MediaObject class hierarchy should be
re-structured to allow:

- The creation of multiple context by allowing opening a media device
  multiple times
  - Acquiring a shared media graph can be performed multiple times
    - Each time `MediaDevice::acquire()` is performed during the
      pipeline handler `match()`, the media device is opened again,
creating a new context
  - flock for a shared media device doesn't make much sense
    - The "context" on which libcamera operates will be isolated from
      the ones created by other applications in the system

It is required to redirect the ioctls performed on a media device (only
`MEDIA_IOC_SETUP_LINK` and `MEDIA_IOC_REQUEST_ALLOC` at the moment) on
the `context` the pipeline handler is operating on. Currently the media
device is only opened once, and all the ioctls are directed to the file
descriptor created at `aquire()` time. Once we'll allow to open a media
device at every new `acquire()` each pipeline handler will operate on a
different context, with its own file descriptor.

A new class is introduced: `MediaContext`

It will represent a "mutable" `MediaDevice` which will allow pipeline
handlers to:
- Bind video device/sub-devices to it
- Iterate the list of entities and MediaObjects like the MediaDevice
  class did
- Perform ioctls (LINK_SETUP and REQUEST_ALLOC) on the open file
  descriptor

In order to allow pipeline handlers to operate on new contexts, the
prototype of the `PipelineHandler::acquire()` function needs to change:

```
 43 ⋮    │-        std::shared_ptr<MediaDevice>
 44 ⋮    │-                acquireMediaContext(DeviceEnumerator *enumerator,
 45 ⋮    │-                                    const DeviceMatch &dm);
    ⋮ 43 │+        MediaContext *acquireMediaDevice(DeviceEnumerator *enumerator,
    ⋮ 44 │+                                         const DeviceMatch &dm);
```

The `MediaDevice::acquire()` operation will now create a new
MediaContext and this will be given to pipeline handlers to operate on.

This design applies to legacy (non-multicontext) MediaDevices as well.
They will only be able to create one MediaContext.

In order to implement proper support for both context and non-context
media devices the MediaDevice class performs access arbitration:

- Sharabled (== multicontext) media devices are identified at creation
  time through the `MEDIA_DEVICE_FL_CONTEXT` flag.
- If a MediaDevice is sharable, then it gets opened once and a
  MediaContext is created with the newly obtained file descriptor
- If a MediaDevice is not sharable, a single MediaContext can be created

A similar reasoning goes for locking. At `MediaDevice::lock()` time, if
`MEDIA_DEVICE_FL_CONTEXT` is _not_ set then a file lock is created,
otherwise it is skipped

`MediaDevice` instances are created in the DeviceEnumertor and held as
`shared_ptr<>` in a class member vector.

`std::vector<std::shared_ptr<MediaDevice>> devices_;`

All the other users of the class receive a new `shared_ptr<>` to
guarantee the MediaDevice stays alive until there are users owning a
reference to it. Pipeline handlers are a good example of this pattern:

```
std::shared_ptr<MediaDevice>
PipelineHandler::acquireMediaDevice(DeviceEnumerator *enumerator,
				    const DeviceMatch &dm)
{
	std::shared_ptr<MediaDevice> media = enumerator->search(dm);
	if (!media)
		return nullptr;

	if (!media->acquire())
		return nullptr;

	mediaDevices_.push_back(media);

	return media;
}
```

The newly introduced `MediaContext` is created from a `MediaDevice` and
requires that the `MediaDevice` lifetime is tied to its own (iow _the
MediaDevice lives as long as it has Contexts referencing it_). For this
reason the `MediaContext` will retain a `shared_ptr<>` reference to the
MediaDevice that created it.

The ownership of the `MediaContext` is passed to the PipelineHandler
base class, which will maintain it alive as long as they need.

To implement this

1) Make MediaDevice inherit `enable_shared_from_this` ``` +class
   MediaDevice : public std::enable_shared_from_this<MediaDevice>, ```

To make sure this is safe the MediaDevice constructor will be made
private and a new ``` static std::shared_ptr<MediaDevice>
MediaDevice::create() ```

function needs to be added

2) Pass to the newly created `MediaContext` a reference to the
   `MediaDevice` by increasing the `shared_ptr<>` usage count using
`shared_from_this()`

``` MediaContext c(shared_from_this(), std::move(fd)) ```

3) Create a MediaContext as a `unique_ptr<>` to pass its ownership to
   pipeline handlers

```
    ⋮ 745│+std::unique_ptr<MediaContext> MediaDevice::acquire()
    ⋮ 746│+{

    ...

    ⋮ 749│+
    ⋮ 750│+        UniqueFD fd = open();
    ⋮ 751│+        if (!fd.isValid())
    ⋮ 752│+                return false;
    ⋮ 753│+
    ⋮ 754│+        acquired_ = true;
    ⋮ 755│+        return std::make_unique<MediaContext>(shared_from_this(), std::move(fd));
    ⋮ 756│+}
```

This ties the MediaDevice lifetime to the one of its contexts and gives
to pipelines ownership of the `MediaContext` they acquired.

The newly created MediaContext is stored in the PipelineHandler base
class and given to PipelineHandlers as a raw pointer (or reference).

In the PipelineHandler base class we can now replace mediaDevices_ with
mediaContexts_.

So far, it's all good.

`MediaContext` implements the same enumeration capabilities (entities(),
getEntityByName(), link() etc) of the existing `MediaDevice` class. The
`MediaDevice` class now solely collectes the static information of the
media device (`driver`, `hwVersion` etc) and does not open the media
devnode anymore after the initialization.

Simply `s/MediaDevice/MediaContext` in all pipeline handlers, bind()
where you want to support multi-context and you're done.

There are two main problems still:

- Entities and MediaObject enumeration

Both `DeviceEnumerator` and `PipelineHandlers` need to access entities
and the MediaObject list.

- `DeviceEnumerator` operates on a MediaDevice before any context is
  created
- `PipelineHandler` operates on a MediaContext after the MediaDevice has
  been acquired

The media topology needs to be available both from `MediaDevice` and
from `MediaContext`

- MediaLink handling

As said above, all MediaObjects are created with a reference to the
media device which created them. The action of creating the list of
MediaObjects in a MediaDevice is named `populate()` and is performed by
the `DeviceEnumerator` when the MediaDevice is first created.

The topology need to be immediately available for `DeviceEnumerator` to
perform a match

With this design, all MediaObject will retain a reference to a
MediaDevice, and this is a problem for MediaLink enablement as the
`MediaDevice` class now doesn't have access to an open file descriptor
anymore. As long as we don't have media link state in a context on the
kernel side this is possibily fine, but once support for media link
state handling is supported in the context framework, we'll need to
associate a MediaLink to a MediaContext AND not to a MediaDevice.

The only solution that might help is to re-enumerate all entities at
`MediaContext` creation time and inject the `MediaContext` as the device
on which the link enablement is performed.

To solve both problems, the MediaObjects creation need to be delayed and
be performed by `MediaContext`. This requires to remove all usages of
`MediaEntities` before a `MediaContext` is acquired.

To do so the DeviceEnumerator match() logic and the devnode path
injection need to be reworked to remove dependencies on `MediaEntities`.
-------------------------------------------------------------------------------

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
Jacopo Mondi (6):
      libcamera: Move DeviceMatch::match() to MediaDevice
      libcamera: media_device: Align the two loops in match()
      libcamera: media_device: Store the media v2 topology
      libcamera: media_device: Store entities ids by devnum
      libcamera: device_enumerator: Use devnum to identify entities
      libcamera: media_device: Delay topology creation at acquire()

 include/libcamera/internal/device_enumerator.h     |  10 +-
 .../libcamera/internal/device_enumerator_udev.h    |   3 +-
 include/libcamera/internal/media_device.h          |  74 ++++++++
 src/libcamera/device_enumerator.cpp                |  89 +++------
 src/libcamera/device_enumerator_sysfs.cpp          |  15 +-
 src/libcamera/device_enumerator_udev.cpp           |  29 ++-
 src/libcamera/media_device.cpp                     | 199 +++++++++++++++++----
 test/camera-sensor.cpp                             |   8 +
 test/delayed_controls.cpp                          |  12 ++
 test/libtest/buffer_source.cpp                     |   6 +
 test/v4l2_subdevice/v4l2_subdevice_test.cpp        |   7 +
 test/v4l2_videodevice/v4l2_m2mdevice.cpp           |   8 +
 12 files changed, 328 insertions(+), 132 deletions(-)
---
base-commit: 5efdc682e5b54f7b9f793b1fbf99884d522fd9f9
change-id: 20261119-multicontext-prep-03752c7ed500

Best regards,
--  
Jacopo Mondi <jacopo.mondi@ideasonboard.com>