[v2,0/4] Add Sync Layer

26049 mbox series

This series depends on:
- v6 of "Add Layers support" [0]
(and optionally "libcamera: layer_manager: Add support for global
configuration" [1])

There is a branch available at [2]. There is another version of the
branch available at [3], which is the same branch rebased on top of
Raspberry Pi's next [4] branch with an additional patch [5] on top to
make Raspberry Pi use the Sync Layer instead of RPiSync.

This patch series adds a layer implementation of the Sync algorithm,
where any number of cameras can synchronize the timing of their
captures. This allows any camera to use the Sync algorithm, as long as
it implements the SyncAdjustment control and SensorTimestamp.

I tested this on a pi5 (with an imx219 and imx708) and a Debix SOM (with
an imx335 and imx283) with the following capture scripts, running the
client first before running the server in separate terminals:

$ cat cscript-sync-client.yaml 
frames:
  - 1:
      FrameDurationLimits: [33333, 33333]
      SyncMode: 2

$ cat cscript-sync-server.yaml 
frames:
  - 1:
      FrameDurationLimits: [33333, 33333]
      SyncMode: 1

The below information is also available in the PR at [6] for ease of
access.

I ran:

- pi5 with pure rpi/next (RPiSync) [7]
- pi5 with this branch (epaul/dev/sync on top of rpi/next) (SyncLayer) [8]
- som with pure epaul/dev/sync (SyncLayer) [9]

And I logged the correction time over a single run each of about 10
minutes at 30fps (aiming for 800 corrections).

Here are the descriptive statistics for the correction values (all in
microseconds):

- pi5 with RPiSync
    Min. 1st Qu. Median Mean 3rd Qu. Max.
    238.0 247.0 252.0 251.9 257.0 263.0

- pi5 with SyncLayer
    Min. 1st Qu. Median Mean 3rd Qu. Max.
    239.0 247.0 252.0 251.9 257.0 263.0

- som with SyncLayer
    Min. 1st Qu. Median Mean 3rd Qu. Max.
    -16661 1273 7705 4386 11616 16643

I have graphs of these runs here [10].

The som could use some optimization, but we can see that on the pi5 the
correction times between RPiSync and SyncLayer are nearly identical.

[0] https://patchwork.libcamera.org/project/libcamera/list/?series=5753
[1] https://patchwork.libcamera.org/project/libcamera/list/?series=5758
[2] https://git.ideasonboard.com/epaul/libcamera/src/branch/epaul/dev/sync
[3] https://git.ideasonboard.com/epaul/libcamera/src/branch/epaul/dev/sync-pi5
[4] https://github.com/raspberrypi/libcamera/tree/next
[5] https://git.ideasonboard.com/epaul/libcamera/commit/4c480e1ec43ec3d698a19f444c8994f03b3ed5d6
[6] https://git.ideasonboard.com/epaul/libcamera/pulls/2
[7] https://git.ideasonboard.com/epaul/libcamera/commit/f0e40f1c
[8] https://git.ideasonboard.com/epaul/libcamera/commit/4c480e1e
[9] https://git.ideasonboard.com/epaul/libcamera/commit/0427d359
[10] https://git.ideasonboard.com/epaul/libcamera/pulls/2#issuecomment-26463

Paul Elder (4):
  controls: Add SyncAdjustment and SyncInterface controls
  layer: Add layer that implements the sync algorithm
  libipa: Add SyncHelper
  ipa: rkisp1: agc: Add support for sync

 src/ipa/libipa/meson.build          |   2 +
 src/ipa/libipa/sync_helper.cpp      |  88 ++++++
 src/ipa/libipa/sync_helper.h        |  43 +++
 src/ipa/rkisp1/algorithms/agc.cpp   |  22 +-
 src/ipa/rkisp1/algorithms/agc.h     |   3 +
 src/ipa/rkisp1/ipa_context.h        |   1 +
 src/layer/meson.build               |   1 +
 src/layer/sync/meson.build          |  15 +
 src/layer/sync/sync.cpp             | 463 ++++++++++++++++++++++++++++
 src/layer/sync/sync.h               |  98 ++++++
 src/libcamera/control_ids_core.yaml |  23 ++
 11 files changed, 758 insertions(+), 1 deletion(-)
 create mode 100644 src/ipa/libipa/sync_helper.cpp
 create mode 100644 src/ipa/libipa/sync_helper.h
 create mode 100644 src/layer/sync/meson.build
 create mode 100644 src/layer/sync/sync.cpp
 create mode 100644 src/layer/sync/sync.h