[0/6] Raspberry Pi software camera sync algorithm

21511 mbox series

Hi everyone

This patch set adds a software camera sync algorithm to Raspberry Pi's
libcamera implementation. It is the result of work by our summer
intern Arsen, as well as by Naush and myself.

It consists of a server, which broadcasts timing messages sporadically
(such as once a second) on a socket, and then one or more clients that
listen out for these messages. The clients, which should be running at
the same (or a very close) framerate to the server, will perform a
one-off frame length adjustment in response to each message, causing
the frames to be synchronised with the server.

Although principally designed for identical cameras running on the
same Pi, the system can also work for different kinds of cameras on
different Pis on the same network. In the former case we'd be
expecting sub 100us latencies between frames, and in the latter
probably sub 1ms. On different Pis, of course, one is subject to the
vagaries of NTP (though you have the option to configure it better, or
indeed use something else).

There are 6 patches, consisting of:

* Some new controls.
* Plumbing in the pipeline handler to generate wall clock timestamps.
* Plumbing in the IPAs to support the new "SyncAlgorithm".
* A complete initial implementation of "rpi.sync".
* Updates to the camera tuning files to activate the algorithm.

We've had users wanting this feature for some time, and are keen to
roll it out. As such, perhaps I could encourage reviewers to look at
the new controls (the first patch) first. Once this is agreed, we can
roll the algorithm out without causing a subsequent API breakage for
our users if other details get changed (the implementation underneath,
in this respect, doesn't matter).

Up for discussion, I would imagine, might be what the new controls
mean and do, and indeed whether people would prefer them to be core
controls or not.

Hope that all makes sense. Thanks everyone!

Best regards
David

David Plowman (3):
  controls: rpi: Add controls for the camera sync algorithm
  ipa: rpi: sync: Add an implementation of the camera sync algorithm
  ipa: rpi: vc4: Update all tuning files for sync algorithm

Naushir Patuck (3):
  pipeline: rpi: Add queue of wallclock timestamps
  pipelien: rpi: vc4: Populate the wallclock timestamps queue
  ipa: rpi: Add base classes and plumbing for sync algorithm

 src/ipa/rpi/common/ipa_base.cpp               |  66 ++-
 src/ipa/rpi/common/ipa_base.h                 |   4 +-
 src/ipa/rpi/controller/meson.build            |   2 +
 src/ipa/rpi/controller/rpi/clock_recovery.cpp |  87 ++++
 src/ipa/rpi/controller/rpi/clock_recovery.h   |  55 +++
 src/ipa/rpi/controller/rpi/sync.cpp           | 384 ++++++++++++++++++
 src/ipa/rpi/controller/rpi/sync.h             |  71 ++++
 src/ipa/rpi/controller/sync_algorithm.h       |  30 ++
 src/ipa/rpi/controller/sync_status.h          |  27 ++
 src/ipa/rpi/vc4/data/imx219.json              |  11 +-
 src/ipa/rpi/vc4/data/imx219_noir.json         |  11 +-
 src/ipa/rpi/vc4/data/imx283.json              |  11 +-
 src/ipa/rpi/vc4/data/imx290.json              |  11 +-
 src/ipa/rpi/vc4/data/imx296.json              |  11 +-
 src/ipa/rpi/vc4/data/imx296_mono.json         |  11 +-
 src/ipa/rpi/vc4/data/imx378.json              |  11 +-
 src/ipa/rpi/vc4/data/imx477.json              |  11 +-
 src/ipa/rpi/vc4/data/imx477_noir.json         |  11 +-
 src/ipa/rpi/vc4/data/imx477_scientific.json   |  11 +-
 src/ipa/rpi/vc4/data/imx519.json              |  11 +-
 src/ipa/rpi/vc4/data/imx708.json              |  11 +-
 src/ipa/rpi/vc4/data/imx708_noir.json         |  11 +-
 src/ipa/rpi/vc4/data/imx708_wide.json         |  11 +-
 src/ipa/rpi/vc4/data/imx708_wide_noir.json    |  11 +-
 src/ipa/rpi/vc4/data/ov5647.json              |  11 +-
 src/ipa/rpi/vc4/data/ov5647_noir.json         |  11 +-
 src/ipa/rpi/vc4/data/se327m12.json            |  11 +-
 src/libcamera/control_ids_rpi.yaml            |  76 ++++
 .../pipeline/rpi/common/pipeline_base.cpp     |   9 +
 .../pipeline/rpi/common/pipeline_base.h       |   2 +
 src/libcamera/pipeline/rpi/vc4/vc4.cpp        |  12 +
 31 files changed, 1000 insertions(+), 23 deletions(-)
 create mode 100644 src/ipa/rpi/controller/rpi/clock_recovery.cpp
 create mode 100644 src/ipa/rpi/controller/rpi/clock_recovery.h
 create mode 100644 src/ipa/rpi/controller/rpi/sync.cpp
 create mode 100644 src/ipa/rpi/controller/rpi/sync.h
 create mode 100644 src/ipa/rpi/controller/sync_algorithm.h
 create mode 100644 src/ipa/rpi/controller/sync_status.h