[RFC,0/3] Frame wallclock timestamps and metadata
mbox series

Message ID 20241126121706.4350-1-david.plowman@raspberrypi.com
Headers show
Series
  • Frame wallclock timestamps and metadata
Related show

Message

David Plowman Nov. 26, 2024, 12:17 p.m. UTC 
Hi everyone

This patch set is actually all about the camera sync patches that I
sent several weeks ago. One thing we said back then was that we'd try
and make some of this functionality more accessible to other libcamera
users. So this patch set (which would precede subsequent patches that
actually implement camera sync) moves the recovery of a de-jittered
"wallclock timestamp" into non-Raspberry Pi code. It feels like a
generally useful thing.

I've left it as an RFC for the minute because there are certainly a
few things to think about. Let me talk through the 3 patches.

Patch 1 just defines the new controls that will return the wallclock
timestamps. I've left it so that you get both a "raw" (un-smoothed)
wallclock and the smoothed version. Actually I think this probably
makes sense because how would a developer really understand whether
the de-jittering is working properly for them without the "raw" value.
(It's hard to promise at this point that it will work perfectly on
every camera on every platform forever...)

Patch 2 is the ClockRecovery class that implements fairly
simple-minded clock smoothing. It does have some configuration
parameters, and I do wonder whether the values here will suit
everyone, or whether pipeline handlers will want to customise them. I
think it's easy enough to allow that, but all the more reason to
report the "raw" wallclock as well so that developers can keep an eye
on what's really happening.

Patch 3 adds wallclock timestamps to the V4L2 devices, so that
everyone using V4L2 can access them easily. A slight quirk here is
that the frame start handling is under the control of the V4L2Device,
while the frame end (dequeueBuffer) happens through the
V4L2Videodevice. Another side effect is that you have to register for
frame start events to get the functionality. Perhaps folks can say
what they make of that.

Obviously I have a couple more patches that make the Raspberry Pi PH
output these new wallclock controls. Let me know if you'd like to try
them.

Thanks!
David


David Plowman (3):
  controls: Add FrameWallClock and FrameWallClockRaw controls
  libcamera: clock: Add ClockRecovery class to help generate wallclock
    timestamps
  libcamera: v4l2: Add wallclock metadata to video devices

 include/libcamera/framebuffer.h               |   2 +
 include/libcamera/internal/clock_recovery.h   |  64 ++++++++++
 include/libcamera/internal/meson.build        |   1 +
 include/libcamera/internal/v4l2_device.h      |   4 +
 include/libcamera/internal/v4l2_videodevice.h |   3 +
 src/libcamera/clock_recovery.cpp              | 110 ++++++++++++++++++
 src/libcamera/control_ids_core.yaml           |  29 +++++
 src/libcamera/meson.build                     |   1 +
 src/libcamera/v4l2_device.cpp                 |  22 ++++
 src/libcamera/v4l2_videodevice.cpp            |  29 +++++
 10 files changed, 265 insertions(+)
 create mode 100644 include/libcamera/internal/clock_recovery.h
 create mode 100644 src/libcamera/clock_recovery.cpp