[1/1] controls: Add camera synchronisation controls

22258 diff mbox series

New controls are added to control the camera "sync" algorithm, which
allows different cameras to synchronise their frames.

Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
---
 src/libcamera/control_ids_core.yaml | 104 ++++++++++++++++++++++++++++
 1 file changed, 104 insertions(+)

Replying to my own patch...

On Mon, 9 Dec 2024 at 16:27, David Plowman
<david.plowman@raspberrypi.com> wrote:
>
> New controls are added to control the camera "sync" algorithm, which
> allows different cameras to synchronise their frames.
>
> Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
> ---
>  src/libcamera/control_ids_core.yaml | 104 ++++++++++++++++++++++++++++
>  1 file changed, 104 insertions(+)
>
> diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml
> index d45cf8e5..6d45b757 100644
> --- a/src/libcamera/control_ids_core.yaml
> +++ b/src/libcamera/control_ids_core.yaml
> @@ -971,4 +971,108 @@ controls:
>        description: |
>          Enable or disable the debug metadata.
>
> +  - SyncMode:
> +      type: int32_t
> +      description: |
> +        Enable or disable camera synchronisation ("sync") mode.
> +
> +        When sync mode is enabled, a camera will synchronise frames temporally
> +        with other cameras, either attached to the same device or a different
> +        one. There should be one "server" device, which broadcasts timing
> +        information to one or more "clients".
> +
> +        Sync mode requires all cameras to be running at (as far as possible) the
> +        same fixed framerate, and may continue to make adjustments to keep the
> +        cameras together for the duration of the session.
> +
> +      enum:
> +        - name: SyncModeOff
> +          value: 0
> +          description: Disable sync mode.
> +        - name: SyncModeServer
> +          value: 1
> +          description: |
> +            Enable sync mode, act as server. The server broadcasts timing
> +            messages to any clients that are listening, so that the clients can
> +            synchronise their camera frames with the server's.
> +        - name: SyncModeClient
> +          value: 2
> +          description: |
> +            Enable sync mode, act as client. A client listens for any server
> +            messages, and arranges for its camera frames to synchronise as
> +            closely as possible with the server's. Many clients can listen out
> +            for the same server. Clients can also be started ahead of any
> +            servers, causing them merely to wait for the server to start.
> +
> +        \sa SyncReady
> +        \sa SyncTimer
> +        \sa SyncFrames

These three "\sa" lines need to go above the "enum:", of course. Will
fix that in v2.

Thanks, David!  :)

> +
> +  - SyncReady:
> +      type: bool
> +      description: |
> +        When using the camera synchronisation algorithm, the server broadcasts
> +        timing information to the clients. This also includes the time (some
> +        number of frames in the future, called the "ready time") at which the
> +        server will signal its controlling application, using this control, to
> +        start using the image frames.
> +
> +        The client receives the "ready time" from the server, and will signal
> +        its application to start using the frames at this same moment.
> +
> +        While this control value is false, applications (on both client and
> +        server) should continue to wait, and not use the frames.
> +
> +        Once this value becomes true, it means that this is the first frame
> +        where client and server have agreed that they will both be synchronised
> +        and that applications should begin consuming frames.
> +
> +        \sa SyncMode
> +        \sa SyncTImer
> +        \sa SyncFrames
> +
> +  - SyncTimer:
> +      type: int64_t
> +      description: |
> +        This reports the amount of time, in microseconds, until the "ready
> +        time", at which the server and client will signal their controlling
> +        applications that the frames are now synchronised and should be
> +        used. The value may be refined slightly over time, becoming more precise
> +        as the "ready time" approaches.
> +
> +        Servers always report this value, whereas clients will omit this control
> +        until they have received a message from the server that enables them to
> +        calculate it.
> +
> +        Normally the value will start positive (the "ready time" is in the
> +        future), and decrease towards zero, before becoming negative (the "ready
> +        time" has elapsed). So there should be just one frame where the timer
> +        value is, or is very close to, zero - the one for which the SyncReady
> +        control becomes true. At this moment, the value indicates how closely
> +        synchronised the client believes it is with the server.
> +
> +        But note that if frames are being dropped, then the "near zero" valued
> +        frame, or indeed any other, could be skipped. In these cases the timer
> +        value allows an application to deduce that this has happened.
> +
> +        \sa SyncMode
> +        \sa SyncReady
> +        \sa SyncFrames
> +
> +  - SyncFrames:
> +      type: int32_t
> +      description: |
> +        The number of frames the server should wait, after enabling
> +        SyncModeServer, before signalling (via the SyncReady control) that
> +        frames should be used. This therefore determines the "ready time" for
> +        all synchronised cameras.
> +
> +        This control value should be set only for the device that is to act as
> +        the server, before or at the same moment that SyncModeServer is
> +        enabled.
> +
> +        \sa SyncMode
> +        \sa SyncReady
> +        \sa SyncTimer
> +
>  ...
> --
> 2.39.5
>

Hi David,

On Mon, 9 Dec 2024 at 16:27, David Plowman
<david.plowman@raspberrypi.com> wrote:
>
> New controls are added to control the camera "sync" algorithm, which
> allows different cameras to synchronise their frames.
>
> Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
> ---
>  src/libcamera/control_ids_core.yaml | 104 ++++++++++++++++++++++++++++
>  1 file changed, 104 insertions(+)
>
> diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml
> index d45cf8e5..6d45b757 100644
> --- a/src/libcamera/control_ids_core.yaml
> +++ b/src/libcamera/control_ids_core.yaml
> @@ -971,4 +971,108 @@ controls:
>        description: |
>          Enable or disable the debug metadata.
>
> +  - SyncMode:
> +      type: int32_t
> +      description: |
> +        Enable or disable camera synchronisation ("sync") mode.
> +
> +        When sync mode is enabled, a camera will synchronise frames temporally
> +        with other cameras, either attached to the same device or a different
> +        one. There should be one "server" device, which broadcasts timing
> +        information to one or more "clients".
> +
> +        Sync mode requires all cameras to be running at (as far as possible) the
> +        same fixed framerate, and may continue to make adjustments to keep the
> +        cameras together for the duration of the session.

Perhaps worth noting that only the "client" will make adjustments in
the above paragraph?

> +
> +      enum:
> +        - name: SyncModeOff
> +          value: 0
> +          description: Disable sync mode.
> +        - name: SyncModeServer
> +          value: 1
> +          description: |
> +            Enable sync mode, act as server. The server broadcasts timing
> +            messages to any clients that are listening, so that the clients can
> +            synchronise their camera frames with the server's.
> +        - name: SyncModeClient
> +          value: 2
> +          description: |
> +            Enable sync mode, act as client. A client listens for any server
> +            messages, and arranges for its camera frames to synchronise as
> +            closely as possible with the server's. Many clients can listen out
> +            for the same server. Clients can also be started ahead of any
> +            servers, causing them merely to wait for the server to start.
> +
> +        \sa SyncReady
> +        \sa SyncTimer
> +        \sa SyncFrames
> +
> +  - SyncReady:
> +      type: bool
> +      description: |
> +        When using the camera synchronisation algorithm, the server broadcasts
> +        timing information to the clients. This also includes the time (some
> +        number of frames in the future, called the "ready time") at which the
> +        server will signal its controlling application, using this control, to
> +        start using the image frames.
> +
> +        The client receives the "ready time" from the server, and will signal
> +        its application to start using the frames at this same moment.
> +
> +        While this control value is false, applications (on both client and
> +        server) should continue to wait, and not use the frames.
> +
> +        Once this value becomes true, it means that this is the first frame
> +        where client and server have agreed that they will both be synchronised

s/client/clients/

Perhaps we can clarify if SyncReady remains latched to true after
ready time or if it's a single-shot?

With these minor edits:

Reviewed-by: Naushir Patuck <naush@raspberrypi.com>

> +        and that applications should begin consuming frames.
> +
> +        \sa SyncMode
> +        \sa SyncTImer
> +        \sa SyncFrames
> +
> +  - SyncTimer:
> +      type: int64_t
> +      description: |
> +        This reports the amount of time, in microseconds, until the "ready
> +        time", at which the server and client will signal their controlling
> +        applications that the frames are now synchronised and should be
> +        used. The value may be refined slightly over time, becoming more precise
> +        as the "ready time" approaches.
> +
> +        Servers always report this value, whereas clients will omit this control
> +        until they have received a message from the server that enables them to
> +        calculate it.
> +
> +        Normally the value will start positive (the "ready time" is in the
> +        future), and decrease towards zero, before becoming negative (the "ready
> +        time" has elapsed). So there should be just one frame where the timer
> +        value is, or is very close to, zero - the one for which the SyncReady
> +        control becomes true. At this moment, the value indicates how closely
> +        synchronised the client believes it is with the server.
> +
> +        But note that if frames are being dropped, then the "near zero" valued
> +        frame, or indeed any other, could be skipped. In these cases the timer
> +        value allows an application to deduce that this has happened.
> +
> +        \sa SyncMode
> +        \sa SyncReady
> +        \sa SyncFrames
> +
> +  - SyncFrames:
> +      type: int32_t
> +      description: |
> +        The number of frames the server should wait, after enabling
> +        SyncModeServer, before signalling (via the SyncReady control) that
> +        frames should be used. This therefore determines the "ready time" for
> +        all synchronised cameras.
> +
> +        This control value should be set only for the device that is to act as
> +        the server, before or at the same moment that SyncModeServer is
> +        enabled.
> +
> +        \sa SyncMode
> +        \sa SyncReady
> +        \sa SyncTimer
> +
>  ...
> --
> 2.39.5
>