From patchwork Fri Jan 30 08:09:32 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 26050 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 23B8CC3226 for ; Fri, 30 Jan 2026 08:09:53 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D2DA061FC6; Fri, 30 Jan 2026 09:09:52 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="dW8lDmH+"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1B80661FC6 for ; Fri, 30 Jan 2026 09:09:51 +0100 (CET) Received: from neptunite.hamster-moth.ts.net (unknown [IPv6:2404:7a81:160:2100:ec11:5e0c:deb8:1e2d]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 92369122A; Fri, 30 Jan 2026 09:09:10 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1769760552; bh=Tj7WSS2gUoMIzfcyQsxZhS7kG2I6xoxUFF6NwosZ5e8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dW8lDmH+d9Vz08JpJZF3sbSaSAHK289ZOKBjvDUCHtuCOciWxePJBdT1Cn6D41Et3 oDzwfwP4xg5jNBxmI5INmJecLCmV/R7S/n/qOqOGx8E4i6F2pCs2IhbYbAvtdWtbxo m1h9uDz7DPcpdCWjYXHRaQXcwlrCrjyaJwld5/YM= From: Paul Elder To: libcamera-devel@lists.libcamera.org Cc: Paul Elder , david.plowman@raspberrypi.com, naush@raspberrypi.com, kieran.bingham@ideasonboard.com, stefan.klug@ideasonboard.com, Isaac Scott Subject: [PATCH v2 1/4] controls: Add SyncAdjustment and SyncInterface controls Date: Fri, 30 Jan 2026 17:09:32 +0900 Message-ID: <20260130080935.2569621-2-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20260130080935.2569621-1-paul.elder@ideasonboard.com> References: <20260130080935.2569621-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a couple of controls related to the sync algorithm. The SyncAdjustment control signals to the Camera the amount to adjust the frame duration by for a single frame. Any Camera that implements this control (and FrameDurationLimits and SensorTimestamp metadata) will be supported by the sync algorithm layer. The SyncInterface control signals to the sync layer which network interface to use for listening (on client mode) or for multicasting (on server mode). Given that enabling/disabling sync is done via control (as opposed to always being on via environment variable), choosing the network interface should be chosen in a similar manner. Signed-off-by: Paul Elder Reviewed-by: Isaac Scott --- No change in v2 --- src/libcamera/control_ids_core.yaml | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml index f26a1b1b08bd..2f0eea24850a 100644 --- a/src/libcamera/control_ids_core.yaml +++ b/src/libcamera/control_ids_core.yaml @@ -1361,4 +1361,27 @@ controls: description: | Enable or disable the lens shading correction. + - SyncAdjustment: + type: int32_t + direction: inout + description: | + This is an adjustment value that can be either positive or negative, in + microseconds, that tells the Camera how much to adjust the frame + duration by. This is meant to be used by the Sync layer, and any camera + that implements this control (and FrameDurationLimits) will be + supported by the sync layer. When reported in metadata, this reports + any frame duration adjustment that occurred based on the control value + that was passed in. + + - SyncInterface: + type: string + direction: inout + description: | + This is the IP address of the network interface to use for the Sync + algorithm provided by the Sync layer. When set using the SyncMode + control to client mode, this will be the network interface to listen + on, and for server mode it will multicast from this network interface. + If no network interface is supplied by this control then INADDR_ANY + will be used. + ... From patchwork Fri Jan 30 08:09:33 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 26051 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id BBD91C3226 for ; Fri, 30 Jan 2026 08:09:55 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 7400061FD2; Fri, 30 Jan 2026 09:09:55 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="AbJ8xZB7"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D2F6261FA0 for ; Fri, 30 Jan 2026 09:09:53 +0100 (CET) Received: from neptunite.hamster-moth.ts.net (unknown [IPv6:2404:7a81:160:2100:ec11:5e0c:deb8:1e2d]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 8F1891E23; Fri, 30 Jan 2026 09:09:13 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1769760555; bh=SD6awa9bfRediz1L9MVK8p6/dNUu0WrkLN6isbS636I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AbJ8xZB7SFU9HPpEh5vS9XJDw5COonHP/m+EpU79d6lKZaisvxuLMvjWzYy3gLhGN lKKwJ64vZL6usUCPcJ6SJsslbuSd+eNf9XRBaGgKNX2OA2w1oGLF73BAm0CIzDH63L aUEUL37T6oX3Mzdfxo2YNVHu0ImlXL8OSfVd6b6Y= From: Paul Elder To: libcamera-devel@lists.libcamera.org Cc: Paul Elder , david.plowman@raspberrypi.com, naush@raspberrypi.com, kieran.bingham@ideasonboard.com, stefan.klug@ideasonboard.com Subject: [PATCH v2 2/4] layer: Add layer that implements the sync algorithm Date: Fri, 30 Jan 2026 17:09:33 +0900 Message-ID: <20260130080935.2569621-3-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20260130080935.2569621-1-paul.elder@ideasonboard.com> References: <20260130080935.2569621-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Implement the sync layer, which implements the sync algorithm. Any Camera that supports the SyncAdjustment and FrameDurationLimits controls, and that reports the SensorTimestamp metadata will automatically be supported by this layer. This code is heavily based on Raspberry Pi's sync algorithm implementation, from the following files in commit [1]: - src/ipa/rpi/controller/sync_algorithm.h - src/ipa/rpi/controller/sync_status.h - src/ipa/rpi/controller/rpi/sync.cpp - src/ipa/rpi/controller/rpi/sync.h [1] https://github.com/raspberrypi/libcamera/commit/d1a712060dcb0aab8564e0d1d86efe9ffcfee6b9 Signed-off-by: Paul Elder --- Changes in v2: - use UniqueFD for socket - remove setting nonblock from fcntl() to socket() - add cloexec --- src/layer/meson.build | 1 + src/layer/sync/meson.build | 15 ++ src/layer/sync/sync.cpp | 463 +++++++++++++++++++++++++++++++++++++ src/layer/sync/sync.h | 98 ++++++++ 4 files changed, 577 insertions(+) create mode 100644 src/layer/sync/meson.build create mode 100644 src/layer/sync/sync.cpp create mode 100644 src/layer/sync/sync.h diff --git a/src/layer/meson.build b/src/layer/meson.build index 3d8b70ad2cd2..24012b239eb0 100644 --- a/src/layer/meson.build +++ b/src/layer/meson.build @@ -14,3 +14,4 @@ layers_env.set('LIBCAMERA_LAYER_PATH', meson.current_build_dir()) meson.add_devenv(layers_env) subdir('inject_controls') +subdir('sync') diff --git a/src/layer/sync/meson.build b/src/layer/sync/meson.build new file mode 100644 index 000000000000..acee5bef7aba --- /dev/null +++ b/src/layer/sync/meson.build @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: CC0-1.0 + +layer_name = 'sync' + +sync_sources = files([ + 'sync.cpp', +]) + +mod = shared_module(layer_name, [sync_sources, libcamera_internal_headers], + name_prefix : '', + include_directories : layer_includes, + dependencies : libcamera_private, + gnu_symbol_visibility: 'hidden', + install : true, + install_dir : layer_install_dir) diff --git a/src/layer/sync/sync.cpp b/src/layer/sync/sync.cpp new file mode 100644 index 000000000000..7e19ab72c043 --- /dev/null +++ b/src/layer/sync/sync.cpp @@ -0,0 +1,463 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * Copyright (C) 2025, Ideas On Board Oy + * + * Layer implementation for sync algorithm + */ + +#include "sync.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include +#include + +namespace libcamera { + +LOG_DEFINE_CATEGORY(SyncLayer) + +} /* namespace libcamera */ + +using namespace libcamera; +using namespace std::chrono_literals; + +void *init([[maybe_unused]] const std::string &id) +{ + SyncLayerData *data = new SyncLayerData; + + LOG(SyncLayer, Info) << "Initializing sync layer"; + + const char *kDefaultGroup = "239.255.255.250"; + constexpr unsigned int kDefaultPort = 10000; + constexpr unsigned int kDefaultSyncPeriod = 30; + constexpr unsigned int kDefaultReadyFrame = 100; + constexpr unsigned int kDefaultMinAdjustment = 50; + + /* \todo load these from configuration file */ + data->group = kDefaultGroup; + data->port = kDefaultPort; + data->syncPeriod = kDefaultSyncPeriod; + data->readyFrame = kDefaultReadyFrame; + data->minAdjustment = kDefaultMinAdjustment; + + return data; +} + +void terminate(void *closure) +{ + SyncLayerData *data = static_cast(closure); + + data->socket.reset(); + + delete data; +} + +ControlList requestCompleted(void *closure, Request *request) +{ + SyncLayerData *data = static_cast(closure); + + const ControlList &metadata = request->metadata(); + ControlList ret; + + /* SensorTimestamp is required for sync */ + /* \todo Document this requirement (along with SyncAdjustment) */ + auto sensorTimestamp = metadata.get(controls::SensorTimestamp); + if (sensorTimestamp) { + data->clockRecovery.addSample(); + uint64_t frameWallClock = data->clockRecovery.getOutput(*sensorTimestamp); + ret.set(controls::FrameWallClock, static_cast(frameWallClock)); + if (data->mode != Mode::Off && data->frameDuration) + processFrame(data, frameWallClock, ret); + } + + return ret; +} + +ControlInfoMap::Map updateControls(void *closure, ControlInfoMap &controls) +{ + SyncLayerData *data = static_cast(closure); + + /* + * If the SyncAdjustment control is unavailable then the Camera does + * not support Sync adjustment + */ + auto it = controls.find(&controls::SyncAdjustment); + data->syncAvailable = it != controls.end(); + if (!data->syncAvailable) { + LOG(SyncLayer, Warning) + << "Sync layer is not supported: SyncAdjustment control is not available"; + return {}; + } + + /* + * Save the default FrameDurationLimits. If it's not available then we + * have to wait until one is supplied in a request. We cannot use the + * FrameDuration returned from the first frame as the + * FrameDurationLimits has to be explicitly set by the application, as + * this is the frame rate target that the cameras will sync to. + * \todo Document that FrameDurationLimits is a required control + */ + it = controls.find(&controls::FrameDurationLimits); + if (it != controls.end()) + data->frameDuration = std::chrono::microseconds(it->second.min().get()); + + return { + { &controls::rpi::SyncMode, + ControlInfo(controls::rpi::SyncModeValues) }, + { &controls::rpi::SyncFrames, + ControlInfo(1, 1000000, 100) }, + { &controls::SyncInterface, + ControlInfo(1, 40) } + }; +} + +void queueRequest(void *closure, Request *request) +{ + SyncLayerData *data = static_cast(closure); + if (!data->syncAvailable) + return; + + processControls(data, request->controls()); + + if (data->mode == Mode::Client) { + request->controls().set(controls::SyncAdjustment, + data->frameDurationOffset.count() / 1000); + data->frameDurationOffset = utils::Duration(0); + } +} + +void start(void *closure, ControlList &controls) +{ + SyncLayerData *data = static_cast(closure); + if (!data->syncAvailable) + return; + + reset(data); + data->clockRecovery.addSample(); + processControls(data, controls); +} + +void reset(SyncLayerData *data) +{ + data->syncReady = false; + data->frameCount = 0; + data->serverFrameCountPeriod = 0; + data->serverReadyTime = 0; + data->clientSeenPacket = false; +} + +void initializeSocket(SyncLayerData *data) +{ + ssize_t ret; + struct ip_mreq mreq{}; + socklen_t addrlen; + SyncPayload payload; + unsigned int en = 1; + + if (data->socket.isValid()) { + LOG(SyncLayer, Debug) + << "Socket already exists; not recreating"; + return; + } + + if (!data->networkInterface.empty()) + mreq.imr_interface.s_addr = inet_addr(data->networkInterface.c_str()); + else + mreq.imr_interface.s_addr = htonl(INADDR_ANY); + + int flags = SOCK_CLOEXEC; + if (data->mode == Mode::Client) + flags |= SOCK_NONBLOCK; + + int fd = socket(AF_INET, SOCK_DGRAM | flags, 0); + if (fd < 0) { + LOG(SyncLayer, Error) << "Unable to create socket"; + return; + } + data->socket = UniqueFD(fd); + + struct sockaddr_in &addr = data->addr; + memset(&addr, 0, sizeof(addr)); + addr.sin_family = AF_INET; + addr.sin_addr.s_addr = data->mode == Mode::Client ? htonl(INADDR_ANY) : inet_addr(data->group.c_str()); + addr.sin_port = htons(data->port); + + if (data->mode != Mode::Client) + return; + + if (setsockopt(data->socket.get(), SOL_SOCKET, SO_REUSEADDR, &en, sizeof(en)) < 0) { + LOG(SyncLayer, Error) << "Unable to set socket options: " << strerror(errno); + goto err; + } + + mreq.imr_multiaddr.s_addr = inet_addr(data->group.c_str()); + if (setsockopt(data->socket.get(), IPPROTO_IP, IP_ADD_MEMBERSHIP, &mreq, sizeof(mreq)) < 0) { + LOG(SyncLayer, Error) << "Unable to set socket options: " << strerror(errno); + goto err; + } + + if (bind(data->socket.get(), (struct sockaddr *)&addr, sizeof(addr)) < 0) { + LOG(SyncLayer, Error) << "Unable to bind client socket: " << strerror(errno); + goto err; + } + + /* + * For the client, flush anything in the socket. It might be stale from a previous sync run, + * or we might get another packet in a frame to two before the adjustment caused by this (old) + * packet, although correct, had taken effect. So this keeps things simpler. + */ + addrlen = sizeof(addr); + ret = 0; + while (ret >= 0) { + ret = recvfrom(data->socket.get(), + &payload, sizeof(payload), 0, + (struct sockaddr *)&addr, &addrlen); + } + + return; + +err: + data->socket.reset(); +} + +void processControls(SyncLayerData *data, ControlList &controls) +{ + auto intf = controls.get(controls::SyncInterface); + if (intf) + data->networkInterface = *intf; + + auto mode = controls.get(controls::rpi::SyncMode); + if (mode) { + data->mode = static_cast(*mode); + if (data->mode == Mode::Off) { + reset(data); + } else { + /* + * This goes here instead of init() because we need the control + * to tell us whether we're server or client + */ + initializeSocket(data); + } + } + + auto syncFrames = controls.get(controls::rpi::SyncFrames); + if (syncFrames && *syncFrames > 0) + data->readyFrame = *syncFrames; + + auto frameDurationLimits = controls.get(controls::FrameDurationLimits); + if (frameDurationLimits) + data->frameDuration = std::chrono::microseconds((*frameDurationLimits)[0]); + + /* + * \todo Should we just let SyncAdjustment through as-is if the + * application provides it? Maybe it wants to do sync itself without + * the layer, but the layer has been loaded anyway + */ +} + +/* + * Camera sync algorithm. + * Server - there is a single server that sends framerate timing information over the network to any + * clients that are listening. It also signals when it will send a "everything is synchronised, now go" + * message back to the algorithm. + * Client - there may be many clients, either on the same device or different ones. They match their + * framerates to the server, and indicate when to "go" at the same instant as the server. + */ +void processFrame(SyncLayerData *data, uint64_t frameWallClock, + ControlList &metadata) +{ + /* frameWallClock has already been de-jittered for us. Convert from ns into us. */ + uint64_t wallClockFrameTimestamp = frameWallClock / 1000; + + /* + * This is the headline frame duration in microseconds as programmed into the sensor. Strictly, + * the sensor might not quite match the system clock, but this shouldn't matter for the calculations + * we'll do with it, unless it's a very very long way out! + */ + uint32_t frameDuration = data->frameDuration.get(); + + /* Timestamps tell us if we've dropped any frames, but we still want to count them. */ + unsigned int droppedFrames = 0; + /* Count dropped frames into the frame counter */ + if (data->frameCount) { + wallClockFrameTimestamp = + std::max(wallClockFrameTimestamp, + data->lastWallClockFrameTimestamp + frameDuration / 2); + /* + * Round down here, because data->frameCount gets incremented + * at the end of the function. + */ + droppedFrames = + (wallClockFrameTimestamp - data->lastWallClockFrameTimestamp - frameDuration / 2) / frameDuration; + data->frameCount += droppedFrames; + } + + if (data->mode == Mode::Server) + processFrameServer(data, wallClockFrameTimestamp, + metadata, droppedFrames); + else if (data->mode == Mode::Client) + processFrameClient(data, wallClockFrameTimestamp, + metadata); + + data->lastWallClockFrameTimestamp = wallClockFrameTimestamp; + + metadata.set(controls::rpi::SyncReady, data->syncReady); + + data->frameCount++; +} + +void processFrameServer(SyncLayerData *data, int64_t wallClockFrameTimestamp, + ControlList &metadata, unsigned int droppedFrames) +{ + uint32_t frameDuration = data->frameDuration.get(); + + /* + * Server sends a packet every syncPeriod frames, or as soon after as possible (if any + * frames were dropped). + */ + data->serverFrameCountPeriod += droppedFrames; + + if (data->frameCount == 0) { + data->frameDurationEstimated = frameDuration; + } else { + double diff = (wallClockFrameTimestamp - data->lastWallClockFrameTimestamp) / (1 + droppedFrames); + unsigned int N = std::min(data->frameCount, 99U); + /* + * Smooth out the variance of the frame duration estimation to + * get closer to the true frame duration + */ + data->frameDurationEstimated = data->frameCount == 1 ? diff + : (N * data->frameDurationEstimated + diff) / (N + 1); + } + + /* Calculate frames remaining, and therefore "time left until ready". */ + int framesRemaining = data->readyFrame - data->frameCount; + uint64_t wallClockReadyTime = wallClockFrameTimestamp + (int64_t)framesRemaining * data->frameDurationEstimated; + + if (data->serverFrameCountPeriod >= data->syncPeriod) { + /* It's time to sync */ + data->serverFrameCountPeriod = 0; + + SyncPayload payload; + /* round to nearest us */ + payload.frameDuration = data->frameDurationEstimated + .5; + payload.wallClockFrameTimestamp = wallClockFrameTimestamp; + payload.wallClockReadyTime = wallClockReadyTime; + + LOG(SyncLayer, Debug) << "Send packet (frameNumber " << data->frameCount << "):"; + LOG(SyncLayer, Debug) << " frameDuration " << payload.frameDuration; + LOG(SyncLayer, Debug) << " wallClockFrameTimestamp " << wallClockFrameTimestamp + << " (" << wallClockFrameTimestamp - data->lastWallClockFrameTimestamp << ")"; + LOG(SyncLayer, Debug) << " wallClockReadyTime " << wallClockReadyTime; + + if (sendto(data->socket.get(), &payload, sizeof(payload), 0, (const sockaddr *)&data->addr, sizeof(data->addr)) < 0) + LOG(SyncLayer, Error) << "Send error! " << strerror(errno); + } + + int64_t timerValue = static_cast(wallClockReadyTime - wallClockFrameTimestamp); + if (!data->syncReady && wallClockFrameTimestamp + data->frameDurationEstimated / 2 > wallClockReadyTime) { + data->syncReady = true; + LOG(SyncLayer, Info) << "*** Sync achieved! Difference " << timerValue << "us"; + } + + /* Server always reports this */ + metadata.set(controls::rpi::SyncTimer, timerValue); + + data->serverFrameCountPeriod += 1; +} + +void processFrameClient(SyncLayerData *data, int64_t wallClockFrameTimestamp, + ControlList &metadata) +{ + uint64_t serverFrameTimestamp = 0; + SyncPayload payload; + + bool packetReceived = false; + while (true) { + ssize_t ret = recv(data->socket.get(), &payload, sizeof(payload), 0); + + if (ret != sizeof(payload)) + break; + packetReceived = true; + data->clientSeenPacket = true; + + data->frameDurationEstimated = payload.frameDuration; + serverFrameTimestamp = payload.wallClockFrameTimestamp; + data->serverReadyTime = payload.wallClockReadyTime; + } + + if (packetReceived) { + uint64_t clientFrameTimestamp = wallClockFrameTimestamp; + int64_t clientServerDelta = clientFrameTimestamp - serverFrameTimestamp; + /* "A few frames ago" may have better matched the server's frame. Calculate when it was. */ + int framePeriodErrors = (clientServerDelta + data->frameDurationEstimated / 2) / data->frameDurationEstimated; + int64_t clientFrameTimestampNearest = clientFrameTimestamp - framePeriodErrors * data->frameDurationEstimated; + /* We must shorten a single client frame by this amount if it exceeds the minimum: */ + int32_t correction = clientFrameTimestampNearest - serverFrameTimestamp; + if (std::abs(correction) < data->minAdjustment) + correction = 0; + + LOG(SyncLayer, Debug) << "Received packet (frameNumber " << data->frameCount << "):"; + LOG(SyncLayer, Debug) << " serverFrameTimestamp " << serverFrameTimestamp; + LOG(SyncLayer, Debug) << " serverReadyTime " << data->serverReadyTime; + LOG(SyncLayer, Debug) << " clientFrameTimestamp " << clientFrameTimestamp; + LOG(SyncLayer, Debug) << " clientFrameTimestampNearest " << clientFrameTimestampNearest + << " (" << framePeriodErrors << ")"; + LOG(SyncLayer, Debug) << " correction " << correction; + + data->frameDurationOffset = correction * 1us; + } + + int64_t timerValue = static_cast(data->serverReadyTime - wallClockFrameTimestamp); + if (data->clientSeenPacket && !data->syncReady && wallClockFrameTimestamp + data->frameDurationEstimated / 2 > data->serverReadyTime) { + data->syncReady = true; + LOG(SyncLayer, Info) << "*** Sync achieved! Difference " << timerValue << "us"; + } + + /* Client reports this once it receives it from the server */ + if (data->clientSeenPacket) + metadata.set(controls::rpi::SyncTimer, timerValue); +} + +namespace libcamera { + +extern "C" { + +[[gnu::visibility("default")]] +struct LayerInfo layerInfo{ + .name = "sync", + .layerAPIVersion = 1, +}; + +[[gnu::visibility("default")]] +struct LayerInterface layerInterface{ + .init = init, + .terminate = terminate, + .bufferCompleted = nullptr, + .requestCompleted = requestCompleted, + .disconnected = nullptr, + .acquire = nullptr, + .release = nullptr, + .controls = updateControls, + .properties = nullptr, + .configure = nullptr, + .createRequest = nullptr, + .queueRequest = queueRequest, + .start = start, + .stop = nullptr, +}; +} + +} /* namespace libcamera */ diff --git a/src/layer/sync/sync.h b/src/layer/sync/sync.h new file mode 100644 index 000000000000..e507b0efc566 --- /dev/null +++ b/src/layer/sync/sync.h @@ -0,0 +1,98 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2025, Ideas On Board Oy + * + * Layer implementation for sync algorithm + */ + +#pragma once + +#include +#include + +#include +#include + +#include +#include + +#include "libcamera/internal/clock_recovery.h" + +enum class Mode { + Off, + Server, + Client, +}; + +struct SyncPayload { + /* Frame duration in microseconds. */ + uint32_t frameDuration; + /* Server wall clock version of the frame timestamp. */ + uint64_t wallClockFrameTimestamp; + /* Server wall clock version of the sync time. */ + uint64_t wallClockReadyTime; +}; + +struct SyncLayerData { + bool syncAvailable; + + /* Sync algorithm parameters */ + /* IP group address for sync messages */ + std::string group; + /* port number for messages */ + uint16_t port; + /* send a sync message every this many frames */ + uint32_t syncPeriod; + /* don't adjust the client frame length by less than this (us) */ + uint32_t minAdjustment; + /* This is the network interface to listen/multicast on */ + std::string networkInterface; + + /* Sync algorithm controls */ + Mode mode; + libcamera::utils::Duration frameDuration; + /* tell the application we're ready after this many frames */ + uint32_t readyFrame; + + /* Sync algorithm state */ + bool syncReady = false; + unsigned int frameCount = 0; + /* send the next packet when this reaches syncPeriod */ + uint32_t serverFrameCountPeriod = 0; + /* the client's latest value for when the server will be "ready" */ + uint64_t serverReadyTime = 0; + /* whether the client has received a packet yet */ + bool clientSeenPacket = false; + + /* estimate the true frame duration of the sensor (in us) */ + double frameDurationEstimated = 0; + /* wall clock timestamp of previous frame (in us) */ + uint64_t lastWallClockFrameTimestamp; + + /* Frame length correction to apply */ + libcamera::utils::Duration frameDurationOffset; + + /* Infrastructure state */ + libcamera::ClockRecovery clockRecovery; + sockaddr_in addr; + libcamera::UniqueFD socket; +}; + +void *init(const std::string &id); +void terminate(void *closure); +libcamera::ControlList requestCompleted(void *closure, libcamera::Request *request); +libcamera::ControlInfoMap::Map updateControls(void *closure, + libcamera::ControlInfoMap &controls); +void queueRequest(void *closure, libcamera::Request *request); +void start(void *closure, libcamera::ControlList &controls); + +void reset(SyncLayerData *data); +void initializeSocket(SyncLayerData *data); +void processControls(SyncLayerData *data, libcamera::ControlList &controls); +void processFrame(SyncLayerData *data, uint64_t frameWallClock, + libcamera::ControlList &metadata); +void processFrameServer(SyncLayerData *data, int64_t wallClockFrameTimestamp, + libcamera::ControlList &metadata, + unsigned int droppedFrames); +void processFrameClient(SyncLayerData *data, int64_t wallClockFrameTimestamp, + libcamera::ControlList &metadata); From patchwork Fri Jan 30 08:09:34 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 26052 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8C209C3226 for ; Fri, 30 Jan 2026 08:09:58 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4657E61FDA; Fri, 30 Jan 2026 09:09:58 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="bj22LJTe"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8411861FC7 for ; Fri, 30 Jan 2026 09:09:56 +0100 (CET) Received: from neptunite.hamster-moth.ts.net (unknown [IPv6:2404:7a81:160:2100:ec11:5e0c:deb8:1e2d]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 4F8BE122A; Fri, 30 Jan 2026 09:09:16 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1769760558; bh=NTxeZCYj98CSpFNe/g4MpZo45xvZAc6k3TP0XoYClUo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bj22LJTeptHlrALiSUrTH4fzuja7X/6POEY7vNuhgzf6P20T0aHu4G2k3pE6lwP7e pGw5FsE1XE23MGJLTltgN+e+IYwiz94J61fowDtOfO6Qf2OQMYXfTADnKe1EpS1XxW Mo9kKAfsbwr85rctygU+r2p9lUgV9Dg+dKjNflis= From: Paul Elder To: libcamera-devel@lists.libcamera.org Cc: Paul Elder , david.plowman@raspberrypi.com, naush@raspberrypi.com, kieran.bingham@ideasonboard.com, stefan.klug@ideasonboard.com Subject: [PATCH v2 3/4] libipa: Add SyncHelper Date: Fri, 30 Jan 2026 17:09:34 +0900 Message-ID: <20260130080935.2569621-4-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20260130080935.2569621-1-paul.elder@ideasonboard.com> References: <20260130080935.2569621-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The sync layer allows any Camera to get sync support as long as it implements: - SyncAdjustment control - FrameDurationLimits control - SensorTimestamp metadata While FrameDurationLimits is usually implemented by AGC and SensorTimestamp is already implemented by all pipeline handlers, only SyncAdjustment is left that needs to be implemented for all platforms. It naturally would be a good idea to implement it in libipa so all IPAs can easily be supported. Add SyncHelper that wraps handling of the SyncAdjustment control. Although it still needs to be plumbed into each IPA, it should mitigate a lot of future code duplication. Signed-off-by: Paul Elder --- Changes in v2: - move functions to header - make controlInfo static --- src/ipa/libipa/meson.build | 2 + src/ipa/libipa/sync_helper.cpp | 88 ++++++++++++++++++++++++++++++++++ src/ipa/libipa/sync_helper.h | 43 +++++++++++++++++ 3 files changed, 133 insertions(+) create mode 100644 src/ipa/libipa/sync_helper.cpp create mode 100644 src/ipa/libipa/sync_helper.h diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build index 7202df869c2f..44f712879b3f 100644 --- a/src/ipa/libipa/meson.build +++ b/src/ipa/libipa/meson.build @@ -17,6 +17,7 @@ libipa_headers = files([ 'lux.h', 'module.h', 'pwl.h', + 'sync_helper.h', 'v4l2_params.h', ]) @@ -37,6 +38,7 @@ libipa_sources = files([ 'lux.cpp', 'module.cpp', 'pwl.cpp', + 'sync_helper.cpp', 'v4l2_params.cpp', ]) diff --git a/src/ipa/libipa/sync_helper.cpp b/src/ipa/libipa/sync_helper.cpp new file mode 100644 index 000000000000..2df6fb2473be --- /dev/null +++ b/src/ipa/libipa/sync_helper.cpp @@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2025 Ideas on Board Oy + * + * Helper class that handles sync + */ +#include "sync_helper.h" + +#include +#include +#include + +/** + * \file sync_helper.h + * \brief Helper class that encapsulates handling sync + */ + +namespace libcamera { + +namespace ipa { + +/** + * \class SyncHelper + * \brief Class for handling sync + * + * In order for a Camera to support the sync algorithm (via the sync layer), it + * needs to implement the FrameDurationLimits control, the SyncAdjustment + * control, and the SensorTimestamp metadata. The first is handled by AGC, the + * last is handled by the IPA cores, and the second one is handled by this + * helper. It must be plumbed into IPAs, however. + */ + +/** + * \fn SyncHelper::SyncHelper() + * \brief Construct an SyncHelper instance + */ + +/** + * \fn SyncHelper::controlInfo(int64_t maxFrameDuration) + * \brief Return an entry for the ControlInfoMap of the IPA for SyncAdjustment + * \param[in] maxFrameDuration The maximum value for FrameDurationLimits + * + * This function creates an entry for SyncAdjustment that can be insterted + * directly into the ControlInfoMap of the IPA. + * + * The SyncAdjustment limits are computed based on the \a maxFrameDuration. + * Technically the limits of SyncAdjustment depend on the currently set + * FrameDurationLimits, but since they can be set in the same request this + * doesn't really work. Instead report half of the maximum FrameDurationLimits + * as the SyncAdjustment limits. + * + * \return Entry for ControlInfoMap for SyncAdjustment + */ + +/** + * \fn SyncHelper::setSync(int32_t sync, utils::Duration minFrameDuration) + * \brief Set the sync adjustment value + * \param[in] sync The SyncAdjustment value as passed in by the application, in microseconds + * \param[in] minFrameDuration The minimum frame duration as set by FrameDurationLimits + * + * This function takes the value of SyncAdjustment as passed in by the + * application, and saves it to be later read by getSync(). The \a sync value + * will be clamped by \a minFrameDuration. This + * function is meant to be called by the IPA at queueRequest time. + */ + +/** + * \fn SyncHelper::getSync() + * \brief Retrieve the set sync adjustment value + * + * This function returns the SyncAdjustment value stored in setSync(). It is + * meant to be read out by the IPA when the computing the frame duration to + * set, usually by AGC. + * + * \return The amount of time to adjust the frame duration by for sync + */ + +/** + * \fn SyncHelper::resetSync() + * \brief Reset the stored sync adjustment value + * + * This function resets the state of the sync helper, that is it zeros the + * stored frame duration offset. + */ + +} /* namespace ipa */ + +} /* namespace libcamera */ diff --git a/src/ipa/libipa/sync_helper.h b/src/ipa/libipa/sync_helper.h new file mode 100644 index 000000000000..bf01658c259b --- /dev/null +++ b/src/ipa/libipa/sync_helper.h @@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2025 Ideas on Board Oy + * + * Helper class that handles sync + */ + +#pragma once + +#include + +#include + +namespace libcamera { + +namespace ipa { + +class SyncHelper +{ +public: + static ControlInfo controlInfo(int64_t maxFrameDuration) + { + return ControlInfo(static_cast(-maxFrameDuration / 2), + static_cast(maxFrameDuration / 2), 0); + } + + void setSync(int32_t sync, utils::Duration minFrameDuration) + { + utils::Duration value = std::chrono::microseconds(sync); + frameDurationOffset_ = std::clamp(value, + -minFrameDuration, minFrameDuration); + } + + utils::Duration getSync() const { return frameDurationOffset_; } + void resetSync() { frameDurationOffset_ = utils::Duration(0); } + +private: + utils::Duration frameDurationOffset_ = utils::Duration(0); +}; + +} /* namespace ipa */ + +} /* namespace libcamera */ From patchwork Fri Jan 30 08:09:35 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 26053 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 14EF9C3226 for ; Fri, 30 Jan 2026 08:10:01 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id CE35D61FDC; Fri, 30 Jan 2026 09:10:00 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="HBn5eddS"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4EB0661FD1 for ; Fri, 30 Jan 2026 09:09:59 +0100 (CET) Received: from neptunite.hamster-moth.ts.net (unknown [IPv6:2404:7a81:160:2100:ec11:5e0c:deb8:1e2d]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 098FA1E23; Fri, 30 Jan 2026 09:09:18 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1769760561; bh=YkNHzAtu2r5UXKdVQG7lH8PyiJgdwYF3NoBj50v0SYU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HBn5eddSzx/xcQZmdW1Tcp0QKPqmq9dk/8bKJ6mMK1k5qJkZfLOQ5Wk6cYrLD2lAV 2gucpoZd+teRl3/8eMC3CiuUdrHeJK+Q4C2pi1qLp0fUISgt4vWfRUvvg5/dKB/stT /b13dWhD/5BVnZWNsJ+IP7ZDLNRIizCee37bYUVo= From: Paul Elder To: libcamera-devel@lists.libcamera.org Cc: Paul Elder , david.plowman@raspberrypi.com, naush@raspberrypi.com, kieran.bingham@ideasonboard.com, stefan.klug@ideasonboard.com Subject: [PATCH v2 4/4] ipa: rkisp1: agc: Add support for sync Date: Fri, 30 Jan 2026 17:09:35 +0900 Message-ID: <20260130080935.2569621-5-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20260130080935.2569621-1-paul.elder@ideasonboard.com> References: <20260130080935.2569621-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add sync support to the RkISP1 IPA using the SyncHelper from libipa. The syncAdjustment is saved in the frameContext (as opposed to getting it from the SyncHelper) to avoid potential races and ensure that the same value that was set into vblank will be returned in metadata. Signed-off-by: Paul Elder --- Changes in v2: - move SyncHelper from inherited to a member of Agc - cosmetic changes --- src/ipa/rkisp1/algorithms/agc.cpp | 22 +++++++++++++++++++++- src/ipa/rkisp1/algorithms/agc.h | 3 +++ src/ipa/rkisp1/ipa_context.h | 1 + 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 55cadb1793a5..1496a118ae2b 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -156,6 +156,15 @@ int Agc::init(IPAContext &context, const YamlObject &tuningData) ControlValue(controls::AnalogueGainModeManual) } }, ControlValue(controls::AnalogueGainModeAuto)); context.ctrlMap[&controls::ExposureValue] = ControlInfo(-8.0f, 8.0f, 0.0f); + /* + * Insert the controlInfo for sync. Since FrameDurationLimits is only + * set *after* the algorithms are initialized, we have no information + * on it here. Use a sensible default here and update it later in + * configure(). + * \todo Move FrameDurationLimits from the base IPA to AGC + */ + context.ctrlMap[&controls::SyncAdjustment] = SyncHelper::controlInfo(120'000); + /* Insert the controls for agc */ context.ctrlMap.merge(controls()); return 0; @@ -208,7 +217,11 @@ int Agc::configure(IPAContext &context, const IPACameraSensorInfo &configInfo) context.activeState.agc.automatic.yTarget = effectiveYTarget(); + context.ctrlMap[&controls::SyncAdjustment] = + SyncHelper::controlInfo(frameDurationLimits.max().get()); + resetFrameCount(); + sync_.resetSync(); return 0; } @@ -334,6 +347,10 @@ void Agc::queueRequest(IPAContext &context, } frameContext.agc.minFrameDuration = agc.minFrameDuration; frameContext.agc.maxFrameDuration = agc.maxFrameDuration; + + const auto &sync = controls.get(controls::SyncAdjustment); + if (sync) + sync_.setSync(*sync, frameContext.agc.minFrameDuration); } /** @@ -453,6 +470,7 @@ void Agc::fillMetadata(IPAContext &context, IPAFrameContext &frameContext, metadata.set(controls::AeExposureMode, frameContext.agc.exposureMode); metadata.set(controls::AeConstraintMode, frameContext.agc.constraintMode); metadata.set(controls::ExposureValue, frameContext.agc.exposureValue); + metadata.set(controls::SyncAdjustment, frameContext.agc.syncAdjustment.count()); } /** @@ -511,7 +529,9 @@ void Agc::processFrameDuration(IPAContext &context, IPACameraSensorInfo &sensorInfo = context.sensorInfo; utils::Duration lineDuration = context.configuration.sensor.lineDuration; - frameContext.agc.vblank = (frameDuration / lineDuration) - sensorInfo.outputSize.height; + utils::Duration sync = sync_.getSync(); + frameContext.agc.vblank = ((frameDuration + sync) / lineDuration) - sensorInfo.outputSize.height; + frameContext.agc.syncAdjustment = sync; /* Update frame duration accounting for line length quantization. */ frameContext.agc.frameDuration = (sensorInfo.outputSize.height + frameContext.agc.vblank) * lineDuration; diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h index 7867eed9c4e3..c6cc20679cf7 100644 --- a/src/ipa/rkisp1/algorithms/agc.h +++ b/src/ipa/rkisp1/algorithms/agc.h @@ -15,6 +15,7 @@ #include #include "libipa/agc_mean_luminance.h" +#include "libipa/sync_helper.h" #include "algorithm.h" @@ -58,6 +59,8 @@ private: Span weights_; std::map> meteringModes_; + + SyncHelper sync_; }; } /* namespace ipa::rkisp1::algorithms */ diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h index fa748811be74..3a5b499e088e 100644 --- a/src/ipa/rkisp1/ipa_context.h +++ b/src/ipa/rkisp1/ipa_context.h @@ -164,6 +164,7 @@ struct IPAFrameContext : public FrameContext { bool updateMetering; bool autoExposureModeChange; bool autoGainModeChange; + utils::Duration syncAdjustment; } agc; struct {