From patchwork Wed Dec 18 18:03:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 22399 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E8DF5C3301 for ; Wed, 18 Dec 2024 18:03:20 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9AFC668457; Wed, 18 Dec 2024 19:03:20 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="MK+m6+7z"; dkim-atps=neutral Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1F7B3680B6 for ; Wed, 18 Dec 2024 19:03:17 +0100 (CET) Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-43618283d48so49304825e9.1 for ; Wed, 18 Dec 2024 10:03:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1734544996; x=1735149796; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MJxK/bYe/NGW7I0pZiRtUEp8+1iDVycrVP02cMFKu1Q=; b=MK+m6+7zFYEvphKWCdlOKmR/YriAxhJewL6Dx21gyVX3bixTsuDd8q9xL3eELv+/LK OBwPwnzBqb+fQqZvq5/pN3lUmWHhDh7G4IXGw0pdABE5CIceIU0ioZw4WXpE+PD1ZIEz 3xKFquqrbRaxHVNNfZFL9iuIbZ+UG7ONX7Mr8s9dp2/PcNewXr4GU9neVlF+1cgMwM9K zszzCJ3vmNe6gAj1J4D3Ao1nhDx2RGPB6FedfjC4KeHc3b0SdlsKO7xnLqZKWyedKiRk qojTm/IuczRS/POt+5t3ZzztTHRSn0/qXi3T2OQ+iUr1OjnWmCNfeRlIjPBGYoIFNSJz VHtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734544996; x=1735149796; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MJxK/bYe/NGW7I0pZiRtUEp8+1iDVycrVP02cMFKu1Q=; b=QvPoSQZ1TT4Brwd5TlTjLV5M8rWm7srsyold5X8TLRI33yyATES0QflagKpKdrBsOe RLNBBrEHwDQYftAXn9gmUYel4Ue5CITgARQbSUw1HUn1g85MjB+RsOSCx8y8xNai6dV8 3KYnHYiR1vzRiWqXcAn+a6UhI38hnZQjjEgCWcDMKeQBAepDPwL7M/3WUj8hnS5nMjOs qwjJpZYaOMYHrLXvtslRRzvBRzwNkNTnDetpmIMniP7BwRs9anPEg8kQtk0N4oiMwcmM HeMDaU1AJyOVnpg7ye5OUJi7xqA8fhtAktkrh9Ic3cZDxOCJC4vzUlDq1D0MEzV1eh66 /2DQ== X-Gm-Message-State: AOJu0YyEMY1e54E6ysGQUxzkL3LgPAxNmB0Sycfq64SF2mKxZApiB/Yi yYaQF4JCuXtUCfV/xAabgRiuWrPB2pcv8iWcQce1JMUKOdADDig9DcvdYRmkFRSQaFM7jvBTuqh z X-Gm-Gg: ASbGnctZM7F7Dr0sEiOYi/WWr4LWTwNO2VVW6ay1EEvRJbi6/ZQ3yxVnCJBKf1ql5p5 HKUEUAd9OoVBNVmKOvFCI5CeWHC6V6+F76Vagukzf6wBBpGyOTBoVt7gTut6E+XaLKDW0mv1Tx7 /TRqJdgh6KhM2JObf9ANRl+G7lJiMu4iX19vy/BVkkdvx42Ko/65Hv/H34HKAZ8iDLWKfX2k4iC vGVR4T8rxLPn8ZuXukL8Z0TVcUAFX2krk50FosPykHxygewbevdMQd+hJE/K+dvA+Zwy13q4gLs jml/JCismxL9 X-Google-Smtp-Source: AGHT+IGbJf7SVMJ0gji+EXBevVf0v7oTanxvjySagrNgvUwbEUvLcc3XYdzn6iBuSZmdIxfqmT741w== X-Received: by 2002:a05:600c:1d1f:b0:434:f131:1e64 with SMTP id 5b1f17b1804b1-4365535de80mr35777565e9.9.1734544995927; Wed, 18 Dec 2024 10:03:15 -0800 (PST) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:c68a:6be1:5ba3:eddd]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-43656b01c88sm27927285e9.17.2024.12.18.10.03.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 10:03:14 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman , Kieran Bingham , Naushir Patuck , Laurent Pinchart Subject: [PATCH v2 1/3] controls: Add FrameWallClock control Date: Wed, 18 Dec 2024 18:03:08 +0000 Message-Id: <20241218180310.7824-2-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241218180310.7824-1-david.plowman@raspberrypi.com> References: <20241218180310.7824-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a FrameWallClock control that reports the same moment as the frame's SensorTimestamp, but in wallclock units. Signed-off-by: David Plowman Reviewed-by: Kieran Bingham Reviewed-by: Naushir Patuck Reviewed-by: Laurent Pinchart --- src/libcamera/control_ids_core.yaml | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml index 073e0611..8485f7e8 100644 --- a/src/libcamera/control_ids_core.yaml +++ b/src/libcamera/control_ids_core.yaml @@ -1,6 +1,4 @@ -# SPDX-License-Identifier: LGPL-2.1-or-later -# -# Copyright (C) 2019, Google Inc. + # %YAML 1.1 --- @@ -1015,4 +1013,18 @@ controls: description: | Enable or disable the debug metadata. + - FrameWallClock: + type: int64_t + description: | + This timestamp corresponds to the same moment in time as the + SensorTimestamp, but is represented as a wall clock time as measured by + the CLOCK_REALTIME clock. + + Being a wall clock measurement, it can be used to synchronise timing + across different devices. + + \sa SensorTimestamp + + The FrameWallClock control can only be returned in metadata. + ... From patchwork Wed Dec 18 18:03:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 22400 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E5885C3301 for ; Wed, 18 Dec 2024 18:03:22 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BEF1B68455; Wed, 18 Dec 2024 19:03:21 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="HVvnhWCn"; dkim-atps=neutral Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4F58D680A2 for ; Wed, 18 Dec 2024 19:03:18 +0100 (CET) Received: by mail-wm1-x330.google.com with SMTP id 5b1f17b1804b1-4361f664af5so73192935e9.1 for ; Wed, 18 Dec 2024 10:03:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1734544997; x=1735149797; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0nAyOHSpY0zR0dhTRQ+G7CBUy1LrRHRDo0Ax+OefIcQ=; b=HVvnhWCnXdUBOgbxVTuOHwFd2H5lymLHdjRDC0/qUzRzVWayGMj+wGIGph1YXtboJi EOTy9+o63c0B/qTzR3MHrkrI6XE3QpJEVE9vT9WR+S7B2J6uEFaWLiqNnKD68iDbZM1Z ZhjoHoEbEqNgEFBvKfU89bgSVi5VjDpH/iafHaL0Kigj4cEmrTegjI8Om+bmKN+/fm1R bwP2QUwVOAOIvcsUbmGje1VIavFlP2wGN4OhPq4Xeui2OFalQ08VfWlqDrYYW7/eGRBU ZxyuvvQfnBrjBZKxPRUe8g8fLPUBpkSA4MeGKylaOGTodY68hGhHC25MqWR+Q2OzHuA0 wA5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734544997; x=1735149797; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0nAyOHSpY0zR0dhTRQ+G7CBUy1LrRHRDo0Ax+OefIcQ=; b=j8pLuKADuXpCi8D8RTnIcD9U68Ryi9mCYG5/LVJZ77Bd8zg182S2F6MHRNTyKtJEKQ qFU89oefm4yAHgUzYVcgBfAHVkCCnMSmdO1g9qOR3W1zc4N1L2jQzszkAaXwB5F2N5VE hBSF8dtCZZJeIGBOvJfTlQEMXZ7J3GJXhMfJ9+wGRG1BSZUBwEVKqyizC6eY//a+soQc qxQxR8DWjXBu+VMKDURC6yU2NYLEg+oOq5VF5/gUwWjVbRwj1ZccSDxF1nV2ccAHcIB1 7cjAnLQVr7m0wnaFZuygJcxpe97VZyQX+828NNvtsVvGClWJhrQi7+so2UBpH9+33jK8 fQ7w== X-Gm-Message-State: AOJu0YziyURM2piRM+cGUUj+WfcimIGYB0p2ODRdJ+Ntacjc4wxG/mRu 6S049syGJ1DKcOjBjTwBkKviOflHwFN0z09hNp4+CLhnvEFneesayInyGXxDcppTootnhCesv8S l X-Gm-Gg: ASbGncvGi3g7Ya/LS/Wl2IN+DH6iekS0BJQQ/u6I6bgr0yebSh7mkyDtrdlQTO6stXz Jdwt1dUcdD1q0/HBxK4tNcer45Wrw/H8dX3kWeZBRxjp8EG7Z55VqSFlgi9IlcqOHLjPbm+102C TM1Q6NsQnlS49C/pa2PBuYNg/ZEwYWMYUZGxH+DIgpEjdFNwy3nZcw9+5pn6CWhS36i69D4HU1+ hPWRdvdhXwPlhKrO7LG9TPyfYKSgbm3XjcTiIaWgJ273518jN1LJ8QrhmTi+vb0dusVI9FJDg2G Zu7KDbizPXWo X-Google-Smtp-Source: AGHT+IFnFcnl88wnpzmtDr6HE90SnlOpY6Qyv0Z3CHCrNotGYMqzlLqqHtvqzkdyfWSHwmgfnDRtlw== X-Received: by 2002:a05:600c:1c10:b0:434:a815:2b5d with SMTP id 5b1f17b1804b1-436553ea71amr35360585e9.24.1734544996673; Wed, 18 Dec 2024 10:03:16 -0800 (PST) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:c68a:6be1:5ba3:eddd]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-43656b01c88sm27927285e9.17.2024.12.18.10.03.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 10:03:16 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman Subject: [PATCH v2 2/3] libcamera: Add ClockRecovery class to generate wallclock timestamps Date: Wed, 18 Dec 2024 18:03:09 +0000 Message-Id: <20241218180310.7824-3-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241218180310.7824-1-david.plowman@raspberrypi.com> References: <20241218180310.7824-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The ClockRecovery class takes pairs of timestamps from two different clocks, and models the second ("output") clock from the first ("input") clock. We can use it, in particular, to get a good wallclock estimate for a frame's SensorTimestamp. Signed-off-by: David Plowman --- include/libcamera/internal/clock_recovery.h | 68 ++++++ include/libcamera/internal/meson.build | 1 + src/libcamera/clock_recovery.cpp | 230 ++++++++++++++++++++ src/libcamera/meson.build | 1 + 4 files changed, 300 insertions(+) create mode 100644 include/libcamera/internal/clock_recovery.h create mode 100644 src/libcamera/clock_recovery.cpp diff --git a/include/libcamera/internal/clock_recovery.h b/include/libcamera/internal/clock_recovery.h new file mode 100644 index 00000000..43e46b7d --- /dev/null +++ b/include/libcamera/internal/clock_recovery.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * Camera recovery algorithm + */ +#pragma once + +#include + +namespace libcamera { + +class ClockRecovery +{ +public: + ClockRecovery(); + + void configure(unsigned int numSamples = 100, unsigned int maxJitter = 2000, + unsigned int minSamples = 10, unsigned int errorThreshold = 50000); + void reset(); + + void addSample(); + void addSample(uint64_t input, uint64_t output); + + uint64_t getOutput(uint64_t input); + +private: + /* Approximate number of samples over which the model state persists. */ + unsigned int numSamples_; + /* Remove any output jitter larger than this immediately. */ + unsigned int maxJitter_; + /* Number of samples required before we start to use model estimates. */ + unsigned int minSamples_; + /* Threshold above which we assume the wallclock has been reset. */ + unsigned int errorThreshold_; + + /* How many samples seen (up to numSamples_). */ + unsigned int count_; + /* This gets subtracted from all input values, just to make the numbers easier. */ + uint64_t inputBase_; + /* As above, for the output. */ + uint64_t outputBase_; + /* The previous input sample. */ + uint64_t lastInput_; + /* The previous output sample. */ + uint64_t lastOutput_; + + /* Average x value seen so far. */ + double xAve_; + /* Average y value seen so far */ + double yAve_; + /* Average x^2 value seen so far. */ + double x2Ave_; + /* Average x*y value seen so far. */ + double xyAve_; + + /* + * The latest estimate of linear parameters to derive the output clock + * from the input. + */ + double slope_; + double offset_; + + /* Use this cumulative error to monitor for spontaneous clock updates. */ + double error_; +}; + +} /* namespace libcamera */ diff --git a/include/libcamera/internal/meson.build b/include/libcamera/internal/meson.build index 7d6aa8b7..41500636 100644 --- a/include/libcamera/internal/meson.build +++ b/include/libcamera/internal/meson.build @@ -11,6 +11,7 @@ libcamera_internal_headers = files([ 'camera_manager.h', 'camera_sensor.h', 'camera_sensor_properties.h', + 'clock_recovery.h', 'control_serializer.h', 'control_validator.h', 'converter.h', diff --git a/src/libcamera/clock_recovery.cpp b/src/libcamera/clock_recovery.cpp new file mode 100644 index 00000000..abacf444 --- /dev/null +++ b/src/libcamera/clock_recovery.cpp @@ -0,0 +1,230 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * Clock recovery algorithm + */ + +#include "libcamera/internal/clock_recovery.h" + +#include + +#include + +/** + * \file clock_recovery.h + * \brief Clock recovery - deriving one clock from another independent clock + */ + +namespace libcamera { + +LOG_DEFINE_CATEGORY(ClockRec) + +/** + * \class ClockRecovery + * \brief Recover an output clock from an input clock + * + * The ClockRecovery class derives an output clock from an input clock, + * modelling the output clock as being linearly related to the input clock. + * For example, we may use it to derive wall clock timestamps from timestamps + * measured by the internal system clock which counts local time since boot. + * + * When pairs of corresponding input and output timestamps are available, + * they should be submitted to the model with addSample(). The model will + * update, and output clock values for known input clock values can be + * obtained using getOutput(). + * + * As a convenience, if the input clock is indeed the time since boot, and the + * output clock represents a real wallclock time, then addSample() can be + * called with no arguments, and a pair of timestamps will be captured at + * that moment. + * + * The configure() function accepts some configuration parameters to control + * the linear fitting process. + */ + +/** + * \brief Construct a ClockRecovery + */ +ClockRecovery::ClockRecovery() +{ + configure(); + reset(); +} + +/** + * \brief Set configuration parameters + * \param[in] numSamples The approximate duration for which the state of the model + * is persistent + * \param[in] maxJitter New output samples are clamped to no more than this + * amount of jitter, to prevent sudden swings from having a large effect + * \param[in] minSamples The fitted clock model is not used to generate outputs + * until this many samples have been received + * \param[in] errorThreshold If the accumulated differences between input and + * output clocks reaches this amount over a few frames, the model is reset + */ +void ClockRecovery::configure(unsigned int numSamples, unsigned int maxJitter, + unsigned int minSamples, unsigned int errorThreshold) +{ + LOG(ClockRec, Debug) + << "configure " << numSamples << " " << maxJitter << " " << minSamples << " " << errorThreshold; + + numSamples_ = numSamples; + maxJitter_ = maxJitter; + minSamples_ = minSamples; + errorThreshold_ = errorThreshold; +} + +/** + * \brief Reset the clock recovery model and start again from scratch + */ +void ClockRecovery::reset() +{ + LOG(ClockRec, Debug) << "reset"; + + lastInput_ = 0; + lastOutput_ = 0; + xAve_ = 0; + yAve_ = 0; + x2Ave_ = 0; + xyAve_ = 0; + count_ = 0; + error_ = 0.0; + /* + * Setting slope_ and offset_ to zero initially means that the clocks + * advance at exactly the same rate. + */ + slope_ = 0.0; + offset_ = 0.0; +} + +/** + * \brief Add a sample point to the clock recovery model, for recovering a wall + * clock value from the internal system time since boot + * + * This is a convenience function to make it easy to derive a wall clock value + * (using the Linux CLOCK_REALTIME) from the time since the system started + * (measured by CLOCK_BOOTTIME). + */ +void ClockRecovery::addSample() +{ + LOG(ClockRec, Debug) << "addSample"; + + struct timespec bootTime1; + struct timespec bootTime2; + struct timespec wallTime; + + /* Get boot and wall clocks in microseconds. */ + clock_gettime(CLOCK_BOOTTIME, &bootTime1); + clock_gettime(CLOCK_REALTIME, &wallTime); + clock_gettime(CLOCK_BOOTTIME, &bootTime2); + uint64_t boot1 = bootTime1.tv_sec * 1000000ULL + bootTime1.tv_nsec / 1000; + uint64_t boot2 = bootTime2.tv_sec * 1000000ULL + bootTime2.tv_nsec / 1000; + uint64_t boot = (boot1 + boot2) / 2; + uint64_t wall = wallTime.tv_sec * 1000000ULL + wallTime.tv_nsec / 1000; + + addSample(boot, wall); +} + +/** + * \brief Add a sample point to the clock recovery model, specifying the exact + * input and output clock values + * \param[in] input The input clock value + * \param[in] output The value of the output clock at the same moment, as far + * as possible, that the input clock was sampled + * + * This function should be used for corresponding clocks other than the Linux + * BOOTTIME and REALTIME clocks. + */ +void ClockRecovery::addSample(uint64_t input, uint64_t output) +{ + LOG(ClockRec, Debug) << "addSample " << input << " " << output; + + if (count_ == 0) { + inputBase_ = input; + outputBase_ = output; + } + + /* + * We keep an eye on cumulative drift over the last several frames. If this exceeds a + * threshold, then probably the system clock has been updated and we're going to have to + * reset everything and start over. + */ + if (lastOutput_) { + int64_t inputDiff = getOutput(input) - getOutput(lastInput_); + int64_t outputDiff = output - lastOutput_; + error_ = error_ * 0.95 + (outputDiff - inputDiff); + if (std::abs(error_) > errorThreshold_) { + reset(); + inputBase_ = input; + outputBase_ = output; + } + } + lastInput_ = input; + lastOutput_ = output; + + /* + * Never let the new output value be more than maxJitter_ away from what + * we would have expected. This is just to reduce the effect of sudden + * large delays in the measured output. + */ + uint64_t expectedOutput = getOutput(input); + output = std::clamp(output, expectedOutput - maxJitter_, expectedOutput + maxJitter_); + + /* + * We use x, y, x^2 and x*y sums to calculate the best fit line. Here we + * update them by pretending we have count_ samples at the previous fit, + * and now one new one. Gradually the effect of the older values gets + * lost. This is a very simple way of updating the fit (there are much + * more complicated ones!), but it works well enough. Using averages + * instead of sums makes the relative effect of old values and the new + * sample clearer. + */ + double x = static_cast(input - inputBase_); + double y = static_cast(output - outputBase_) - x; + unsigned int count1 = count_ + 1; + xAve_ = (count_ * xAve_ + x) / count1; + yAve_ = (count_ * yAve_ + y) / count1; + x2Ave_ = (count_ * x2Ave_ + x * x) / count1; + xyAve_ = (count_ * xyAve_ + x * y) / count1; + + /* + * Don't update slope and offset until we've seen "enough" sample + * points. Note that the initial settings for slope_ and offset_ + * ensures that the wallclock advances at the same rate as the realtime + * clock (but with their respective initial offsets). + */ + if (count_ > minSamples_) { + /* These are the standard equations for least squares linear regression. */ + slope_ = (count1 * count1 * xyAve_ - count1 * xAve_ * count1 * yAve_) / + (count1 * count1 * x2Ave_ - count1 * xAve_ * count1 * xAve_); + offset_ = yAve_ - slope_ * xAve_; + } + + /* + * Don't increase count_ above numSamples_, as this controls the long-term + * amount of the residual fit. + */ + if (count1 < numSamples_) + count_++; +} + +/** + * \brief Calculate the output clock value according to the model from an input + * clock value + * \param[in] input The input clock value + * + * \return Output clock value + */ +uint64_t ClockRecovery::getOutput(uint64_t input) +{ + double x = static_cast(input - inputBase_); + double y = slope_ * x + offset_; + uint64_t output = y + x + outputBase_; + + LOG(ClockRec, Debug) << "getOutput " << input << " " << output; + + return output; +} + +} /* namespace libcamera */ diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build index 57fde8a8..4eaa1c8e 100644 --- a/src/libcamera/meson.build +++ b/src/libcamera/meson.build @@ -21,6 +21,7 @@ libcamera_internal_sources = files([ 'byte_stream_buffer.cpp', 'camera_controls.cpp', 'camera_lens.cpp', + 'clock_recovery.cpp', 'control_serializer.cpp', 'control_validator.cpp', 'converter.cpp', From patchwork Wed Dec 18 18:03:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 22401 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0D2CAC3306 for ; Wed, 18 Dec 2024 18:03:25 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 047FB680B2; Wed, 18 Dec 2024 19:03:24 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="ZWbFTrII"; dkim-atps=neutral Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DCF70680A2 for ; Wed, 18 Dec 2024 19:03:19 +0100 (CET) Received: by mail-wm1-x32b.google.com with SMTP id 5b1f17b1804b1-436249df846so47462965e9.3 for ; Wed, 18 Dec 2024 10:03:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1734544999; x=1735149799; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=a34Z2mQqo6HWxuMMY6rFsnAumW5AmxjBsIlATHxwa6E=; b=ZWbFTrIIThmceZf7LlTOneW0sIXVWYLI4+hH7khMR8394KCkzJ5ZVi4I+0vqa7A6D8 Cx61h8NXd5j5G16Rzd0Ba08kX/2KnFjmg7IKcB9HjvHrjuJzvu7ccvB/tooVCFe9TqXb f/y02UV7GDDnUMmwT6i7p97AtMtIqDBr0Y4kZnIaLSnqLH/+NHbWd3O9siCbDNhCdbEk JhRILxAtSc/z+zY1c6aQAnpbjvbG+LRksQ2fOADP6dgKzYpLJw+mRc0HdY4uTTe3rzyX jvO8ow2zc9bY97jK7RoXwcrk23ThcbY395ETqCWl7G+tZhIV9WqfbfvU1aq+nStHLNdr p0EQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734544999; x=1735149799; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=a34Z2mQqo6HWxuMMY6rFsnAumW5AmxjBsIlATHxwa6E=; b=LftXvkmXWIGcGsABowoValkJUUyGLFv+/mVvk9r2jkpjx6IDf6Q9Oe+6WLoyN3zO8u L858hmppw+k8PViWm16UeJu/F2sRlgj94fnRdxWIt7zut27eXGepczNUYCRuFZ6eMx7D CFi4EIA++lMbR3JHqfy7Pmv4gF/RF48+mFi7cqooTXaiva/kIBrn1IKGKLSf9aFNxAS4 5va4ih6xUXN99cyuqUWY/XYYrd+pRGyg1SB4UjGjHveyAK5PQ5bbfNSy8PXZ6hdnp6ym GYAqcjy7iL3djpcvViVQkQ8oN5/r7jbfbHUyJ401g7bZLPs5NjJzmx6+s+NiRvLZY54Z NSdQ== X-Gm-Message-State: AOJu0YzlWzg0Z42HNbivQpwyYV/3pVCx2KJhLqMpB8cinujA0QlymjD3 Yeg01JizWkxM/foXxXkEf0foB/KY9IGrUQnup7weohAnNRkeFsltgp+0U3JVZp6ldfP6DTTY+hN s X-Gm-Gg: ASbGncu7MicWzGzkcTuTrEqvzzedsU55Scy+hTkCIJuayzPUPzUrcrFekJhIrplHJjH 0iD89Ru2rIrhPlWG7dcvUf5T4fH1ItaFfgwoyTl0qsLkpmMbcjLt3493Ww0DKmTcOGKyo8uW2LM lGwf3FC6iTjmcY9O9Q9SFfM+OolLGFrXKlASgvY4Cr0CJBAkzzNHpocPZtOoEaXUaucxK3ZPTVW XoQvKNXkce4PlBzEMqIypyL/UC5B85iEcgg5qHHCanG0kVIoD2nYRK0jrlfrsgjTmdvD+1/V9w6 3fqgN1Ey/NhO X-Google-Smtp-Source: AGHT+IEKmxBmx+hLTo1JCDz8CxHKYx3Zp20fMcF3FPJu+dG7jwOiamkKJ7vGc/lOEgawGggH3bO3rQ== X-Received: by 2002:a05:600c:1d15:b0:436:46f9:4fc6 with SMTP id 5b1f17b1804b1-4365c78124fmr2228985e9.8.1734544998786; Wed, 18 Dec 2024 10:03:18 -0800 (PST) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:c68a:6be1:5ba3:eddd]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-43656b01c88sm27927285e9.17.2024.12.18.10.03.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 10:03:16 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman , Naushir Patuck Subject: [PATCH v2 3/3] controls: Add camera synchronisation controls Date: Wed, 18 Dec 2024 18:03:10 +0000 Message-Id: <20241218180310.7824-4-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241218180310.7824-1-david.plowman@raspberrypi.com> References: <20241218180310.7824-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" New controls are added to control the camera "sync" algorithm, which allows different cameras to synchronise their frames. Signed-off-by: David Plowman Reviewed-by: Naushir Patuck --- src/libcamera/control_ids_core.yaml | 109 ++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml index 8485f7e8..cf502944 100644 --- a/src/libcamera/control_ids_core.yaml +++ b/src/libcamera/control_ids_core.yaml @@ -1027,4 +1027,113 @@ controls: The FrameWallClock control can only be returned in metadata. + - SyncMode: + type: int32_t + description: | + Enable or disable camera synchronisation ("sync") mode. + + When sync mode is enabled, a camera will synchronise frames temporally + with other cameras, either attached to the same device or a different + one. There should be one "server" device, which broadcasts timing + information to one or more "clients". Communication is one-way, from + server to clients only, and it is only clients that adjust their frame + timings to match the server. + + Sync mode requires all cameras to be running at (as far as possible) the + same fixed framerate. Clients may continue to make adjustments to keep + their cameras synchronised with the server for the duration of the + session, though any updates after the initial ones should remain small. + + \sa SyncReady + \sa SyncTimer + \sa SyncFrames + + enum: + - name: SyncModeOff + value: 0 + description: Disable sync mode. + - name: SyncModeServer + value: 1 + description: | + Enable sync mode, act as server. The server broadcasts timing + messages to any clients that are listening, so that the clients can + synchronise their camera frames with the server's. + - name: SyncModeClient + value: 2 + description: | + Enable sync mode, act as client. A client listens for any server + messages, and arranges for its camera frames to synchronise as + closely as possible with the server's. Many clients can listen out + for the same server. Clients can also be started ahead of any + servers, causing them merely to wait for the server to start. + + - SyncReady: + type: bool + description: | + When using the camera synchronisation algorithm, the server broadcasts + timing information to the clients. This also includes the time (some + number of frames in the future, called the "ready time") at which the + server will signal its controlling application, using this control, to + start using the image frames. + + The client receives the "ready time" from the server, and will signal + its application to start using the frames at this same moment. + + While this control value is false, applications (on both client and + server) should continue to wait, and not use the frames. + + Once this value becomes true, it means that this is the first frame + where the server and its clients have agreed that they will both be + synchronised and that applications should begin consuming frames. + Thereafter, this control will continue to signal the value true for + the rest of the session. + + \sa SyncMode + \sa SyncTImer + \sa SyncFrames + + - SyncTimer: + type: int64_t + description: | + This reports the amount of time, in microseconds, until the "ready + time", at which the server and client will signal their controlling + applications that the frames are now synchronised and should be + used. The value may be refined slightly over time, becoming more precise + as the "ready time" approaches. + + Servers always report this value, whereas clients will omit this control + until they have received a message from the server that enables them to + calculate it. + + Normally the value will start positive (the "ready time" is in the + future), and decrease towards zero, before becoming negative (the "ready + time" has elapsed). So there should be just one frame where the timer + value is, or is very close to, zero - the one for which the SyncReady + control becomes true. At this moment, the value indicates how closely + synchronised the client believes it is with the server. + + But note that if frames are being dropped, then the "near zero" valued + frame, or indeed any other, could be skipped. In these cases the timer + value allows an application to deduce that this has happened. + + \sa SyncMode + \sa SyncReady + \sa SyncFrames + + - SyncFrames: + type: int32_t + description: | + The number of frames the server should wait, after enabling + SyncModeServer, before signalling (via the SyncReady control) that + frames should be used. This therefore determines the "ready time" for + all synchronised cameras. + + This control value should be set only for the device that is to act as + the server, before or at the same moment at which SyncModeServer is + enabled. + + \sa SyncMode + \sa SyncReady + \sa SyncTimer + ...