From patchwork Thu Jun 19 10:05:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 23602 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A8D07C3237 for ; Thu, 19 Jun 2025 10:09:08 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 786BC68DEC; Thu, 19 Jun 2025 12:09:04 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="ik8LmVIL"; dkim-atps=neutral Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5E66E68DC6 for ; Thu, 19 Jun 2025 12:09:02 +0200 (CEST) Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-3a577f164c8so107972f8f.2 for ; Thu, 19 Jun 2025 03:09:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1750327742; x=1750932542; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yu9r/67Lvp8keDcehgTMXQn4KOvwJQfKdTVVKeCET/8=; b=ik8LmVIL625xCWe3NiMMfHxsdTncq+RfrCFEU2+3R/72Uc2tKZ/2gREM/qK9ez+WMi T6r80wXJdMcWOLWHFiNLN8KMN0cpn8OadmMw7ARSTgQt54U6PlXkqu379RiQtATKPhDV rmHirhMekhfXvqkKC1IjPAzR1kLZuX49bDIgQSjXNAIhGyM2LwsoBij9L/irEF8fWigt OJ2rIDNSVsGpv5EpztqrASiUBcnFOirFB5xxTDS3Jo21VdAxM3Iqa0zicJ0kWpmCRn2S gznOl+RKK//by2IWe/FeLHAuv5AMSL1lXhAtY6t6t+3laEBeUXGFME3vz1mSSuQMxytx TibA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750327742; x=1750932542; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yu9r/67Lvp8keDcehgTMXQn4KOvwJQfKdTVVKeCET/8=; b=APp6y6frADTX/Ks/X6Sk8SLNq2BX2nMBkRLDZmN+V3nBTBEYKffLyaKHlNVGkeL5lw VCfPLfI7dt4JPkjEiyRawJ5gxzska2XbkakYg3WqoLGDdvFARYPeASm2fjv2htPYfXw+ O+WHaArZ4MeXf40bqShTrNEpQGtoOwAubvKSROMfwzGJvk7PSu/ot0sohS+NkmsGF75V glGKwvCW3+Ud8jrFChW4voCyX0quAa4R0W5awoIUjtTzQdtKXiDI8XaEIfQMAI5DjdHA kdBYQhD/6W4OYiVuEyf9Myq0ue8myyiGUVoqhpLG+mAA/m8nADcxM5VAlHYNsFLT7EpT 75KA== X-Gm-Message-State: AOJu0Yw8jLiCI8LywY38ys5bFoiNXcmc+nL8LoJQPzpHaaakKC22cJhg g/8QTnIF1MHPwSbTmPRQy7hVTKVsNfJv6yd6BXltaomJQZVehsa9dsQCSDGISkgV/u4ufY8BHHu XfEuQ X-Gm-Gg: ASbGnct6+kbWEslPMI3ztSgfs9pbN+6KkNB18WpjBjwYKZXNzm5aR329DeP7v5fLeOI Ohq7W9zGgO/e3OumMLMYuXTx3dWoRWpqi5aedTHpFi6ZecmOVMpHAF0xJiYIsB8FKOi75Yyk+yL 93i9lcQUdwj6fakoL2j6e+p5/zmO4MdnOeAhDUSvRA4BSO0N/n5hC/1cAXFETRSfty1fFbXDmui jly2I64rXNd4rKh/CY6zDobf7lwAe3lxzyWPJXkD5v2Mt1EEJNMJW3HYj3ZuNcbe4Y6fHh0LTWW 4DL/4SHcb0bq4H4ma+wmhKnZ7HxcXl2x0A5+CJaT+ujAy59nMguSOC5lZxcyWbbmWUcEFl3LkR+ HGNiK7g== X-Google-Smtp-Source: AGHT+IEoX/KjeRaXExa/3w7gA+EpGErMs+VzpzTLQtmc9m9rAVASeGnTn9sqFZWTMzEvm/Pl+fSyCQ== X-Received: by 2002:a05:600c:8b5b:b0:439:8c80:6aee with SMTP id 5b1f17b1804b1-45352401529mr37962045e9.4.1750327741498; Thu, 19 Jun 2025 03:09:01 -0700 (PDT) Received: from NAUSH-P-DELL.pitowers.org ([93.93.133.154]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4535ebcecb5sm23926555e9.37.2025.06.19.03.09.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Jun 2025 03:09:01 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Cc: kieran.bingham@ideasonboard.com, David Plowman , Naushir Patuck , Laurent Pinchart Subject: [PATCH 1/4] controls: Add FrameWallClock control Date: Thu, 19 Jun 2025 11:05:53 +0100 Message-ID: <20250619100857.124809-2-naush@raspberrypi.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250619100857.124809-1-naush@raspberrypi.com> References: <20250619100857.124809-1-naush@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: David Plowman Add a FrameWallClock control that reports the same moment as the frame's SensorTimestamp, but in wallclock units. Signed-off-by: David Plowman Reviewed-by: Kieran Bingham Reviewed-by: Naushir Patuck Reviewed-by: Laurent Pinchart Signed-off-by: Naushir Patuck --- src/libcamera/control_ids_core.yaml | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml index aa7448645880..028919ef3d3e 100644 --- a/src/libcamera/control_ids_core.yaml +++ b/src/libcamera/control_ids_core.yaml @@ -1,6 +1,4 @@ -# SPDX-License-Identifier: LGPL-2.1-or-later -# -# Copyright (C) 2019, Google Inc. + # %YAML 1.1 --- @@ -1268,4 +1266,19 @@ controls: description: | Enable or disable the debug metadata. + - FrameWallClock: + type: int64_t + direction: out + description: | + This timestamp corresponds to the same moment in time as the + SensorTimestamp, but is represented as a wall clock time as measured by + the CLOCK_REALTIME clock. + + Being a wall clock measurement, it can be used to synchronise timing + across different devices. + + \sa SensorTimestamp + + The FrameWallClock control can only be returned in metadata. + ... From patchwork Thu Jun 19 10:05:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 23603 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 50C4BC3237 for ; Thu, 19 Jun 2025 10:09:10 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C0BE268DC6; Thu, 19 Jun 2025 12:09:06 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="LK4yHjBY"; dkim-atps=neutral Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id EDB5968DC6 for ; Thu, 19 Jun 2025 12:09:02 +0200 (CEST) Received: by mail-wm1-x32e.google.com with SMTP id 5b1f17b1804b1-450ddb35583so534915e9.1 for ; Thu, 19 Jun 2025 03:09:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1750327742; x=1750932542; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LpSvAHfLJ3s+U22kjYDMqV0WiShYD8LM29GMSrXDETg=; b=LK4yHjBYtNdg/2i6fv2SpyBcjtb0j7HXculovw3BAYYwqG6EL81YK/EPHdAwRgfRmT sxQWExxde7O1iVGaBJbuqwhrkJoWLK5+MjiiWUfcjlD7iOZdsJQjV2JvY8rCX4wlfmJd C9FS2aGFiIH5l/DMBd42/IfBxeHUeat4bUbx35PRkbu1JEjnzz2f3+dNbcrSLBnhnYBE 4SFzct4B7iAMo/4rbeR/Vlyq5TNTu+dsuJPn9LnoDN/q0fyvGSIaRs/c/SuKka1wq86W ClEpWzI05tjKQBEu3/ioKNnB8XbIN5p1pwbMoeyZX8t5bfBQFQxRkmXPCGh+wZ/Qjr1c /R1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750327742; x=1750932542; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LpSvAHfLJ3s+U22kjYDMqV0WiShYD8LM29GMSrXDETg=; b=CSl4cODw4p9V8plpobQLMfER1Gqw0o9k2h5Ly9GqQquNNasTUGFpvaeYppSFv5zYvZ cZ/mwbQu9F3X/svrWeNOCP8BT3D1REy8vFJQRMijQ+Rn64QGA+yvThQEyTZ54NA/zQFE UpXBZTDpD5j6ZWK1G/CTZaIvnzfpuVPnQhUCIDHkMH3VkWn9MvLxQVPmXNOo7NJ05tMs Vxc+N6RE3en5h22vUm1j9+6C5pKb76rkCvkLZJd8KFc/vHaRa3jSe5nSnsT82IiEjNfx I/SaDvhV9QnUkQjleIXHKPFr/NPH7iofJOKnA8TnhweniUZx1nggbSpwId5wt/EJiuRV mX0g== X-Gm-Message-State: AOJu0YxWl8GU95Q73k0B1Z0nDVT2FUjvVE8m2CyLFP6Skxv+0OM/3czT 6GGrxLxdGR/5xrf5AgXjnmxwGgxzpjMAQ8FKsjSH0vTPfg073+dLwqQnZ82Nyq40nmxSnnPUqlV bASDE X-Gm-Gg: ASbGnctuxPIvnF4pmj+i92cq3duovRLd03Nu+1GtPtPVj1LZcvCQAhsqgTezDvQKC5q mtaKwaPiw+fqXUIOFgs6zGFGjCg0KBHCiqNgBu08YWxuBwozBwBpzZ4YrxgB2RjJtM8AR86OL6c O78wVAo1VKY4QVZ209Nc2rD/itCFthjBV1Jp39ukQxuOKJ4g2L24uaj/lhe0FeLeGGLE4WFswSE /0mPM5JbYjZv0nkvyGV3z5mfhoC6PeAUw/3nKHOe99Cj5kH+gCOaO7+AFhvs2dBLvi+6K3cD/s9 qzh4sIRsKAI4Ytk9rqK06hp5KE4q5sFlBnv/yYCbBcBh8V3GJYhdMmfuxy2GwzhTrORYkosduWX Qxyek/Q== X-Google-Smtp-Source: AGHT+IHFlnDKgmvT9vjneCyg/TA3DOrvI/cwCPwmA+rGb91gd97oMReSGqXRRcxCre5RqlilBEWiRA== X-Received: by 2002:a05:600c:a402:b0:451:dee4:cd07 with SMTP id 5b1f17b1804b1-4535eee3c98mr7918565e9.0.1750327742109; Thu, 19 Jun 2025 03:09:02 -0700 (PDT) Received: from NAUSH-P-DELL.pitowers.org ([93.93.133.154]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4535ebcecb5sm23926555e9.37.2025.06.19.03.09.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Jun 2025 03:09:01 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Cc: kieran.bingham@ideasonboard.com, David Plowman , Naushir Patuck Subject: [PATCH 2/4] libcamera: Add ClockRecovery class to generate wallclock timestamps Date: Thu, 19 Jun 2025 11:05:54 +0100 Message-ID: <20250619100857.124809-3-naush@raspberrypi.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250619100857.124809-1-naush@raspberrypi.com> References: <20250619100857.124809-1-naush@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: David Plowman The ClockRecovery class takes pairs of timestamps from two different clocks, and models the second ("output") clock from the first ("input") clock. We can use it, in particular, to get a good wallclock estimate for a frame's SensorTimestamp. Signed-off-by: David Plowman Reviewed-by: Kieran Bingham Reviewed-by: Naushir Patuck Signed-off-by: Naushir Patuck --- include/libcamera/internal/clock_recovery.h | 68 ++++++ include/libcamera/internal/meson.build | 1 + src/libcamera/clock_recovery.cpp | 230 ++++++++++++++++++++ src/libcamera/meson.build | 1 + 4 files changed, 300 insertions(+) create mode 100644 include/libcamera/internal/clock_recovery.h create mode 100644 src/libcamera/clock_recovery.cpp diff --git a/include/libcamera/internal/clock_recovery.h b/include/libcamera/internal/clock_recovery.h new file mode 100644 index 000000000000..43e46b7dcf9f --- /dev/null +++ b/include/libcamera/internal/clock_recovery.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * Camera recovery algorithm + */ +#pragma once + +#include + +namespace libcamera { + +class ClockRecovery +{ +public: + ClockRecovery(); + + void configure(unsigned int numSamples = 100, unsigned int maxJitter = 2000, + unsigned int minSamples = 10, unsigned int errorThreshold = 50000); + void reset(); + + void addSample(); + void addSample(uint64_t input, uint64_t output); + + uint64_t getOutput(uint64_t input); + +private: + /* Approximate number of samples over which the model state persists. */ + unsigned int numSamples_; + /* Remove any output jitter larger than this immediately. */ + unsigned int maxJitter_; + /* Number of samples required before we start to use model estimates. */ + unsigned int minSamples_; + /* Threshold above which we assume the wallclock has been reset. */ + unsigned int errorThreshold_; + + /* How many samples seen (up to numSamples_). */ + unsigned int count_; + /* This gets subtracted from all input values, just to make the numbers easier. */ + uint64_t inputBase_; + /* As above, for the output. */ + uint64_t outputBase_; + /* The previous input sample. */ + uint64_t lastInput_; + /* The previous output sample. */ + uint64_t lastOutput_; + + /* Average x value seen so far. */ + double xAve_; + /* Average y value seen so far */ + double yAve_; + /* Average x^2 value seen so far. */ + double x2Ave_; + /* Average x*y value seen so far. */ + double xyAve_; + + /* + * The latest estimate of linear parameters to derive the output clock + * from the input. + */ + double slope_; + double offset_; + + /* Use this cumulative error to monitor for spontaneous clock updates. */ + double error_; +}; + +} /* namespace libcamera */ diff --git a/include/libcamera/internal/meson.build b/include/libcamera/internal/meson.build index 33f318b2b602..5c80a28c4cbe 100644 --- a/include/libcamera/internal/meson.build +++ b/include/libcamera/internal/meson.build @@ -11,6 +11,7 @@ libcamera_internal_headers = files([ 'camera_manager.h', 'camera_sensor.h', 'camera_sensor_properties.h', + 'clock_recovery.h', 'control_serializer.h', 'control_validator.h', 'converter.h', diff --git a/src/libcamera/clock_recovery.cpp b/src/libcamera/clock_recovery.cpp new file mode 100644 index 000000000000..abacf444fbf8 --- /dev/null +++ b/src/libcamera/clock_recovery.cpp @@ -0,0 +1,230 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * Clock recovery algorithm + */ + +#include "libcamera/internal/clock_recovery.h" + +#include + +#include + +/** + * \file clock_recovery.h + * \brief Clock recovery - deriving one clock from another independent clock + */ + +namespace libcamera { + +LOG_DEFINE_CATEGORY(ClockRec) + +/** + * \class ClockRecovery + * \brief Recover an output clock from an input clock + * + * The ClockRecovery class derives an output clock from an input clock, + * modelling the output clock as being linearly related to the input clock. + * For example, we may use it to derive wall clock timestamps from timestamps + * measured by the internal system clock which counts local time since boot. + * + * When pairs of corresponding input and output timestamps are available, + * they should be submitted to the model with addSample(). The model will + * update, and output clock values for known input clock values can be + * obtained using getOutput(). + * + * As a convenience, if the input clock is indeed the time since boot, and the + * output clock represents a real wallclock time, then addSample() can be + * called with no arguments, and a pair of timestamps will be captured at + * that moment. + * + * The configure() function accepts some configuration parameters to control + * the linear fitting process. + */ + +/** + * \brief Construct a ClockRecovery + */ +ClockRecovery::ClockRecovery() +{ + configure(); + reset(); +} + +/** + * \brief Set configuration parameters + * \param[in] numSamples The approximate duration for which the state of the model + * is persistent + * \param[in] maxJitter New output samples are clamped to no more than this + * amount of jitter, to prevent sudden swings from having a large effect + * \param[in] minSamples The fitted clock model is not used to generate outputs + * until this many samples have been received + * \param[in] errorThreshold If the accumulated differences between input and + * output clocks reaches this amount over a few frames, the model is reset + */ +void ClockRecovery::configure(unsigned int numSamples, unsigned int maxJitter, + unsigned int minSamples, unsigned int errorThreshold) +{ + LOG(ClockRec, Debug) + << "configure " << numSamples << " " << maxJitter << " " << minSamples << " " << errorThreshold; + + numSamples_ = numSamples; + maxJitter_ = maxJitter; + minSamples_ = minSamples; + errorThreshold_ = errorThreshold; +} + +/** + * \brief Reset the clock recovery model and start again from scratch + */ +void ClockRecovery::reset() +{ + LOG(ClockRec, Debug) << "reset"; + + lastInput_ = 0; + lastOutput_ = 0; + xAve_ = 0; + yAve_ = 0; + x2Ave_ = 0; + xyAve_ = 0; + count_ = 0; + error_ = 0.0; + /* + * Setting slope_ and offset_ to zero initially means that the clocks + * advance at exactly the same rate. + */ + slope_ = 0.0; + offset_ = 0.0; +} + +/** + * \brief Add a sample point to the clock recovery model, for recovering a wall + * clock value from the internal system time since boot + * + * This is a convenience function to make it easy to derive a wall clock value + * (using the Linux CLOCK_REALTIME) from the time since the system started + * (measured by CLOCK_BOOTTIME). + */ +void ClockRecovery::addSample() +{ + LOG(ClockRec, Debug) << "addSample"; + + struct timespec bootTime1; + struct timespec bootTime2; + struct timespec wallTime; + + /* Get boot and wall clocks in microseconds. */ + clock_gettime(CLOCK_BOOTTIME, &bootTime1); + clock_gettime(CLOCK_REALTIME, &wallTime); + clock_gettime(CLOCK_BOOTTIME, &bootTime2); + uint64_t boot1 = bootTime1.tv_sec * 1000000ULL + bootTime1.tv_nsec / 1000; + uint64_t boot2 = bootTime2.tv_sec * 1000000ULL + bootTime2.tv_nsec / 1000; + uint64_t boot = (boot1 + boot2) / 2; + uint64_t wall = wallTime.tv_sec * 1000000ULL + wallTime.tv_nsec / 1000; + + addSample(boot, wall); +} + +/** + * \brief Add a sample point to the clock recovery model, specifying the exact + * input and output clock values + * \param[in] input The input clock value + * \param[in] output The value of the output clock at the same moment, as far + * as possible, that the input clock was sampled + * + * This function should be used for corresponding clocks other than the Linux + * BOOTTIME and REALTIME clocks. + */ +void ClockRecovery::addSample(uint64_t input, uint64_t output) +{ + LOG(ClockRec, Debug) << "addSample " << input << " " << output; + + if (count_ == 0) { + inputBase_ = input; + outputBase_ = output; + } + + /* + * We keep an eye on cumulative drift over the last several frames. If this exceeds a + * threshold, then probably the system clock has been updated and we're going to have to + * reset everything and start over. + */ + if (lastOutput_) { + int64_t inputDiff = getOutput(input) - getOutput(lastInput_); + int64_t outputDiff = output - lastOutput_; + error_ = error_ * 0.95 + (outputDiff - inputDiff); + if (std::abs(error_) > errorThreshold_) { + reset(); + inputBase_ = input; + outputBase_ = output; + } + } + lastInput_ = input; + lastOutput_ = output; + + /* + * Never let the new output value be more than maxJitter_ away from what + * we would have expected. This is just to reduce the effect of sudden + * large delays in the measured output. + */ + uint64_t expectedOutput = getOutput(input); + output = std::clamp(output, expectedOutput - maxJitter_, expectedOutput + maxJitter_); + + /* + * We use x, y, x^2 and x*y sums to calculate the best fit line. Here we + * update them by pretending we have count_ samples at the previous fit, + * and now one new one. Gradually the effect of the older values gets + * lost. This is a very simple way of updating the fit (there are much + * more complicated ones!), but it works well enough. Using averages + * instead of sums makes the relative effect of old values and the new + * sample clearer. + */ + double x = static_cast(input - inputBase_); + double y = static_cast(output - outputBase_) - x; + unsigned int count1 = count_ + 1; + xAve_ = (count_ * xAve_ + x) / count1; + yAve_ = (count_ * yAve_ + y) / count1; + x2Ave_ = (count_ * x2Ave_ + x * x) / count1; + xyAve_ = (count_ * xyAve_ + x * y) / count1; + + /* + * Don't update slope and offset until we've seen "enough" sample + * points. Note that the initial settings for slope_ and offset_ + * ensures that the wallclock advances at the same rate as the realtime + * clock (but with their respective initial offsets). + */ + if (count_ > minSamples_) { + /* These are the standard equations for least squares linear regression. */ + slope_ = (count1 * count1 * xyAve_ - count1 * xAve_ * count1 * yAve_) / + (count1 * count1 * x2Ave_ - count1 * xAve_ * count1 * xAve_); + offset_ = yAve_ - slope_ * xAve_; + } + + /* + * Don't increase count_ above numSamples_, as this controls the long-term + * amount of the residual fit. + */ + if (count1 < numSamples_) + count_++; +} + +/** + * \brief Calculate the output clock value according to the model from an input + * clock value + * \param[in] input The input clock value + * + * \return Output clock value + */ +uint64_t ClockRecovery::getOutput(uint64_t input) +{ + double x = static_cast(input - inputBase_); + double y = slope_ * x + offset_; + uint64_t output = y + x + outputBase_; + + LOG(ClockRec, Debug) << "getOutput " << input << " " << output; + + return output; +} + +} /* namespace libcamera */ diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build index 202db1efe844..28a3b0f36d53 100644 --- a/src/libcamera/meson.build +++ b/src/libcamera/meson.build @@ -21,6 +21,7 @@ libcamera_internal_sources = files([ 'byte_stream_buffer.cpp', 'camera_controls.cpp', 'camera_lens.cpp', + 'clock_recovery.cpp', 'control_serializer.cpp', 'control_validator.cpp', 'converter.cpp', From patchwork Thu Jun 19 10:05:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 23604 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1214AC3237 for ; Thu, 19 Jun 2025 10:09:12 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5C18968DF0; Thu, 19 Jun 2025 12:09:07 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="rFII36hj"; dkim-atps=neutral Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7D65E68DC6 for ; Thu, 19 Jun 2025 12:09:03 +0200 (CEST) Received: by mail-wr1-x42a.google.com with SMTP id ffacd0b85a97d-3a52878d37aso106284f8f.2 for ; Thu, 19 Jun 2025 03:09:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1750327743; x=1750932543; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VZqGcoOUmsa4WYZ1V8vAzxzJY09zdLnQpU1CTW+aVRU=; b=rFII36hjAYf+OKlst3sjnVkEx6ifCOz3iY6NSYItDSRySEM4l4uwWSswR3KESglsa5 w32jYtLw+QPiFJy9XLLeos1eBDs8Yo7eBQmQpT5GHX1AiVYiIR7JeHieAUi7Lgnhvnrj O1Jn90zBWd0Ew7FUTMNGMFlYCikxGV9fSXpXQuSlci7b0XOl8CqTDtF1Yr2/27tFcabL 2F2QnILAuLnuVlBYmjs/G5KBRkMxGA/4BL4tzlDLHkyhixnQEevAM7j0wfWU0c6Xz9jp YGKkTbMaKk3CWjEp2yOs3UYqk79lGEvbRcRH67z0UWZcXloAyhrBAKluUDnmyrLSF85U I+qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750327743; x=1750932543; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VZqGcoOUmsa4WYZ1V8vAzxzJY09zdLnQpU1CTW+aVRU=; b=f6hj4HlQDhI7TJMGYy5WQeoToWiWZvgk3cMdAokU3gvpbL3Kyqwq+XkuvWMMhS6nwQ 1edCg+Z9QvWxcIgLG3IEwLbRH1t3JLw63SS0wRCFwi4aDSyHS2t+Gw/MWH7dJ5Gv8yBL AcFmWQjvKT42L0L25rP6j2WrRuN/9zimXZppoyIBTYWf7GWhl+QTGiHFa2q4fMy/RLMA S2cC4J+NSMWBsuo2w2AHmNGtXx+JX88WQBcyyHtQ6DZrQcEzdB3/1IXZ9etjElVNkfc4 E2Iups7T0KaUUmLa/ITXl961SEf3YH0DnWH3BYa2t7uEzltpfEQZh1iG4Vl3ue5T/9/+ +rzg== X-Gm-Message-State: AOJu0YxrOegZWDnW9s6uVxy0k7LL3Cte/V8q8oW3EmQv3T3JfcD6WC4i a3FTCH/rexmqp/F4GHZRg8GrT8nT8pDUE/sXBKkXF8qJPPMX2Gn1m+8xSQa/pw0zuub2iTTzR6F XI9GU X-Gm-Gg: ASbGncsb2sksO26B93+G6xaJeLSRpLxk2iCR6XhNW21fA5NJFet1p8UBiniGOKPljEz wcGf7CeXCQ7DEJ34/r2RZAXZ9YJ8hpF4NnDStqfimGFq5vZVeJnsRlz3lps/5uniZtmJcFITbHU 3zrqxI9WiUJ8ufLkjeHCdmHg0pNo+OsFl52G74ueRRdSk+SHEk/dWTwODrxvL3bCY08eDN/JnjG haV/eVWt6C+kNxN/KzgPtIsuijN6vzMrf22Khh6nOYc9MFYFVuTZmDWhMtqDEWFaSaZUtcueSxC UwNUbB6EugjX4u1rjxKdkCkur8qM8vwOd6zwadh+tLNg86u5fH9eLTTJ5f4fqfmGDDtxWAWCV9L Ztj5Ssg== X-Google-Smtp-Source: AGHT+IHbi7QHunVNcRrGYVkHjsesT6nScoXaHyPitcmFHcI8uRd+G+lhGL2rISJ/nkTiBxZJCGvaKg== X-Received: by 2002:a05:6000:2002:b0:3a3:61ab:86c2 with SMTP id ffacd0b85a97d-3a6c98fbfc7mr836626f8f.7.1750327742709; Thu, 19 Jun 2025 03:09:02 -0700 (PDT) Received: from NAUSH-P-DELL.pitowers.org ([93.93.133.154]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4535ebcecb5sm23926555e9.37.2025.06.19.03.09.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Jun 2025 03:09:02 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Cc: kieran.bingham@ideasonboard.com, David Plowman , Naushir Patuck Subject: [PATCH 3/4] controls: Add camera synchronisation controls for Raspberry Pi Date: Thu, 19 Jun 2025 11:05:55 +0100 Message-ID: <20250619100857.124809-4-naush@raspberrypi.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250619100857.124809-1-naush@raspberrypi.com> References: <20250619100857.124809-1-naush@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: David Plowman New controls are added to control the camera "sync" algorithm, which allows different cameras to synchronise their frames. For the time being, the controls are Raspberry Pi specific, though this is expected to change in future. Signed-off-by: David Plowman Reviewed-by: Naushir Patuck Signed-off-by: Naushir Patuck --- src/libcamera/control_ids_rpi.yaml | 112 +++++++++++++++++++++++++++++ 1 file changed, 112 insertions(+) diff --git a/src/libcamera/control_ids_rpi.yaml b/src/libcamera/control_ids_rpi.yaml index 8d1e8b47c821..a861511233b6 100644 --- a/src/libcamera/control_ids_rpi.yaml +++ b/src/libcamera/control_ids_rpi.yaml @@ -71,4 +71,116 @@ controls: \sa StatsOutputEnable + - SyncMode: + type: int32_t + direction: in + description: | + Enable or disable camera synchronisation ("sync") mode. + + When sync mode is enabled, a camera will synchronise frames temporally + with other cameras, either attached to the same device or a different + one. There should be one "server" device, which broadcasts timing + information to one or more "clients". Communication is one-way, from + server to clients only, and it is only clients that adjust their frame + timings to match the server. + + Sync mode requires all cameras to be running at (as far as possible) the + same fixed framerate. Clients may continue to make adjustments to keep + their cameras synchronised with the server for the duration of the + session, though any updates after the initial ones should remain small. + + \sa SyncReady + \sa SyncTimer + \sa SyncFrames + + enum: + - name: SyncModeOff + value: 0 + description: Disable sync mode. + - name: SyncModeServer + value: 1 + description: | + Enable sync mode, act as server. The server broadcasts timing + messages to any clients that are listening, so that the clients can + synchronise their camera frames with the server's. + - name: SyncModeClient + value: 2 + description: | + Enable sync mode, act as client. A client listens for any server + messages, and arranges for its camera frames to synchronise as + closely as possible with the server's. Many clients can listen out + for the same server. Clients can also be started ahead of any + servers, causing them merely to wait for the server to start. + + - SyncReady: + type: bool + direction: out + description: | + When using the camera synchronisation algorithm, the server broadcasts + timing information to the clients. This also includes the time (some + number of frames in the future, called the "ready time") at which the + server will signal its controlling application, using this control, to + start using the image frames. + + The client receives the "ready time" from the server, and will signal + its application to start using the frames at this same moment. + + While this control value is false, applications (on both client and + server) should continue to wait, and not use the frames. + + Once this value becomes true, it means that this is the first frame + where the server and its clients have agreed that they will both be + synchronised and that applications should begin consuming frames. + Thereafter, this control will continue to signal the value true for + the rest of the session. + + \sa SyncMode + \sa SyncTimer + \sa SyncFrames + + - SyncTimer: + type: int64_t + direction: out + description: | + This reports the amount of time, in microseconds, until the "ready + time", at which the server and client will signal their controlling + applications that the frames are now synchronised and should be + used. The value may be refined slightly over time, becoming more precise + as the "ready time" approaches. + + Servers always report this value, whereas clients will omit this control + until they have received a message from the server that enables them to + calculate it. + + Normally the value will start positive (the "ready time" is in the + future), and decrease towards zero, before becoming negative (the "ready + time" has elapsed). So there should be just one frame where the timer + value is, or is very close to, zero - the one for which the SyncReady + control becomes true. At this moment, the value indicates how closely + synchronised the client believes it is with the server. + + But note that if frames are being dropped, then the "near zero" valued + frame, or indeed any other, could be skipped. In these cases the timer + value allows an application to deduce that this has happened. + + \sa SyncMode + \sa SyncReady + \sa SyncFrames + + - SyncFrames: + type: int32_t + direction: in + description: | + The number of frames the server should wait, after enabling + SyncModeServer, before signalling (via the SyncReady control) that + frames should be used. This therefore determines the "ready time" for + all synchronised cameras. + + This control value should be set only for the device that is to act as + the server, before or at the same moment at which SyncModeServer is + enabled. + + \sa SyncMode + \sa SyncReady + \sa SyncTimer ... From patchwork Thu Jun 19 10:05:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naushir Patuck X-Patchwork-Id: 23605 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 02661C3240 for ; Thu, 19 Jun 2025 10:09:13 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 495D268DEB; Thu, 19 Jun 2025 12:09:10 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="QCAiWwYw"; dkim-atps=neutral Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 48A3368DEB for ; Thu, 19 Jun 2025 12:09:04 +0200 (CEST) Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-4519dd6523dso338225e9.1 for ; Thu, 19 Jun 2025 03:09:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1750327743; x=1750932543; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UsJG0sUjwmdzpZDVOl09h+1erDYt2a8aPHUb1k2opsw=; b=QCAiWwYwu9D+PTr80zQCcp1mhpNPW+Dj1ZpbJ5XlxTK2+7T8RJlLaQCqZnUB9Jxqb5 JLjCyw19h65ZCfDP2qJ+mKrJOefknlN6Bf9EZu6x44Umc7w4RnX3wOrwEByFAXMDw5nL 4e0Q/74VQX2qNjObRq+szCwmK1KdOLxMTML/v+k0aCFdXag0ttZ/PMpApVj7o4OAZb4N 5CH3tY5kdqfwobqmshb7BiyfA1nJMPFbHMOz5sBWy1f0HGXDew3vN2gabrM5EJJ6Yp8+ /zTxp2yA4uLNdSmn6UkozjUkoyTA0eCHGqslV61JHH9oTTn4wDnjYeGsCw7EQNd1X9ye POpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750327743; x=1750932543; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UsJG0sUjwmdzpZDVOl09h+1erDYt2a8aPHUb1k2opsw=; b=QDD86eQQ4Sx1WmUVTOVm50gmUSoMR3qGGvLgG/mjHQWgnVUZexW9BjEPEGNogVwtUh tEFs2kzkd9WDfPGVuD+ixA7df0nvQn8OaWC2lrCsqx4dbxsVefX0lxTSSuXjqedhBu+U UVcOMKgXXimQ3QL4rTJx8ttYMJ9mqJs1SlhNw4xjEd1ZoTwNi0sz1G/o8ra4odoqjhhH BXlLgdHz5tUOs6hiTIp1oGlEk7HuyrDH9HxkEoX25zIphyBk5hZa4IJW993OlSRblf5M Swp8sHq3lXERZl4WpaflazkmjB4HFTLan7DmnvLQ3UzuvblttAOODgGi8rcAST1lsQVT HCBw== X-Gm-Message-State: AOJu0Yw0WhQBLQPT0JnKcclFhpej08CJaRvPrpv8zYNCdpovNsijpaWu JZrxDwodHkuXEN0/3I6KFobaWXTSF6i5rdLHw3qH24j5FwnwoQFgQK2v49pvjDWnpWMPE+U1x16 Evw6J X-Gm-Gg: ASbGncv7k9fz54HQYb/B2Leh44c0ACN8qbPRCEPdPIbZpZZRnXua7DrjbwXu05K60rj qyIEbj5zWpOT90+L7n1Vh8yuJLQN1/MN/7us5Nl+NYPZ+yIcTnMT5c8oG4s+8h9OF/f8s9rRZbu LgP0p/wse8AJBC6f2oIoQWRW1eLO1CoMbm+AZLRty/yjOVZoySNqXvNrZScXGUXfKbYxx98xLcW djbrn+BV3ZtE+UV4/LYRAE2T0sVCkuN9TK/wmVAJttNuF6rLQT2K23yx6TbEoGqdnX0JcSvPhh2 AP0DA6uhY8VprazNoC60OLIR7NA1TNrGibOXTFxmh8aIR/b07eEsPE7GaKzjwOFH95bCz5nQ1N7 CPL5B3A== X-Google-Smtp-Source: AGHT+IHgI0VtbGGbbHClENwshKCGdK1RN+jS9GJWJ7LFEJt/F6GFyT7tKbxuPjOg+H+e/zVa8yio0w== X-Received: by 2002:a05:600c:b95:b0:43e:94fa:4aef with SMTP id 5b1f17b1804b1-4535efa1e0cmr8297005e9.8.1750327743248; Thu, 19 Jun 2025 03:09:03 -0700 (PDT) Received: from NAUSH-P-DELL.pitowers.org ([93.93.133.154]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4535ebcecb5sm23926555e9.37.2025.06.19.03.09.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Jun 2025 03:09:02 -0700 (PDT) From: Naushir Patuck To: libcamera-devel@lists.libcamera.org Cc: kieran.bingham@ideasonboard.com, David Plowman , Naushir Patuck Subject: [PATCH 4/4] pipeline: rpi: Add wallclock timestamp support Date: Thu, 19 Jun 2025 11:05:56 +0100 Message-ID: <20250619100857.124809-5-naush@raspberrypi.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250619100857.124809-1-naush@raspberrypi.com> References: <20250619100857.124809-1-naush@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: David Plowman A ClockRecovery object is added for derived classes to use, and wallclock timestamps are copied into the request metadata for applications. Wallclock timestamps are derived corresponding to the sensor timestamp, and made available to the base pipeline handler class and to IPAs, for both vc4 and pisp platforms. Signed-off-by: David Plowman Reviewed-by: Kieran Bingham Reviewed-by: Naushir Patuck Signed-off-by: Naushir Patuck --- src/libcamera/pipeline/rpi/common/pipeline_base.cpp | 5 +++++ src/libcamera/pipeline/rpi/common/pipeline_base.h | 3 +++ src/libcamera/pipeline/rpi/pisp/pisp.cpp | 10 ++++++++-- src/libcamera/pipeline/rpi/vc4/vc4.cpp | 10 ++++++++-- 4 files changed, 24 insertions(+), 4 deletions(-) diff --git a/src/libcamera/pipeline/rpi/common/pipeline_base.cpp b/src/libcamera/pipeline/rpi/common/pipeline_base.cpp index d8c7ca9352de..e14d3b913aaa 100644 --- a/src/libcamera/pipeline/rpi/common/pipeline_base.cpp +++ b/src/libcamera/pipeline/rpi/common/pipeline_base.cpp @@ -685,6 +685,9 @@ int PipelineHandlerBase::start(Camera *camera, const ControlList *controls) return ret; } + /* A good moment to add an initial clock sample. */ + data->wallClockRecovery_.addSample(); + /* * Reset the delayed controls with the gain and exposure values set by * the IPA. @@ -1482,6 +1485,8 @@ void CameraData::fillRequestMetadata(const ControlList &bufferControls, Request { request->metadata().set(controls::SensorTimestamp, bufferControls.get(controls::SensorTimestamp).value_or(0)); + request->metadata().set(controls::FrameWallClock, + bufferControls.get(controls::FrameWallClock).value_or(0)); if (cropParams_.size()) { std::vector crops; diff --git a/src/libcamera/pipeline/rpi/common/pipeline_base.h b/src/libcamera/pipeline/rpi/common/pipeline_base.h index 898f31577059..4c5743e04f86 100644 --- a/src/libcamera/pipeline/rpi/common/pipeline_base.h +++ b/src/libcamera/pipeline/rpi/common/pipeline_base.h @@ -20,6 +20,7 @@ #include "libcamera/internal/bayer_format.h" #include "libcamera/internal/camera.h" #include "libcamera/internal/camera_sensor.h" +#include "libcamera/internal/clock_recovery.h" #include "libcamera/internal/framebuffer.h" #include "libcamera/internal/media_device.h" #include "libcamera/internal/media_object.h" @@ -172,6 +173,8 @@ public: Config config_; + ClockRecovery wallClockRecovery_; + protected: void fillRequestMetadata(const ControlList &bufferControls, Request *request); diff --git a/src/libcamera/pipeline/rpi/pisp/pisp.cpp b/src/libcamera/pipeline/rpi/pisp/pisp.cpp index ccf135c3d8ce..2df91bacf3be 100644 --- a/src/libcamera/pipeline/rpi/pisp/pisp.cpp +++ b/src/libcamera/pipeline/rpi/pisp/pisp.cpp @@ -1755,9 +1755,15 @@ void PiSPCameraData::cfeBufferDequeue(FrameBuffer *buffer) auto [ctrl, delayContext] = delayedCtrls_->get(buffer->metadata().sequence); /* * Add the frame timestamp to the ControlList for the IPA to use - * as it does not receive the FrameBuffer object. + * as it does not receive the FrameBuffer object. Also derive a + * corresponding wallclock value. */ - ctrl.set(controls::SensorTimestamp, buffer->metadata().timestamp); + wallClockRecovery_.addSample(); + uint64_t sensorTimestamp = buffer->metadata().timestamp; + uint64_t wallClockTimestamp = wallClockRecovery_.getOutput(sensorTimestamp / 1000); + + ctrl.set(controls::SensorTimestamp, sensorTimestamp); + ctrl.set(controls::FrameWallClock, wallClockTimestamp); job.sensorControls = std::move(ctrl); job.delayContext = delayContext; } else if (stream == &cfe_[Cfe::Config]) { diff --git a/src/libcamera/pipeline/rpi/vc4/vc4.cpp b/src/libcamera/pipeline/rpi/vc4/vc4.cpp index ac6dab814d35..e99a7edf809c 100644 --- a/src/libcamera/pipeline/rpi/vc4/vc4.cpp +++ b/src/libcamera/pipeline/rpi/vc4/vc4.cpp @@ -773,9 +773,15 @@ void Vc4CameraData::unicamBufferDequeue(FrameBuffer *buffer) auto [ctrl, delayContext] = delayedCtrls_->get(buffer->metadata().sequence); /* * Add the frame timestamp to the ControlList for the IPA to use - * as it does not receive the FrameBuffer object. + * as it does not receive the FrameBuffer object. Also derive a + * corresponding wallclock value. */ - ctrl.set(controls::SensorTimestamp, buffer->metadata().timestamp); + wallClockRecovery_.addSample(); + uint64_t sensorTimestamp = buffer->metadata().timestamp; + uint64_t wallClockTimestamp = wallClockRecovery_.getOutput(sensorTimestamp / 1000); + + ctrl.set(controls::SensorTimestamp, sensorTimestamp); + ctrl.set(controls::FrameWallClock, wallClockTimestamp); bayerQueue_.push({ buffer, std::move(ctrl), delayContext }); } else { embeddedQueue_.push(buffer);