{"id":22098,"url":"https://patchwork.libcamera.org/api/1.1/patches/22098/?format=json","web_url":"https://patchwork.libcamera.org/patch/22098/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20241126121706.4350-3-david.plowman@raspberrypi.com>","date":"2024-11-26T12:17:05","name":"[RFC,2/3] libcamera: clock: Add ClockRecovery class to help generate wallclock timestamps","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"10421797d732b89b09e9f2c78436970d4876f768","submitter":{"id":42,"url":"https://patchwork.libcamera.org/api/1.1/people/42/?format=json","name":"David Plowman","email":"david.plowman@raspberrypi.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/22098/mbox/","series":[{"id":4822,"url":"https://patchwork.libcamera.org/api/1.1/series/4822/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=4822","date":"2024-11-26T12:17:03","name":"Frame wallclock timestamps and metadata","version":1,"mbox":"https://patchwork.libcamera.org/series/4822/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/22098/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/22098/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id DB681C0DA4\n\tfor ;\n\tTue, 26 Nov 2024 12:17:18 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id ECC636606D;\n\tTue, 26 Nov 2024 13:17:15 +0100 (CET)","from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com\n\t[IPv6:2a00:1450:4864:20::32e])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 5648C65FBA\n\tfor ;\n\tTue, 26 Nov 2024 13:17:11 +0100 (CET)","by mail-wm1-x32e.google.com with SMTP id\n\t5b1f17b1804b1-4315e9e9642so51086395e9.0\n\tfor ;\n\tTue, 26 Nov 2024 04:17:11 -0800 (PST)","from raspberrypi.pitowers.org\n\t([2a00:1098:3142:1f:c68a:6be1:5ba3:eddd])\n\tby smtp.gmail.com with ESMTPSA id\n\t5b1f17b1804b1-434a15d86a4sm51070325e9.36.2024.11.26.04.17.10\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tTue, 26 Nov 2024 04:17:10 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=raspberrypi.com header.i=@raspberrypi.com\n\theader.b=\"RYEupJZA\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=raspberrypi.com; s=google; t=1732623431; x=1733228231;\n\tdarn=lists.libcamera.org; \n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:cc:to:from:from:to:cc:subject:date\n\t:message-id:reply-to;\n\tbh=W9JLKZInM78lcIWFvLsDR1/OYxWXBHffYFHvcJoTyBc=;\n\tb=RYEupJZAuwms8/j7eRWBVKWHsECiJ9fFQMBqZIBWfAtZKRRi/dDSiL29c5aUE/33j3\n\tlpXMGQfHNQ8ZXRXu9q0aHj6zsC/phZhooW+JVh3nBLMleOj7g1SZOahgiv7EONvgcK2J\n\tcF48/g3fZMxLbZ4G4lOFlpzbwzb0nfB/IcuBi+GniIhS1ZvjROj7qTwqjoembruZDBGH\n\tdQPY6QtkmQzAySDf7bpZx8tSyHwZ3mwnjPD0PN1FnCOrvUHXvFPPEMtIAsmVlIljrEDH\n\tkvnv9Yk6aSl05PIN19d5jHkwhoTESjZSksmXD+klC7VEQ0YE0DzYOFvcC2gdhItMdaQW\n\tRfcQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1732623431; x=1733228231;\n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc\n\t:subject:date:message-id:reply-to;\n\tbh=W9JLKZInM78lcIWFvLsDR1/OYxWXBHffYFHvcJoTyBc=;\n\tb=n2WIl1aL94GvHgpj486xM4EuSuzGEH6bfxWD6ZRt6h2xg5hpEuN9RRM8jbh18q7vXj\n\tJXsEDOuAWHsPTi4W2+YBCPh8lrw5msQoTonc03XvhqpvnhF1CMZRc7RS/whiBw4t0b/M\n\tYEQqoeMX5iGIe08EFy9JXLaIuP5WHqOwDr08AtUQPGdAqQ4vE6GH0luJv8uTci+Jg4O7\n\tEd/wXDsPLSC9Ucsg2wjKrflv/owc1muzNJ10f/hW6i/fs8GT2b32ZB4b5DuS+LzlZ/te\n\tsicq4xVlV2LGH/ccxcXswKZQJKi3nBu/54idisjn+dz2IM6xyhQqA76AaTYZth4uGSNe\n\tGtvg==","X-Gm-Message-State":"AOJu0YxFAldcQIQV9/aGtStLM1igR3SxGBzBsu+FeRU5utioh5ecd9mk\n\tW4qHMWxmlpBMtn5TiTB2ZxVP7urbifBS733TL+O8M/d7+YzcZf1vxoc0F3El30z1GSGQJGbR2sX\n\t8","X-Gm-Gg":"ASbGncueHchzolrWpFwcuN+DkVkOCavAlw+02Q/Hb/20zTStL43bhMYZuCtpRbDGTCY\n\tRNu02qA+O3hPNAp30jCS6rWI/gUOyMW0U/+A9N3U4QXSwV6FFOBy2spG6cBjwU43DQHbXofCr7I\n\td09F/CsyK1Wxd3XPXoA/zefdX84E2MhXhZCK6s7wcuagqswKtLf7ZoRI3V956qK1sj2LsU0euF9\n\tDPfvCoscTZual/+I/oSpPhWOepp0IQf3jSFZjNhsv2jwa3lrjEd7lqHT86ZvdzSE7FnpxUBTWi1\n\tP8+CJA==","X-Google-Smtp-Source":"AGHT+IEjkPgkaxXgjAINVcD1BiADVPpjUMrvQUtubDs2dS0VsKAZhOtK11KYiUikSFPWsf5tVVTTlA==","X-Received":"by 2002:a05:600c:5251:b0:434:a75b:5f6c with SMTP id\n\t5b1f17b1804b1-434a75b6173mr17365145e9.10.1732623430660; \n\tTue, 26 Nov 2024 04:17:10 -0800 (PST)","From":"David Plowman ","To":"libcamera-devel@lists.libcamera.org","Cc":"David Plowman ","Subject":"[RFC PATCH 2/3] libcamera: clock: Add ClockRecovery class to help\n\tgenerate wallclock timestamps","Date":"Tue, 26 Nov 2024 12:17:05 +0000","Message-Id":"<20241126121706.4350-3-david.plowman@raspberrypi.com>","X-Mailer":"git-send-email 2.39.5","In-Reply-To":"<20241126121706.4350-1-david.plowman@raspberrypi.com>","References":"<20241126121706.4350-1-david.plowman@raspberrypi.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Sampling the system clock is susceptible to many milliseconds of\njitter, dependent on system load and other factors.\n\nThe ClockRecovery class takes pairs of kernel timestamps (which\nexhitbit much less jitter) and wallclock readings, and returns a\nsmoother version of the wallclock timestamps.\n\nSigned-off-by: David Plowman \n---\n include/libcamera/internal/clock_recovery.h | 64 ++++++++++++\n include/libcamera/internal/meson.build | 1 +\n src/libcamera/clock_recovery.cpp | 110 ++++++++++++++++++++\n src/libcamera/meson.build | 1 +\n 4 files changed, 176 insertions(+)\n create mode 100644 include/libcamera/internal/clock_recovery.h\n create mode 100644 src/libcamera/clock_recovery.cpp","diff":"diff --git a/include/libcamera/internal/clock_recovery.h b/include/libcamera/internal/clock_recovery.h\nnew file mode 100644\nindex 00000000..49747747\n--- /dev/null\n+++ b/include/libcamera/internal/clock_recovery.h\n@@ -0,0 +1,64 @@\n+/* SPDX-License-Identifier: BSD-2-Clause */\n+/*\n+ * Copyright (C) 2024, Raspberry Pi Ltd\n+ *\n+ * Camera sync control algorithm\n+ */\n+#pragma once\n+\n+#include \n+\n+namespace libcamera {\n+\n+class ClockRecovery\n+{\n+public:\n+\tClockRecovery();\n+\n+\t/* Initialise with configuration parameters and restart the fitting process. */\n+\tvoid initialise(unsigned int numPts = 100, unsigned int maxJitter = 2000, unsigned int minPts = 10,\n+\t\t\tunsigned int errorThreshold = 50000);\n+\t/* Erase all history and restart the fitting process. */\n+\tvoid reset();\n+\n+\t// Add a new input clock / output clock sample. */\n+\tvoid addSample(uint64_t input, uint64_t output);\n+\t/* Calculate the output clock value for this input. */\n+\tuint64_t getOutput(uint64_t input);\n+\n+private:\n+\tunsigned int numPts_; /* how many samples contribute to the history */\n+\tunsigned int maxJitter_; /* smooth out any jitter larger than this immediately */\n+\tunsigned int minPts_; /* number of samples below which we treat clocks as 1:1 */\n+\tunsigned int errorThreshold_; /* reset everything when the error exceeds this */\n+\n+\tunsigned int count_; /* how many samples seen (up to numPts_) */\n+\tuint64_t inputBase_; /* subtract this from all input values, just to make the numbers easier */\n+\tuint64_t outputBase_; /* as above, for the output */\n+\n+\tuint64_t lastInput_; /* the previous input sample */\n+\tuint64_t lastOutput_; /* the previous output sample */\n+\n+\t/*\n+\t * We do a linear regression of y against x, where:\n+\t * x is the value input - inputBase_, and\n+\t * y is the value output - outputBase_ - x.\n+\t * We additionally subtract x from y so that y \"should\" be zero, again making the numnbers easier.\n+\t */\n+\tdouble xAve_; /* average x value seen so far */\n+\tdouble yAve_; /* average y value seen so far */\n+\tdouble x2Ave_; /* average x^2 value seen so far */\n+\tdouble xyAve_; /* average x*y value seen so far */\n+\n+\t/*\n+\t * Once we've seen more than minPts_ samples, we recalculate the slope and offset according\n+\t * to the linear regression normal equations.\n+\t */\n+\tdouble slope_; /* latest slope value */\n+\tdouble offset_; /* latest offset value */\n+\n+\t/* We use this cumulative error to monitor spontaneous system clock updates. */\n+\tdouble error_;\n+};\n+\n+} //namespace libcamera\ndiff --git a/include/libcamera/internal/meson.build b/include/libcamera/internal/meson.build\nindex 1dddcd50..b6271ee1 100644\n--- a/include/libcamera/internal/meson.build\n+++ b/include/libcamera/internal/meson.build\n@@ -11,6 +11,7 @@ libcamera_internal_headers = files([\n 'camera_manager.h',\n 'camera_sensor.h',\n 'camera_sensor_properties.h',\n+ 'clock_recovery.h',\n 'control_serializer.h',\n 'control_validator.h',\n 'converter.h',\ndiff --git a/src/libcamera/clock_recovery.cpp b/src/libcamera/clock_recovery.cpp\nnew file mode 100644\nindex 00000000..6dec8cb3\n--- /dev/null\n+++ b/src/libcamera/clock_recovery.cpp\n@@ -0,0 +1,110 @@\n+/* SPDX-License-Identifier: BSD-2-Clause */\n+/*\n+ * Copyright (C) 2024, Raspberry Pi Ltd\n+ *\n+ * Camera sync control algorithm\n+ */\n+\n+#include \"libcamera/internal/clock_recovery.h\"\n+\n+#include \n+\n+using namespace libcamera;\n+\n+LOG_DEFINE_CATEGORY(RPiClockRec)\n+\n+ClockRecovery::ClockRecovery()\n+{\n+\tinitialise();\n+}\n+\n+void ClockRecovery::initialise(unsigned int numPts, unsigned int maxJitter, unsigned int minPts,\n+\t\t\t unsigned int errorThreshold)\n+{\n+\tnumPts_ = numPts;\n+\tmaxJitter_ = maxJitter;\n+\tminPts_ = minPts;\n+\terrorThreshold_ = errorThreshold;\n+\treset();\n+}\n+\n+void ClockRecovery::reset()\n+{\n+\tlastInput_ = 0;\n+\tlastOutput_ = 0;\n+\txAve_ = 0;\n+\tyAve_ = 0;\n+\tx2Ave_ = 0;\n+\txyAve_ = 0;\n+\tcount_ = 0;\n+\tslope_ = 0.0;\n+\toffset_ = 0.0;\n+\terror_ = 0.0;\n+}\n+\n+void ClockRecovery::addSample(uint64_t input, uint64_t output)\n+{\n+\tif (count_ == 0) {\n+\t\tinputBase_ = input;\n+\t\toutputBase_ = output;\n+\t}\n+\n+\t/*\n+\t * We keep an eye on cumulative drift over the last several frames. If this exceeds a\n+\t * threshold, then probably the system clock has been updated and we're going to have to\n+\t * reset everything and start over.\n+\t */\n+\tif (lastOutput_) {\n+\t\tint64_t inputDiff = getOutput(input) - getOutput(lastInput_);\n+\t\tint64_t outputDiff = output - lastOutput_;\n+\t\terror_ = error_ * 0.95 + (outputDiff - inputDiff);\n+\t\tif (std::abs(error_) > errorThreshold_) {\n+\t\t\treset();\n+\t\t\tinputBase_ = input;\n+\t\t\toutputBase_ = output;\n+\t\t}\n+\t}\n+\tlastInput_ = input;\n+\tlastOutput_ = output;\n+\n+\t/*\n+\t * Never let the new output value be more than maxJitter_ away from what we would have expected.\n+\t * This is just to reduce the effect of sudden large delays in the measured output.\n+\t */\n+\tuint64_t expectedOutput = getOutput(input);\n+\toutput = std::clamp(output, expectedOutput - maxJitter_, expectedOutput + maxJitter_);\n+\n+\t/*\n+\t * We use x, y, x^2 and x*y sums to calculate the best fit line. Here we update them by\n+\t * pretending we have count_ samples at the previous fit, and now one new one. Gradually\n+\t * the effect of the older values gets lost. This is a very simple way of updating the\n+\t * fit (there are much more complicated ones!), but it works well enough. Using averages\n+\t * instead of sums makes the relative effect of old values and the new sample clearer.\n+\t */\n+\tdouble x = input - inputBase_;\n+\tdouble y = output - outputBase_ - x;\n+\tunsigned int count1 = count_ + 1;\n+\txAve_ = (count_ * xAve_ + x) / count1;\n+\tyAve_ = (count_ * yAve_ + y) / count1;\n+\tx2Ave_ = (count_ * x2Ave_ + x * x) / count1;\n+\txyAve_ = (count_ * xyAve_ + x * y) / count1;\n+\n+\t/* Don't update slope and offset until we've seen \"enough\" sample points. */\n+\tif (count_ > minPts_) {\n+\t\t/* These are the standard equations for least squares linear regression. */\n+\t\tslope_ = (count1 * count1 * xyAve_ - count1 * xAve_ * count1 * yAve_) /\n+\t\t\t (count1 * count1 * x2Ave_ - count1 * xAve_ * count1 * xAve_);\n+\t\toffset_ = yAve_ - slope_ * xAve_;\n+\t}\n+\n+\t/* Don't increase count_ above numPts_, as this controls the long-term amount of the residual fit. */\n+\tif (count1 < numPts_)\n+\t\tcount_++;\n+}\n+\n+uint64_t ClockRecovery::getOutput(uint64_t input)\n+{\n+\tdouble x = input - inputBase_;\n+\tdouble y = slope_ * x + offset_;\n+\treturn y + x + outputBase_;\n+}\ndiff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\nindex 21cae117..f221590c 100644\n--- a/src/libcamera/meson.build\n+++ b/src/libcamera/meson.build\n@@ -21,6 +21,7 @@ libcamera_internal_sources = files([\n 'byte_stream_buffer.cpp',\n 'camera_controls.cpp',\n 'camera_lens.cpp',\n+ 'clock_recovery.cpp',\n 'control_serializer.cpp',\n 'control_validator.cpp',\n 'converter.cpp',\n","prefixes":["RFC","2/3"]}