From patchwork Thu Jan 9 14:32:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 22500 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5DE00C32EA for ; Thu, 9 Jan 2025 14:32:21 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 34EDD684EA; Thu, 9 Jan 2025 15:32:18 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="pH2dbdOs"; dkim-atps=neutral Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E24D2684E0 for ; Thu, 9 Jan 2025 15:32:15 +0100 (CET) Received: by mail-wm1-x32e.google.com with SMTP id 5b1f17b1804b1-436249df846so7781685e9.3 for ; Thu, 09 Jan 2025 06:32:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1736433135; x=1737037935; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FK56TuW/KQijiRb8LKpul7gor7QkK2FNyQNIdQ+Swkk=; b=pH2dbdOsf3S/OhZVqewff8gmW1DkUkZgZHzxd9sXdzG2jlNzyNrNcNYA6t78iX3tZz eI6s02yb/hjpXVctB0G+PQjyLBVUyKV/RqmnasRsyNXSKVnGw0psHD8N9Vf/Nqby+ycU LHrrebcbcSNAWRrPzII8HF6lV4+hu6D9lHaXcV4qhpGT/At43IFfDGnXP+pjO7HZl0lB jYn78uZfT3SR6hwmjyH0X//wPvbygEaLkOuwqshMr1Qc5z4yK3t0TtU+NviW7txt0Prn z2YOw3GXmJwaGws2DUg2Yueo+Y1yPW4Oetgamufa63ttNahAUZle2IFX4CT4ZrwF8V+v 7DcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736433135; x=1737037935; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FK56TuW/KQijiRb8LKpul7gor7QkK2FNyQNIdQ+Swkk=; b=P25YHmbltffUgpHWMURD0t6RIAfLieqLZLTBHZ7EaKt83ZsrexxjWaoEAa/0nV6JEC TSU5JSelvdGIKrv3fvHDbxwGwkimD9mB8dhbQPoifFoByYDokF4Zunky3v9gNGvWc1wI pROv6kdN1i0S2ngYznX/VX/zPzMoKu5Vek3s3ZPNu0CZcW3CfWWJMQ2wGOV6NeoqWHPK f97JBuhVnq6n/ax+9neUmemtwxKca8dJF3UUkAfmLv1SvF16MBiF1GQsvkLmG9ii9Nqj jVMAAYejtQefE9DCa+SJMEvMMED3x6KjD2wSsq5AH8XNXqot0jJVFoJ+310C+8pprNO4 DRMQ== X-Gm-Message-State: AOJu0YwHWAJ5LNBfz7skKfGyxtm6rZCOYZesmC3QhZ+XtW/CVaHog+Tl AgR8pQEBKnUunUPEGnDNmTt+hNdnShfCioto3a2xdcIcG3Mm5O4IcMpjni0G4mCXfdzq8uzBIfc Y X-Gm-Gg: ASbGnctdqZkvl5F5i0MvYX6MAZWWbzH4Zx3VJeYQ6jqQdJdh8akDqEpbzECf8R4uz6Z 8HPp8ZRGhBp7bim8dmtgBGq2zufSQxmNKKFpLPS+emzOrVVAwgUgH9lX/iXhbMy4vJSZPG04wod 6tRECQBuQ4/bHYzlZU0fS07xvmjQvmK92Lzk2HlABXJXmeeeYivwqzsZp6TC7+Ci2RdZmn3E61O /DOILIxZzVncjxeaOkkcnytDZ50t4/yEYU465yGq1PURiDzFJhIu3OywaRZyuazlYQYi5AWzFoU 6jpBzEPG33ah X-Google-Smtp-Source: AGHT+IEO3VqYmxG0ReB1StjX43HzdN+UpSjR0NN3IrPy2vGUx4dvngLFcoTItOI0Tgmbb4K8G6EZ5w== X-Received: by 2002:a05:600c:5698:b0:434:a7f1:6545 with SMTP id 5b1f17b1804b1-436e2d91910mr56188355e9.27.1736433135221; Thu, 09 Jan 2025 06:32:15 -0800 (PST) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:ffc9:aff6:7f7f:893b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38a8e4b80b2sm1952569f8f.80.2025.01.09.06.32.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Jan 2025 06:32:14 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman , Kieran Bingham , Naushir Patuck , Laurent Pinchart Subject: [PATCH v3 1/7] controls: Add FrameWallClock control Date: Thu, 9 Jan 2025 14:32:05 +0000 Message-Id: <20250109143211.11939-2-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250109143211.11939-1-david.plowman@raspberrypi.com> References: <20250109143211.11939-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a FrameWallClock control that reports the same moment as the frame's SensorTimestamp, but in wallclock units. Signed-off-by: David Plowman Reviewed-by: Kieran Bingham Reviewed-by: Naushir Patuck Reviewed-by: Laurent Pinchart --- src/libcamera/control_ids_core.yaml | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml index 1dfaee0c..b0555d76 100644 --- a/src/libcamera/control_ids_core.yaml +++ b/src/libcamera/control_ids_core.yaml @@ -1049,4 +1049,19 @@ controls: description: | Enable or disable the debug metadata. + - FrameWallClock: + type: int64_t + direction: out + description: | + This timestamp corresponds to the same moment in time as the + SensorTimestamp, but is represented as a wall clock time as measured by + the CLOCK_REALTIME clock. + + Being a wall clock measurement, it can be used to synchronise timing + across different devices. + + \sa SensorTimestamp + + The FrameWallClock control can only be returned in metadata. + ... From patchwork Thu Jan 9 14:32:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 22501 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9DE80C32EA for ; Thu, 9 Jan 2025 14:32:23 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 19E7A68519; Thu, 9 Jan 2025 15:32:22 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="tX6tTd1x"; dkim-atps=neutral Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B076761891 for ; Thu, 9 Jan 2025 15:32:18 +0100 (CET) Received: by mail-wr1-x42e.google.com with SMTP id ffacd0b85a97d-3862ca8e0bbso789770f8f.0 for ; Thu, 09 Jan 2025 06:32:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1736433138; x=1737037938; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0nAyOHSpY0zR0dhTRQ+G7CBUy1LrRHRDo0Ax+OefIcQ=; b=tX6tTd1xzISIIAgCHuomiLnN+VCzxyXC8SEDTELmxcR0yP+XzhMfBBsuD/haVXVY9o MM3+VmdA2KmhLpfCPv2Lb1SmhcFw4SnAMD0Fo1gLHb/wNmnOnCifkDch4vK7BRn5Jq/L cetBi1OyvOAYlcUKMLWVksEuOGdHmfVv8rPgh/7dCLKbpr9QIofK4YODNhFaEWQH8Top +saP526GVcr8cI26h+9GH1QWcfnjkQdH+Q55oMAhLa2Mg9iDeLqSIwlPUDBas1nMzlPR p9EgzJWSOh6kOvADJiih0UB0LeeW9T9OQ2K2lNa21lBR5MXjgrGWP5xM+4FAABo6eWRw 8eBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736433138; x=1737037938; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0nAyOHSpY0zR0dhTRQ+G7CBUy1LrRHRDo0Ax+OefIcQ=; b=pe+9bTfuo4qVFrO2tmj431ArOYxrIO4a61fWsss1dzK+B49r5q3FsDbCyLB7zJHnKE UQxzvW4p0OpttIJk1YRNN86rs77RrU1e/ifHFV84zZiAUUjkrcQC4humVpfOy3z8yb76 zHR070M6I75MlaIWbyZ27zAUm7XxE2S0IBK2zFeZt/6iWuD0QOrJQhxfOi78yzGv8AvQ GjLZiwGdBUYjmlWrknoDH1II5TY+Z1EICqNXuLgZWyWiKbJtY30HQJX/os5W+Fulmv2F mo4yhQ4cr23EA5ftvUIxOi0yEw9vEp3l/mVIQHl2rjz4maJj/FYtEhgZpYt3xDuJsQs3 ya+A== X-Gm-Message-State: AOJu0Yy4X/J3WQbbz0Bl+w+azqwndF40k9LxeVDMp7Fy+Kp4aoW0cGIZ krh6yIsUhulz6Z8zULFJnqevCgjX7f0F0qqrEsax5aKnhAsq0cwNuvVIMY4sylUBQbSIFfve6dU N X-Gm-Gg: ASbGncuqxHU9aeZW+D+tWZDjEKw/D6Fg0SLOJSsjLRIFRr/B0idJKkgHa0cMxqRFtC6 waW+h4qylH5GuzPIgDY/cMnURr4TZpxTqfhka8ZbLB4pwGfqAvJbPDl3VbrBd6Ljj/243LEKRT2 iFdarFYbTTY6E6dOSwKap7fr6G2wA3HIUsaw8a/lUmlK9N8MO7izH8LcjSZMC6GjreB3Zz2h8mO mYkDXEX3mZeyrQPcGxU7/oJncCff0mBJsaWq64FxNVFySmeK42JCOrb0OdiN+5aXF8avNe3cGuz 4+65v3UkB1hG X-Google-Smtp-Source: AGHT+IFQHPPP8RzdaZ4vBMZyoHmASzE3LdKdlCx7Hz2xeyFMjdNn2DGxdW6qY8MC+lfnDr9/J2zQ0A== X-Received: by 2002:a5d:59ab:0:b0:385:eb7c:5d0f with SMTP id ffacd0b85a97d-38a8730db6cmr6795818f8f.26.1736433137830; Thu, 09 Jan 2025 06:32:17 -0800 (PST) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:ffc9:aff6:7f7f:893b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38a8e4b80b2sm1952569f8f.80.2025.01.09.06.32.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Jan 2025 06:32:15 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman Subject: [PATCH v3 2/7] libcamera: Add ClockRecovery class to generate wallclock timestamps Date: Thu, 9 Jan 2025 14:32:06 +0000 Message-Id: <20250109143211.11939-3-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250109143211.11939-1-david.plowman@raspberrypi.com> References: <20250109143211.11939-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The ClockRecovery class takes pairs of timestamps from two different clocks, and models the second ("output") clock from the first ("input") clock. We can use it, in particular, to get a good wallclock estimate for a frame's SensorTimestamp. Signed-off-by: David Plowman Reviewed-by: Kieran Bingham --- include/libcamera/internal/clock_recovery.h | 68 ++++++ include/libcamera/internal/meson.build | 1 + src/libcamera/clock_recovery.cpp | 230 ++++++++++++++++++++ src/libcamera/meson.build | 1 + 4 files changed, 300 insertions(+) create mode 100644 include/libcamera/internal/clock_recovery.h create mode 100644 src/libcamera/clock_recovery.cpp diff --git a/include/libcamera/internal/clock_recovery.h b/include/libcamera/internal/clock_recovery.h new file mode 100644 index 00000000..43e46b7d --- /dev/null +++ b/include/libcamera/internal/clock_recovery.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * Camera recovery algorithm + */ +#pragma once + +#include + +namespace libcamera { + +class ClockRecovery +{ +public: + ClockRecovery(); + + void configure(unsigned int numSamples = 100, unsigned int maxJitter = 2000, + unsigned int minSamples = 10, unsigned int errorThreshold = 50000); + void reset(); + + void addSample(); + void addSample(uint64_t input, uint64_t output); + + uint64_t getOutput(uint64_t input); + +private: + /* Approximate number of samples over which the model state persists. */ + unsigned int numSamples_; + /* Remove any output jitter larger than this immediately. */ + unsigned int maxJitter_; + /* Number of samples required before we start to use model estimates. */ + unsigned int minSamples_; + /* Threshold above which we assume the wallclock has been reset. */ + unsigned int errorThreshold_; + + /* How many samples seen (up to numSamples_). */ + unsigned int count_; + /* This gets subtracted from all input values, just to make the numbers easier. */ + uint64_t inputBase_; + /* As above, for the output. */ + uint64_t outputBase_; + /* The previous input sample. */ + uint64_t lastInput_; + /* The previous output sample. */ + uint64_t lastOutput_; + + /* Average x value seen so far. */ + double xAve_; + /* Average y value seen so far */ + double yAve_; + /* Average x^2 value seen so far. */ + double x2Ave_; + /* Average x*y value seen so far. */ + double xyAve_; + + /* + * The latest estimate of linear parameters to derive the output clock + * from the input. + */ + double slope_; + double offset_; + + /* Use this cumulative error to monitor for spontaneous clock updates. */ + double error_; +}; + +} /* namespace libcamera */ diff --git a/include/libcamera/internal/meson.build b/include/libcamera/internal/meson.build index 7d6aa8b7..41500636 100644 --- a/include/libcamera/internal/meson.build +++ b/include/libcamera/internal/meson.build @@ -11,6 +11,7 @@ libcamera_internal_headers = files([ 'camera_manager.h', 'camera_sensor.h', 'camera_sensor_properties.h', + 'clock_recovery.h', 'control_serializer.h', 'control_validator.h', 'converter.h', diff --git a/src/libcamera/clock_recovery.cpp b/src/libcamera/clock_recovery.cpp new file mode 100644 index 00000000..abacf444 --- /dev/null +++ b/src/libcamera/clock_recovery.cpp @@ -0,0 +1,230 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * Clock recovery algorithm + */ + +#include "libcamera/internal/clock_recovery.h" + +#include + +#include + +/** + * \file clock_recovery.h + * \brief Clock recovery - deriving one clock from another independent clock + */ + +namespace libcamera { + +LOG_DEFINE_CATEGORY(ClockRec) + +/** + * \class ClockRecovery + * \brief Recover an output clock from an input clock + * + * The ClockRecovery class derives an output clock from an input clock, + * modelling the output clock as being linearly related to the input clock. + * For example, we may use it to derive wall clock timestamps from timestamps + * measured by the internal system clock which counts local time since boot. + * + * When pairs of corresponding input and output timestamps are available, + * they should be submitted to the model with addSample(). The model will + * update, and output clock values for known input clock values can be + * obtained using getOutput(). + * + * As a convenience, if the input clock is indeed the time since boot, and the + * output clock represents a real wallclock time, then addSample() can be + * called with no arguments, and a pair of timestamps will be captured at + * that moment. + * + * The configure() function accepts some configuration parameters to control + * the linear fitting process. + */ + +/** + * \brief Construct a ClockRecovery + */ +ClockRecovery::ClockRecovery() +{ + configure(); + reset(); +} + +/** + * \brief Set configuration parameters + * \param[in] numSamples The approximate duration for which the state of the model + * is persistent + * \param[in] maxJitter New output samples are clamped to no more than this + * amount of jitter, to prevent sudden swings from having a large effect + * \param[in] minSamples The fitted clock model is not used to generate outputs + * until this many samples have been received + * \param[in] errorThreshold If the accumulated differences between input and + * output clocks reaches this amount over a few frames, the model is reset + */ +void ClockRecovery::configure(unsigned int numSamples, unsigned int maxJitter, + unsigned int minSamples, unsigned int errorThreshold) +{ + LOG(ClockRec, Debug) + << "configure " << numSamples << " " << maxJitter << " " << minSamples << " " << errorThreshold; + + numSamples_ = numSamples; + maxJitter_ = maxJitter; + minSamples_ = minSamples; + errorThreshold_ = errorThreshold; +} + +/** + * \brief Reset the clock recovery model and start again from scratch + */ +void ClockRecovery::reset() +{ + LOG(ClockRec, Debug) << "reset"; + + lastInput_ = 0; + lastOutput_ = 0; + xAve_ = 0; + yAve_ = 0; + x2Ave_ = 0; + xyAve_ = 0; + count_ = 0; + error_ = 0.0; + /* + * Setting slope_ and offset_ to zero initially means that the clocks + * advance at exactly the same rate. + */ + slope_ = 0.0; + offset_ = 0.0; +} + +/** + * \brief Add a sample point to the clock recovery model, for recovering a wall + * clock value from the internal system time since boot + * + * This is a convenience function to make it easy to derive a wall clock value + * (using the Linux CLOCK_REALTIME) from the time since the system started + * (measured by CLOCK_BOOTTIME). + */ +void ClockRecovery::addSample() +{ + LOG(ClockRec, Debug) << "addSample"; + + struct timespec bootTime1; + struct timespec bootTime2; + struct timespec wallTime; + + /* Get boot and wall clocks in microseconds. */ + clock_gettime(CLOCK_BOOTTIME, &bootTime1); + clock_gettime(CLOCK_REALTIME, &wallTime); + clock_gettime(CLOCK_BOOTTIME, &bootTime2); + uint64_t boot1 = bootTime1.tv_sec * 1000000ULL + bootTime1.tv_nsec / 1000; + uint64_t boot2 = bootTime2.tv_sec * 1000000ULL + bootTime2.tv_nsec / 1000; + uint64_t boot = (boot1 + boot2) / 2; + uint64_t wall = wallTime.tv_sec * 1000000ULL + wallTime.tv_nsec / 1000; + + addSample(boot, wall); +} + +/** + * \brief Add a sample point to the clock recovery model, specifying the exact + * input and output clock values + * \param[in] input The input clock value + * \param[in] output The value of the output clock at the same moment, as far + * as possible, that the input clock was sampled + * + * This function should be used for corresponding clocks other than the Linux + * BOOTTIME and REALTIME clocks. + */ +void ClockRecovery::addSample(uint64_t input, uint64_t output) +{ + LOG(ClockRec, Debug) << "addSample " << input << " " << output; + + if (count_ == 0) { + inputBase_ = input; + outputBase_ = output; + } + + /* + * We keep an eye on cumulative drift over the last several frames. If this exceeds a + * threshold, then probably the system clock has been updated and we're going to have to + * reset everything and start over. + */ + if (lastOutput_) { + int64_t inputDiff = getOutput(input) - getOutput(lastInput_); + int64_t outputDiff = output - lastOutput_; + error_ = error_ * 0.95 + (outputDiff - inputDiff); + if (std::abs(error_) > errorThreshold_) { + reset(); + inputBase_ = input; + outputBase_ = output; + } + } + lastInput_ = input; + lastOutput_ = output; + + /* + * Never let the new output value be more than maxJitter_ away from what + * we would have expected. This is just to reduce the effect of sudden + * large delays in the measured output. + */ + uint64_t expectedOutput = getOutput(input); + output = std::clamp(output, expectedOutput - maxJitter_, expectedOutput + maxJitter_); + + /* + * We use x, y, x^2 and x*y sums to calculate the best fit line. Here we + * update them by pretending we have count_ samples at the previous fit, + * and now one new one. Gradually the effect of the older values gets + * lost. This is a very simple way of updating the fit (there are much + * more complicated ones!), but it works well enough. Using averages + * instead of sums makes the relative effect of old values and the new + * sample clearer. + */ + double x = static_cast(input - inputBase_); + double y = static_cast(output - outputBase_) - x; + unsigned int count1 = count_ + 1; + xAve_ = (count_ * xAve_ + x) / count1; + yAve_ = (count_ * yAve_ + y) / count1; + x2Ave_ = (count_ * x2Ave_ + x * x) / count1; + xyAve_ = (count_ * xyAve_ + x * y) / count1; + + /* + * Don't update slope and offset until we've seen "enough" sample + * points. Note that the initial settings for slope_ and offset_ + * ensures that the wallclock advances at the same rate as the realtime + * clock (but with their respective initial offsets). + */ + if (count_ > minSamples_) { + /* These are the standard equations for least squares linear regression. */ + slope_ = (count1 * count1 * xyAve_ - count1 * xAve_ * count1 * yAve_) / + (count1 * count1 * x2Ave_ - count1 * xAve_ * count1 * xAve_); + offset_ = yAve_ - slope_ * xAve_; + } + + /* + * Don't increase count_ above numSamples_, as this controls the long-term + * amount of the residual fit. + */ + if (count1 < numSamples_) + count_++; +} + +/** + * \brief Calculate the output clock value according to the model from an input + * clock value + * \param[in] input The input clock value + * + * \return Output clock value + */ +uint64_t ClockRecovery::getOutput(uint64_t input) +{ + double x = static_cast(input - inputBase_); + double y = slope_ * x + offset_; + uint64_t output = y + x + outputBase_; + + LOG(ClockRec, Debug) << "getOutput " << input << " " << output; + + return output; +} + +} /* namespace libcamera */ diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build index 57fde8a8..4eaa1c8e 100644 --- a/src/libcamera/meson.build +++ b/src/libcamera/meson.build @@ -21,6 +21,7 @@ libcamera_internal_sources = files([ 'byte_stream_buffer.cpp', 'camera_controls.cpp', 'camera_lens.cpp', + 'clock_recovery.cpp', 'control_serializer.cpp', 'control_validator.cpp', 'converter.cpp', From patchwork Thu Jan 9 14:32:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 22503 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id AE24EC32EA for ; Thu, 9 Jan 2025 14:32:28 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E8B8E68549; Thu, 9 Jan 2025 15:32:27 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="hAJqMt+F"; dkim-atps=neutral Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C1040684E4 for ; Thu, 9 Jan 2025 15:32:20 +0100 (CET) Received: by mail-wr1-x42c.google.com with SMTP id ffacd0b85a97d-3862d161947so538006f8f.3 for ; Thu, 09 Jan 2025 06:32:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1736433140; x=1737037940; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O7R7laaCtTqG8D2rZUcpq6OOb09xjPw1X7M9gCXqZtw=; b=hAJqMt+FHYhhU4Y+eilTljb/iRaAN4wesE7ralJM5ZnbjKb8wwJ7GW2BthPZByp6Yo HfT396V8B5gLyqfeThO5LvI0gMV8xezfrPnm90YyHxFBc3VTzdrklSDG9oQKIo99Vay3 4uMQa6KE+V9DOGRMJ4CQ7jEhyWnmOpI2sfJLozDdBAYtZashMj+t0kUP2VQO4uc11igI IMUnGWs/SEHA6Co5OybJDiHaqjP99EsGIdUuuzyj3SPr7IxfMjI2n3GXQj8G7hxqE+gl 2sQU1A2jBCheta75SCbW+Yo0rygEIPI1K4YM0T5ySvZxm735E1CDZ6/fE9JiolSGgs+x IA0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736433140; x=1737037940; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O7R7laaCtTqG8D2rZUcpq6OOb09xjPw1X7M9gCXqZtw=; b=RcyzCGYTn48giDek+tdsnqvSV7IQLPJf6gAG22vPm6ZZgIfKExoagTtLgQzzZ7eWPm gt+gyeUTII7HL2G9rCo2rmii+DwdwzyLbB0OCx2rNkCsEgtee6I3qGC13IPiOXwChP8E KixpU7xhlGo2i7QHJN/dXCOySA8y2L8jnppxneISOqx5PeU4SeQ47uo118N3D7PRt47O WlTc6ZGH/o8p06t4OQ4CKoCgXEaGx4IhRCgRB7q1YJKFwSuvL+m/XkZ+q6/hkx0TZiCo lRkHdd0XY+piFyae4tgMpp8wJ/2lW1Nhvx5ciQWN4JVGzNUds/OtAaAu3P1laNegUvBQ KqJg== X-Gm-Message-State: AOJu0Yw9vukROhIJzdUkVogCLfjDwOJhXS4bP4L8tQ9uMldLDMTfG194 Nd04YKyHor5RdTwuwqkh0sEqP5aUO6DNhFsP1suigI+yi1m0hQmRJafg9IgXwkZ2xp3srrCNqjC 8 X-Gm-Gg: ASbGncvaJ+dzR5W7R6+qA5l37SejQrXHhAqQ69UjraInNGJPWw7tAKrU4oN0SqkXfSv b55UbC/D0X5mA7hhjlb+lcPKc+TXVHWAoNJ8Q3zfCQ/jWTgK1U7ds0YYRV/GpBELRitZ9X5YL5c 6Bix7xcB3gHTUnno7CUk1ZjKCdAL2iNRF8tXejn6NUmo0ldCMOdqLtRMIca6QX71BSC7qYwYAzD n7r7GZPAjwLEvXyi+uznb6PM4knMP1AoOBI8fpAQtxxQ53P2AQqoV0Ljp0U3F98Mxl84e2imk7o H8LVv5dqYF++ X-Google-Smtp-Source: AGHT+IEbXmlWX0F2jBQLlopT9Vq/ymdnV3x4n7BJ+aoon3seQwb0gOP2XIO0+VPWbceYbojkgfubog== X-Received: by 2002:a5d:6d8e:0:b0:38a:88be:bcb4 with SMTP id ffacd0b85a97d-38a88bebd23mr5435050f8f.29.1736433138555; Thu, 09 Jan 2025 06:32:18 -0800 (PST) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:ffc9:aff6:7f7f:893b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38a8e4b80b2sm1952569f8f.80.2025.01.09.06.32.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Jan 2025 06:32:18 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman , Naushir Patuck Subject: [PATCH v3 3/7] controls: Add camera synchronisation controls for Raspberry Pi Date: Thu, 9 Jan 2025 14:32:07 +0000 Message-Id: <20250109143211.11939-4-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250109143211.11939-1-david.plowman@raspberrypi.com> References: <20250109143211.11939-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" New controls are added to control the camera "sync" algorithm, which allows different cameras to synchronise their frames. For the time being, the controls are Raspberry Pi specific, though this is expected to change in future. Signed-off-by: David Plowman Reviewed-by: Naushir Patuck Acked-by: Kieran Bingham --- src/libcamera/control_ids_rpi.yaml | 113 +++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) diff --git a/src/libcamera/control_ids_rpi.yaml b/src/libcamera/control_ids_rpi.yaml index 7524c5d2..5c2721c0 100644 --- a/src/libcamera/control_ids_rpi.yaml +++ b/src/libcamera/control_ids_rpi.yaml @@ -58,4 +58,117 @@ controls: official libcamera API support for per-stream controls in the future. \sa ScalerCrop + + - SyncMode: + type: int32_t + direction: in + description: | + Enable or disable camera synchronisation ("sync") mode. + + When sync mode is enabled, a camera will synchronise frames temporally + with other cameras, either attached to the same device or a different + one. There should be one "server" device, which broadcasts timing + information to one or more "clients". Communication is one-way, from + server to clients only, and it is only clients that adjust their frame + timings to match the server. + + Sync mode requires all cameras to be running at (as far as possible) the + same fixed framerate. Clients may continue to make adjustments to keep + their cameras synchronised with the server for the duration of the + session, though any updates after the initial ones should remain small. + + \sa SyncReady + \sa SyncTimer + \sa SyncFrames + + enum: + - name: SyncModeOff + value: 0 + description: Disable sync mode. + - name: SyncModeServer + value: 1 + description: | + Enable sync mode, act as server. The server broadcasts timing + messages to any clients that are listening, so that the clients can + synchronise their camera frames with the server's. + - name: SyncModeClient + value: 2 + description: | + Enable sync mode, act as client. A client listens for any server + messages, and arranges for its camera frames to synchronise as + closely as possible with the server's. Many clients can listen out + for the same server. Clients can also be started ahead of any + servers, causing them merely to wait for the server to start. + + - SyncReady: + type: bool + direction: out + description: | + When using the camera synchronisation algorithm, the server broadcasts + timing information to the clients. This also includes the time (some + number of frames in the future, called the "ready time") at which the + server will signal its controlling application, using this control, to + start using the image frames. + + The client receives the "ready time" from the server, and will signal + its application to start using the frames at this same moment. + + While this control value is false, applications (on both client and + server) should continue to wait, and not use the frames. + + Once this value becomes true, it means that this is the first frame + where the server and its clients have agreed that they will both be + synchronised and that applications should begin consuming frames. + Thereafter, this control will continue to signal the value true for + the rest of the session. + + \sa SyncMode + \sa SyncTimer + \sa SyncFrames + + - SyncTimer: + type: int64_t + direction: out + description: | + This reports the amount of time, in microseconds, until the "ready + time", at which the server and client will signal their controlling + applications that the frames are now synchronised and should be + used. The value may be refined slightly over time, becoming more precise + as the "ready time" approaches. + + Servers always report this value, whereas clients will omit this control + until they have received a message from the server that enables them to + calculate it. + + Normally the value will start positive (the "ready time" is in the + future), and decrease towards zero, before becoming negative (the "ready + time" has elapsed). So there should be just one frame where the timer + value is, or is very close to, zero - the one for which the SyncReady + control becomes true. At this moment, the value indicates how closely + synchronised the client believes it is with the server. + + But note that if frames are being dropped, then the "near zero" valued + frame, or indeed any other, could be skipped. In these cases the timer + value allows an application to deduce that this has happened. + + \sa SyncMode + \sa SyncReady + \sa SyncFrames + + - SyncFrames: + type: int32_t + direction: in + description: | + The number of frames the server should wait, after enabling + SyncModeServer, before signalling (via the SyncReady control) that + frames should be used. This therefore determines the "ready time" for + all synchronised cameras. + + This control value should be set only for the device that is to act as + the server, before or at the same moment at which SyncModeServer is + enabled. + + \sa SyncMode + \sa SyncReady + \sa SyncTimer ... From patchwork Thu Jan 9 14:32:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 22502 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id BFE9EC32EA for ; Thu, 9 Jan 2025 14:32:26 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 678F468536; Thu, 9 Jan 2025 15:32:26 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="AvJDToAo"; dkim-atps=neutral Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id EF5F5684E2 for ; Thu, 9 Jan 2025 15:32:19 +0100 (CET) Received: by mail-wr1-x42a.google.com with SMTP id ffacd0b85a97d-3862b40a6e0so571433f8f.0 for ; Thu, 09 Jan 2025 06:32:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1736433139; x=1737037939; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MyqtSsrjam7Je3P1fuyUgnWz9ifuQumfixpMt1SesR8=; b=AvJDToAoHZSSO8mAaGWWBClu+O2VO6im+UE5Iau8nrVQKPdvcOw0hu588fzi5HJaT4 +YO35Q4vAGVNG3xhUm6PXtJxH7HClxo5CwG0F3wcABBC4mEMl79d0KCETyfmmKTw0+rt RWSpLJZcPDtvQE9AybcY2nFQGUPbMfiJjnuM6FbX51dzBVTzfJY5Bh2UriDDMJsnCXEP AbyE1z4XPw0bATJd4UH/fnbjm9z5qQ7X8ce4R7c4qDJ7HP2f1JXlXJinqHLS4fLLtkha UjUni+zFaF6TlN9b5zynQAHj/g79R75iA3b5NdVV+h5Bl+1GMC3of/NEd0kpQpl9PdSh QEsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736433139; x=1737037939; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MyqtSsrjam7Je3P1fuyUgnWz9ifuQumfixpMt1SesR8=; b=o/FofA25uSJwUgqfj57cbG09lDSNU+byEA4O0f/yk9vwE6vXUt/Z2PjUlpACuE0PYJ Nb8qrYyl/yJHw5NvpJ9WueXJ19GflRwvVdrymP3Gq/pVInTjvaxobT/AJRfrDSeu32TS W3lfX21OLJhwjP8YrsKZFj3EGze9r+j9GmQcd/9h8XFkhYkNqXHO5lclhN3S3vhRj8Pb VaoqQ8sEPksWkNjioII45NMIn4qLy75X33DoRRIw0uISKg8MKJkz+Lbaa1pzkv5qQNTP rLtccg6NkaQyUUt1Er1V/7MFUvb9oI7MFHwhsANwet23rdRT4o6Ob+3s7aFlKZcO5QSl b3lg== X-Gm-Message-State: AOJu0YzGU5sc4387VxI8aRuk2ON5R3sBqP84I8tJ0iGA4SVDWYeRN+KZ Vx8MHpeRMvcMGic5KU+EEYvD2ULSnWgQFvyhIoedEc2kmuUT4MXKJdMG944koC4F98OJvbl9x9E l X-Gm-Gg: ASbGncsmC3AhbB8JPyrt5xBvFLWR+m8i1SdGoOfovx2IdskxZJa/uDfHPJ4hjj0AUEm 2Vivc69O3Zs/pA98drpQWHD1YyBRSU6+n5V9Tbh/jA2w591hogCdQPxT0pFTXbYPO0ZOay8dYri SAnYa/Ya5oHANHvzJnd1zuX+2taDZ/NUaCqoZhGSH1k8n8YqCU6K2HUlF0b2dJMwcS1gwD+HGMh HGaW2W3Hl7aA6RSOonWV+QXrvxDJ182yWbfXbxJsIzEt9hfde8GgFndspKOhxl3blt03chOQ6nU YWcSEv+r2gXC X-Google-Smtp-Source: AGHT+IEIIp/CBeq9Y5yD9yFHh7cQhsebEOG/gj/u8RRQhR/Xpq/nx7Mog1Bh2vaL9BTZPcUVKlJoXQ== X-Received: by 2002:a05:6000:18a8:b0:388:c790:1dff with SMTP id ffacd0b85a97d-38a8733fd71mr4965527f8f.47.1736433139182; Thu, 09 Jan 2025 06:32:19 -0800 (PST) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:ffc9:aff6:7f7f:893b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38a8e4b80b2sm1952569f8f.80.2025.01.09.06.32.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Jan 2025 06:32:18 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman Subject: [PATCH v3 4/7] pipeline: rpi:: Add wallclock timestamp support Date: Thu, 9 Jan 2025 14:32:08 +0000 Message-Id: <20250109143211.11939-5-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250109143211.11939-1-david.plowman@raspberrypi.com> References: <20250109143211.11939-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" A ClockRecovery object is added for derived classes to use, and wallclock timetamps are copied into the request metadata for applications. In the derived class, wallclock timestamps are derived corresponding to the sensor timestamp, and made available to the base pipeline handler class. Signed-off-by: David Plowman Reviewed-by: Kieran Bingham --- src/libcamera/pipeline/rpi/common/pipeline_base.cpp | 5 +++++ src/libcamera/pipeline/rpi/common/pipeline_base.h | 3 +++ src/libcamera/pipeline/rpi/vc4/vc4.cpp | 10 ++++++++-- 3 files changed, 16 insertions(+), 2 deletions(-) diff --git a/src/libcamera/pipeline/rpi/common/pipeline_base.cpp b/src/libcamera/pipeline/rpi/common/pipeline_base.cpp index 4b147fdb..56f3c4df 100644 --- a/src/libcamera/pipeline/rpi/common/pipeline_base.cpp +++ b/src/libcamera/pipeline/rpi/common/pipeline_base.cpp @@ -686,6 +686,9 @@ int PipelineHandlerBase::start(Camera *camera, const ControlList *controls) return ret; } + /* A good moment to add an initial clock sample. */ + data->wallClockRecovery_.addSample(); + /* * Reset the delayed controls with the gain and exposure values set by * the IPA. @@ -1510,6 +1513,8 @@ void CameraData::fillRequestMetadata(const ControlList &bufferControls, Request { request->metadata().set(controls::SensorTimestamp, bufferControls.get(controls::SensorTimestamp).value_or(0)); + request->metadata().set(controls::FrameWallClock, + bufferControls.get(controls::FrameWallClock).value_or(0)); if (cropParams_.size()) { std::vector crops; diff --git a/src/libcamera/pipeline/rpi/common/pipeline_base.h b/src/libcamera/pipeline/rpi/common/pipeline_base.h index aae0c2f3..ca706bf3 100644 --- a/src/libcamera/pipeline/rpi/common/pipeline_base.h +++ b/src/libcamera/pipeline/rpi/common/pipeline_base.h @@ -20,6 +20,7 @@ #include "libcamera/internal/bayer_format.h" #include "libcamera/internal/camera.h" #include "libcamera/internal/camera_sensor.h" +#include "libcamera/internal/clock_recovery.h" #include "libcamera/internal/framebuffer.h" #include "libcamera/internal/media_device.h" #include "libcamera/internal/media_object.h" @@ -177,6 +178,8 @@ public: Config config_; + ClockRecovery wallClockRecovery_; + protected: void fillRequestMetadata(const ControlList &bufferControls, Request *request); diff --git a/src/libcamera/pipeline/rpi/vc4/vc4.cpp b/src/libcamera/pipeline/rpi/vc4/vc4.cpp index fd8d84b1..3d8c65c6 100644 --- a/src/libcamera/pipeline/rpi/vc4/vc4.cpp +++ b/src/libcamera/pipeline/rpi/vc4/vc4.cpp @@ -781,9 +781,15 @@ void Vc4CameraData::unicamBufferDequeue(FrameBuffer *buffer) auto [ctrl, delayContext] = delayedCtrls_->get(buffer->metadata().sequence); /* * Add the frame timestamp to the ControlList for the IPA to use - * as it does not receive the FrameBuffer object. + * as it does not receive the FrameBuffer object. Also derive a + * corresponding wallclock value. */ - ctrl.set(controls::SensorTimestamp, buffer->metadata().timestamp); + wallClockRecovery_.addSample(); + uint64_t sensorTimestamp = buffer->metadata().timestamp; + uint64_t wallClockTimestamp = wallClockRecovery_.getOutput(sensorTimestamp / 1000); + + ctrl.set(controls::SensorTimestamp, sensorTimestamp); + ctrl.set(controls::FrameWallClock, wallClockTimestamp); bayerQueue_.push({ buffer, std::move(ctrl), delayContext }); } else { embeddedQueue_.push(buffer); From patchwork Thu Jan 9 14:32:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 22505 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E33E6C32EA for ; Thu, 9 Jan 2025 14:32:33 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 6B11968503; Thu, 9 Jan 2025 15:32:33 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="TTT8QR2K"; dkim-atps=neutral Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0F01F68503 for ; Thu, 9 Jan 2025 15:32:22 +0100 (CET) Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-385e87b25f0so1402834f8f.0 for ; Thu, 09 Jan 2025 06:32:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1736433141; x=1737037941; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yKDZcqefLc17FC1rrbX68afdo5XuZS6xj1yrJzIz0E8=; b=TTT8QR2KNPGsiR3lUhfdGNeXtV8TUe+btohZoqywftlnZZGsMy9FpsOFmyDlWD1kNF qFPnwrRP7ELH06fpIJu73/5EROJ/547XVNUUJU5z99R1ZIpE480FJ1yoq5UGJJ6cQ2Dn 10+VFRmJuwruLoLMx6XQm2PCoU56kyfIrIDiQhfF5POzQZ/4T1xoeyjeFfhVer1ZcfjP 7fqTkRVXjO3/Ftie6dVBH8CPHBZbn23n4TlYmQYPsM4Aazo6vVr1KmzAX1/k1DEaJx6m nR1h66SdxjrV3vqrNHikF253pwStHyG04ERyqlyHrh4LBbxyydJgrNK0vmTi9ZqUlQiE laag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736433141; x=1737037941; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yKDZcqefLc17FC1rrbX68afdo5XuZS6xj1yrJzIz0E8=; b=vmFcpuxvP/plzEfHDjg0sKzQiFSEaelPGMYu27LVLQFyLxrlRw27Erz+GcGck2h4BX 7y5b1c+7kzKfUQyYmL5KAJ8ooLyKPdclFRpazxhZegD0DzHNEQxmt2WQJzD93KrogFFc jENCslNbdwq/7GRk66dfyKqQQzglJ+FdjckaWhxb7G5I3Kr+ug0OH9pP7NlwJjNh/Y3m bQ7HExFqcGSveVUcpQUP/uCwGW7Csx0qbBy2O8u9slaDwZjOfEznkkOIlvaWbrJGI3AD kjQH64/Hg+iskC14NcQXxLQ87Fk34spl7EyxiX14o+gFfllcwUUfvxxG3/Il9cTfj1ei mG2A== X-Gm-Message-State: AOJu0Yy79YLY+Npp3ufd3H6mTfGlHa6MMXfR97L4t+QeK+yE1itZQCvV Ch1q+o9DfIbzrPqDOAJlXflFrLuvH4JtAeq25BlwP6Hd/k7OZmmbh6MKbyTFfqs7BclNwmKqkEr F X-Gm-Gg: ASbGncs4V4pcXS+tyd7x2Q0znZFH9PxQ8aJ37EKcCbe2+EBfY4Od0lz77cTcP8eUTG0 TGQoDFwhrzjp/cXZt4uqk8nP7QhqqFT2IMGqMkkWS3xrxEjQvtTCW5DIFrDWnxKZ4oesanXdHK7 uLYNO50iMiVa5K38gQxgG4EdjCzmb8iS3qWq2VctSUOZs+raxz2G1znPIgBhNBY7qjTjJ4TeLJv cPRT2CfOnFQ8z6ah5q9QF3y6g2TJ0igo11YzhQQnroVneqDdI9cMmohJ6e9H8fOPfKC76aMaeao 7PcYs8Ey8LSV X-Google-Smtp-Source: AGHT+IHq0xP6QSB6NgJEmFnmTsPGaBBNH1bK4P6K3KOrhv/7jSLPXI3Ao01cKK+ef4v6MdJmF++pJA== X-Received: by 2002:a05:6000:1f8a:b0:385:faec:d945 with SMTP id ffacd0b85a97d-38a8b0d3202mr3391183f8f.9.1736433139842; Thu, 09 Jan 2025 06:32:19 -0800 (PST) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:ffc9:aff6:7f7f:893b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38a8e4b80b2sm1952569f8f.80.2025.01.09.06.32.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Jan 2025 06:32:19 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: Naushir Patuck Subject: [PATCH v3 5/7] ipa: rpi: Add base classes and plumbing for sync algorithm Date: Thu, 9 Jan 2025 14:32:09 +0000 Message-Id: <20250109143211.11939-6-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250109143211.11939-1-david.plowman@raspberrypi.com> References: <20250109143211.11939-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Naushir Patuck We add a base class for a "sync algorithm", and define its inputs and outputs in the SyncStatus class. We add the necessary plumbing to the base IPA code so as to arrange for the necessary parameters to be made available to such an algorithm, and also to handle the return values, passing them back as necessary to the pipeline handler. Signed-off-by: Naushir Patuck --- src/ipa/rpi/common/ipa_base.cpp | 78 +++++++++++++++++++++++-- src/ipa/rpi/common/ipa_base.h | 4 +- src/ipa/rpi/controller/sync_algorithm.h | 31 ++++++++++ src/ipa/rpi/controller/sync_status.h | 27 +++++++++ 4 files changed, 135 insertions(+), 5 deletions(-) create mode 100644 src/ipa/rpi/controller/sync_algorithm.h create mode 100644 src/ipa/rpi/controller/sync_status.h diff --git a/src/ipa/rpi/common/ipa_base.cpp b/src/ipa/rpi/common/ipa_base.cpp index 6ff1e22b..f1e4a161 100644 --- a/src/ipa/rpi/common/ipa_base.cpp +++ b/src/ipa/rpi/common/ipa_base.cpp @@ -28,6 +28,8 @@ #include "controller/lux_status.h" #include "controller/sharpen_algorithm.h" #include "controller/statistics.h" +#include "controller/sync_algorithm.h" +#include "controller/sync_status.h" namespace libcamera { @@ -72,6 +74,8 @@ const ControlInfoMap::Map ipaControls{ { &controls::Sharpness, ControlInfo(0.0f, 16.0f, 1.0f) }, { &controls::ScalerCrop, ControlInfo(Rectangle{}, Rectangle(65535, 65535, 65535, 65535), Rectangle{}) }, { &controls::FrameDurationLimits, ControlInfo(INT64_C(33333), INT64_C(120000)) }, + { &controls::rpi::SyncMode, ControlInfo(controls::rpi::SyncModeValues) }, + { &controls::rpi::SyncFrames, ControlInfo(1, 1000000, 100) }, { &controls::draft::NoiseReductionMode, ControlInfo(controls::draft::NoiseReductionModeValues) }, { &controls::rpi::StatsOutputEnable, ControlInfo(false, true, false) }, }; @@ -390,6 +394,7 @@ void IpaBase::prepareIsp(const PrepareParams ¶ms) rpiMetadata.clear(); fillDeviceStatus(params.sensorControls, ipaContext); + fillSyncParams(params, ipaContext); if (params.buffers.embedded) { /* @@ -488,10 +493,23 @@ void IpaBase::processStats(const ProcessParams ¶ms) helper_->process(statistics, rpiMetadata); controller_.process(statistics, &rpiMetadata); + /* Send any sync algorithm outputs back to the pipeline handler */ + Duration offset(0s); + struct SyncStatus syncStatus; + if (rpiMetadata.get("sync.status", syncStatus) == 0) { + if (minFrameDuration_ != maxFrameDuration_) + LOG(IPARPI, Error) << "Sync algorithm enabled with variable framerate. " << minFrameDuration_ << " " << maxFrameDuration_; + offset = syncStatus.frameDurationOffset; + + libcameraMetadata_.set(controls::rpi::SyncReady, syncStatus.ready); + if (syncStatus.timerKnown) + libcameraMetadata_.set(controls::rpi::SyncTimer, syncStatus.timerValue); + } + struct AgcStatus agcStatus; if (rpiMetadata.get("agc.status", agcStatus) == 0) { ControlList ctrls(sensorCtrls_); - applyAGC(&agcStatus, ctrls); + applyAGC(&agcStatus, ctrls, offset); setDelayedControls.emit(ctrls, ipaContext); setCameraTimeoutValue(); } @@ -728,6 +746,7 @@ void IpaBase::applyControls(const ControlList &controls) using RPiController::ContrastAlgorithm; using RPiController::DenoiseAlgorithm; using RPiController::HdrAlgorithm; + using RPiController::SyncAlgorithm; /* Clear the return metadata buffer. */ libcameraMetadata_.clear(); @@ -1274,6 +1293,35 @@ void IpaBase::applyControls(const ControlList &controls) statsMetadataOutput_ = ctrl.second.get(); break; + case controls::rpi::SYNC_MODE: { + SyncAlgorithm *sync = dynamic_cast(controller_.getAlgorithm("sync")); + + if (sync) { + int mode = ctrl.second.get(); + SyncAlgorithm::Mode m = SyncAlgorithm::Mode::Off; + if (mode == controls::rpi::SyncModeServer) { + m = SyncAlgorithm::Mode::Server; + LOG(IPARPI, Info) << "Sync mode set to server"; + } else if (mode == controls::rpi::SyncModeClient) { + m = SyncAlgorithm::Mode::Client; + LOG(IPARPI, Info) << "Sync mode set to client"; + } + sync->setMode(m); + } + break; + } + + case controls::rpi::SYNC_FRAMES: { + SyncAlgorithm *sync = dynamic_cast(controller_.getAlgorithm("sync")); + + if (sync) { + int frames = ctrl.second.get(); + if (frames > 0) + sync->setReadyFrame(frames); + } + break; + } + default: LOG(IPARPI, Warning) << "Ctrl " << controls::controls.at(ctrl.first)->name() @@ -1310,6 +1358,19 @@ void IpaBase::fillDeviceStatus(const ControlList &sensorControls, unsigned int i rpiMetadata_[ipaContext].set("device.status", deviceStatus); } +void IpaBase::fillSyncParams(const PrepareParams ¶ms, unsigned int ipaContext) +{ + RPiController::SyncAlgorithm *sync = dynamic_cast( + controller_.getAlgorithm("sync")); + if (!sync) + return; + + SyncParams syncParams; + syncParams.wallClock = *params.sensorControls.get(controls::FrameWallClock); + syncParams.sensorTimestamp = *params.sensorControls.get(controls::SensorTimestamp); + rpiMetadata_[ipaContext].set("sync.params", syncParams); +} + void IpaBase::reportMetadata(unsigned int ipaContext) { RPiController::Metadata &rpiMetadata = rpiMetadata_[ipaContext]; @@ -1478,14 +1539,22 @@ void IpaBase::applyFrameDurations(Duration minFrameDuration, Duration maxFrameDu * value possible. */ Duration maxExposureTime = Duration::max(); - helper_->getBlanking(maxExposureTime, minFrameDuration_, maxFrameDuration_); + auto [vblank, hblank] = helper_->getBlanking(maxExposureTime, minFrameDuration_, maxFrameDuration_); RPiController::AgcAlgorithm *agc = dynamic_cast( controller_.getAlgorithm("agc")); agc->setMaxExposureTime(maxExposureTime); + + RPiController::SyncAlgorithm *sync = dynamic_cast( + controller_.getAlgorithm("sync")); + if (sync) { + Duration duration = (mode_.height + vblank) * ((mode_.width + hblank) * 1.0s / mode_.pixelRate); + LOG(IPARPI, Debug) << "setting sync frame duration to " << duration; + sync->setFrameDuration(duration); + } } -void IpaBase::applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls) +void IpaBase::applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls, Duration frameDurationOffset) { const int32_t minGainCode = helper_->gainCode(mode_.minAnalogueGain); const int32_t maxGainCode = helper_->gainCode(mode_.maxAnalogueGain); @@ -1500,7 +1569,8 @@ void IpaBase::applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls) /* getBlanking might clip exposure time to the fps limits. */ Duration exposure = agcStatus->exposureTime; - auto [vblank, hblank] = helper_->getBlanking(exposure, minFrameDuration_, maxFrameDuration_); + auto [vblank, hblank] = helper_->getBlanking(exposure, minFrameDuration_ - frameDurationOffset, + maxFrameDuration_ - frameDurationOffset); int32_t exposureLines = helper_->exposureLines(exposure, helper_->hblankToLineLength(hblank)); diff --git a/src/ipa/rpi/common/ipa_base.h b/src/ipa/rpi/common/ipa_base.h index 1a811beb..b53d0bfb 100644 --- a/src/ipa/rpi/common/ipa_base.h +++ b/src/ipa/rpi/common/ipa_base.h @@ -95,9 +95,11 @@ private: void applyControls(const ControlList &controls); virtual void handleControls(const ControlList &controls) = 0; void fillDeviceStatus(const ControlList &sensorControls, unsigned int ipaContext); + void fillSyncParams(const PrepareParams ¶ms, unsigned int ipaContext); void reportMetadata(unsigned int ipaContext); void applyFrameDurations(utils::Duration minFrameDuration, utils::Duration maxFrameDuration); - void applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls); + void applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls, + utils::Duration frameDurationOffset = utils::Duration(0)); std::map buffers_; diff --git a/src/ipa/rpi/controller/sync_algorithm.h b/src/ipa/rpi/controller/sync_algorithm.h new file mode 100644 index 00000000..c242def6 --- /dev/null +++ b/src/ipa/rpi/controller/sync_algorithm.h @@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * sync_algorithm.h - Camera sync algorithm interface + */ +#pragma once + +#include + +#include "algorithm.h" + +namespace RPiController { + +class SyncAlgorithm : public Algorithm +{ +public: + enum class Mode { + Off, + Server, + Client, + }; + + SyncAlgorithm(Controller *controller) + : Algorithm(controller) {} + virtual void setFrameDuration(libcamera::utils::Duration frameDuration) = 0; + virtual void setReadyFrame(unsigned int frame) = 0; + virtual void setMode(Mode mode) = 0; +}; + +} /* namespace RPiController */ diff --git a/src/ipa/rpi/controller/sync_status.h b/src/ipa/rpi/controller/sync_status.h new file mode 100644 index 00000000..10f97502 --- /dev/null +++ b/src/ipa/rpi/controller/sync_status.h @@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * sync_status.h - Sync algorithm params and status structures + */ +#pragma once + +#include + +struct SyncParams { + /* Wall clock time for this frame */ + uint64_t wallClock; + /* Kernel timestamp for this frame */ + uint64_t sensorTimestamp; +}; + +struct SyncStatus { + /* Frame length correction to apply */ + libcamera::utils::Duration frameDurationOffset; + /* Whether the "ready time" has been reached */ + bool ready; + /* Time until the "ready time" */ + int64_t timerValue; + /* Whether timerValue is known (client has to wait for a server message) */ + bool timerKnown; +}; From patchwork Thu Jan 9 14:32:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 22504 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 62ABEC32EA for ; Thu, 9 Jan 2025 14:32:32 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DBA4F684CD; Thu, 9 Jan 2025 15:32:31 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="b3bQWcYd"; dkim-atps=neutral Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7DB4668506 for ; Thu, 9 Jan 2025 15:32:21 +0100 (CET) Received: by mail-wm1-x335.google.com with SMTP id 5b1f17b1804b1-4361c705434so8093165e9.3 for ; Thu, 09 Jan 2025 06:32:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1736433141; x=1737037941; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sDULZN4NqTcgv4C9d2RSWDOhuj4WFpuSUlo+qOGh2Q0=; b=b3bQWcYdR/DLnLuxED2LIRpEdGT+nsg8ThLZ4EycZubpwKzYLBlBOS2E0jZ9bxKNTa YZ8oHHRJtBPQlG0hMnle9krZkfcEbdvhUUtuq/riNnAiZwBk3uW8KwkmbnEz2+8+Yemc H/RowZRY/rI6NfRJOlSV26preybpl9bwW1ecAh2tupnTlyZbXsmARLe+LXumlqRI+THI tFCGYcxmxPymwtoHcUMVP5PToRWNaBdusu3+aRe1+SlfgNOJJdnqc5LbAGBrTRG8c50M jKweKZlacIn4MKjPOWOkHd5gBK1l/aDMkN5nazBZQ2N7lF8Ha0YYiD4LxKw00NvEkO9+ lP+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736433141; x=1737037941; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sDULZN4NqTcgv4C9d2RSWDOhuj4WFpuSUlo+qOGh2Q0=; b=S1GD3RNwvBdeqhzqUPIv+WzsJ+qUGomMxW6xPosL8/EX6V5ZH6jY64bcMEvJISaRtk aD+AxIA/VwSfp7aueXJ1404PYMLPIPxizeD87p7e9MuKdgZJiA5Zq2t4MZ1/iR4f77kt gNWn07KJ5i8G/xMqEVfQyf7fvlm0MqYe6iGwOTbzI37iVPedCGQllYvXv/eX6sDtX/40 bR3yRlljUlpWi39kSq0skUzDeD4STjzBiUlkiXOdw/D049PnNHu43gXqFX7qPR/hHOcM dtMUB5iaM6YQ0MUa3b+Y7BsjpvYB0g7vmpZwe70KfY59m8KaXJPxAhZtAe0mbMLgKr7d q+ZA== X-Gm-Message-State: AOJu0YzzUBGIIhlmrrP9wMkHZMY2l6JvWUVituA/R1cRZVcVNgqd6jtH mh7pZ1gc8AAh61V/bHmFVhR/ax5QPlHtKg0rUhrE0a3nPyy2PX7IBPO1Lr9DrsUmDkHucTeMefV 8 X-Gm-Gg: ASbGncutqnw5XpkzGElGeFHq0GRPDTKQy8iJmXDUJNencf4jDTauMCwJXEEhAgKGurE qGDt1Dmmuv1d2oyMs94z1tAho9KUJFBRaGluYuXT7vobAMAAa6H391VEqoq6Uhqg549xlLkCtdr ymQZsIHGVCO8HUq58imDRBq1FuMZRaX43jtJG934MC8+ongAO1OZQvd7KpAqgb9MIxeOkKFRlKd j70i4eGZpjMz2Ones5RDpliQ3evYANI+vTcsTN9JAWSEWBGKPNLFEEa0x5w12bjZd3i3oPkNmUZ QcD9pleX6/y8 X-Google-Smtp-Source: AGHT+IG9HGOJEJ5VvNjG6tS7QNzc4aJZsMKTpJgSZXTtU2nb+R7Hha9yN5k78dvc8u5qBoxsEekEFg== X-Received: by 2002:a05:6000:1789:b0:38a:5122:5a07 with SMTP id ffacd0b85a97d-38a8730394cmr6842668f8f.15.1736433140644; Thu, 09 Jan 2025 06:32:20 -0800 (PST) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:ffc9:aff6:7f7f:893b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38a8e4b80b2sm1952569f8f.80.2025.01.09.06.32.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Jan 2025 06:32:20 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman , Arsen Mikovic , Naushir Patuck Subject: [PATCH v3 6/7] ipa: rpi: sync: Add an implementation of the camera sync algorithm Date: Thu, 9 Jan 2025 14:32:10 +0000 Message-Id: <20250109143211.11939-7-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250109143211.11939-1-david.plowman@raspberrypi.com> References: <20250109143211.11939-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In this implementation, the server sends data packets out onto the network every 30 frames or so. Clients listening for this packet will send frame length deltas back to the pipeline handler to match the synchronisation of the server. We use wallclock timestamps, passed to us from the pipeline handler, that have been de-jittered appropriately, meaning that the synchronisation will actually work across networked devices. When the server's advertised "ready time" is reached, both client and server will signal this through metadata back to their respective controlling applications. Signed-off-by: David Plowman Signed-off-by: Arsen Mikovic Signed-off-by: Naushir Patuck --- src/ipa/rpi/controller/meson.build | 1 + src/ipa/rpi/controller/rpi/sync.cpp | 330 ++++++++++++++++++++++++++++ src/ipa/rpi/controller/rpi/sync.h | 68 ++++++ 3 files changed, 399 insertions(+) create mode 100644 src/ipa/rpi/controller/rpi/sync.cpp create mode 100644 src/ipa/rpi/controller/rpi/sync.h diff --git a/src/ipa/rpi/controller/meson.build b/src/ipa/rpi/controller/meson.build index 74b74888..dde4ac12 100644 --- a/src/ipa/rpi/controller/meson.build +++ b/src/ipa/rpi/controller/meson.build @@ -23,6 +23,7 @@ rpi_ipa_controller_sources = files([ 'rpi/saturation.cpp', 'rpi/sdn.cpp', 'rpi/sharpen.cpp', + 'rpi/sync.cpp', 'rpi/tonemap.cpp', ]) diff --git a/src/ipa/rpi/controller/rpi/sync.cpp b/src/ipa/rpi/controller/rpi/sync.cpp new file mode 100644 index 00000000..43a8cbe6 --- /dev/null +++ b/src/ipa/rpi/controller/rpi/sync.cpp @@ -0,0 +1,330 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * sync.cpp - sync algorithm + */ +#include "sync.h" + +#include +#include +#include +#include +#include + +#include + +#include + +#include "sync_status.h" + +using namespace std; +using namespace std::chrono_literals; +using namespace RPiController; +using namespace libcamera; + +LOG_DEFINE_CATEGORY(RPiSync) + +#define NAME "rpi.sync" + +const char *kDefaultGroup = "239.255.255.250"; +constexpr unsigned int kDefaultPort = 10000; +constexpr unsigned int kDefaultSyncPeriod = 30; +constexpr unsigned int kDefaultReadyFrame = 100; +constexpr unsigned int kDefaultMinAdjustment = 50; + +Sync::Sync(Controller *controller) + : SyncAlgorithm(controller), mode_(Mode::Off), socket_(-1), frameDuration_(0s), frameCount_(0) +{ +} + +Sync::~Sync() +{ + if (socket_ >= 0) + close(socket_); +} + +char const *Sync::name() const +{ + return NAME; +} + +/* This reads from json file and intitiaises server and client */ +int Sync::read(const libcamera::YamlObject ¶ms) +{ + /* Socket on which to communicate. */ + group_ = params["group"].get(kDefaultGroup); + port_ = params["port"].get(kDefaultPort); + /* Send a sync message every this many frames. */ + syncPeriod_ = params["sync_period"].get(kDefaultSyncPeriod); + /* Application will be told we're ready after this many frames. */ + readyFrame_ = params["ready_frame"].get(kDefaultReadyFrame); + /* Don't change client frame length unless the change exceeds this amount (microseconds). */ + minAdjustment_ = params["min_adjustment"].get(kDefaultMinAdjustment); + + return 0; +} + +void Sync::initialiseSocket() +{ + socket_ = socket(AF_INET, SOCK_DGRAM, 0); + if (socket_ < 0) { + LOG(RPiSync, Error) << "Unable to create socket"; + return; + } + + memset(&addr_, 0, sizeof(addr_)); + addr_.sin_family = AF_INET; + addr_.sin_addr.s_addr = mode_ == Mode::Client ? htonl(INADDR_ANY) : inet_addr(group_.c_str()); + addr_.sin_port = htons(port_); + + if (mode_ == Mode::Client) { + /* Set to non-blocking. */ + int flags = fcntl(socket_, F_GETFL, 0); + fcntl(socket_, F_SETFL, flags | O_NONBLOCK); + + unsigned int en = 1; + if (setsockopt(socket_, SOL_SOCKET, SO_REUSEADDR, &en, sizeof(en)) < 0) { + LOG(RPiSync, Error) << "Unable to set socket options"; + goto err; + } + + struct ip_mreq mreq { + }; + mreq.imr_multiaddr.s_addr = inet_addr(group_.c_str()); + mreq.imr_interface.s_addr = htonl(INADDR_ANY); + if (setsockopt(socket_, IPPROTO_IP, IP_ADD_MEMBERSHIP, &mreq, sizeof(mreq)) < 0) { + LOG(RPiSync, Error) << "Unable to set socket options"; + goto err; + } + + if (bind(socket_, (struct sockaddr *)&addr_, sizeof(addr_)) < 0) { + LOG(RPiSync, Error) << "Unable to bind client socket"; + goto err; + } + } + + return; + +err: + close(socket_); + socket_ = -1; +} + +void Sync::switchMode([[maybe_unused]] CameraMode const &cameraMode, [[maybe_unused]] Metadata *metadata) +{ + /* + * A mode switch means the camera has stopped, so synchronisation will be lost. + * Reset all the internal state so that we start over. + */ + reset(); +} + +/* + * Camera sync algorithm. + * Server - there is a single server that sends framerate timing information over the network to any + * clients that are listening. It also signals when it will send a "everything is synchronised, now go" + * message back to the algorithm. + * Client - there may be many clients, either on the same Pi or different ones. They match their + * framerates to the server, and indicate when to "go" at the same instant as the server. + */ +void Sync::process([[maybe_unused]] StatisticsPtr &stats, Metadata *imageMetadata) +{ + SyncPayload payload; + SyncParams local{}; + SyncStatus status{}; + bool timerKnown = true; + + if (mode_ == Mode::Off) + return; + + if (!frameDuration_) { + LOG(RPiSync, Error) << "Sync frame duration not set!"; + return; + } + + if (socket_ < 0) { + initialiseSocket(); + + if (socket_ < 0) + return; + + /* + * For the client, flush anything in the socket. It might be stale from a previous sync run, + * or we might get another packet in a frame to two before the adjustment caused by this (old) + * packet, although correct, had taken effect. So this keeps things simpler. + */ + if (mode_ == Mode::Client) { + socklen_t addrlen = sizeof(addr_); + int ret = 0; + while (ret >= 0) + ret = recvfrom(socket_, &payload, sizeof(payload), 0, (struct sockaddr *)&addr_, &addrlen); + } + } + + imageMetadata->get("sync.params", local); + + /* The wallclock has already been de-jittered for us. */ + uint64_t wallClockFrameTimestamp = local.wallClock; + + /* + * This is the headline frame duration in microseconds as programmed into the sensor. Strictly, + * the sensor might not quite match the system clock, but this shouldn't matter for the calculations + * we'll do with it, unless it's a very very long way out! + */ + uint32_t frameDuration = frameDuration_.get(); + + /* Timestamps tell us if we've dropped any frames, but we still want to count them. */ + int droppedFrames = 0; + if (frameCount_) { + /* + * Round down here, because frameCount_ gets incremented at the end of the function. Also + * ensure droppedFrames can't go negative. It shouldn't, but things would go badly wrong + * if it did. + */ + wallClockFrameTimestamp = std::max(wallClockFrameTimestamp, lastWallClockFrameTimestamp_ + frameDuration / 2); + droppedFrames = (wallClockFrameTimestamp - lastWallClockFrameTimestamp_ - frameDuration / 2) / frameDuration; + frameCount_ += droppedFrames; + } + + if (mode_ == Mode::Server) { + /* + * Server sends a packet every syncPeriod_ frames, or as soon after as possible (if any + * frames were dropped). + */ + serverFrameCountPeriod_ += droppedFrames; + + /* + * The client may want a better idea of the true frame duration. Any error would feed straight + * into the correction term because of how it uses it to get the "nearest" frame. + */ + if (frameCount_ == 0) + frameDurationEstimated_ = frameDuration; + else { + double diff = (wallClockFrameTimestamp - lastWallClockFrameTimestamp_) / (1 + droppedFrames); + int N = std::min(frameCount_, 99U); + frameDurationEstimated_ = frameCount_ == 1 ? diff : (N * frameDurationEstimated_ + diff) / (N + 1); + } + + /* Calculate frames remaining, and therefore "time left until ready". */ + int framesRemaining = readyFrame_ - frameCount_; + uint64_t wallClockReadyTime = wallClockFrameTimestamp + (int64_t)framesRemaining * frameDurationEstimated_; + + if (serverFrameCountPeriod_ >= syncPeriod_) { + serverFrameCountPeriod_ = 0; + + payload.frameDuration = frameDurationEstimated_ + .5; /* round to nearest */ + payload.wallClockFrameTimestamp = wallClockFrameTimestamp; + payload.wallClockReadyTime = wallClockReadyTime; + + LOG(RPiSync, Debug) << "Send packet (frameNumber " << frameCount_ << "):"; + LOG(RPiSync, Debug) << " frameDuration " << payload.frameDuration; + LOG(RPiSync, Debug) << " wallClockFrameTimestamp " << wallClockFrameTimestamp + << " (" << wallClockFrameTimestamp - lastWallClockFrameTimestamp_ << ")"; + LOG(RPiSync, Debug) << " wallClockReadyTime " << wallClockReadyTime; + + if (sendto(socket_, &payload, sizeof(payload), 0, (const sockaddr *)&addr_, sizeof(addr_)) < 0) + LOG(RPiSync, Error) << "Send error! " << strerror(errno); + } + + timerValue_ = static_cast(wallClockReadyTime - wallClockFrameTimestamp); + if (!syncReady_ && wallClockFrameTimestamp + frameDurationEstimated_ / 2 > wallClockReadyTime) { + syncReady_ = true; + LOG(RPiSync, Info) << "*** Sync achieved! Difference " << timerValue_ << "us"; + } + + serverFrameCountPeriod_ += 1; + + } else if (mode_ == Mode::Client) { + uint64_t serverFrameTimestamp = 0; + + bool packetReceived = false; + while (true) { + socklen_t addrlen = sizeof(addr_); + int ret = recvfrom(socket_, &payload, sizeof(payload), 0, (struct sockaddr *)&addr_, &addrlen); + + if (ret < 0) + break; + packetReceived = (ret > 0); + clientSeenPacket_ = true; + + frameDurationEstimated_ = payload.frameDuration; + serverFrameTimestamp = payload.wallClockFrameTimestamp; + serverReadyTime_ = payload.wallClockReadyTime; + } + + if (packetReceived) { + uint64_t clientFrameTimestamp = wallClockFrameTimestamp; + int64_t clientServerDelta = clientFrameTimestamp - serverFrameTimestamp; + /* "A few frames ago" may have better matched the server's frame. Calculate when it was. */ + int framePeriodErrors = (clientServerDelta + frameDurationEstimated_ / 2) / frameDurationEstimated_; + int64_t clientFrameTimestampNearest = clientFrameTimestamp - framePeriodErrors * frameDurationEstimated_; + /* We must shorten a single client frame by this amount if it exceeds the minimum: */ + int32_t correction = clientFrameTimestampNearest - serverFrameTimestamp; + if (std::abs(correction) < minAdjustment_) + correction = 0; + + LOG(RPiSync, Debug) << "Received packet (frameNumber " << frameCount_ << "):"; + LOG(RPiSync, Debug) << " serverFrameTimestamp " << serverFrameTimestamp; + LOG(RPiSync, Debug) << " serverReadyTime " << serverReadyTime_; + LOG(RPiSync, Debug) << " clientFrameTimestamp " << clientFrameTimestamp; + LOG(RPiSync, Debug) << " clientFrameTimestampNearest " << clientFrameTimestampNearest + << " (" << framePeriodErrors << ")"; + LOG(RPiSync, Debug) << " correction " << correction; + + status.frameDurationOffset = correction * 1us; + } + + timerValue_ = static_cast(serverReadyTime_ - wallClockFrameTimestamp); + timerKnown = clientSeenPacket_; /* client must receive a packet before the timer value is correct */ + if (clientSeenPacket_ && !syncReady_ && wallClockFrameTimestamp + frameDurationEstimated_ / 2 > serverReadyTime_) { + syncReady_ = true; + LOG(RPiSync, Info) << "*** Sync achieved! Difference " << timerValue_ << "us"; + } + } + + lastWallClockFrameTimestamp_ = wallClockFrameTimestamp; + + status.ready = syncReady_; + status.timerValue = timerValue_; + status.timerKnown = timerKnown; + imageMetadata->set("sync.status", status); + frameCount_++; +} + +void Sync::reset() +{ + /* This resets the state so that the synchronisation procedure will start over. */ + syncReady_ = false; + frameCount_ = 0; + timerValue_ = 0; + serverFrameCountPeriod_ = 0; + serverReadyTime_ = 0; + clientSeenPacket_ = false; +} + +void Sync::setMode(Mode mode) +{ + mode_ = mode; + + /* Another "sync session" can be started by turning it off and on again. */ + if (mode == Mode::Off) + reset(); +} + +void Sync::setFrameDuration(libcamera::utils::Duration frameDuration) +{ + frameDuration_ = frameDuration; +}; + +void Sync::setReadyFrame(unsigned int frame) +{ + readyFrame_ = frame; +}; + +/* Register algorithm with the system. */ +static Algorithm *create(Controller *controller) +{ + return (Algorithm *)new Sync(controller); +} +static RegisterAlgorithm reg(NAME, &create); diff --git a/src/ipa/rpi/controller/rpi/sync.h b/src/ipa/rpi/controller/rpi/sync.h new file mode 100644 index 00000000..d3c79b7a --- /dev/null +++ b/src/ipa/rpi/controller/rpi/sync.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * sync.h - sync algorithm + */ +#pragma once + +#include + +#include "../sync_algorithm.h" + +namespace RPiController { + +struct SyncPayload { + /* Frame duration in microseconds. */ + uint32_t frameDuration; + /* Server system (kernel) frame timestamp. */ + uint64_t systemFrameTimestamp; + /* Server wall clock version of the frame timestamp. */ + uint64_t wallClockFrameTimestamp; + /* Server system (kernel) sync time (the time at which frames are marked ready). */ + uint64_t systemReadyTime; + /* Server wall clock version of the sync time. */ + uint64_t wallClockReadyTime; +}; + +class Sync : public SyncAlgorithm +{ +public: + Sync(Controller *controller); + ~Sync(); + char const *name() const override; + int read(const libcamera::YamlObject ¶ms) override; + void setMode(Mode mode) override; + void initialiseSocket(); + void switchMode(CameraMode const &cameraMode, Metadata *metadata) override; + void process(StatisticsPtr &stats, Metadata *imageMetadata) override; + void setFrameDuration(libcamera::utils::Duration frameDuration) override; + void setReadyFrame(unsigned int frame) override; + +private: + void reset(); /* reset internal state and start over */ + + Mode mode_; /* server or client */ + std::string group_; /* IP group address for sync messages */ + uint16_t port_; /* port number for messages */ + uint32_t syncPeriod_; /* send a sync message every this many frames */ + uint32_t readyFrame_; /* tell the application we're ready after this many frames */ + uint32_t minAdjustment_; /* don't adjust the client frame length by less than this */ + + struct sockaddr_in addr_; + int socket_ = -1; + libcamera::utils::Duration frameDuration_; + unsigned int frameCount_; + bool syncReady_; + int64_t timerValue_ = 0; /* time until "ready time" */ + + double frameDurationEstimated_ = 0; /* estimate the true frame duration of the sensor */ + uint64_t lastWallClockFrameTimestamp_; /* wall clock timestamp of previous frame */ + + uint32_t serverFrameCountPeriod_ = 0; /* send the next packet when this reaches syncPeriod_ */ + + bool clientSeenPacket_ = false; /* whether the client has received a packet yet */ + uint64_t serverReadyTime_ = 0; /* the client's latest value for when the server will be "ready" */ +}; + +} /* namespace RPiController */ From patchwork Thu Jan 9 14:32:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 22506 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 21CF0C32F7 for ; Thu, 9 Jan 2025 14:32:35 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 7CE716851D; Thu, 9 Jan 2025 15:32:34 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="BRGVoNZP"; dkim-atps=neutral Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 82B7868516 for ; Thu, 9 Jan 2025 15:32:23 +0100 (CET) Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-385d7f19f20so490601f8f.1 for ; Thu, 09 Jan 2025 06:32:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1736433143; x=1737037943; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qcY4uAmG5C+sI2GY5kJK1UPtgl4qHQf8ffKUnSUWfXc=; b=BRGVoNZPdWXv/9uIo6vg5Yh0zDmorgCh3YmexyvT1JpQ8YSG8b4kesDX4Rl0cepZe0 /Q0XWq8y6eAILQSV27WufDlfxVKV6PPFv727dDTFF+5NVxF6ZLyjQiBk9WjemmsJs+h2 tdr5wxALSIwIlvV9+MKjm3jqnKfy1EplGY7bS7U4UUbGk50R5D5zQ3jbjjM5UZYkWrBh ayxbh439ZAVQFnykNHTLHyzbeCShUjWxdmNi5zowJRZ9zOm6yozX6W7HkYQ+34Ux0Hwx Na78enyauk4umC65arGOGze9pghETIYzUOdwDAV1BGQa7HHNSccDmaJXj/bDK05/80U0 fKRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736433143; x=1737037943; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qcY4uAmG5C+sI2GY5kJK1UPtgl4qHQf8ffKUnSUWfXc=; b=SzYTZiauPqIvgpCFpRseXWMWlyap1Gie4OSZ7IBUb/Y9T8mXBSjXcmItYdUzg+j8qZ tHP55836rwekpnnVG2BTVbvCQlNxhC8M1JPGXfKuhHsIanIhchIciVwZBLlXBPzrrrhR ZQHIxSJIfn32eCNw5qlTkI0aQoNBLJ8gyDOq/42KU9ooCsYgEV9XlBo6ReZFXPO6xuKt tyvGpHe1kBLKrtIkpw0toBnv9JENKqWMNvUEBKBk50RdXdgxrUiWX5XVU9CDNbKgrcIf TPHvRyvDiSlX28+HMAiCs5Azj1TlMDpgiAjk+LlpPO7UT8A1MDu2qvl0187i07Smf00u aM3w== X-Gm-Message-State: AOJu0YxOXTliUm/3xfRtsDCPTcD133VS2VsHoMQErYRkOrTzG8K4OdHC 4qoZ/tWKySc/h3oIaVXShPpwz5jL5zePNMR4CAEVmeKErBg8jOqG67bvNdZL17/c+LO1TpsiflH f X-Gm-Gg: ASbGncv/6n7nkGueztrhSlVOeoaDIdpmHb0VVz6JOOyrAlWsAoF3J0I2b2qQ/wXGbDI u65yg6fgxP9GPviEw+29ZawNReR01BpTbG+F1e5gGrEO3B20OmM3UdGoW/D5isU+tQznT4b9esk Lwq8xA29J5hqVxCckHoo+CvKBsuas6WHqV0wcI1sL6oS5rs4awAkPpX+P3JlYeOhxeoSCacbEvS Q92r5HXsbWfs/qp93N9aOlrDicgFGeArztXB4Hjny0gOGwGM173pC+F9zLQfYyqLTgLjFH7nN0Y xefJVCQQIqKA X-Google-Smtp-Source: AGHT+IEQXPN4GEsArl8feajvg6VsG7wcnxxv7jmImlep4QRFDVebZZn5QVmju+oIDdUNeykgaAqkQw== X-Received: by 2002:a05:6000:1f88:b0:385:fc97:9c63 with SMTP id ffacd0b85a97d-38a872f6915mr5631573f8f.9.1736433141334; Thu, 09 Jan 2025 06:32:21 -0800 (PST) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:ffc9:aff6:7f7f:893b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38a8e4b80b2sm1952569f8f.80.2025.01.09.06.32.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Jan 2025 06:32:20 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman Subject: [PATCH v3 7/7] ipa: rpi: vc4: Update all tuning files for sync algorithm Date: Thu, 9 Jan 2025 14:32:11 +0000 Message-Id: <20250109143211.11939-8-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250109143211.11939-1-david.plowman@raspberrypi.com> References: <20250109143211.11939-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Default to all the standard parameter values. Signed-off-by: David Plowman --- src/ipa/rpi/vc4/data/imx219.json | 7 ++++++- src/ipa/rpi/vc4/data/imx219_noir.json | 7 ++++++- src/ipa/rpi/vc4/data/imx283.json | 7 ++++++- src/ipa/rpi/vc4/data/imx290.json | 7 ++++++- src/ipa/rpi/vc4/data/imx296.json | 7 ++++++- src/ipa/rpi/vc4/data/imx296_mono.json | 7 ++++++- src/ipa/rpi/vc4/data/imx378.json | 7 ++++++- src/ipa/rpi/vc4/data/imx477.json | 7 ++++++- src/ipa/rpi/vc4/data/imx477_noir.json | 7 ++++++- src/ipa/rpi/vc4/data/imx477_scientific.json | 7 ++++++- src/ipa/rpi/vc4/data/imx519.json | 7 ++++++- src/ipa/rpi/vc4/data/imx708.json | 7 ++++++- src/ipa/rpi/vc4/data/imx708_noir.json | 7 ++++++- src/ipa/rpi/vc4/data/imx708_wide.json | 7 ++++++- src/ipa/rpi/vc4/data/imx708_wide_noir.json | 7 ++++++- src/ipa/rpi/vc4/data/ov5647.json | 7 ++++++- src/ipa/rpi/vc4/data/ov5647_noir.json | 7 ++++++- src/ipa/rpi/vc4/data/se327m12.json | 7 ++++++- 18 files changed, 108 insertions(+), 18 deletions(-) diff --git a/src/ipa/rpi/vc4/data/imx219.json b/src/ipa/rpi/vc4/data/imx219.json index a020b12f..21f82dd2 100644 --- a/src/ipa/rpi/vc4/data/imx219.json +++ b/src/ipa/rpi/vc4/data/imx219.json @@ -690,6 +690,11 @@ } } } - } + }, + { + "rpi.sync": + { + } + } ] } diff --git a/src/ipa/rpi/vc4/data/imx219_noir.json b/src/ipa/rpi/vc4/data/imx219_noir.json index d8bc9639..a51a5387 100644 --- a/src/ipa/rpi/vc4/data/imx219_noir.json +++ b/src/ipa/rpi/vc4/data/imx219_noir.json @@ -624,6 +624,11 @@ } } } - } + }, + { + "rpi.sync": + { + } + } ] } diff --git a/src/ipa/rpi/vc4/data/imx283.json b/src/ipa/rpi/vc4/data/imx283.json index bfacecc8..d98f3f08 100644 --- a/src/ipa/rpi/vc4/data/imx283.json +++ b/src/ipa/rpi/vc4/data/imx283.json @@ -308,6 +308,11 @@ }, { "rpi.sharpen": { } - } + }, + { + "rpi.sync": + { + } + } ] } diff --git a/src/ipa/rpi/vc4/data/imx290.json b/src/ipa/rpi/vc4/data/imx290.json index 8f41bf51..b2757e29 100644 --- a/src/ipa/rpi/vc4/data/imx290.json +++ b/src/ipa/rpi/vc4/data/imx290.json @@ -209,6 +209,11 @@ } ] } - } + }, + { + "rpi.sync": + { + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx296.json b/src/ipa/rpi/vc4/data/imx296.json index 8f24ce5b..222cb372 100644 --- a/src/ipa/rpi/vc4/data/imx296.json +++ b/src/ipa/rpi/vc4/data/imx296.json @@ -438,6 +438,11 @@ "strength": 1.0, "limit": 0.18 } - } + }, + { + "rpi.sync": + { + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx296_mono.json b/src/ipa/rpi/vc4/data/imx296_mono.json index fe331569..bbf78b3b 100644 --- a/src/ipa/rpi/vc4/data/imx296_mono.json +++ b/src/ipa/rpi/vc4/data/imx296_mono.json @@ -235,6 +235,11 @@ "strength": 1.0, "limit": 0.18 } - } + }, + { + "rpi.sync": + { + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx378.json b/src/ipa/rpi/vc4/data/imx378.json index 363b47e1..e4276732 100644 --- a/src/ipa/rpi/vc4/data/imx378.json +++ b/src/ipa/rpi/vc4/data/imx378.json @@ -422,6 +422,11 @@ }, { "rpi.sharpen": { } - } + }, + { + "rpi.sync": + { + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx477.json b/src/ipa/rpi/vc4/data/imx477.json index fa25ee86..ece91db9 100644 --- a/src/ipa/rpi/vc4/data/imx477.json +++ b/src/ipa/rpi/vc4/data/imx477.json @@ -695,6 +695,11 @@ } } } - } + }, + { + "rpi.sync": + { + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx477_noir.json b/src/ipa/rpi/vc4/data/imx477_noir.json index 472f33fe..13426ee3 100644 --- a/src/ipa/rpi/vc4/data/imx477_noir.json +++ b/src/ipa/rpi/vc4/data/imx477_noir.json @@ -651,6 +651,11 @@ } } } - } + }, + { + "rpi.sync": + { + } + } ] } diff --git a/src/ipa/rpi/vc4/data/imx477_scientific.json b/src/ipa/rpi/vc4/data/imx477_scientific.json index 9dc32eb1..de028986 100644 --- a/src/ipa/rpi/vc4/data/imx477_scientific.json +++ b/src/ipa/rpi/vc4/data/imx477_scientific.json @@ -483,6 +483,11 @@ }, { "rpi.sharpen": { } - } + }, + { + "rpi.sync": + { + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx519.json b/src/ipa/rpi/vc4/data/imx519.json index ce194256..d7e33f9f 100644 --- a/src/ipa/rpi/vc4/data/imx519.json +++ b/src/ipa/rpi/vc4/data/imx519.json @@ -422,6 +422,11 @@ }, { "rpi.sharpen": { } - } + }, + { + "rpi.sync": + { + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx708.json b/src/ipa/rpi/vc4/data/imx708.json index 4de6f079..35c60509 100644 --- a/src/ipa/rpi/vc4/data/imx708.json +++ b/src/ipa/rpi/vc4/data/imx708.json @@ -666,6 +666,11 @@ } } } - } + }, + { + "rpi.sync": + { + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx708_noir.json b/src/ipa/rpi/vc4/data/imx708_noir.json index 7b7ee874..495a2a9e 100644 --- a/src/ipa/rpi/vc4/data/imx708_noir.json +++ b/src/ipa/rpi/vc4/data/imx708_noir.json @@ -765,6 +765,11 @@ } } } - } + }, + { + "rpi.sync": + { + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx708_wide.json b/src/ipa/rpi/vc4/data/imx708_wide.json index 6f45aafc..077f740e 100644 --- a/src/ipa/rpi/vc4/data/imx708_wide.json +++ b/src/ipa/rpi/vc4/data/imx708_wide.json @@ -677,6 +677,11 @@ } } } - } + }, + { + "rpi.sync": + { + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx708_wide_noir.json b/src/ipa/rpi/vc4/data/imx708_wide_noir.json index b9a5227e..3760925d 100644 --- a/src/ipa/rpi/vc4/data/imx708_wide_noir.json +++ b/src/ipa/rpi/vc4/data/imx708_wide_noir.json @@ -668,6 +668,11 @@ } } } - } + }, + { + "rpi.sync": + { + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/ov5647.json b/src/ipa/rpi/vc4/data/ov5647.json index 40c6059c..d8241004 100644 --- a/src/ipa/rpi/vc4/data/ov5647.json +++ b/src/ipa/rpi/vc4/data/ov5647.json @@ -691,6 +691,11 @@ } } } - } + }, + { + "rpi.sync": + { + } + } ] } diff --git a/src/ipa/rpi/vc4/data/ov5647_noir.json b/src/ipa/rpi/vc4/data/ov5647_noir.json index 488b7119..accceb8b 100644 --- a/src/ipa/rpi/vc4/data/ov5647_noir.json +++ b/src/ipa/rpi/vc4/data/ov5647_noir.json @@ -407,6 +407,11 @@ }, { "rpi.sharpen": { } - } + }, + { + "rpi.sync": + { + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/se327m12.json b/src/ipa/rpi/vc4/data/se327m12.json index 948169db..48512b50 100644 --- a/src/ipa/rpi/vc4/data/se327m12.json +++ b/src/ipa/rpi/vc4/data/se327m12.json @@ -427,6 +427,11 @@ "strength": 0.5, "limit": 0.5 } - } + }, + { + "rpi.sync": + { + } + } ] } \ No newline at end of file