From patchwork Tue Nov 26 12:17:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 22099 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D69B1C0DA4 for ; Tue, 26 Nov 2024 12:17:19 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9D10466070; Tue, 26 Nov 2024 13:17:16 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="R6p3T/dr"; dkim-atps=neutral Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D337B66069 for ; Tue, 26 Nov 2024 13:17:12 +0100 (CET) Received: by mail-wr1-x435.google.com with SMTP id ffacd0b85a97d-3825c05cc90so3607333f8f.1 for ; Tue, 26 Nov 2024 04:17:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1732623432; x=1733228232; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=E3I3DGbsQGhBGHn93NSpiB8CG8LBqlqFxrF4Mwh1Yc8=; b=R6p3T/druiQLii0WSZfSN9owqcYdtlvTzYQwMwGwZhue+ZGnM81WMwIPwrFnsYRuN3 kOcY19N6FZTPLb0XaGs9EaP7HOaqOhO6BQDIjF98gI7EpztTB3BJ60/6FVHZ+UyxEOrN r5pRxkwifUH+cqBL8jOzU+EPEh4G7BJZvaRb98E69nm8y+WhsSNLSB+abnmzIuTLqAKY Vwavd9qt+s+VshCvUj86OGNt90WJHNSsCSGQI3GuaJk1lyXL+30pWSaqPOd/rYfxo842 LdQpocX3T/Yx5xRoGxDB1iLWbGhp0qq2Oa6C0ls/I3r5Pu6kTKPDjKcrvBPf+7Vf8BvZ pKxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732623432; x=1733228232; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=E3I3DGbsQGhBGHn93NSpiB8CG8LBqlqFxrF4Mwh1Yc8=; b=UpfjG0M8F9wHvpbQr5ujGakgGB1GqSrMcV3+Po7qysEnSkjiKux4Dkf1CJZOAsHjoQ hpc/C38xxawQAyUFCG+WYu+6c9uGgp1jf47S7/fhbz5VCCQB+6UODW8u+vv8LWLjFUVs pA8PSg4z533ypYatngAHOaRTfqBbZNJjxjRFNM5Z9IatbMUwodw+tcwE1z5zq3S+EedD ZE7JdzB8NY65E1xfPQ/HECK6sbBEi5P9PE2KXh9MN8IY7yMcdH6XK1I7YOdWb6qw1601 wlphrQvAkglf+n55HQGA2ANkssdb7WPz1QxgIrltRovvNUwAKh1glTox/qNOw7cX6WXl RuuQ== X-Gm-Message-State: AOJu0Yz3Dkmov1j+ohE69opIYIy44/OJrAcorVnMG+TfUujez7ZuEY+G UYcUl+b7qbbmvG+IHGcw1T6CW32+37M8Ia9mkA0OzfOwDV7c6Er/dxrf8lbsnSPOgyK7BccEC0B n X-Gm-Gg: ASbGnctcVo/cJu+KPkuvyZ9btq30SuqdwBrFhxSUOMhRcp+CAPUVa4qlZhehWS9Luff RUwUNoG1z66Pa+DWaO8yNuQ235JwkNnvI+r4ekTFSjxekUFSlNdxB9AcuZxfoE309gdSHIs1dzx zWZ2HwX65s7ANBma9VKGTh0F4uqYnzNYvAOQkMTvInPti2oI7cVI+xO3SoXsU/Ok2GupNlJCObk DdmMa/4aLgGNkvUhQ3+b8+kwyuonxJDjOzjiveGGi+iw1iogZBTybU3D5R881apIhUgJ07GlSdc ScezKA== X-Google-Smtp-Source: AGHT+IF52tpJMal+t5FhVB6TTekZ3zvTRlSt2Lz523dIcLU961Hvjyta3OOiGBUagtca1/Fek5Sadg== X-Received: by 2002:a5d:6d82:0:b0:382:4b6f:24f5 with SMTP id ffacd0b85a97d-38260bce4femr14359745f8f.41.1732623431742; Tue, 26 Nov 2024 04:17:11 -0800 (PST) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:c68a:6be1:5ba3:eddd]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-434a15d86a4sm51070325e9.36.2024.11.26.04.17.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Nov 2024 04:17:10 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman Subject: [RFC PATCH 3/3] libcamera: v4l2: Add wallclock metadata to video devices Date: Tue, 26 Nov 2024 12:17:06 +0000 Message-Id: <20241126121706.4350-4-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241126121706.4350-1-david.plowman@raspberrypi.com> References: <20241126121706.4350-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" FrameMetadata is extended to include wallclock timestamps, both "raw" (unsmoothed) and de-jittered versions. The "raw" wallclock timestamps are recorded when frames start, and at frame end we generate a de-jittered version of this wallclock, handing both out to pipeline handlers. Signed-off-by: David Plowman --- include/libcamera/framebuffer.h | 2 ++ include/libcamera/internal/v4l2_device.h | 4 +++ include/libcamera/internal/v4l2_videodevice.h | 3 ++ src/libcamera/v4l2_device.cpp | 22 ++++++++++++++ src/libcamera/v4l2_videodevice.cpp | 29 +++++++++++++++++++ 5 files changed, 60 insertions(+) diff --git a/include/libcamera/framebuffer.h b/include/libcamera/framebuffer.h index ff839243..a181d97a 100644 --- a/include/libcamera/framebuffer.h +++ b/include/libcamera/framebuffer.h @@ -35,6 +35,8 @@ struct FrameMetadata { Status status; unsigned int sequence; uint64_t timestamp; + uint64_t wallClockRaw; + uint64_t wallClock; Span planes() { return planes_; } Span planes() const { return planes_; } diff --git a/include/libcamera/internal/v4l2_device.h b/include/libcamera/internal/v4l2_device.h index f5aa5024..90dbc4a8 100644 --- a/include/libcamera/internal/v4l2_device.h +++ b/include/libcamera/internal/v4l2_device.h @@ -10,6 +10,7 @@ #include #include #include +#include #include #include @@ -67,6 +68,9 @@ protected: template static int fromColorSpace(const std::optional &colorSpace, T &v4l2Format); + std::queue> wallClockQueue_; + bool frameStartEnabled() const { return frameStartEnabled_; } + private: static ControlType v4l2CtrlType(uint32_t ctrlType); static std::unique_ptr v4l2ControlId(const v4l2_query_ext_ctrl &ctrl); diff --git a/include/libcamera/internal/v4l2_videodevice.h b/include/libcamera/internal/v4l2_videodevice.h index f021c2a0..d6127928 100644 --- a/include/libcamera/internal/v4l2_videodevice.h +++ b/include/libcamera/internal/v4l2_videodevice.h @@ -31,6 +31,7 @@ #include #include +#include "libcamera/internal/clock_recovery.h" #include "libcamera/internal/formats.h" #include "libcamera/internal/v4l2_device.h" #include "libcamera/internal/v4l2_pixelformat.h" @@ -290,6 +291,8 @@ private: Timer watchdog_; utils::Duration watchdogDuration_; + + ClockRecovery wallClockRecovery_; }; class V4L2M2MDevice diff --git a/src/libcamera/v4l2_device.cpp b/src/libcamera/v4l2_device.cpp index 7d21cf15..285527b4 100644 --- a/src/libcamera/v4l2_device.cpp +++ b/src/libcamera/v4l2_device.cpp @@ -474,6 +474,11 @@ int V4L2Device::ioctl(unsigned long request, void *argp) * \return The V4L2 device file descriptor, -1 if the device node is not open */ +/** + * \fn V4L2Device::frameStartEnabled() + * \return True if frame start notifications are enabled, otherwise false + */ + /** * \brief Retrieve the libcamera control type associated with the V4L2 control * \param[in] ctrlType The V4L2 control type @@ -761,6 +766,23 @@ void V4L2Device::eventAvailable() return; } + /* + * Record this frame (by its sequence number) and its corresponding wallclock value. + * Use a queue as these two events may not interleave perfectly. + */ + auto now = std::chrono::system_clock::now(); + uint64_t wallClock = std::chrono::duration_cast(now.time_since_epoch()).count(); + + wallClockQueue_.emplace(event.u.frame_sync.frame_sequence, wallClock); + + /* + * Also avoid growing the queue indefiniteily. It seems highly unlikely that you could + * get more than a few "frame starts" being processed without a "frame end", so the value + * 5, while arbitrary, appears to be more than enough in practice. + */ + while (wallClockQueue_.size() > 5) + wallClockQueue_.pop(); + frameStart.emit(event.u.frame_sync.frame_sequence); } diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp index 14eba056..5c5dfc79 100644 --- a/src/libcamera/v4l2_videodevice.cpp +++ b/src/libcamera/v4l2_videodevice.cpp @@ -1865,6 +1865,33 @@ FrameBuffer *V4L2VideoDevice::dequeueBuffer() metadata.timestamp = buf.timestamp.tv_sec * 1000000000ULL + buf.timestamp.tv_usec * 1000ULL; + if (frameStartEnabled()) { + /* + * Find the wallclock that should have been recorded for this frame, discarding any + * stale frames on the way. + */ + while (!wallClockQueue_.empty() && wallClockQueue_.front().first < buf.sequence) + wallClockQueue_.pop(); + + if (!wallClockQueue_.empty() && wallClockQueue_.front().first == buf.sequence) { + metadata.wallClockRaw = wallClockQueue_.front().second; + wallClockQueue_.pop(); + } else { + /* + * At higher framerates it can happen that this gets handled before the frame + * start event, meaning there's no wallclock time in the queue. So the best we + * can do is sample the wallclock now. (The frame start will subsequently add + * another wallclock timestamp, but this will get safely discarded.) + */ + auto now = std::chrono::system_clock::now(); + metadata.wallClockRaw = std::chrono::duration_cast(now.time_since_epoch()).count(); + } + + /* Now recover a de-jittered wallclock value. Everything must be in microseconds. */ + wallClockRecovery_.addSample(metadata.timestamp / 1000, metadata.wallClockRaw); + metadata.wallClock = wallClockRecovery_.getOutput(metadata.timestamp / 1000); + } + if (V4L2_TYPE_IS_OUTPUT(buf.type)) return buffer; @@ -1958,6 +1985,8 @@ int V4L2VideoDevice::streamOn() firstFrame_.reset(); + wallClockQueue_ = {}; + ret = ioctl(VIDIOC_STREAMON, &bufferType_); if (ret < 0) { LOG(V4L2, Error)