From patchwork Fri Dec 6 14:27:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 22214 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E8252BE173 for ; Fri, 6 Dec 2024 14:27:57 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id CFC3A6616B; Fri, 6 Dec 2024 15:27:52 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="qJOVOanT"; dkim-atps=neutral Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D6964618B3 for ; Fri, 6 Dec 2024 15:27:47 +0100 (CET) Received: by mail-wr1-x435.google.com with SMTP id ffacd0b85a97d-385e3621518so1580564f8f.1 for ; Fri, 06 Dec 2024 06:27:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1733495267; x=1734100067; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oAcvVu9/nphoPJeM1gayrL8QAg+lnY0IAKNaVkIKf7A=; b=qJOVOanTCDD+m8M+e3GDldf3NtTRsNcOXJJxH5vTgInNTbyXakbneTTau0jJQJVXUb 7u0Dc4PZ6PsYnXw5Wx5s9YhYe4dEdEcLd+p3SBhUMHcRTM7yr7J+RU0Cc+iGfaHutTN4 1opTccEOXvaoq2geHaq9lKG4kzoaJedeTONu04nyE3dts8FIKVIGRrJp+Nnds9X7DUVt imrz4u6yBbdXrjxgb/x5WOHnkjF09cUvrAka3qHiwiOYYY+eDuzAd66xlW/chp3iXBQ4 p3LdRkSlGYJyaezQHcD61PkbC7XIWa9lz/qKgvJutylrVQDXWPGhZaakfESlSts86J6h Oxgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733495267; x=1734100067; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oAcvVu9/nphoPJeM1gayrL8QAg+lnY0IAKNaVkIKf7A=; b=TSx666v22bdnPVydcV5GUX8S1oeunokVFgXQuVmVFTgwYWUdDIqAnDcKlGcpaxSd0O IvZ8La30isbDdCDQLSyLleNnKmYQRfv5cchP+xXYIE/dk9nLt+kh4GHcm5g8S9GR+EvT YwxCHDoqEzIEero6bpKehm58LQxZKnhc4hk0XSdnn27HMBIt3vfgwe6qK0kd4zfx9aYq Y+EmtZ9byo2U8uRQNkWjOBPWksR/K0YveExR2h3DcC3+cm+fdlJuDXWDw3DA3drGK2LJ WczD5pk6SeXH12DCZIASE0EvXm+jeQQime4fo7HZFBrRYDklRyG+hk85kvrscLiXPwkz oPhA== X-Gm-Message-State: AOJu0Yzc+IRPKUiBPgfESQbfWckNYfvqKPPRH8Rv4GaP2mzGkBbtP/wU qko8jZ0at3UrO4EkEYzWX5vDnfz2jJZKq+e6dWd9AfWxGw9e5V8Cw/NtEe+al8vow/ZZrPQGP8E f X-Gm-Gg: ASbGnct/WZ1rn9yEK2AOlxmqHOvzbnLsha3jV1xQ0nxPMC87dpPmb0ljofnTOZoFELv 93uOTSz3lZSu2d8p9R3bZl3qW6ef1atduvzj1aCFg9YHqGD7rnXweajMg1Ahzeax483HJm7SyIF ZK8anipuum03CxTDQueYP7c3JB6m1+O6UBObtx6qI7oE0HBg6ey7KOFUYlbkW+FVgwlH4IgEVnt d7zrB8HHRZdvnI/Hh1Ea2rip9HLNQtg+JW2P7gPGoMHEL1IYsmG+XrRcGOVxUmzmxIY5JAF5GU= X-Google-Smtp-Source: AGHT+IFRtD17/etU97O2m2ykLDQVmAa7qfsAw76noXUeAIF4ce7P+5BSMmzss4DB32GAfb9qYcfHMQ== X-Received: by 2002:a05:6000:717:b0:385:f4b0:596 with SMTP id ffacd0b85a97d-3862b3632ecmr2536499f8f.18.1733495267051; Fri, 06 Dec 2024 06:27:47 -0800 (PST) Received: from localhost.localdomain ([88.202.252.90]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-386220b071dsm4608219f8f.101.2024.12.06.06.27.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Dec 2024 06:27:46 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman Subject: [PATCH 3/5] libcamera: v4l2: Add wallclock metadata to video devices Date: Fri, 6 Dec 2024 14:27:40 +0000 Message-Id: <20241206142742.7931-4-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241206142742.7931-1-david.plowman@raspberrypi.com> References: <20241206142742.7931-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" FrameMetadata is extended to include a wallclock timestamp. When a frame is dequeued, we use the ClockRecovery class to generate a good estimate of a wallclock timestamp to correspond to the frame's SensorTimestamp. Pipeline handlers must enable wallclocks for chosen devices by passing an appropriate ClockRecovery object. Signed-off-by: David Plowman --- include/libcamera/framebuffer.h | 1 + include/libcamera/internal/v4l2_videodevice.h | 5 +++ src/libcamera/v4l2_videodevice.cpp | 36 ++++++++++++++++++- 3 files changed, 41 insertions(+), 1 deletion(-) diff --git a/include/libcamera/framebuffer.h b/include/libcamera/framebuffer.h index ff839243..4579d0c6 100644 --- a/include/libcamera/framebuffer.h +++ b/include/libcamera/framebuffer.h @@ -35,6 +35,7 @@ struct FrameMetadata { Status status; unsigned int sequence; uint64_t timestamp; + uint64_t wallClock; Span planes() { return planes_; } Span planes() const { return planes_; } diff --git a/include/libcamera/internal/v4l2_videodevice.h b/include/libcamera/internal/v4l2_videodevice.h index f021c2a0..5327c112 100644 --- a/include/libcamera/internal/v4l2_videodevice.h +++ b/include/libcamera/internal/v4l2_videodevice.h @@ -37,6 +37,7 @@ namespace libcamera { +class ClockRecovery; class EventNotifier; class MediaDevice; class MediaEntity; @@ -232,6 +233,8 @@ public: V4L2PixelFormat toV4L2PixelFormat(const PixelFormat &pixelFormat) const; + void enableWallClock(ClockRecovery *wallClockRecovery); + protected: std::string logPrefix() const override; @@ -290,6 +293,8 @@ private: Timer watchdog_; utils::Duration watchdogDuration_; + + ClockRecovery *wallClockRecovery_; }; class V4L2M2MDevice diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp index a5cf6784..2007dffc 100644 --- a/src/libcamera/v4l2_videodevice.cpp +++ b/src/libcamera/v4l2_videodevice.cpp @@ -26,6 +26,7 @@ #include #include +#include "libcamera/internal/clock_recovery.h" #include "libcamera/internal/formats.h" #include "libcamera/internal/framebuffer.h" #include "libcamera/internal/media_device.h" @@ -534,7 +535,7 @@ std::ostream &operator<<(std::ostream &out, const V4L2DeviceFormat &f) V4L2VideoDevice::V4L2VideoDevice(const std::string &deviceNode) : V4L2Device(deviceNode), formatInfo_(nullptr), cache_(nullptr), fdBufferNotifier_(nullptr), state_(State::Stopped), - watchdogDuration_(0.0) + watchdogDuration_(0.0), wallClockRecovery_(nullptr) { /* * We default to an MMAP based CAPTURE video device, however this will @@ -1865,6 +1866,17 @@ FrameBuffer *V4L2VideoDevice::dequeueBuffer() metadata.timestamp = buf.timestamp.tv_sec * 1000000000ULL + buf.timestamp.tv_usec * 1000ULL; + metadata.wallClock = 0; + if (wallClockRecovery_) { + /* + * Sample the internal (CLOCK_BOOTTIME) and realtime (CLOCK_REALTIME) clocks and + * update the clock recovery model. Then derive the wallclock estimate for the + * frame timestamp. + */ + wallClockRecovery_->addSample(); + metadata.wallClock = wallClockRecovery_->getOutput(metadata.timestamp / 1000); + } + if (V4L2_TYPE_IS_OUTPUT(buf.type)) return buffer; @@ -1965,6 +1977,11 @@ int V4L2VideoDevice::streamOn() return ret; } + if (wallClockRecovery_) { + /* A good moment to sample the clocks to improve the clock recovery model. */ + wallClockRecovery_->addSample(); + } + state_ = State::Streaming; if (watchdogDuration_ && !queuedBuffers_.empty()) watchdog_.start(std::chrono::duration_cast(watchdogDuration_)); @@ -2120,6 +2137,23 @@ V4L2PixelFormat V4L2VideoDevice::toV4L2PixelFormat(const PixelFormat &pixelForma return {}; } +/** + * \brief Enable wall clock timestamps for this device + * \param[in] wallClockRecovery an appropriately configured ClockRecovery, or + * nullptr to disable wall clocks + * + * When buffers are dequeued, wall clock timestamps will be generated that + * correspond to the frame's timestamp, as returned by V4l2. + */ +void V4L2VideoDevice::enableWallClock(ClockRecovery *wallClockRecovery) +{ + wallClockRecovery_ = wallClockRecovery; + + /* Also a reasonable moment to sample the two clocks. */ + if (wallClockRecovery_) + wallClockRecovery_->addSample(); +} + /** * \class V4L2M2MDevice * \brief Memory-to-Memory video device