From patchwork Tue Nov 26 12:17:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 22096 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 60AD6C0DA4 for ; Tue, 26 Nov 2024 12:17:13 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4F63966063; Tue, 26 Nov 2024 13:17:12 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="StF9x7ZK"; dkim-atps=neutral Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5057F65FBA for ; Tue, 26 Nov 2024 13:17:10 +0100 (CET) Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-4316f3d3c21so48531465e9.3 for ; Tue, 26 Nov 2024 04:17:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1732623429; x=1733228229; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=CDhzpiee5IpMCJf78wYq9db/CW1GUElLMV2tv19EzCs=; b=StF9x7ZKvclykOUxZuBPmNdoO99LCB+dykGWeCqeeeOnethk9APFXE8+/xSbXA4OrB qkJ1nejcSVaFjWieWN8i/zECHBgiyTvqRh6jgavAoEcPalrLNs9fNC4KvaxqmyRaNv0W 58sGN7pUVzm4i3FsvDCYAcJb3thX5BqmdyWjEoQbRKC+r701s5Nf2mtbU04mVguXxBlZ ypYrBUcdUhvxdGnsExiCQffwaAFmIIEQWp+GDn7q+An0gGUoJa8+mJ+/jz8eqMy/gp/6 KLAzLfuqxZuwlWLi32uin3iX1UdfqN9kCHFKAVGOPO36ejsyIPDayI+Xl7k1L96gPDSQ IOJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732623429; x=1733228229; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=CDhzpiee5IpMCJf78wYq9db/CW1GUElLMV2tv19EzCs=; b=mSp9JI4eJTTRwvI846KU/lYDw501E/nIxSD7wfTlsOCLR1ke7mIiUbcrQVu4oHySQO apabgiU2ZbOiq7bqUwxiSe0987pNNm4uwBC4b9K3NWA7R9Ixm4EPeB8qQWEZbdvwW2CR 6I+VbhFhdqyQ+qj60Q2uW3kVyhgGCQP28fGBEFEmRo5s2UN5eOMr9l9v25rBjCqO9ND0 UjARweNxHKPsZ9fuxoqtLGqXiglgjeR27KpYsjby1X7ATrYxDCS9zmYIhOiBIhBP/PKF PNCeTIDY/V8QASnP4L6wVCUTQDLzTHZJ+7ODvUc0OQIlebSppXvEZaISXc2f9ezBEZuX UjRw== X-Gm-Message-State: AOJu0Yw0L6foLGby69A66zQj8iIAXj2L8griwjjjjVx5FvQZPPMCcKoh Rz2TAX73VZk0bMU1la9mPECJztCoGxwfPkj5L31gSqsP8HvzT9kaiphj5CRGmXH2CjXfx2tOYHp 4 X-Gm-Gg: ASbGncs3cwL7goejwOxaeDJ1m6aZGaIuYpTheqC6Bcmb/kOEdob1BVjbyeUCseztO5C Jqo8X/0T5TAXJWQ/NeV7jAyM7N3b+R1BLGHCE++ESibXKgf+GWO/vaoYpoX6csQ6eFAsFxL+Sp8 lC/PmfsF3joCDrWGB4P0T/RxsKjPKrtoi3qkXA5Vsj9MN+I/YFlmBmXWGbtjhA+/opqOsoww+L4 PJT4HmxAqymxSXxzp26NcX2dZkUaM0Jog5CcokTk8GRKk7prn4QciAqrRPAv8jNQnHymqgvmwXE MKrHRA== X-Google-Smtp-Source: AGHT+IHh11Bwm8LJ6JkFih9haHKL4wTF8llywituy6FEUuQQuv/M8SjvYK17tPW9r+PcTify26ziZA== X-Received: by 2002:a05:600c:45cd:b0:434:a529:3b87 with SMTP id 5b1f17b1804b1-434a5293ebemr28739405e9.10.1732623429459; Tue, 26 Nov 2024 04:17:09 -0800 (PST) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:c68a:6be1:5ba3:eddd]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-434a15d86a4sm51070325e9.36.2024.11.26.04.17.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Nov 2024 04:17:09 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman Subject: [RFC PATCH 0/3] Frame wallclock timestamps and metadata Date: Tue, 26 Nov 2024 12:17:03 +0000 Message-Id: <20241126121706.4350-1-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Hi everyone This patch set is actually all about the camera sync patches that I sent several weeks ago. One thing we said back then was that we'd try and make some of this functionality more accessible to other libcamera users. So this patch set (which would precede subsequent patches that actually implement camera sync) moves the recovery of a de-jittered "wallclock timestamp" into non-Raspberry Pi code. It feels like a generally useful thing. I've left it as an RFC for the minute because there are certainly a few things to think about. Let me talk through the 3 patches. Patch 1 just defines the new controls that will return the wallclock timestamps. I've left it so that you get both a "raw" (un-smoothed) wallclock and the smoothed version. Actually I think this probably makes sense because how would a developer really understand whether the de-jittering is working properly for them without the "raw" value. (It's hard to promise at this point that it will work perfectly on every camera on every platform forever...) Patch 2 is the ClockRecovery class that implements fairly simple-minded clock smoothing. It does have some configuration parameters, and I do wonder whether the values here will suit everyone, or whether pipeline handlers will want to customise them. I think it's easy enough to allow that, but all the more reason to report the "raw" wallclock as well so that developers can keep an eye on what's really happening. Patch 3 adds wallclock timestamps to the V4L2 devices, so that everyone using V4L2 can access them easily. A slight quirk here is that the frame start handling is under the control of the V4L2Device, while the frame end (dequeueBuffer) happens through the V4L2Videodevice. Another side effect is that you have to register for frame start events to get the functionality. Perhaps folks can say what they make of that. Obviously I have a couple more patches that make the Raspberry Pi PH output these new wallclock controls. Let me know if you'd like to try them. Thanks! David David Plowman (3): controls: Add FrameWallClock and FrameWallClockRaw controls libcamera: clock: Add ClockRecovery class to help generate wallclock timestamps libcamera: v4l2: Add wallclock metadata to video devices include/libcamera/framebuffer.h | 2 + include/libcamera/internal/clock_recovery.h | 64 ++++++++++ include/libcamera/internal/meson.build | 1 + include/libcamera/internal/v4l2_device.h | 4 + include/libcamera/internal/v4l2_videodevice.h | 3 + src/libcamera/clock_recovery.cpp | 110 ++++++++++++++++++ src/libcamera/control_ids_core.yaml | 29 +++++ src/libcamera/meson.build | 1 + src/libcamera/v4l2_device.cpp | 22 ++++ src/libcamera/v4l2_videodevice.cpp | 29 +++++ 10 files changed, 265 insertions(+) create mode 100644 include/libcamera/internal/clock_recovery.h create mode 100644 src/libcamera/clock_recovery.cpp