From patchwork Fri Oct 4 11:55:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 21511 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id ACC5DBD80A for ; Fri, 4 Oct 2024 11:56:06 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B39CE63527; Fri, 4 Oct 2024 13:56:04 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="Mh/q6Ia6"; dkim-atps=neutral Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 0665F62C8F for ; Fri, 4 Oct 2024 13:56:02 +0200 (CEST) Received: by mail-wr1-x431.google.com with SMTP id ffacd0b85a97d-37cc60c9838so1208563f8f.1 for ; Fri, 04 Oct 2024 04:56:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1728042962; x=1728647762; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=etoJ/Ta82BZMOQg7/ruzN2mxWes+BNuLhWq4YwV+EVc=; b=Mh/q6Ia6BUB3ebap5sApVwGLLAnBo7tkVI+EygryHlJ+Mp2OElVkyvQq7rdFanzMTO /tK2izXVLbZUWK5ocxgHAe7A9wmCsaa/POXOI68mNahyIIwjnR0ehvAJyKAAKFhG03IP 2Iy4VcOKWDwBqIAivtl8VwzRg85DO/VIy9XS+A7HPAEOL2VNElfCUTBCYvr6bjbxfBgi HV+AvhHT40CzC33ruKTU1wGhtZ1OqgV1HUmq+ShfuD51zJVrwvjTU9Ui168SEwChMWeG AkOfhszybeTm0heQEizeGvY3j/udJ/SV/ZaHBoqg2YKUVAE9wmh3xfBWWuIR+y1xCixF 4lQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728042962; x=1728647762; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=etoJ/Ta82BZMOQg7/ruzN2mxWes+BNuLhWq4YwV+EVc=; b=usNKhtYsEIhlkHBZlYGAS0UxY/Sg6O8kpdQ3UBh9kpkmfcUs0fV6syO+XDpJBce6aZ kE9eqliQfYJ3RtId0MydnqozP2n6lS/u7FNjtpXmKrk52/A4R9yeVCxkHTiYkt2Fszwe elm9M+LrzfsxEEbLm0ulmi605f5Pxs2a4pld6Bds5ZiCLfoCInUNmeIxlC5ZGFd5GrTE 2oaVKGFl5wPBz83SjSSmxlt+bpleIVurTwJ/lsYXDCe2zRC2BtitOl52CQUxeIBv9ty1 GXFqhTO29Cl622LeABdhVs51SpK4DsaVsjDJC5OtNcnV4DuAKJ36AJYM7lZ85WZynvWI GS8A== X-Gm-Message-State: AOJu0YyrbBwYL03yIiAlIm6KXiKFjtGwKqsNsykgBGKHRYMoEz1H9zU+ 5TEGVDQwbT3dcULvDTZlHpn/bw3/pfxj+0+ekm57AfKojDvDhpQ0mIB0dtknBuBhLo8zFTV5M/4 + X-Google-Smtp-Source: AGHT+IHDKRbezWvgh3YGSmUV0BpmHG7scayI2jQ+sS9A7UM5nGtd1eqkJ4IRvzEXRj7qO8Mw9K9C+w== X-Received: by 2002:a5d:4a12:0:b0:374:bf6b:1021 with SMTP id ffacd0b85a97d-37d04a7b139mr4409007f8f.27.1728042962104; Fri, 04 Oct 2024 04:56:02 -0700 (PDT) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:daa2:371b:a97:3e3e]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37d081f743esm3107147f8f.21.2024.10.04.04.56.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Oct 2024 04:56:01 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman Subject: [PATCH 0/6] Raspberry Pi software camera sync algorithm Date: Fri, 4 Oct 2024 12:55:52 +0100 Message-Id: <20241004115558.9166-1-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Hi everyone This patch set adds a software camera sync algorithm to Raspberry Pi's libcamera implementation. It is the result of work by our summer intern Arsen, as well as by Naush and myself. It consists of a server, which broadcasts timing messages sporadically (such as once a second) on a socket, and then one or more clients that listen out for these messages. The clients, which should be running at the same (or a very close) framerate to the server, will perform a one-off frame length adjustment in response to each message, causing the frames to be synchronised with the server. Although principally designed for identical cameras running on the same Pi, the system can also work for different kinds of cameras on different Pis on the same network. In the former case we'd be expecting sub 100us latencies between frames, and in the latter probably sub 1ms. On different Pis, of course, one is subject to the vagaries of NTP (though you have the option to configure it better, or indeed use something else). There are 6 patches, consisting of: * Some new controls. * Plumbing in the pipeline handler to generate wall clock timestamps. * Plumbing in the IPAs to support the new "SyncAlgorithm". * A complete initial implementation of "rpi.sync". * Updates to the camera tuning files to activate the algorithm. We've had users wanting this feature for some time, and are keen to roll it out. As such, perhaps I could encourage reviewers to look at the new controls (the first patch) first. Once this is agreed, we can roll the algorithm out without causing a subsequent API breakage for our users if other details get changed (the implementation underneath, in this respect, doesn't matter). Up for discussion, I would imagine, might be what the new controls mean and do, and indeed whether people would prefer them to be core controls or not. Hope that all makes sense. Thanks everyone! Best regards David David Plowman (3): controls: rpi: Add controls for the camera sync algorithm ipa: rpi: sync: Add an implementation of the camera sync algorithm ipa: rpi: vc4: Update all tuning files for sync algorithm Naushir Patuck (3): pipeline: rpi: Add queue of wallclock timestamps pipelien: rpi: vc4: Populate the wallclock timestamps queue ipa: rpi: Add base classes and plumbing for sync algorithm src/ipa/rpi/common/ipa_base.cpp | 66 ++- src/ipa/rpi/common/ipa_base.h | 4 +- src/ipa/rpi/controller/meson.build | 2 + src/ipa/rpi/controller/rpi/clock_recovery.cpp | 87 ++++ src/ipa/rpi/controller/rpi/clock_recovery.h | 55 +++ src/ipa/rpi/controller/rpi/sync.cpp | 384 ++++++++++++++++++ src/ipa/rpi/controller/rpi/sync.h | 71 ++++ src/ipa/rpi/controller/sync_algorithm.h | 30 ++ src/ipa/rpi/controller/sync_status.h | 27 ++ src/ipa/rpi/vc4/data/imx219.json | 11 +- src/ipa/rpi/vc4/data/imx219_noir.json | 11 +- src/ipa/rpi/vc4/data/imx283.json | 11 +- src/ipa/rpi/vc4/data/imx290.json | 11 +- src/ipa/rpi/vc4/data/imx296.json | 11 +- src/ipa/rpi/vc4/data/imx296_mono.json | 11 +- src/ipa/rpi/vc4/data/imx378.json | 11 +- src/ipa/rpi/vc4/data/imx477.json | 11 +- src/ipa/rpi/vc4/data/imx477_noir.json | 11 +- src/ipa/rpi/vc4/data/imx477_scientific.json | 11 +- src/ipa/rpi/vc4/data/imx519.json | 11 +- src/ipa/rpi/vc4/data/imx708.json | 11 +- src/ipa/rpi/vc4/data/imx708_noir.json | 11 +- src/ipa/rpi/vc4/data/imx708_wide.json | 11 +- src/ipa/rpi/vc4/data/imx708_wide_noir.json | 11 +- src/ipa/rpi/vc4/data/ov5647.json | 11 +- src/ipa/rpi/vc4/data/ov5647_noir.json | 11 +- src/ipa/rpi/vc4/data/se327m12.json | 11 +- src/libcamera/control_ids_rpi.yaml | 76 ++++ .../pipeline/rpi/common/pipeline_base.cpp | 9 + .../pipeline/rpi/common/pipeline_base.h | 2 + src/libcamera/pipeline/rpi/vc4/vc4.cpp | 12 + 31 files changed, 1000 insertions(+), 23 deletions(-) create mode 100644 src/ipa/rpi/controller/rpi/clock_recovery.cpp create mode 100644 src/ipa/rpi/controller/rpi/clock_recovery.h create mode 100644 src/ipa/rpi/controller/rpi/sync.cpp create mode 100644 src/ipa/rpi/controller/rpi/sync.h create mode 100644 src/ipa/rpi/controller/sync_algorithm.h create mode 100644 src/ipa/rpi/controller/sync_status.h