From patchwork Fri Oct 4 11:55:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 21512 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id DEAB0BD80A for ; Fri, 4 Oct 2024 11:56:11 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B58B76352C; Fri, 4 Oct 2024 13:56:06 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="dN8Y26yz"; dkim-atps=neutral Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 898FB62C8F for ; Fri, 4 Oct 2024 13:56:03 +0200 (CEST) Received: by mail-wr1-x435.google.com with SMTP id ffacd0b85a97d-37cca239886so1274633f8f.2 for ; Fri, 04 Oct 2024 04:56:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1728042963; x=1728647763; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/+QeznJmVtfJhdoE5niV18EGr1SvQlkkgJ2zxe7NDGY=; b=dN8Y26yzt5WjmV3vtX6+wWAWwaIjeMxzxqYUcdDZSlqyJdfKzlNeTuJYJOXXXBVpY7 6yhrFf3U3VuWPIskENVscuhkRl5hLFfxuCuPrL8LMFoVOr9XPTiZQKQSTpTAF+wv2CN6 Qp9Q6KnsjQ4i99JK3fkJVrAIttnQvNYOrY+XbVqtylTT/vnBC6+YtbFn4g58oA++xw5b XJ+SmLGOnZ1LF8QERzII5rd24zWdovadvkCbhkRgGIRLdBYFZlY72s1pPdwQDBJ7KdDQ 8WkmNtV5bRi21En4YBYsTNRtd5VCa0CKAy86LWUm4Eu4/JgFtHu9Zpqa6BeNjNe0cee9 91xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728042963; x=1728647763; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/+QeznJmVtfJhdoE5niV18EGr1SvQlkkgJ2zxe7NDGY=; b=ZhpNTEHvVwwNvSVe/hru7mC0xvLbjwYgXPPisclU2wkqX7M5rPdQqcBpIEszz/4HuK rOGwN7/hMkS4OkJTWrqmeBgrEA+MpVPv+2CkJ/ygD1bb0xfRbwlTiFgvNGOm8MBRAPSW tfSSCYdyekslmUmFPC6kwvEy63jNb+cI3WhT85TfBm3pcAmagPIItc/vFPXo8r4RgLnH rrq1VS5xTSC4oPpAOiwsAdmZTMbGjU+mWwaojvdO/GV74uoawe7NpcOL/yhHOC0jM54U nQSb7bismNt9jO3yUf3vuBnq6WQS14MtLHRsbP7tyO4wYe7SSLR4JYSyU49jfStNG9Hk K+ww== X-Gm-Message-State: AOJu0YwRC6Pe7aUGTuAQl8NDd5yHfXo8wvOYzGfKqbcEa5VHrp/bSmI3 lN5+DIdPM1bRajmwDbDYppvPwAa+pAIv5bI6asMloV0As4X+0cfGgzWVDBemWaL1jXpue/mcVFL U X-Google-Smtp-Source: AGHT+IGjx0fS5C7bqF5N73ZKjOKeyVP4nzVMjBqPgZBJgDMLhh1GbZFL4SOIQS4Yj8q/aJdRBuTelQ== X-Received: by 2002:a5d:64c8:0:b0:368:5ba0:622 with SMTP id ffacd0b85a97d-37d0eafa3a2mr2132091f8f.44.1728042962696; Fri, 04 Oct 2024 04:56:02 -0700 (PDT) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:daa2:371b:a97:3e3e]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37d081f743esm3107147f8f.21.2024.10.04.04.56.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Oct 2024 04:56:02 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman , Arsen Mikovic , Naushir Patuck Subject: [PATCH 1/6] controls: rpi: Add controls for the camera sync algorithm Date: Fri, 4 Oct 2024 12:55:53 +0100 Message-Id: <20241004115558.9166-2-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241004115558.9166-1-david.plowman@raspberrypi.com> References: <20241004115558.9166-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The camera sync algorithm uses the following new controls: SyncMode - a camera can be a server or client SyncWait - whether the sync point has been reached SyncLag - how far away from synchronisation a camera was SyncFrameWallClock - for passing wall clock time to the IPA. Signed-off-by: David Plowman Signed-off-by: Arsen Mikovic Reviewed-by: Naushir Patuck --- src/libcamera/control_ids_rpi.yaml | 76 ++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/src/libcamera/control_ids_rpi.yaml b/src/libcamera/control_ids_rpi.yaml index 42c4bf5d..b5605dfa 100644 --- a/src/libcamera/control_ids_rpi.yaml +++ b/src/libcamera/control_ids_rpi.yaml @@ -30,4 +30,80 @@ controls: \sa StatsOutputEnable + - FrameWallClock: + type: int64_t + description: | + Control that returns the wall clock timestamp of a frame. This + is the "time since epoch" value obtained from the system, in + microseconds. This value is likely to be subject to + significantly more jitter than the recorded SensorTimestamp. + + - SyncMode: + type: int32_t + description: | + Puts the camera system into sync mode, so that frames can be + temporally synchronised with another camera, either on the same + device, or on a different one. + enum: + - name: SyncModeOff + value: 0 + description: Sync not in use. + - name: SyncModeServer + value: 1 + description: | + Sync on, act as server. The server broadcasts timing + messages to any clients that are listening, so that the + clients can synchronise their camera frames with the + server's. + - name: SyncModeClient + value: 2 + description: | + Sync on, act as client. A client listens for any server + messages, and arranges for its camera frames to synchronise + as closely as possible with the server's. Many clients + can listen out for the same server. + + - SyncWait: + type: bool + description: | + When using the camera syncrhonisation algorithm, the server + broadcasts timing information to the client. This also includes + the time (some number of frames in the future) at which it will + tell the application running on the server when to start using + the image frames (the "ready time"). + + The client receives the "ready time" from the server, and will + tell the application on its end to start using the frames at + this same moment. + + While this control value is true, applications (on both client + and server) should continue to wait. + + Once this value is false, it means that this is the frame where + client and server have agreed that it is the first synchronised + frame that should be used by the application. + + - SyncLag: + type: int64_t + description: | + The lag is the amount of time since the "ready time", at which + the server and client will signal their controlling applications + that the frames are now synchronised and should be used. + + Normally, therefore, the value will start negative (the "ready + time" is in the future), and increase towards zero, before + becoming positive (the "ready time" has elapsed). + + Servers always report this value; clients will omit this control + until they have received a message from the server that enables + them to calculate it. + + Normally there will be just one frame where the lag value is, or + is very close to, zero - the one for which SyncWait becomes + false. But note that if frames are being dropped, then the "near + zero" value, or indeed any other, could be skipped. In these + cases the lag value allows an application to work out exactly + what has happened. + + \sa SyncWait ... From patchwork Fri Oct 4 11:55:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 21513 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 88654BD80A for ; Fri, 4 Oct 2024 11:56:14 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5CAA863530; Fri, 4 Oct 2024 13:56:12 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="cvKvB1Lz"; dkim-atps=neutral Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4C3FD63524 for ; Fri, 4 Oct 2024 13:56:04 +0200 (CEST) Received: by mail-wm1-x329.google.com with SMTP id 5b1f17b1804b1-42e748f78d6so18538215e9.0 for ; Fri, 04 Oct 2024 04:56:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1728042963; x=1728647763; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=m60dYGWd9WItWVLwXEyJ3xnk5Efq/SYRBpGuZAumOCc=; b=cvKvB1LztbUKQHOzW+i69zeTVKC6rfU6S4AkNS1zMalubwvVFilVcfGSXoZlsVa8C+ mge5irmj1Dn1bFnEeuyLPUvp8r9ScJcIcxOGFPWcFicH0KPRpvfXy03u5K3OlpxqeXz5 1Ob05EFyc9ebNuo97O4RTaM8NpjF+MhRfXXnoDCDGbC1irFMlRApIjFL7DQLBS6m0C2b WBwKb03duGs7WMGtz62XTpRAUZ+FDQLwwAYyMR+ZRdMgB7ysRiUKotp3T62LuaRoDfiP zAH+Xrg8cbis9NJjgyzkPat+yrxNoQ/+McxdcrIkwALmigKyih8JmjSTmFg0ZDFyemMo fIBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728042963; x=1728647763; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m60dYGWd9WItWVLwXEyJ3xnk5Efq/SYRBpGuZAumOCc=; b=R3pr72F23ID8bygG0wSNjYV20LPPGcGQ387qD5fZpR6NdrubQmy50FJgPkF7W12e0K XVwwNQsf3ua786MfUJQDTzfIYdzHvIIcgUmOktVihVB7ZWpY4aCozirZ/X6sIpsnX7zh vCQxXMG4bkANhpizFQLFZyVw/NIzjqIKC95tBLLjj/VWjZ+h/dn7+LDyKhXZ7nN7obrk uFcLiQo4pVePZ6CJ/yggSVdDZdaxnDcUkoKXp0Ay/D8iFmbns/p1k2uvfwXwO21UAPZ0 BYcpxRp14ENx4xhjWs14TbbcKZ06ykuoeAV3htgynuDTHCtdQbDGnO43YrssfdEu6uHi K11Q== X-Gm-Message-State: AOJu0Yx9+YzbMFJBYTM2oiLDgtHWD9fVVAcdbm6qLqe44nBsnKJwOfHS lKtAqxp8cXmyCSQFQpxUjx8uP1pNgRHun7d/KsJWbf380oHPhXOKbNF98x/NemIcVOKBiaG3TSe C X-Google-Smtp-Source: AGHT+IHz1gvW0tNf8QpT1yvW+NcTJuny1KzCpe5X12VUmadgkeykxN67q3LVJYvONwJf5K9ml2wK0A== X-Received: by 2002:adf:a18e:0:b0:37c:d54c:a41 with SMTP id ffacd0b85a97d-37d0e7d53f9mr1229535f8f.54.1728042963361; Fri, 04 Oct 2024 04:56:03 -0700 (PDT) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:daa2:371b:a97:3e3e]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37d081f743esm3107147f8f.21.2024.10.04.04.56.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Oct 2024 04:56:02 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: Naushir Patuck , David Plowman Subject: [PATCH 2/6] pipeline: rpi: Add queue of wallclock timestamps Date: Fri, 4 Oct 2024 12:55:54 +0100 Message-Id: <20241004115558.9166-3-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241004115558.9166-1-david.plowman@raspberrypi.com> References: <20241004115558.9166-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Naushir Patuck Subsequent commits will add actual values to this queue, which we can then use for sending wallclock timestamps over to the IPAs, where a future "synchonisation algorithm" can use them. Also add code to return the wallclock time to the application through the frame metadata. Signed-off-by: Naushir Patuck Reviewed-by: David Plowman --- src/libcamera/pipeline/rpi/common/pipeline_base.cpp | 9 +++++++++ src/libcamera/pipeline/rpi/common/pipeline_base.h | 2 ++ 2 files changed, 11 insertions(+) diff --git a/src/libcamera/pipeline/rpi/common/pipeline_base.cpp b/src/libcamera/pipeline/rpi/common/pipeline_base.cpp index 3041fd1e..253fac96 100644 --- a/src/libcamera/pipeline/rpi/common/pipeline_base.cpp +++ b/src/libcamera/pipeline/rpi/common/pipeline_base.cpp @@ -676,6 +676,7 @@ int PipelineHandlerBase::start(Camera *camera, const ControlList *controls) */ data->delayedCtrls_->reset(0); data->state_ = CameraData::State::Idle; + data->frameWallClock_ = {}; /* Enable SOF event generation. */ data->frontendDevice()->setFrameStartEnabled(true); @@ -1353,6 +1354,11 @@ void CameraData::cameraTimeout() void CameraData::frameStarted(uint32_t sequence) { + /* Get frame wall clock. */ + auto now = std::chrono::system_clock::now(); + auto durNow = std::chrono::duration_cast(now.time_since_epoch()); + frameWallClock_.emplace(sequence, durNow); + LOG(RPI, Debug) << "Frame start " << sequence; /* Write any controls for the next frame as soon as we can. */ @@ -1483,6 +1489,9 @@ void CameraData::fillRequestMetadata(const ControlList &bufferControls, Request request->metadata().set(controls::SensorTimestamp, bufferControls.get(controls::SensorTimestamp).value_or(0)); + request->metadata().set(controls::rpi::FrameWallClock, + bufferControls.get(controls::rpi::FrameWallClock).value_or(0)); + request->metadata().set(controls::ScalerCrop, scalerCrop_); } diff --git a/src/libcamera/pipeline/rpi/common/pipeline_base.h b/src/libcamera/pipeline/rpi/common/pipeline_base.h index f9cecf70..b7f5470f 100644 --- a/src/libcamera/pipeline/rpi/common/pipeline_base.h +++ b/src/libcamera/pipeline/rpi/common/pipeline_base.h @@ -163,6 +163,8 @@ public: Config config_; + std::queue> frameWallClock_; + protected: void fillRequestMetadata(const ControlList &bufferControls, Request *request); From patchwork Fri Oct 4 11:55:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 21514 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D2293BD80A for ; Fri, 4 Oct 2024 11:56:16 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2CCD36352B; Fri, 4 Oct 2024 13:56:16 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="Z8bxFDNN"; dkim-atps=neutral Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 69B0B6352B for ; Fri, 4 Oct 2024 13:56:05 +0200 (CEST) Received: by mail-wr1-x431.google.com with SMTP id ffacd0b85a97d-37cd8a5aac9so1204579f8f.2 for ; Fri, 04 Oct 2024 04:56:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1728042964; x=1728647764; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lxFKq/MpaE936mX2wEmbFqI5SPb5I+EyRyQlidCho7Q=; b=Z8bxFDNNBqM2QpxyKXH8c5K1HnKfVs47cvuJGPlfLYq9dIqt1291w7eX17k4wsNGmz Rfs7IwsfA5egFRW6fO1w4nbCXGtyRTYkDCnzlQ3jsq8wq+eDgqyTWM0SL82HCoUDhTdA SaqWoJkG7Isnse2XsAhe8R/Ffbob9WOIW1cjPNMAUOBhSVJ9Z1azL67Y1d1R78vc746A qU9X1blScKYLgEGANW8CtJOAUJCKP1yl726+9pPhU46mU8kOYyACZOWs/EH96aZRJTZv 0M1YpLK9TJ2qodG/zeDZ9USu0sfgL86peb5E2x9F8dxiYThWh22cls1QYnGFaE4u2UT/ ogiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728042964; x=1728647764; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lxFKq/MpaE936mX2wEmbFqI5SPb5I+EyRyQlidCho7Q=; b=MSYGdzsO7heLDg9VTvxofGltWT1BkwUI0IPHIq78LLqkby0x4o4JHWwMvl0G9EIPZ4 QwT0xWnxNwIz2SIWZLKTmJPMBhqrWzinrQFq68sh7pvKPWEL+B6a42Pg7hZ2sCVXYo6g AGDd99zfuCFpSP1xr23F7hY1E0AEkhKjaRVC7fP7Fz5Z3byyzB8nTc1WrCji26D/jhV1 QDDwK8zLeN8NDFxk+0a+cQ6WSn9y+HlEwTFwjO37bRE1PVFaP0WaAkqkUpNc2eSgWP56 mN+LXdi2eEAeHre4CyMIjSniEvqmm/Tj98shNI10HJgWOAwvQqbLeH1peXb5oFcpA11f y5Hg== X-Gm-Message-State: AOJu0Yw0OO9Udb9EnsYwa23ZqB0/QpyKRL2IBUFIW1nw/abKwxKuNV3C Rs5VIGXV7hNqKWa8ap0p43ZjyQIpvkj3G9kFuTLf6T4uWdStE9TtLPbBFg3WVAQjGRy6ScvqyQy A X-Google-Smtp-Source: AGHT+IHsc4JYwDumkCtd0Xc0FYqSHhljBWcmdw9FUfXMHgHhFQ5MHX8WgKG8u3MzrMa+STV7MYAsFQ== X-Received: by 2002:a5d:4e45:0:b0:374:c0c5:3c05 with SMTP id ffacd0b85a97d-37d0e7baf3fmr1409348f8f.42.1728042964037; Fri, 04 Oct 2024 04:56:04 -0700 (PDT) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:daa2:371b:a97:3e3e]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37d081f743esm3107147f8f.21.2024.10.04.04.56.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Oct 2024 04:56:03 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: Naushir Patuck Subject: [PATCH 3/6] pipelien: rpi: vc4: Populate the wallclock timestamps queue Date: Fri, 4 Oct 2024 12:55:55 +0100 Message-Id: <20241004115558.9166-4-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241004115558.9166-1-david.plowman@raspberrypi.com> References: <20241004115558.9166-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Naushir Patuck We add wallclock timestamps to the queue when we dequeue a camera buffer from Unicam. Signed-off-by: Naushir Patuck --- src/libcamera/pipeline/rpi/vc4/vc4.cpp | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/src/libcamera/pipeline/rpi/vc4/vc4.cpp b/src/libcamera/pipeline/rpi/vc4/vc4.cpp index e5b6ef2b..754cfdde 100644 --- a/src/libcamera/pipeline/rpi/vc4/vc4.cpp +++ b/src/libcamera/pipeline/rpi/vc4/vc4.cpp @@ -777,6 +777,18 @@ void Vc4CameraData::unicamBufferDequeue(FrameBuffer *buffer) * as it does not receive the FrameBuffer object. */ ctrl.set(controls::SensorTimestamp, buffer->metadata().timestamp); + + /* Also record a wall-clock timestamp that can be passed to IPAs. */ + while (!frameWallClock_.empty() && + frameWallClock_.front().first < buffer->metadata().sequence) + frameWallClock_.pop(); + + if (!frameWallClock_.empty() && + frameWallClock_.front().first == buffer->metadata().sequence) { + ctrl.set(controls::rpi::FrameWallClock, frameWallClock_.front().second.get()); + frameWallClock_.pop(); + } + bayerQueue_.push({ buffer, std::move(ctrl), delayContext }); } else { embeddedQueue_.push(buffer); From patchwork Fri Oct 4 11:55:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 21515 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7AC8AC3263 for ; Fri, 4 Oct 2024 11:56:19 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A82FB6352E; Fri, 4 Oct 2024 13:56:18 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="LmtZAUU+"; dkim-atps=neutral Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A90FC62C8F for ; Fri, 4 Oct 2024 13:56:05 +0200 (CEST) Received: by mail-wr1-x432.google.com with SMTP id ffacd0b85a97d-37cc60c9838so1208580f8f.1 for ; Fri, 04 Oct 2024 04:56:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1728042965; x=1728647765; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vTL6DimDDA9r/zGhGutNdHJ4z5oVSpHpu6UwaAbb2SM=; b=LmtZAUU+E8MQmocUmAklarujj0MEbG0P0pc8r7ULItYQ/0JC5R5TtYJhH2EsmiJ1xu neeN2iXnMeDIbQFGM1urv1OKYoO+oGUmPbpk3zdiCYMsoBTEQ2uOLzdWQNkrnrRrjT42 gJiYe7tty8x/m+twbH88lOwLD1NUNU7PM8UQLq2qol9057AnNJxZ8FqTIRkiQYv7qGyh YQGqUVXc2t0Q97cBMYPVpkF84n+ZsmZH2Fv6UTwy/gpXJCZeeynI/Zt8tRH+WHqytfb8 qFvJPLcD+add6aKwFq0nVCGcRHx1B0esgsPGPQBl15RLzH2tVqgNqS1HySwih2CKznoh Hx0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728042965; x=1728647765; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vTL6DimDDA9r/zGhGutNdHJ4z5oVSpHpu6UwaAbb2SM=; b=nln00oqug7Cjp+MsNuf9jB0ptJmCrpAVnH6nKEB8BaOuKXPR4A0bSx34FH6Mknxmuv atgv3ET/ihOaHnZnZ/w1t2fqZaX8oPioWCoJkMrksV+pO2EwrpfzMHsZi3Cbaysgwj+K +6drmJ5E+7K6g3l4hITDMsD2Lbk2Q58mNVW8hlONlIZFfZbK/id/D6CCQcZDv9yH8mFl qJNhya3VZTLw494jUzqTVT7hBmApyz70pfSClzdZ2fRiiyPvNx6rMniOjEnZV+p//tWI vTLYTZ8/UtixWVf3+co/ngbQEgoa8QH+lkLt3bm4PZ0C4+AQed+Ro+WvmEb3TQmLii9L n1tg== X-Gm-Message-State: AOJu0Yxrw0cbTbwkm7DWkz3XAibCW0VDN3vpoXN7f2pwmYByYJPj9zuQ jSCSnlKz7a1JnQS891k/KT90AS5tYILd5NbG1k7qFtokLEN34Cga7JToD5rKzmgu952eoZVPKx0 B X-Google-Smtp-Source: AGHT+IGWtR7pK77SJNmhk/qhH4mOhADTbEAGgq3yA8ofYsOo/aumQKRwPNN3g1WWUZDhCDe3t9ivaA== X-Received: by 2002:a05:6000:1569:b0:374:c6b6:c656 with SMTP id ffacd0b85a97d-37d0f70ca1emr1793382f8f.21.1728042964666; Fri, 04 Oct 2024 04:56:04 -0700 (PDT) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:daa2:371b:a97:3e3e]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37d081f743esm3107147f8f.21.2024.10.04.04.56.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Oct 2024 04:56:04 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: Naushir Patuck , David Plowman Subject: [PATCH 4/6] ipa: rpi: Add base classes and plumbing for sync algorithm Date: Fri, 4 Oct 2024 12:55:56 +0100 Message-Id: <20241004115558.9166-5-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241004115558.9166-1-david.plowman@raspberrypi.com> References: <20241004115558.9166-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Naushir Patuck We add a base class for a "sync algorithm", and define its inputs and outputs in the SyncStatus class. We add the necessary plumbing to the base IPA code so as to arrange for the necessary parameters to be made available to such an algorithm, and also to handle the return values, passing them back as necessary to the pipeline handler. Signed-off-by: Naushir Patuck Reviewed-by: David Plowman --- src/ipa/rpi/common/ipa_base.cpp | 66 +++++++++++++++++++++++-- src/ipa/rpi/common/ipa_base.h | 4 +- src/ipa/rpi/controller/sync_algorithm.h | 30 +++++++++++ src/ipa/rpi/controller/sync_status.h | 27 ++++++++++ 4 files changed, 122 insertions(+), 5 deletions(-) create mode 100644 src/ipa/rpi/controller/sync_algorithm.h create mode 100644 src/ipa/rpi/controller/sync_status.h diff --git a/src/ipa/rpi/common/ipa_base.cpp b/src/ipa/rpi/common/ipa_base.cpp index ee3848b5..ac228791 100644 --- a/src/ipa/rpi/common/ipa_base.cpp +++ b/src/ipa/rpi/common/ipa_base.cpp @@ -28,6 +28,8 @@ #include "controller/lux_status.h" #include "controller/sharpen_algorithm.h" #include "controller/statistics.h" +#include "controller/sync_algorithm.h" +#include "controller/sync_status.h" namespace libcamera { @@ -74,6 +76,7 @@ const ControlInfoMap::Map ipaControls{ { &controls::FrameDurationLimits, ControlInfo(INT64_C(33333), INT64_C(120000)) }, { &controls::draft::NoiseReductionMode, ControlInfo(controls::draft::NoiseReductionModeValues) }, { &controls::rpi::StatsOutputEnable, ControlInfo(false, true, false) }, + { &controls::rpi::SyncMode, ControlInfo(controls::rpi::SyncModeValues) }, }; /* IPA controls handled conditionally, if the sensor is not mono */ @@ -387,6 +390,7 @@ void IpaBase::prepareIsp(const PrepareParams ¶ms) rpiMetadata.clear(); fillDeviceStatus(params.sensorControls, ipaContext); + fillSyncParams(params, ipaContext); if (params.buffers.embedded) { /* @@ -485,10 +489,23 @@ void IpaBase::processStats(const ProcessParams ¶ms) helper_->process(statistics, rpiMetadata); controller_.process(statistics, &rpiMetadata); + /* Send any sync algorithm outputs back to the pipeline handler */ + Duration offset(0s); + struct SyncStatus syncStatus; + if (rpiMetadata.get("sync.status", syncStatus) == 0) { + if (minFrameDuration_ != maxFrameDuration_) + LOG(IPARPI, Error) << "Sync algorithm enabled with variable framerate. " << minFrameDuration_ << " " << maxFrameDuration_; + offset = syncStatus.frameDurationOffset; + + libcameraMetadata_.set(controls::rpi::SyncWait, !syncStatus.ready); + if (syncStatus.lagKnown) + libcameraMetadata_.set(controls::rpi::SyncLag, syncStatus.lag); + } + struct AgcStatus agcStatus; if (rpiMetadata.get("agc.status", agcStatus) == 0) { ControlList ctrls(sensorCtrls_); - applyAGC(&agcStatus, ctrls); + applyAGC(&agcStatus, ctrls, offset); setDelayedControls.emit(ctrls, ipaContext); setCameraTimeoutValue(); } @@ -721,6 +738,7 @@ void IpaBase::applyControls(const ControlList &controls) using RPiController::ContrastAlgorithm; using RPiController::DenoiseAlgorithm; using RPiController::HdrAlgorithm; + using RPiController::SyncAlgorithm; /* Clear the return metadata buffer. */ libcameraMetadata_.clear(); @@ -1247,6 +1265,24 @@ void IpaBase::applyControls(const ControlList &controls) statsMetadataOutput_ = ctrl.second.get(); break; + case controls::rpi::SYNC_MODE: { + SyncAlgorithm *sync = dynamic_cast(controller_.getAlgorithm("sync")); + + if (sync) { + int mode = ctrl.second.get(); + SyncAlgorithm::Mode m = SyncAlgorithm::Mode::Off; + if (mode == controls::rpi::SyncModeServer) { + m = SyncAlgorithm::Mode::Server; + LOG(IPARPI, Info) << "Sync mode set to server"; + } else if (mode == controls::rpi::SyncModeClient) { + m = SyncAlgorithm::Mode::Client; + LOG(IPARPI, Info) << "Sync mode set to client"; + } + sync->setMode(m); + } + break; + } + default: LOG(IPARPI, Warning) << "Ctrl " << controls::controls.at(ctrl.first)->name() @@ -1283,6 +1319,19 @@ void IpaBase::fillDeviceStatus(const ControlList &sensorControls, unsigned int i rpiMetadata_[ipaContext].set("device.status", deviceStatus); } +void IpaBase::fillSyncParams(const PrepareParams ¶ms, unsigned int ipaContext) +{ + RPiController::SyncAlgorithm *sync = dynamic_cast( + controller_.getAlgorithm("sync")); + if (!sync) + return; + + SyncParams syncParams; + syncParams.wallClock = *params.sensorControls.get(controls::rpi::FrameWallClock); + syncParams.sensorTimestamp = *params.sensorControls.get(controls::SensorTimestamp); + rpiMetadata_[ipaContext].set("sync.params", syncParams); +} + void IpaBase::reportMetadata(unsigned int ipaContext) { RPiController::Metadata &rpiMetadata = rpiMetadata_[ipaContext]; @@ -1451,14 +1500,22 @@ void IpaBase::applyFrameDurations(Duration minFrameDuration, Duration maxFrameDu * value possible. */ Duration maxShutter = Duration::max(); - helper_->getBlanking(maxShutter, minFrameDuration_, maxFrameDuration_); + auto [vblank, hblank] = helper_->getBlanking(maxShutter, minFrameDuration_, maxFrameDuration_); RPiController::AgcAlgorithm *agc = dynamic_cast( controller_.getAlgorithm("agc")); agc->setMaxShutter(maxShutter); + + RPiController::SyncAlgorithm *sync = dynamic_cast( + controller_.getAlgorithm("sync")); + if (sync) { + Duration duration = (mode_.height + vblank) * ((mode_.width + hblank) * 1.0s / mode_.pixelRate); + LOG(IPARPI, Debug) << "setting sync frame duration to " << duration; + sync->setFrameDuration(duration); + } } -void IpaBase::applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls) +void IpaBase::applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls, Duration frameDurationOffset) { const int32_t minGainCode = helper_->gainCode(mode_.minAnalogueGain); const int32_t maxGainCode = helper_->gainCode(mode_.maxAnalogueGain); @@ -1473,7 +1530,8 @@ void IpaBase::applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls) /* getBlanking might clip exposure time to the fps limits. */ Duration exposure = agcStatus->shutterTime; - auto [vblank, hblank] = helper_->getBlanking(exposure, minFrameDuration_, maxFrameDuration_); + auto [vblank, hblank] = helper_->getBlanking(exposure, minFrameDuration_ - frameDurationOffset, + maxFrameDuration_ - frameDurationOffset); int32_t exposureLines = helper_->exposureLines(exposure, helper_->hblankToLineLength(hblank)); diff --git a/src/ipa/rpi/common/ipa_base.h b/src/ipa/rpi/common/ipa_base.h index 1a811beb..b53d0bfb 100644 --- a/src/ipa/rpi/common/ipa_base.h +++ b/src/ipa/rpi/common/ipa_base.h @@ -95,9 +95,11 @@ private: void applyControls(const ControlList &controls); virtual void handleControls(const ControlList &controls) = 0; void fillDeviceStatus(const ControlList &sensorControls, unsigned int ipaContext); + void fillSyncParams(const PrepareParams ¶ms, unsigned int ipaContext); void reportMetadata(unsigned int ipaContext); void applyFrameDurations(utils::Duration minFrameDuration, utils::Duration maxFrameDuration); - void applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls); + void applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls, + utils::Duration frameDurationOffset = utils::Duration(0)); std::map buffers_; diff --git a/src/ipa/rpi/controller/sync_algorithm.h b/src/ipa/rpi/controller/sync_algorithm.h new file mode 100644 index 00000000..298b5951 --- /dev/null +++ b/src/ipa/rpi/controller/sync_algorithm.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * sync_algorithm.h - Camera sync algorithm interface + */ +#pragma once + +#include + +#include "algorithm.h" + +namespace RPiController { + +class SyncAlgorithm : public Algorithm +{ +public: + enum class Mode { + Off, + Server, + Client, + }; + + SyncAlgorithm(Controller *controller) + : Algorithm(controller) {} + virtual void setFrameDuration(libcamera::utils::Duration frameDuration) = 0; + virtual void setMode(Mode mode) = 0; +}; + +} /* namespace RPiController */ diff --git a/src/ipa/rpi/controller/sync_status.h b/src/ipa/rpi/controller/sync_status.h new file mode 100644 index 00000000..289f0182 --- /dev/null +++ b/src/ipa/rpi/controller/sync_status.h @@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * sync_status.h - Sync algorithm params and status structures + */ +#pragma once + +#include + +struct SyncParams { + /* Wall clock time for this frame */ + uint64_t wallClock; + /* Kernel timestamp for this frame */ + uint64_t sensorTimestamp; +}; + +struct SyncStatus { + /* Frame length correction to apply */ + libcamera::utils::Duration frameDurationOffset; + /* Whether the "ready time" has been reached */ + bool ready; + /* Lag between camera frame and the "ready time" */ + int64_t lag; + /* Whether lag is known (client has to wait for a server message) */ + bool lagKnown; +}; From patchwork Fri Oct 4 11:55:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 21516 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id EF1ACBD80A for ; Fri, 4 Oct 2024 11:56:21 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 272206352E; Fri, 4 Oct 2024 13:56:21 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="BobiUisv"; dkim-atps=neutral Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 88B8B62C92 for ; Fri, 4 Oct 2024 13:56:06 +0200 (CEST) Received: by mail-wr1-x42e.google.com with SMTP id ffacd0b85a97d-37cc84c12c2so1134770f8f.3 for ; Fri, 04 Oct 2024 04:56:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1728042966; x=1728647766; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=obcW8TllxKY4racC770ZVBJlg3ef+KFaFjujkzVUGIA=; b=BobiUisvPdaXruLLn5OdPd1qknjJMoFA+DEqITTSamFgRoygIA+YPidzeYZk1979Au hr9R0kRJGgQU8cGS8LUbXd4xnnFRJhHB/GaWT1movUJ8KENQxPM+KSeB/dcyJs42Mc6N 6J5UQlSWJjZnOsqZuwkyAvRW0Ulk3ta+xgEiNIyg+lz4eolHLwmByb36K4enlq01Nhku 7aDAML+T2SMsuo/ouMczogSB9oWFyFVuPlxyM4j7ffUttvP7viOpvYsmty1nzdRMdlyH Xmw1kO1AhqebKlJTJDBwU0ZllwiIpL/zfkd05bck8S/RFoEuLNJ2lJnMXeKY3vEK3huB Rlrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728042966; x=1728647766; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=obcW8TllxKY4racC770ZVBJlg3ef+KFaFjujkzVUGIA=; b=A094ZGMPlYwp8CW4bfLfScQs3c6ZvM4W3N6w76tc3ZuAIRmB4GpIR1ScTAd0QlsjLW aODZua+Y7zHbCJGryk8pj/6SjeCNWK1gXc04aOtp4pm1U7VTpJEo1jRyHFSMX4R1L+X8 Tz6TDfCqEhAiqUdMBY7blfS1rN4RMO7cz0Jb+b1BqlPiQaKfrrmldN+H4XoA1Vu4b718 mLWbWtcKjw4IOQr2iuAiHgmlt3uJrMZPp5z6Me1/ZPZ4n2ftwEfSSxVlwLguqjBccrhO 9XtJL6+fKnnQZwwGmUypKGrTvppY/+uTotQKkQFYi2BcIP5vVsvj3ifXR35cXS+S8HL7 OXlQ== X-Gm-Message-State: AOJu0YzVEyE0PKxyHa1artPw4XhFd7sNABVn2HvkT8UGvr9RtDbzLfiF +b1+kK7OGy3lt/iCJG2bi1ZkHAfPbdeeNRzGwZdIyY1qb/uGqYrY41BOOQAbwKUaaX/635mATOV W X-Google-Smtp-Source: AGHT+IFfVtQZ1tC5ApzCGAa1QQ6y8mfMrURmjQUNfW1wEarklQHspQn0wpXJIfvZPkAyzmTHjZ25yA== X-Received: by 2002:adf:ce12:0:b0:371:9121:5643 with SMTP id ffacd0b85a97d-37d0e6f247cmr1557654f8f.12.1728042965359; Fri, 04 Oct 2024 04:56:05 -0700 (PDT) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:daa2:371b:a97:3e3e]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37d081f743esm3107147f8f.21.2024.10.04.04.56.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Oct 2024 04:56:05 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman , Arsen Mikovic , Naushir Patuck Subject: [PATCH 5/6] ipa: rpi: sync: Add an implementation of the camera sync algorithm Date: Fri, 4 Oct 2024 12:55:57 +0100 Message-Id: <20241004115558.9166-6-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241004115558.9166-1-david.plowman@raspberrypi.com> References: <20241004115558.9166-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In this implementation, the server sends data packets out onto the network every 30 frames or so. Clients listening for this packet will send frame length deltas back to the pipeline handler to match the synchronisation of the server. We can use wallclock timestamps so that the process will actually work across networked Pis, but it does really on those wallclocks being properly synchronised. We de-jitter our wallclock measurements (as they're made in userspace) to match the more accurate kernel SensorTimestamp value. When the server's advertised "ready time" is reached, both client and server will signal this through metadata back to their respective controlling applications. Signed-off-by: David Plowman Signed-off-by: Arsen Mikovic Signed-off-by: Naushir Patuck --- src/ipa/rpi/controller/meson.build | 2 + src/ipa/rpi/controller/rpi/clock_recovery.cpp | 87 ++++ src/ipa/rpi/controller/rpi/clock_recovery.h | 55 +++ src/ipa/rpi/controller/rpi/sync.cpp | 384 ++++++++++++++++++ src/ipa/rpi/controller/rpi/sync.h | 71 ++++ 5 files changed, 599 insertions(+) create mode 100644 src/ipa/rpi/controller/rpi/clock_recovery.cpp create mode 100644 src/ipa/rpi/controller/rpi/clock_recovery.h create mode 100644 src/ipa/rpi/controller/rpi/sync.cpp create mode 100644 src/ipa/rpi/controller/rpi/sync.h diff --git a/src/ipa/rpi/controller/meson.build b/src/ipa/rpi/controller/meson.build index 74b74888..8df38a0c 100644 --- a/src/ipa/rpi/controller/meson.build +++ b/src/ipa/rpi/controller/meson.build @@ -13,6 +13,7 @@ rpi_ipa_controller_sources = files([ 'rpi/black_level.cpp', 'rpi/cac.cpp', 'rpi/ccm.cpp', + 'rpi/clock_recovery.cpp', 'rpi/contrast.cpp', 'rpi/denoise.cpp', 'rpi/dpc.cpp', @@ -23,6 +24,7 @@ rpi_ipa_controller_sources = files([ 'rpi/saturation.cpp', 'rpi/sdn.cpp', 'rpi/sharpen.cpp', + 'rpi/sync.cpp', 'rpi/tonemap.cpp', ]) diff --git a/src/ipa/rpi/controller/rpi/clock_recovery.cpp b/src/ipa/rpi/controller/rpi/clock_recovery.cpp new file mode 100644 index 00000000..1ccbf9e9 --- /dev/null +++ b/src/ipa/rpi/controller/rpi/clock_recovery.cpp @@ -0,0 +1,87 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * Camera sync control algorithm + */ +#include "clock_recovery.h" + +#include + +using namespace RPiController; +using namespace libcamera; + +LOG_DEFINE_CATEGORY(RPiClockRec) + +ClockRecovery::ClockRecovery() +{ + initialise(); +} + +void ClockRecovery::initialise(unsigned int numPts, unsigned int maxJitter, unsigned int minPts) +{ + numPts_ = numPts; + maxJitter_ = maxJitter; + minPts_ = minPts; + reset(); +} + +void ClockRecovery::reset() +{ + xAve_ = 0; + yAve_ = 0; + x2Ave_ = 0; + xyAve_ = 0; + count_ = 0; + slope_ = 0.0; + offset_ = 0.0; +} + +void ClockRecovery::addSample(uint64_t input, uint64_t output) +{ + if (count_ == 0) { + inputBase_ = input; + outputBase_ = output; + } + + /* + * Never let the new output value be more than maxJitter_ away from what we would have expected. + * This is just to filter out any rare but really crazy values. + */ + uint64_t expectedOutput = getOutput(input); + output = std::clamp(output, expectedOutput - maxJitter_, expectedOutput + maxJitter_); + double x = input - inputBase_; + double y = output - outputBase_ - x; + + /* + * We use x, y, x^2 and x*y sums to calculate the best fit line. Here we update them by + * pretending we have count_ samples at the previous fit, and now one new one. Gradually + * the effect of the older values gets lost. This is a very simple way of updating the + * fit (there are much more complicated ones!), but it works well enough. Using averages + * instead of sums makes the relative effect of old values and the new sample clearer. + */ + unsigned int count1 = count_ + 1; + xAve_ = (count_ * xAve_ + x) / count1; + yAve_ = (count_ * yAve_ + y) / count1; + x2Ave_ = (count_ * x2Ave_ + x * x) / count1; + xyAve_ = (count_ * xyAve_ + x * y) / count1; + + /* Don't update slope and offset until we've seen "enough" sample points. */ + if (count_ > minPts_) { + /* These are the standard equations for least squares linear regressions. */ + slope_ = (count1 * count1 * xyAve_ - count1 * xAve_ * count1 * yAve_) / + (count1 * count1 * x2Ave_ - count1 * xAve_ * count1 * xAve_); + offset_ = yAve_ - slope_ * xAve_; + } + + /* Don't increase count_ above numPts_, as this controls the long-term amount of the residual fit. */ + if (count1 < numPts_) + count_++; +} + +uint64_t ClockRecovery::getOutput(uint64_t input) +{ + double x = input - inputBase_; + double y = slope_ * x + offset_; + return y + x + outputBase_; +} diff --git a/src/ipa/rpi/controller/rpi/clock_recovery.h b/src/ipa/rpi/controller/rpi/clock_recovery.h new file mode 100644 index 00000000..dd05dd97 --- /dev/null +++ b/src/ipa/rpi/controller/rpi/clock_recovery.h @@ -0,0 +1,55 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * Camera sync control algorithm + */ +#pragma once + +#include + +namespace RPiController { + +class ClockRecovery +{ +public: + ClockRecovery(); + + /* Initialise with configuration parameters and restart the fitting process. */ + void initialise(unsigned int numPts = 100, unsigned int maxJitter = 100000, unsigned int minPts = 10); + /* Erase all history and restart the fitting process. */ + void reset(); + + // Add a new input clock / output clock sample. */ + void addSample(uint64_t input, uint64_t output); + /* Calculate the output clock value for this input. */ + uint64_t getOutput(uint64_t input); + +private: + unsigned int numPts_; /* how many samples contribute to the history */ + unsigned int maxJitter_; /* smooth out any jitter larger than this immediately */ + unsigned int minPts_; /* number of samples below which we treat clocks as 1:1 */ + unsigned int count_; /* how many samples seen (up to numPts_) */ + uint64_t inputBase_; /* subtract this from all input values, just to make the numbers easier */ + uint64_t outputBase_; /* as above, for the output */ + + /* + * We do a linear regression of y against x, where: + * x is the value input - inputBase_, and + * y is the value output - outputBase_ - x. + * We additionally subtract x from y so that y "should" be zero, again making the numnbers easier. + */ + double xAve_; /* average x value seen so far */ + double yAve_; /* average y value seen so far */ + double x2Ave_; /* average x^2 value seen so far */ + double xyAve_; /* average x*y value seen so far */ + + /* + * Once we've seen more than minPts_ samples, we recalculate the slope and offset according + * to the linear regression normal equations. + */ + double slope_; /* latest slope value */ + double offset_; /* latest offset value */ +}; + +} //namespace RPiController diff --git a/src/ipa/rpi/controller/rpi/sync.cpp b/src/ipa/rpi/controller/rpi/sync.cpp new file mode 100644 index 00000000..9e76d879 --- /dev/null +++ b/src/ipa/rpi/controller/rpi/sync.cpp @@ -0,0 +1,384 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * sync.cpp - sync algorithm + */ +#include "sync.h" + +#include +#include +#include +#include +#include + +#include + +#include + +#include "sync_status.h" + +using namespace std; +using namespace std::chrono_literals; +using namespace RPiController; +using namespace libcamera; + +LOG_DEFINE_CATEGORY(RPiSync) + +#define NAME "rpi.sync" + +const char *kDefaultGroup = "239.255.255.250"; +constexpr unsigned int kDefaultPort = 10000; +constexpr unsigned int kDefaultSyncPeriod = 30; +constexpr unsigned int kDefaultReadyFrame = 1000; +constexpr unsigned int kDefaultMinAdjustment = 50; +constexpr unsigned int kDefaultFitNumPts = 100; +constexpr unsigned int kDefaultFitMaxJitter = 100000; +constexpr unsigned int kDefaultFitMinPts = 10; + +/* Returns IP address of the device we are on. */ +static std::string local_address_IP() +{ + const char *google_dns_server = "8.8.8.8"; + int dns_port = 53; + + struct sockaddr_in serv; + int sock = socket(AF_INET, SOCK_DGRAM, 0); + + if (sock < 0) + LOG(RPiSync, Error) << "Socket error"; + + memset(&serv, 0, sizeof(serv)); + serv.sin_family = AF_INET; + serv.sin_addr.s_addr = inet_addr(google_dns_server); + serv.sin_port = htons(dns_port); + + int err = connect(sock, (const struct sockaddr *)&serv, sizeof(serv)); + if (err < 0) + LOG(RPiSync, Error) << "Socket connect error"; + + struct sockaddr_in name; + socklen_t namelen = sizeof(name); + err = getsockname(sock, (struct sockaddr *)&name, &namelen); + + char buffer[80]; + (void)inet_ntop(AF_INET, &name.sin_addr, buffer, 80); + close(sock); + return buffer; +} + +Sync::Sync(Controller *controller) + : SyncAlgorithm(controller), mode_(Mode::Off), socket_(-1), frameDuration_(0s), frameCount_(0) +{ +} + +Sync::~Sync() +{ + if (socket_ >= 0) + close(socket_); +} + +char const *Sync::name() const +{ + return NAME; +} + +/* This reads from json file and intitiaises server and client */ +int Sync::read(const libcamera::YamlObject ¶ms) +{ + /* Socket on which to communicate. */ + group_ = params["group"].get(kDefaultGroup); + port_ = params["port"].get(kDefaultPort); + /* Send a sync message every this many frames. */ + syncPeriod_ = params["sync_period"].get(kDefaultSyncPeriod); + /* Application will be told we're ready after this many frames. */ + readyFrame_ = params["ready_frame"].get(kDefaultReadyFrame); + /* Don't change client frame length unless the change exceeds this amount (microseconds). */ + minAdjustment_ = params["min_adjustment"].get(kDefaultMinAdjustment); + + /* Parameters controlling the clock fitting. */ + uint32_t fitNumPts = params["fit_num_pts"].get(kDefaultFitNumPts); + uint32_t fitMaxJitter = params["fit_max_jitter"].get(kDefaultFitMaxJitter); + uint32_t fitMinPts = params["fit_min_pts"].get(kDefaultFitMinPts); + systemToWallClock_.initialise(fitNumPts, fitMaxJitter, fitMinPts); + + return 0; +} + +void Sync::initialiseSocket() +{ + socket_ = socket(AF_INET, SOCK_DGRAM, 0); + if (socket_ < 0) { + LOG(RPiSync, Error) << "Unable to create socket"; + return; + } + + memset(&addr_, 0, sizeof(addr_)); + addr_.sin_family = AF_INET; + addr_.sin_addr.s_addr = mode_ == Mode::Client ? htonl(INADDR_ANY) : inet_addr(group_.c_str()); + addr_.sin_port = htons(port_); + + if (mode_ == Mode::Client) { + /* Set to non-blocking. */ + int flags = fcntl(socket_, F_GETFL, 0); + fcntl(socket_, F_SETFL, flags | O_NONBLOCK); + + unsigned int en = 1; + if (setsockopt(socket_, SOL_SOCKET, SO_REUSEADDR, &en, sizeof(en)) < 0) { + LOG(RPiSync, Error) << "Unable to set socket options"; + goto err; + } + + struct ip_mreq mreq { + }; + mreq.imr_multiaddr.s_addr = inet_addr(group_.c_str()); + mreq.imr_interface.s_addr = htonl(INADDR_ANY); + if (setsockopt(socket_, IPPROTO_IP, IP_ADD_MEMBERSHIP, &mreq, sizeof(mreq)) < 0) { + LOG(RPiSync, Error) << "Unable to set socket options"; + goto err; + } + + if (bind(socket_, (struct sockaddr *)&addr_, sizeof(addr_)) < 0) { + LOG(RPiSync, Error) << "Unable to bind client socket"; + goto err; + } + } + + return; + +err: + close(socket_); + socket_ = -1; +} + +void Sync::switchMode([[maybe_unused]] CameraMode const &cameraMode, [[maybe_unused]] Metadata *metadata) +{ + syncReady_ = false; + frameCount_ = 0; + firstFrame_ = true; + lag_ = 0; + serverFrameCountPeriod_ = 0; + clientServerReadyTime_ = 0; + clientSeenPacket_ = false; +} + +/* + * Camera sync algorithm. + * Server - there is a single server that sends framerate timing information over the network to any + * clients that are listening. It also signals when it will send a "everything is synchronised, now go" + * message back to the algorithm. + * Client - there may be many clients, either on the same Pi or different ones. They match their + * framerates to the server, and indicate when to "go" at the same instant as the server. + */ +void Sync::process([[maybe_unused]] StatisticsPtr &stats, Metadata *imageMetadata) +{ + SyncPayload payload; + SyncParams local{}; + SyncStatus status{}; + bool lagKnown = true; + + imageMetadata->get("sync.params", local); + + if (!frameDuration_) { + LOG(RPiSync, Error) << "Sync frame duration not set!"; + return; + } + + if (mode_ == Mode::Off) + return; + + if (socket_ < 0) + initialiseSocket(); + + /* The local wallclock for the very first frame can be a bit off, so ignore it. */ + if (firstFrame_) { + firstFrame_ = false; + + /* + * For the client, flush anything in the socket. It might be stale from a previous sync run, + * or we might get another packet in a frame to two before the adjustment caused by this (old) + * packet, although correct, had taken effect. So this keeps things simpler. + */ + if (mode_ == Mode::Client) { + socklen_t addrlen = sizeof(addr_); + int ret = 0; + while (ret >= 0) + ret = recvfrom(socket_, &payload, sizeof(payload), 0, (struct sockaddr *)&addr_, &addrlen); + } + + return; + } + + /* + * It might be possible for a frame not to have a valid wallclock, in which case don't let it + * get into the clock recovery as it would totally throw it off. + */ + if (local.wallClock == 0) { + LOG(RPiSync, Debug) << "Zero-valued wallclock - ignoring"; + return; + } + + /* Derive a de-jittered version of wall clock. sensorTimestamp needs converting from ns to us. */ + uint64_t systemFrameTimestamp = local.sensorTimestamp / 1000; + systemToWallClock_.addSample(systemFrameTimestamp, local.wallClock); + uint64_t wallClockFrameTimestamp = systemToWallClock_.getOutput(systemFrameTimestamp); + + /* + * This is the headline frame duration in microseconds as programmed into the sensor. Strictly, + * the sensor might not quite match the system clock, but this shouldn't matter for the calculations + * we'll do with it, unless it's a very very long way out! + */ + uint32_t frameDuration = frameDuration_.get(); + + /* Timestamps tell us if we've dropped any frames, but we still want to count them. */ + int droppedFrames = 0; + if (frameCount_) { + /* + * Round down here, because frameCount_ gets incremented at the end of the function. Also + * ensure droppedFrames can't go negative. It shouldn't, but things would go badly wrong + * if it did. + */ + wallClockFrameTimestamp = std::max(wallClockFrameTimestamp, lastWallClockFrameTimestamp_ + frameDuration / 2); + droppedFrames = (wallClockFrameTimestamp - lastWallClockFrameTimestamp_ - frameDuration / 2) / frameDuration; + frameCount_ += droppedFrames; + } + + if (mode_ == Mode::Server) { + /* + * Server sends a packet every syncPeriod_ frames, or as soon after as possible (if any + * frames were dropped). + */ + serverFrameCountPeriod_ += droppedFrames; + + /* + * The client may want a better idea of the true frame duration. Any error would feed straight + * into the correction term because of how it uses it to get the "nearest" frame. + */ + if (frameCount_ == 0) + frameDurationEstimated_ = frameDuration; + else { + double diff = (systemFrameTimestamp - lastSystemFrameTimestamp_) / (1 + droppedFrames); + int N = std::min(frameCount_, 99U); + frameDurationEstimated_ = frameCount_ == 1 ? diff : (N * frameDurationEstimated_ + diff) / (N + 1); + } + + /* Calculate frames remaining, and therefore "time left until ready". */ + int framesRemaining = readyFrame_ - frameCount_; + uint64_t systemReadyTime = systemFrameTimestamp + (int64_t)framesRemaining * frameDurationEstimated_; + uint64_t wallClockReadyTime = systemToWallClock_.getOutput(systemReadyTime); + + if (serverFrameCountPeriod_ >= syncPeriod_) { + serverFrameCountPeriod_ = 0; + + payload.frameDuration = frameDurationEstimated_ + .5; /* round to nearest */ + payload.systemFrameTimestamp = systemFrameTimestamp; + payload.wallClockFrameTimestamp = wallClockFrameTimestamp; + payload.systemReadyTime = systemReadyTime; + payload.wallClockReadyTime = wallClockReadyTime; + + LOG(RPiSync, Debug) << "Send packet (frameNumber " << frameCount_ << "):"; + LOG(RPiSync, Debug) << " frameDuration " << payload.frameDuration; + LOG(RPiSync, Debug) << " systemFrameTimestamp " << systemFrameTimestamp + << " (" << systemFrameTimestamp - lastSystemFrameTimestamp_ << ")"; + LOG(RPiSync, Debug) << " wallClockFrameTimestamp " << wallClockFrameTimestamp + << " (" << wallClockFrameTimestamp - lastWallClockFrameTimestamp_ << ")"; + LOG(RPiSync, Debug) << " systemReadyTime " << systemReadyTime; + LOG(RPiSync, Debug) << " wallClockReadyTime " << wallClockReadyTime; + + if (sendto(socket_, &payload, sizeof(payload), 0, (const sockaddr *)&addr_, sizeof(addr_)) < 0) + LOG(RPiSync, Error) << "Send error! " << strerror(errno); + } + + lag_ = (int64_t)wallClockFrameTimestamp - (int64_t)wallClockReadyTime; + if (!syncReady_ && wallClockFrameTimestamp + frameDurationEstimated_ / 2 > wallClockReadyTime) { + syncReady_ = true; + LOG(RPiSync, Info) << "*** Sync achieved! Lag " << lag_; + } + + serverFrameCountPeriod_ += 1; + + } else if (mode_ == Mode::Client) { + uint64_t serverFrameTimestamp = 0; + + bool packetReceived = false; + while (true) { + socklen_t addrlen = sizeof(addr_); + int ret = recvfrom(socket_, &payload, sizeof(payload), 0, (struct sockaddr *)&addr_, &addrlen); + + if (ret < 0) + break; + packetReceived = (ret > 0); + clientSeenPacket_ = true; + + if (!IPCheck_) { + IPCheck_ = true; + char srcIP[INET_ADDRSTRLEN]; + inet_ntop(AF_INET, &(addr_.sin_addr), srcIP, INET_ADDRSTRLEN); + clientSamePi_ = (local_address_IP() == srcIP); + LOG(RPiSync, Debug) << "Server is " << (clientSamePi_ ? "same" : "different"); + } + + frameDurationEstimated_ = payload.frameDuration; + if (clientSamePi_) { + serverFrameTimestamp = payload.systemFrameTimestamp; + clientServerReadyTime_ = payload.systemReadyTime; + } else { + serverFrameTimestamp = payload.wallClockFrameTimestamp; + clientServerReadyTime_ = payload.wallClockReadyTime; + } + } + + if (packetReceived) { + uint64_t clientFrameTimestamp = clientSamePi_ ? systemFrameTimestamp : wallClockFrameTimestamp; + int64_t clientServerDelta = clientFrameTimestamp - serverFrameTimestamp; + /* "A few frames ago" may have better matched the server's frame. Calculate when it was. */ + int framePeriodErrors = (clientServerDelta + frameDurationEstimated_ / 2) / frameDurationEstimated_; + int64_t clientFrameTimestampNearest = clientFrameTimestamp - framePeriodErrors * frameDurationEstimated_; + /* We must shorten a single client frame by this amount if it exceeds the minimum: */ + int32_t correction = clientFrameTimestampNearest - serverFrameTimestamp; + if (std::abs(correction) < minAdjustment_) + correction = 0; + + LOG(RPiSync, Debug) << "Received packet (frameNumber " << frameCount_ << "):"; + LOG(RPiSync, Debug) << " serverFrameTimestamp " << serverFrameTimestamp; + LOG(RPiSync, Debug) << " serverReadyTime " << clientServerReadyTime_; + LOG(RPiSync, Debug) << " clientFrameTimestamp " << clientFrameTimestamp; + LOG(RPiSync, Debug) << " clientFrameTimestampNearest " << clientFrameTimestampNearest + << " (" << framePeriodErrors << ")"; + LOG(RPiSync, Debug) << " systemFrameTimestamp " << systemFrameTimestamp + << " (" << systemFrameTimestamp - lastSystemFrameTimestamp_ << ")"; + LOG(RPiSync, Debug) << " correction " << correction; + + status.frameDurationOffset = correction * 1us; + } + + uint64_t clientFrameTimestamp = clientSamePi_ ? systemFrameTimestamp : wallClockFrameTimestamp; + lag_ = (int64_t)clientFrameTimestamp - (int64_t)clientServerReadyTime_; + lagKnown = clientSeenPacket_; /* client must receive a packet before the lag is correct */ + if (clientSeenPacket_ && !syncReady_ && clientFrameTimestamp + frameDurationEstimated_ / 2 > clientServerReadyTime_) { + syncReady_ = true; + LOG(RPiSync, Info) << "*** Sync achieved! Lag " << lag_; + } + } + + lastSystemFrameTimestamp_ = systemFrameTimestamp; + lastWallClockFrameTimestamp_ = wallClockFrameTimestamp; + + status.ready = syncReady_; + status.lag = lag_; + status.lagKnown = lagKnown; + imageMetadata->set("sync.status", status); + frameCount_++; +} + +void Sync::setFrameDuration(libcamera::utils::Duration frameDuration) +{ + frameDuration_ = frameDuration; +}; + +/* Register algorithm with the system. */ +static Algorithm *create(Controller *controller) +{ + return (Algorithm *)new Sync(controller); +} +static RegisterAlgorithm reg(NAME, &create); diff --git a/src/ipa/rpi/controller/rpi/sync.h b/src/ipa/rpi/controller/rpi/sync.h new file mode 100644 index 00000000..15427adb --- /dev/null +++ b/src/ipa/rpi/controller/rpi/sync.h @@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * sync.h - sync algorithm + */ +#pragma once + +#include + +#include "../sync_algorithm.h" +#include "clock_recovery.h" + +namespace RPiController { + +struct SyncPayload { + /* Frame duration in microseconds. */ + uint32_t frameDuration; + /* Server system (kernel) frame timestamp. */ + uint64_t systemFrameTimestamp; + /* Server wall clock version of the frame timestamp. */ + uint64_t wallClockFrameTimestamp; + /* Server system (kernel) sync time (the time at which frames are marked ready). */ + uint64_t systemReadyTime; + /* Server wall clock version of the sync time. */ + uint64_t wallClockReadyTime; +}; + +class Sync : public SyncAlgorithm +{ +public: + Sync(Controller *controller); + ~Sync(); + char const *name() const override; + int read(const libcamera::YamlObject ¶ms) override; + void setMode(Mode mode) override { mode_ = mode; } + void initialiseSocket(); + void switchMode(CameraMode const &cameraMode, Metadata *metadata) override; + void process(StatisticsPtr &stats, Metadata *imageMetadata) override; + void setFrameDuration(libcamera::utils::Duration frameDuration) override; + +private: + Mode mode_; /* server or client */ + std::string group_; /* IP group address for sync messages */ + uint16_t port_; /* port number for messages */ + uint32_t syncPeriod_; /* send a sync message every this many frames */ + uint32_t readyFrame_; /* tell the application we're ready after this many frames */ + uint32_t minAdjustment_; /* don't adjust the client frame length by less than this */ + + struct sockaddr_in addr_; + int socket_ = -1; + libcamera::utils::Duration frameDuration_; + unsigned int frameCount_; + bool syncReady_; + int64_t lag_ = 0; + bool IPCheck_ = false; + bool firstFrame_ = true; + + double frameDurationEstimated_ = 0; /* estimate the true frame duration of the sensor */ + ClockRecovery systemToWallClock_; /* for deriving a de-jittered wall clock time */ + uint64_t lastSystemFrameTimestamp_; /* system timestamp of previous frame */ + uint64_t lastWallClockFrameTimestamp_; /* wall clock timestamp of previous frame */ + + uint32_t serverFrameCountPeriod_ = 0; /* send the next packet when this reaches syncPeriod_ */ + + bool clientSeenPacket_ = false; /* whether the client has received a packet yet */ + bool clientSamePi_ = false; /* whether server running on the same Pi as client */ + uint64_t clientServerReadyTime_ = 0; /* the client's latest value for when the server will be "ready" */ +}; + +} /* namespace RPiController */ From patchwork Fri Oct 4 11:55:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 21517 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E5A39BD80A for ; Fri, 4 Oct 2024 11:56:23 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B75D763534; Fri, 4 Oct 2024 13:56:22 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="XR4MMk2o"; dkim-atps=neutral Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 393756352E for ; Fri, 4 Oct 2024 13:56:07 +0200 (CEST) Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-42e82f7f36aso17865675e9.0 for ; Fri, 04 Oct 2024 04:56:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1728042966; x=1728647766; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=p1YK8DKtA0NpWinHIZZXLF01L8APL7ANiW8Nc1Y2jjk=; b=XR4MMk2obLEhpD9/V2SVjWiWG1xI8NV6aAMhuHbygqe1PNnO3AieZGv6yPUJiJr4yo NGwV9ZI9Rt54Z7TJLoTpyiSDXTdwJ3Yurb/Li4/UhM11aOqc6lbvfQPiyQh5zxHc5q91 fH1RUZ9GP4QveIM5ztysIg0KHLkWCXQ4+iO2JSv3Bm0wBeXsBob4eXvtmbmXq+J+IFgT 6q+anDohPfAM9e804Ygg33F+WLp4DorV+ghV2m5Gakt5OHIOzlHrpSQ22NZLW1KmlAOE 3diBtZKTUECLKngU5V34FStZ0K7bhR3F1gVMdpAyDjPd3yC1KEPq03HpmCncx4F8flqu lGsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728042966; x=1728647766; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p1YK8DKtA0NpWinHIZZXLF01L8APL7ANiW8Nc1Y2jjk=; b=lmocA/HYO38tXPTNnRsJK4ejPxzQmK2mMDvOuRRgPA4sfgrM20R1k5H+FzS01HbWjh M06nqIsXbWN+N14/h3A2kxwde+VHBtuVIdwQXfNB7t8I51Td6WJ7Mo9V+rGvAuDZXqsw UHfz+W/0ZHB57quNwLcjHq7dYK2iOUU3WVbYTDqv7RPvdUUx+BH/5gwPjDANVW55lkCD LXiHKmiZ9vEiKijpFq7753M2fVxQxwCn9uPIlV6KryHDhs2JZ5vXav7rEG4Uvh5OxMSU rumm0nCE8v0g75RzjO4qq/dKYqo/FengSA19IIa+CyC3HvPhzfi4nwBiLSgGaSXDLYbj BItw== X-Gm-Message-State: AOJu0YyD6QUxKtwENsFedi/+ZhKBqL9eSQE6o6Bmz2xuJ3wfMyxc+zUR IAdBqAz3pr6BWZ2x9rcstOXvDm+LIlRp+uzxmU134MZTKDgi8Z8BAOFLR+qKM5JhEDrP997NiDS G X-Google-Smtp-Source: AGHT+IGE1L28GFMwWh5mWhM1yGDNVLzsLUh2JSy0avwY1Xg0X1ctGSyX/Hob9nFuQ7CFFskGa2HESg== X-Received: by 2002:a5d:4305:0:b0:374:bad2:6a5e with SMTP id ffacd0b85a97d-37d0e777812mr1433641f8f.28.1728042966264; Fri, 04 Oct 2024 04:56:06 -0700 (PDT) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:daa2:371b:a97:3e3e]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37d081f743esm3107147f8f.21.2024.10.04.04.56.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Oct 2024 04:56:05 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman , Naushir Patuck Subject: [PATCH 6/6] ipa: rpi: vc4: Update all tuning files for sync algorithm Date: Fri, 4 Oct 2024 12:55:58 +0100 Message-Id: <20241004115558.9166-7-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241004115558.9166-1-david.plowman@raspberrypi.com> References: <20241004115558.9166-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Standard sync parameters are added to all tuning files. Signed-off-by: David Plowman Reviewed-by: Naushir Patuck --- src/ipa/rpi/vc4/data/imx219.json | 11 ++++++++++- src/ipa/rpi/vc4/data/imx219_noir.json | 11 ++++++++++- src/ipa/rpi/vc4/data/imx283.json | 11 ++++++++++- src/ipa/rpi/vc4/data/imx290.json | 11 ++++++++++- src/ipa/rpi/vc4/data/imx296.json | 11 ++++++++++- src/ipa/rpi/vc4/data/imx296_mono.json | 11 ++++++++++- src/ipa/rpi/vc4/data/imx378.json | 11 ++++++++++- src/ipa/rpi/vc4/data/imx477.json | 11 ++++++++++- src/ipa/rpi/vc4/data/imx477_noir.json | 11 ++++++++++- src/ipa/rpi/vc4/data/imx477_scientific.json | 11 ++++++++++- src/ipa/rpi/vc4/data/imx519.json | 11 ++++++++++- src/ipa/rpi/vc4/data/imx708.json | 11 ++++++++++- src/ipa/rpi/vc4/data/imx708_noir.json | 11 ++++++++++- src/ipa/rpi/vc4/data/imx708_wide.json | 11 ++++++++++- src/ipa/rpi/vc4/data/imx708_wide_noir.json | 11 ++++++++++- src/ipa/rpi/vc4/data/ov5647.json | 11 ++++++++++- src/ipa/rpi/vc4/data/ov5647_noir.json | 11 ++++++++++- src/ipa/rpi/vc4/data/se327m12.json | 11 ++++++++++- 18 files changed, 180 insertions(+), 18 deletions(-) diff --git a/src/ipa/rpi/vc4/data/imx219.json b/src/ipa/rpi/vc4/data/imx219.json index a020b12f..808581f0 100644 --- a/src/ipa/rpi/vc4/data/imx219.json +++ b/src/ipa/rpi/vc4/data/imx219.json @@ -690,6 +690,15 @@ } } } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } diff --git a/src/ipa/rpi/vc4/data/imx219_noir.json b/src/ipa/rpi/vc4/data/imx219_noir.json index d8bc9639..e9706eb7 100644 --- a/src/ipa/rpi/vc4/data/imx219_noir.json +++ b/src/ipa/rpi/vc4/data/imx219_noir.json @@ -624,6 +624,15 @@ } } } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } diff --git a/src/ipa/rpi/vc4/data/imx283.json b/src/ipa/rpi/vc4/data/imx283.json index bfacecc8..6d7bec0b 100644 --- a/src/ipa/rpi/vc4/data/imx283.json +++ b/src/ipa/rpi/vc4/data/imx283.json @@ -308,6 +308,15 @@ }, { "rpi.sharpen": { } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } diff --git a/src/ipa/rpi/vc4/data/imx290.json b/src/ipa/rpi/vc4/data/imx290.json index 8f41bf51..c63f7487 100644 --- a/src/ipa/rpi/vc4/data/imx290.json +++ b/src/ipa/rpi/vc4/data/imx290.json @@ -209,6 +209,15 @@ } ] } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx296.json b/src/ipa/rpi/vc4/data/imx296.json index 8f24ce5b..f6638003 100644 --- a/src/ipa/rpi/vc4/data/imx296.json +++ b/src/ipa/rpi/vc4/data/imx296.json @@ -438,6 +438,15 @@ "strength": 1.0, "limit": 0.18 } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx296_mono.json b/src/ipa/rpi/vc4/data/imx296_mono.json index fe331569..f05698ef 100644 --- a/src/ipa/rpi/vc4/data/imx296_mono.json +++ b/src/ipa/rpi/vc4/data/imx296_mono.json @@ -235,6 +235,15 @@ "strength": 1.0, "limit": 0.18 } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx378.json b/src/ipa/rpi/vc4/data/imx378.json index 363b47e1..9cc76c97 100644 --- a/src/ipa/rpi/vc4/data/imx378.json +++ b/src/ipa/rpi/vc4/data/imx378.json @@ -422,6 +422,15 @@ }, { "rpi.sharpen": { } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx477.json b/src/ipa/rpi/vc4/data/imx477.json index fa25ee86..750c087f 100644 --- a/src/ipa/rpi/vc4/data/imx477.json +++ b/src/ipa/rpi/vc4/data/imx477.json @@ -695,6 +695,15 @@ } } } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx477_noir.json b/src/ipa/rpi/vc4/data/imx477_noir.json index 472f33fe..97f4e2a5 100644 --- a/src/ipa/rpi/vc4/data/imx477_noir.json +++ b/src/ipa/rpi/vc4/data/imx477_noir.json @@ -651,6 +651,15 @@ } } } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } diff --git a/src/ipa/rpi/vc4/data/imx477_scientific.json b/src/ipa/rpi/vc4/data/imx477_scientific.json index 9dc32eb1..b3d7a4ff 100644 --- a/src/ipa/rpi/vc4/data/imx477_scientific.json +++ b/src/ipa/rpi/vc4/data/imx477_scientific.json @@ -483,6 +483,15 @@ }, { "rpi.sharpen": { } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx519.json b/src/ipa/rpi/vc4/data/imx519.json index ce194256..54bdc796 100644 --- a/src/ipa/rpi/vc4/data/imx519.json +++ b/src/ipa/rpi/vc4/data/imx519.json @@ -422,6 +422,15 @@ }, { "rpi.sharpen": { } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx708.json b/src/ipa/rpi/vc4/data/imx708.json index 4de6f079..56b7af8a 100644 --- a/src/ipa/rpi/vc4/data/imx708.json +++ b/src/ipa/rpi/vc4/data/imx708.json @@ -666,6 +666,15 @@ } } } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx708_noir.json b/src/ipa/rpi/vc4/data/imx708_noir.json index 7b7ee874..7d123a64 100644 --- a/src/ipa/rpi/vc4/data/imx708_noir.json +++ b/src/ipa/rpi/vc4/data/imx708_noir.json @@ -765,6 +765,15 @@ } } } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx708_wide.json b/src/ipa/rpi/vc4/data/imx708_wide.json index 6f45aafc..0435706b 100644 --- a/src/ipa/rpi/vc4/data/imx708_wide.json +++ b/src/ipa/rpi/vc4/data/imx708_wide.json @@ -677,6 +677,15 @@ } } } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/imx708_wide_noir.json b/src/ipa/rpi/vc4/data/imx708_wide_noir.json index b9a5227e..a8c3f15b 100644 --- a/src/ipa/rpi/vc4/data/imx708_wide_noir.json +++ b/src/ipa/rpi/vc4/data/imx708_wide_noir.json @@ -668,6 +668,15 @@ } } } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/ov5647.json b/src/ipa/rpi/vc4/data/ov5647.json index 40c6059c..c65173d5 100644 --- a/src/ipa/rpi/vc4/data/ov5647.json +++ b/src/ipa/rpi/vc4/data/ov5647.json @@ -691,6 +691,15 @@ } } } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } diff --git a/src/ipa/rpi/vc4/data/ov5647_noir.json b/src/ipa/rpi/vc4/data/ov5647_noir.json index 488b7119..3b7510e1 100644 --- a/src/ipa/rpi/vc4/data/ov5647_noir.json +++ b/src/ipa/rpi/vc4/data/ov5647_noir.json @@ -407,6 +407,15 @@ }, { "rpi.sharpen": { } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } \ No newline at end of file diff --git a/src/ipa/rpi/vc4/data/se327m12.json b/src/ipa/rpi/vc4/data/se327m12.json index 948169db..d3bedb95 100644 --- a/src/ipa/rpi/vc4/data/se327m12.json +++ b/src/ipa/rpi/vc4/data/se327m12.json @@ -427,6 +427,15 @@ "strength": 0.5, "limit": 0.5 } - } + }, + { + "rpi.sync": + { + "group": "224.0.0.1", + "port": 12345, + "sync_period": 30, + "ready_frame": 1000 + } + } ] } \ No newline at end of file