From patchwork Wed Jan 22 14:53:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 22613 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 3E9C5C3319 for ; Wed, 22 Jan 2025 14:54:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E982C68567; Wed, 22 Jan 2025 15:54:07 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="lC/b6pRD"; dkim-atps=neutral Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E09FA68559 for ; Wed, 22 Jan 2025 15:53:56 +0100 (CET) Received: by mail-wr1-x42c.google.com with SMTP id ffacd0b85a97d-38634c35129so5941096f8f.3 for ; Wed, 22 Jan 2025 06:53:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1737557636; x=1738162436; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dZRsfHRIhVt9X4suCupUrPLxrY9XKlNXZm04oJrTHek=; b=lC/b6pRDQc5GBYRQxFMdA3LuenWBkRtCRnnn9klPHISXSJbFCJDI9hBysmFRJqqiag 0XXdcfhX8zJm8Q53sfaJX5mwtkErcCyqnDT7ah2tiV05O0xi5EsfMIPFobzsl63G1l1w 9VKPRKBOPPZc/G7D76H4GTOpX6uPD5+KPlts04bjWcg875C1jEHS9uEkDahvJpHE7FRs HTNRYFQBZw5gHM35UzNK9yzrI3VTDt+d8j+OjpN1Sf4p1rgM6Z3w79dzqQFMFhpafHGe B3fd4yUe73mzyUizLTuslFwCSIdR3bBTibdcf0B3y1AvghQYr931cuEP15L4sNWHz6we usVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737557636; x=1738162436; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dZRsfHRIhVt9X4suCupUrPLxrY9XKlNXZm04oJrTHek=; b=AvBQe6sw/aIlVuNOqcT24WGKuG3blgVR59JgkPwJNI+MxWocBmmvQCGJ+6fOHtdZy5 EQbgoaXUNXhKgC0naFR9SLpdrhO41MzboRuDb2ucgYGqkJzMIreUfTq2YXcIzv7y75aV ZiCI8CkFfnSW9W+QcwHCkd485j46tzLFdxvzH8hAqxXyFX7PyL0M9SNK7o+2pm0ZKndb gcUEX+KLNy4jyQchylA96543RYvnaV+6bxYNOIR2tiM6QRy7oBVbgzOtN13+Z+36SYSl osjuRskYIIzAaUhoW5bCqkfMJfQEUlTJNiwsDOAP9q55k1HFl1JoA6upznW36zrKetZA aBSQ== X-Gm-Message-State: AOJu0YxsO3WbpFKTdOoy6fbkT7yNwQYLfaqlo+e5o3ma1lZl5IYOUymj R/vCwacT4daEGNg436927h9xUpARUlVVmxA3M01Wo2IPlL8EpsuHuLaYvAYoL6E7Ms8SOzZeX6B s X-Gm-Gg: ASbGncskZg21wIMvl46r3joKM/W+ii7UnVPk9AICgFlv05aC4T44MihR/QsGx+7WZnm 7CaTOeA59LbufdOFauYhJlcipr8L+ST5b0uk3IRhTVMzFVYkjy/KV3gjuGNnHvgpW7oPBZDHJ3y 7WTzj5iurGo7YfV1Gd4dSz9VBw0r1Hgyet47lLpCeiz9VGIp/OPfv9cdDSZViIxJr65lerwbVxj WvHGtIQUJ/FI9jSh32BhHRECfMtIIPE+5E4OT7duNzdhhyvBk25HZXIfSLp940g6lP9ySsnZt4D qgW3afrssMXiIXVFwinaOTYvcw== X-Google-Smtp-Source: AGHT+IEm1yvPzu9FMPWb+e6Pqx1T79DY1cxylGpCoBXikxCqCCijptf6vT9/iW6s60rPQjVFjIsudg== X-Received: by 2002:adf:f141:0:b0:38a:418e:1171 with SMTP id ffacd0b85a97d-38bf58e88admr14678658f8f.37.1737557636073; Wed, 22 Jan 2025 06:53:56 -0800 (PST) Received: from raspberrypi.pitowers.org ([2a00:1098:3142:1f:ffc9:aff6:7f7f:893b]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-438b66dc08bsm11551395e9.37.2025.01.22.06.53.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jan 2025 06:53:55 -0800 (PST) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: Naushir Patuck , David Plowman Subject: [PATCH v4 5/7] ipa: rpi: Add base classes and plumbing for sync algorithm Date: Wed, 22 Jan 2025 14:53:47 +0000 Message-Id: <20250122145349.7220-6-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250122145349.7220-1-david.plowman@raspberrypi.com> References: <20250122145349.7220-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Naushir Patuck We add a base class for a "sync algorithm", and define its inputs and outputs in the SyncStatus class. We add the necessary plumbing to the base IPA code so as to arrange for the necessary parameters to be made available to such an algorithm, and also to handle the return values, passing them back as necessary to the pipeline handler. Signed-off-by: Naushir Patuck Signed-off-by: David Plowman --- src/ipa/rpi/common/ipa_base.cpp | 61 +++++++++++++++++++++++-- src/ipa/rpi/common/ipa_base.h | 4 +- src/ipa/rpi/controller/sync_algorithm.h | 31 +++++++++++++ src/ipa/rpi/controller/sync_status.h | 27 +++++++++++ 4 files changed, 118 insertions(+), 5 deletions(-) create mode 100644 src/ipa/rpi/controller/sync_algorithm.h create mode 100644 src/ipa/rpi/controller/sync_status.h diff --git a/src/ipa/rpi/common/ipa_base.cpp b/src/ipa/rpi/common/ipa_base.cpp index bd3c2200..528e163c 100644 --- a/src/ipa/rpi/common/ipa_base.cpp +++ b/src/ipa/rpi/common/ipa_base.cpp @@ -28,6 +28,8 @@ #include "controller/lux_status.h" #include "controller/sharpen_algorithm.h" #include "controller/statistics.h" +#include "controller/sync_algorithm.h" +#include "controller/sync_status.h" namespace libcamera { @@ -81,6 +83,8 @@ const ControlInfoMap::Map ipaControls{ { &controls::Sharpness, ControlInfo(0.0f, 16.0f, 1.0f) }, { &controls::ScalerCrop, ControlInfo(Rectangle{}, Rectangle(65535, 65535, 65535, 65535), Rectangle{}) }, { &controls::FrameDurationLimits, ControlInfo(INT64_C(33333), INT64_C(120000)) }, + { &controls::rpi::SyncMode, ControlInfo(controls::rpi::SyncModeValues) }, + { &controls::rpi::SyncFrames, ControlInfo(1, 1000000, 100) }, { &controls::draft::NoiseReductionMode, ControlInfo(controls::draft::NoiseReductionModeValues) }, { &controls::rpi::StatsOutputEnable, ControlInfo(false, true, false) }, }; @@ -399,6 +403,7 @@ void IpaBase::prepareIsp(const PrepareParams ¶ms) rpiMetadata.clear(); fillDeviceStatus(params.sensorControls, ipaContext); + fillSyncParams(params, ipaContext); if (params.buffers.embedded) { /* @@ -497,10 +502,24 @@ void IpaBase::processStats(const ProcessParams ¶ms) helper_->process(statistics, rpiMetadata); controller_.process(statistics, &rpiMetadata); + /* Send any sync algorithm outputs back to the pipeline handler */ + Duration offset(0s); + struct SyncStatus syncStatus; + if (rpiMetadata.get("sync.status", syncStatus) == 0) { + if (minFrameDuration_ != maxFrameDuration_) + LOG(IPARPI, Error) << "Sync algorithm enabled with variable framerate. " + << minFrameDuration_ << " " << maxFrameDuration_; + offset = syncStatus.frameDurationOffset; + + libcameraMetadata_.set(controls::rpi::SyncReady, syncStatus.ready); + if (syncStatus.timerKnown) + libcameraMetadata_.set(controls::rpi::SyncTimer, syncStatus.timerValue); + } + struct AgcStatus agcStatus; if (rpiMetadata.get("agc.status", agcStatus) == 0) { ControlList ctrls(sensorCtrls_); - applyAGC(&agcStatus, ctrls); + applyAGC(&agcStatus, ctrls, offset); setDelayedControls.emit(ctrls, ipaContext); setCameraTimeoutValue(); } @@ -737,6 +756,7 @@ void IpaBase::applyControls(const ControlList &controls) using RPiController::ContrastAlgorithm; using RPiController::DenoiseAlgorithm; using RPiController::HdrAlgorithm; + using RPiController::SyncAlgorithm; /* Clear the return metadata buffer. */ libcameraMetadata_.clear(); @@ -1328,6 +1348,17 @@ void IpaBase::applyControls(const ControlList &controls) statsMetadataOutput_ = ctrl.second.get(); break; + case controls::rpi::SYNC_FRAMES: { + SyncAlgorithm *sync = dynamic_cast(controller_.getAlgorithm("sync")); + + if (sync) { + int frames = ctrl.second.get(); + if (frames > 0) + sync->setReadyFrame(frames); + } + break; + } + default: LOG(IPARPI, Warning) << "Ctrl " << controls::controls.at(ctrl.first)->name() @@ -1364,6 +1395,19 @@ void IpaBase::fillDeviceStatus(const ControlList &sensorControls, unsigned int i rpiMetadata_[ipaContext].set("device.status", deviceStatus); } +void IpaBase::fillSyncParams(const PrepareParams ¶ms, unsigned int ipaContext) +{ + RPiController::SyncAlgorithm *sync = dynamic_cast( + controller_.getAlgorithm("sync")); + if (!sync) + return; + + SyncParams syncParams; + syncParams.wallClock = *params.sensorControls.get(controls::FrameWallClock); + syncParams.sensorTimestamp = *params.sensorControls.get(controls::SensorTimestamp); + rpiMetadata_[ipaContext].set("sync.params", syncParams); +} + void IpaBase::reportMetadata(unsigned int ipaContext) { RPiController::Metadata &rpiMetadata = rpiMetadata_[ipaContext]; @@ -1542,14 +1586,22 @@ void IpaBase::applyFrameDurations(Duration minFrameDuration, Duration maxFrameDu * value possible. */ Duration maxExposureTime = Duration::max(); - helper_->getBlanking(maxExposureTime, minFrameDuration_, maxFrameDuration_); + auto [vblank, hblank] = helper_->getBlanking(maxExposureTime, minFrameDuration_, maxFrameDuration_); RPiController::AgcAlgorithm *agc = dynamic_cast( controller_.getAlgorithm("agc")); agc->setMaxExposureTime(maxExposureTime); + + RPiController::SyncAlgorithm *sync = dynamic_cast( + controller_.getAlgorithm("sync")); + if (sync) { + Duration duration = (mode_.height + vblank) * ((mode_.width + hblank) * 1.0s / mode_.pixelRate); + LOG(IPARPI, Debug) << "setting sync frame duration to " << duration; + sync->setFrameDuration(duration); + } } -void IpaBase::applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls) +void IpaBase::applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls, Duration frameDurationOffset) { const int32_t minGainCode = helper_->gainCode(mode_.minAnalogueGain); const int32_t maxGainCode = helper_->gainCode(mode_.maxAnalogueGain); @@ -1564,7 +1616,8 @@ void IpaBase::applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls) /* getBlanking might clip exposure time to the fps limits. */ Duration exposure = agcStatus->exposureTime; - auto [vblank, hblank] = helper_->getBlanking(exposure, minFrameDuration_, maxFrameDuration_); + auto [vblank, hblank] = helper_->getBlanking(exposure, minFrameDuration_ - frameDurationOffset, + maxFrameDuration_ - frameDurationOffset); int32_t exposureLines = helper_->exposureLines(exposure, helper_->hblankToLineLength(hblank)); diff --git a/src/ipa/rpi/common/ipa_base.h b/src/ipa/rpi/common/ipa_base.h index 1a811beb..b53d0bfb 100644 --- a/src/ipa/rpi/common/ipa_base.h +++ b/src/ipa/rpi/common/ipa_base.h @@ -95,9 +95,11 @@ private: void applyControls(const ControlList &controls); virtual void handleControls(const ControlList &controls) = 0; void fillDeviceStatus(const ControlList &sensorControls, unsigned int ipaContext); + void fillSyncParams(const PrepareParams ¶ms, unsigned int ipaContext); void reportMetadata(unsigned int ipaContext); void applyFrameDurations(utils::Duration minFrameDuration, utils::Duration maxFrameDuration); - void applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls); + void applyAGC(const struct AgcStatus *agcStatus, ControlList &ctrls, + utils::Duration frameDurationOffset = utils::Duration(0)); std::map buffers_; diff --git a/src/ipa/rpi/controller/sync_algorithm.h b/src/ipa/rpi/controller/sync_algorithm.h new file mode 100644 index 00000000..c242def6 --- /dev/null +++ b/src/ipa/rpi/controller/sync_algorithm.h @@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * sync_algorithm.h - Camera sync algorithm interface + */ +#pragma once + +#include + +#include "algorithm.h" + +namespace RPiController { + +class SyncAlgorithm : public Algorithm +{ +public: + enum class Mode { + Off, + Server, + Client, + }; + + SyncAlgorithm(Controller *controller) + : Algorithm(controller) {} + virtual void setFrameDuration(libcamera::utils::Duration frameDuration) = 0; + virtual void setReadyFrame(unsigned int frame) = 0; + virtual void setMode(Mode mode) = 0; +}; + +} /* namespace RPiController */ diff --git a/src/ipa/rpi/controller/sync_status.h b/src/ipa/rpi/controller/sync_status.h new file mode 100644 index 00000000..10f97502 --- /dev/null +++ b/src/ipa/rpi/controller/sync_status.h @@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Copyright (C) 2024, Raspberry Pi Ltd + * + * sync_status.h - Sync algorithm params and status structures + */ +#pragma once + +#include + +struct SyncParams { + /* Wall clock time for this frame */ + uint64_t wallClock; + /* Kernel timestamp for this frame */ + uint64_t sensorTimestamp; +}; + +struct SyncStatus { + /* Frame length correction to apply */ + libcamera::utils::Duration frameDurationOffset; + /* Whether the "ready time" has been reached */ + bool ready; + /* Time until the "ready time" */ + int64_t timerValue; + /* Whether timerValue is known (client has to wait for a server message) */ + bool timerKnown; +};