From patchwork Fri May 8 12:57:43 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 26693 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id AAFCFC32F6 for ; Fri, 8 May 2026 13:08:13 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2F22363021; Fri, 8 May 2026 15:08:12 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="QjgoVAIx"; dkim-atps=neutral Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id CD62E62FE1 for ; Fri, 8 May 2026 15:08:08 +0200 (CEST) Received: by mail-wm1-x336.google.com with SMTP id 5b1f17b1804b1-4893940bb5eso12252505e9.3 for ; Fri, 08 May 2026 06:08:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; t=1778245688; x=1778850488; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=p0mMKpb1i8B3EQhJz0nd+cLEbb2uFro+HYoXPQ9QXgw=; b=QjgoVAIxa9Pmgypr5d4vOeelXdQwvwntBu4NrBAZBKJ3HQrT/pWwb2CPf9f8Zx38Xn GHlZrtpIP4tXKZxb8li62uJsRHrtQ4P+p3hplAt4Uy3FYRNTEvNIFErTJgKOMtPbLbO+ sIkxHeus0n8PySaYk9Tlzx0VnAH6YfPHxP7KQI+B/P43UFsDatYFoln68VcSQ7g2dRc4 wItpcSi+g36/zkkdBA6YVHQcfHUh8AXv215/XUgpOrwsiXQS1ABT9czRDGnB6mP37Mma T9wmiQUgUtR1eH0uyYsv1rrUQJaWHs1dBahd87erbJ6RCeWl4qvspI4ibNXtjU11fjCi 7Atg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778245688; x=1778850488; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=p0mMKpb1i8B3EQhJz0nd+cLEbb2uFro+HYoXPQ9QXgw=; b=DJrx4jE6QPYo9+WTIjXYrSAyp0pgBJNAflIen72G/Ndf8ABFWL/hv13NouapwbvXYe 1zW+EpQt0Ysb81gM6Sd/D3nqcapfsrRCNnnlB51SBlCoJBB+iHy9OK5f9fGTRraJWykB ShtRJAdj1byARu2fQrwhOC7+GV8d2fmkp/4ITGTM/i9jyh4NIw7FOdGjFTCpmYNKVNJ+ GbKvqWTBI4tcjaDrqnrQT3i/kYyPfavxWs4xZNaCFCplkNjtm0tvKqF57mVlepbRAZKB LqGg8fic6JMt+rvT4/7n7m3d6j3o0Qs5pdO1id3/L1FRUIYYFDpMlXTexctw9wtSLklR 40Ig== X-Gm-Message-State: AOJu0YwP+6V2vy5Irp8StQfwabvTmnqB+L+WsBKlHqzm+dr6nyCQGLeO KqtmkrYEXxg3I2gjY1MCujL49AWY9vm07PIbO45dogUpkUQfqBgaX8B4zTwAF52DrPtcaU/bjQ2 vT8An X-Gm-Gg: AeBDieuXG9KtKxzYExQ4PWQlIp9kmVvlkU+S4Wc/oLkbxgNAkqu1sD8sVkO1x7Yi6vw LIMK9ncyYmjwnCqHnvYmyOLLt6jIi13291UeddFLxMskwdNYJA7z3kVFaEXrHy+dTCEfBM/4CAs ob5FIyB6lNhSiPoIRXDOrhPRnwP6JV6HTA7SR6+FlvMHCFPddIpEPz7RSCmMxdMHiEkYveopKkj a/KBNPKtsMWYeaKPNZt26O9Yl/ZnWoNEjme9UiICM9tertukG4/3IV7VdN0R86XIJJBM12PFTtQ ss8ATFbWsLFiyP8DGCvDVn+XXC60atQBQlozbZAzvqx6ivbTR3R4Xq/ZM3L/U09D80HHjJuAC8H JscViiRTRg0QzIsd2zB2wQdGZpBL78aySFPJECEk0zq7gDvtr7eRI3YV0OrXBsaZOuzVlaXEXav R5CY7aXYbgXQ8VU/fdUdSYF129JXamXEquYoQzza1SeXvs66KJ0GY++OwmfzBQ9Lrjb5yEDt4IM GI1vR6hqu1YxRkeN50i+C/1S5dzPf7aoszACQ== X-Received: by 2002:a05:600c:198b:b0:48a:592c:e63d with SMTP id 5b1f17b1804b1-48e676a5245mr41755705e9.14.1778245687886; Fri, 08 May 2026 06:08:07 -0700 (PDT) Received: from localhost.localdomain ([2a06:61c0:f337:0:9c1f:b517:931a:3b19]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-48e68f5d9b5sm34044905e9.15.2026.05.08.06.08.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 May 2026 06:08:07 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Cc: David Plowman , Naushir Patuck Subject: [PATCH v5 3/3] pipeline: rpi: Make control lists in requests properly atomic Date: Fri, 8 May 2026 13:57:43 +0100 Message-ID: <20260508130804.8238-4-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260508130804.8238-1-david.plowman@raspberrypi.com> References: <20260508130804.8238-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When a request is about to be processed, this commit separates "delayed controls" (camera-related ones that take "a few frames" to apply) from "immediate controls" (ISP-related controls) that will happen instantly. The immediate controls are held back until the delayed controls that they were submitted with, have happened. We merge back together the immediate controls that must happen now, along with the delayed controls that must be forwarded early, the result being that control lists submitted with a request happen atomically. We avoid overwriting the original control list submitted with the request. We therefore already have the sequence number of the request whose controls have just been applied, so we additionally attach this to the current request as "ControlId" metadata. Signed-off-by: David Plowman Reviewed-by: Naushir Patuck --- src/ipa/rpi/common/ipa_base.cpp | 24 ++++++--- .../pipeline/rpi/common/pipeline_base.cpp | 49 +++++++++++++++++++ .../pipeline/rpi/common/pipeline_base.h | 9 ++++ src/libcamera/pipeline/rpi/pisp/pisp.cpp | 11 +++-- src/libcamera/pipeline/rpi/vc4/vc4.cpp | 11 +++-- 5 files changed, 88 insertions(+), 16 deletions(-) diff --git a/src/ipa/rpi/common/ipa_base.cpp b/src/ipa/rpi/common/ipa_base.cpp index faa77719..dacafa57 100644 --- a/src/ipa/rpi/common/ipa_base.cpp +++ b/src/ipa/rpi/common/ipa_base.cpp @@ -431,6 +431,15 @@ void IpaBase::prepareIsp(const PrepareParams ¶ms) rpiMetadata.clear(); fillDeviceStatus(params.sensorControls, ipaContext); + /* + * When there are controls, it's important that we don't skip running the + * IPAs, as that can mess with synchronisation. Crucially though, we need + * to know whether there were controls when this comes back as the + * _delayed_ metadata, hence why we flag this in the metadata itself. + */ + if (!params.requestControls.empty()) + rpiMetadata.set("ipa.request_controls", true); + if (params.buffers.embedded) { /* * Pipeline handler has supplied us with an embedded data buffer, @@ -451,7 +460,7 @@ void IpaBase::prepareIsp(const PrepareParams ¶ms) */ AgcStatus agcStatus; bool hdrChange = false; - RPiController::Metadata &delayedMetadata = rpiMetadata_[params.delayContext]; + RPiController::Metadata &delayedMetadata = rpiMetadata_[params.delayContext % rpiMetadata_.size()]; if (!delayedMetadata.get("agc.status", agcStatus)) { rpiMetadata.set("agc.delayed_status", agcStatus); hdrChange = agcStatus.hdr.mode != hdrStatus_.mode; @@ -464,9 +473,13 @@ void IpaBase::prepareIsp(const PrepareParams ¶ms) */ helper_->prepare(embeddedBuffer, rpiMetadata); + bool delayedRequestControls = false; + delayedMetadata.get("ipa.request_controls", delayedRequestControls); + /* Allow a 10% margin on the comparison below. */ Duration delta = (frameTimestamp - lastRunTimestamp_) * 1.0ns; - if (lastRunTimestamp_ && frameCount_ > invalidCount_ && + if (!delayedRequestControls && params.requestControls.empty() && + lastRunTimestamp_ && frameCount_ > invalidCount_ && delta < controllerMinFrameDuration_ * 0.9 && !hdrChange) { /* * Ensure we merge the previous frame's metadata with the current @@ -535,7 +548,7 @@ void IpaBase::processStats(const ProcessParams ¶ms) ControlList ctrls(sensorCtrls_); applyAGC(&agcStatus, ctrls); rpiMetadata.set("agc.status", agcStatus); - setDelayedControls.emit(ctrls, ipaContext); + setDelayedControls.emit(ctrls, params.ipaContext); setCameraTimeoutValue(); } @@ -951,8 +964,6 @@ void IpaBase::applyControls(const ControlList &controls) /* The control provides units of microseconds. */ agc->setFixedExposureTime(0, ctrl.second.get() * 1.0us); - - libcameraMetadata_.set(controls::ExposureTime, ctrl.second.get()); break; } @@ -976,9 +987,6 @@ void IpaBase::applyControls(const ControlList &controls) break; agc->setFixedGain(0, ctrl.second.get()); - - libcameraMetadata_.set(controls::AnalogueGain, - ctrl.second.get()); break; } diff --git a/src/libcamera/pipeline/rpi/common/pipeline_base.cpp b/src/libcamera/pipeline/rpi/common/pipeline_base.cpp index f6ef8a3b..ace38997 100644 --- a/src/libcamera/pipeline/rpi/common/pipeline_base.cpp +++ b/src/libcamera/pipeline/rpi/common/pipeline_base.cpp @@ -1528,4 +1528,53 @@ void CameraData::fillRequestMetadata(const ControlList &bufferControls, Request } } +static bool isControlDelayed(unsigned int id) +{ + return id == controls::ExposureTime || + id == controls::AnalogueGain || + id == controls::FrameDurationLimits || + id == controls::AeEnable || + id == controls::ExposureTimeMode || + id == controls::AnalogueGainMode; +} + +void CameraData::handleControlLists(uint32_t delayContext, ControlList ¶mControls) +{ + /* + * The delayContext is the sequence number after it's gone through the various + * pipeline delays, so that's what gets reported as the "ControlListSequence" + * in the metadata, being the sequence number of the request whose ControlList + * has just been applied. + */ + Request *request = requestQueue_.front(); + request->_d()->metadata().set(controls::rpi::ControlListSequence, delayContext); + + /* + * Controls that take effect immediately (typically ISP controls) have to be + * delayed so as to synchronise with those controls that do get delayed. So we + * must remove them from the current request, and push them onto a queue so + * that they can be used later. + * + * Note that we are given a separate control list (paramControls) so that + * we can pass back the controls that really need to happen now, without + * disturbing the controls that were submitted with the request. + */ + ASSERT(paramControls.empty()); + immediateControls_.push({ request->sequence(), {} }); + for (const auto &ctrl : request->controls()) { + if (isControlDelayed(ctrl.first)) + paramControls.set(ctrl.first, ctrl.second); + else + immediateControls_.back().controls.set(ctrl.first, ctrl.second); + } + + /* "Immediate" controls that have become due are now merged back into this request. */ + while (!immediateControls_.empty() && + immediateControls_.front().controlListId <= delayContext) { + paramControls.merge(immediateControls_.front().controls, + ControlList::MergePolicy::OverwriteExisting); + immediateControls_.pop(); + } +} + } /* namespace libcamera */ diff --git a/src/libcamera/pipeline/rpi/common/pipeline_base.h b/src/libcamera/pipeline/rpi/common/pipeline_base.h index 7bfac33e..758155ee 100644 --- a/src/libcamera/pipeline/rpi/common/pipeline_base.h +++ b/src/libcamera/pipeline/rpi/common/pipeline_base.h @@ -180,10 +180,19 @@ public: ClockRecovery wallClockRecovery_; + struct ImmediateControlsEntry { + uint64_t controlListId; + ControlList controls; + }; + std::queue immediateControls_; + protected: void fillRequestMetadata(const ControlList &bufferControls, Request *request); + void handleControlLists(uint32_t delayContext, + ControlList ¶mControls); + virtual void tryRunPipeline() = 0; private: diff --git a/src/libcamera/pipeline/rpi/pisp/pisp.cpp b/src/libcamera/pipeline/rpi/pisp/pisp.cpp index cc7e32c3..b744c901 100644 --- a/src/libcamera/pipeline/rpi/pisp/pisp.cpp +++ b/src/libcamera/pipeline/rpi/pisp/pisp.cpp @@ -2322,9 +2322,6 @@ void PiSPCameraData::tryRunPipeline() fillRequestMetadata(job.sensorControls, request); - /* Set our state to say the pipeline is active. */ - state_ = State::Busy; - unsigned int bayerId = cfe_[Cfe::Output0].getBufferId(job.buffers[&cfe_[Cfe::Output0]]); unsigned int statsId = cfe_[Cfe::Stats].getBufferId(job.buffers[&cfe_[Cfe::Stats]]); ASSERT(bayerId && statsId); @@ -2341,7 +2338,13 @@ void PiSPCameraData::tryRunPipeline() params.ipaContext = requestQueue_.front()->sequence(); params.delayContext = job.delayContext; params.sensorControls = std::move(job.sensorControls); - params.requestControls = request->controls(); + /* params.requestControls is set by handleControlLists. */ + + /* This sorts out synchronisation with ControlLists in earlier requests. */ + handleControlLists(job.delayContext, params.requestControls); + + /* Set our state to say the pipeline is active. */ + state_ = State::Busy; if (sensorMetadata_) { unsigned int embeddedId = diff --git a/src/libcamera/pipeline/rpi/vc4/vc4.cpp b/src/libcamera/pipeline/rpi/vc4/vc4.cpp index f99cfdbc..3e9a4905 100644 --- a/src/libcamera/pipeline/rpi/vc4/vc4.cpp +++ b/src/libcamera/pipeline/rpi/vc4/vc4.cpp @@ -939,9 +939,6 @@ void Vc4CameraData::tryRunPipeline() fillRequestMetadata(bayerFrame.controls, request); - /* Set our state to say the pipeline is active. */ - state_ = State::Busy; - unsigned int bayer = unicam_[Unicam::Image].getBufferId(bayerFrame.buffer); LOG(RPI, Debug) << "Signalling prepareIsp:" @@ -950,10 +947,16 @@ void Vc4CameraData::tryRunPipeline() ipa::RPi::PrepareParams params; params.buffers.bayer = RPi::MaskBayerData | bayer; params.sensorControls = std::move(bayerFrame.controls); - params.requestControls = request->controls(); params.ipaContext = request->sequence(); params.delayContext = bayerFrame.delayContext; params.buffers.embedded = 0; + /* params.requestControls is set by handleControlLists. */ + + /* This sorts out synchronisation with ControlLists in earlier requests. */ + handleControlLists(bayerFrame.delayContext, params.requestControls); + + /* Set our state to say the pipeline is active. */ + state_ = State::Busy; if (embeddedBuffer) { unsigned int embeddedId = unicam_[Unicam::Embedded].getBufferId(embeddedBuffer);