From patchwork Mon Sep 14 14:02:14 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28245 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 336C3C3226 for ; Mon, 14 Sep 2026 14:03:33 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9C5B2686A0; Mon, 14 Sep 2026 16:03:32 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="pZD2K5nC"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C451B68698 for ; Mon, 14 Sep 2026 16:03:30 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 9162A929; Mon, 14 Sep 2026 16:01:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394510; bh=Z+uBhct/s0Qpbs84ms4PA5Bu69o5BzZM5fAj2JkXUoA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pZD2K5nC/Z0gSu7lleAV03qmw9/glgXdSbGHuvknmm2YCCEePcfFEpkrdTVGXB5KP avKCJ373VDfaCwoDZTKNp9sm+MagcQexpzBLTH8VhhPkwrhcl8g2RhQ5ph42b1yV9e Y/pES8z+RZ2gdkUw1czovzL3flQHp6bR96keWpug= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 01/41] libcamera: delayed_controls: Add push() function that accepts a sequence number Date: Mon, 14 Sep 2026 16:02:14 +0200 Message-ID: <20260914140309.3354666-2-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The push function is asymmetric to the get() and applyControls() function in that it doesn't allow to specify a frame number. This leads to the unfortunate situation that it is very difficult to detect if anything goes out of sync. Add a version of the push() function that takes a sequence parameter and warns when the sequence provided differs from the expected sequence. Don't take any further actions for now to see where issues pop up. Signed-off-by: Stefan Klug --- include/libcamera/internal/delayed_controls.h | 1 + src/libcamera/delayed_controls.cpp | 32 ++++++++++++++++++- 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/include/libcamera/internal/delayed_controls.h b/include/libcamera/internal/delayed_controls.h index b64d8bba7cf7..c650e672d964 100644 --- a/include/libcamera/internal/delayed_controls.h +++ b/include/libcamera/internal/delayed_controls.h @@ -32,6 +32,7 @@ public: void reset(); bool push(const ControlList &controls); + bool push(uint32_t sequence, const ControlList &controls); ControlList get(uint32_t sequence); void applyControls(uint32_t sequence); diff --git a/src/libcamera/delayed_controls.cpp b/src/libcamera/delayed_controls.cpp index 5b3ced4e711d..36d1f20a3680 100644 --- a/src/libcamera/delayed_controls.cpp +++ b/src/libcamera/delayed_controls.cpp @@ -144,10 +144,40 @@ void DelayedControls::reset() * Push a set of controls to the control queue. This increases the control queue * depth by one. * + * \note The usage of this function is discouraged as it does not provide a way + * to detect double pushes for the same sequence. Better use + * DelayedControls::push(uint32_t sequence, const ControlList &controls) + * instead. + * * \returns true if \a controls are accepted, or false otherwise */ bool DelayedControls::push(const ControlList &controls) { + LOG(DelayedControls, Debug) << "Deprecated: Push without sequence number"; + return push(queueCount_, controls); +} + +/** + * \brief Push a set of controls on the queue + * \param[in] sequence The sequence number to push for + * \param[in] controls List of controls to add to the device queue + * + * Push a set of controls to the control queue. This increases the control queue + * depth by one. + * + * The \a sequence number is used to do some sanity checks to detect double + * pushes for the same sequence (either due to a bug or a request underrun). + * + * \returns true if \a controls are accepted, or false otherwise + */ +bool DelayedControls::push(uint32_t sequence, const ControlList &controls) +{ + if (sequence < queueCount_) { + LOG(DelayedControls, Warning) + << "Double push for sequence " << sequence + << " current queue index: " << queueCount_; + } + /* Copy state from previous frame. */ for (auto &ctrl : values_) { Info &info = ctrl.second[queueCount_]; @@ -276,7 +306,7 @@ void DelayedControls::applyControls(uint32_t sequence) while (writeCount_ > queueCount_) { LOG(DelayedControls, Debug) << "Queue is empty, auto queue no-op."; - push({}); + push(queueCount_, {}); } device_->setControls(&out); From patchwork Mon Sep 14 14:02:15 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28246 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8A49FC3200 for ; Mon, 14 Sep 2026 14:03:36 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 31DA1686A5; Mon, 14 Sep 2026 16:03:36 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="RbtrQ/le"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 15A3368670 for ; Mon, 14 Sep 2026 16:03:33 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id F1CFC929; Mon, 14 Sep 2026 16:01:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394513; bh=Fa6CkRFaTm+VXywmGg52of+mJwGGUY47q88jS0QmocU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RbtrQ/le/KMYyplyTd2bA8QuprU3h3L8UAZuZA0Mqxq3YJcIatzuKywtpNqCRh1WK w3CdNnDZcmYDFaPlMjt307SX5ATq8V2iFfZopWTF6zlzqQsss2eq/pBmAT+Xm9VXBA tsu7xsnTs4KJUfiImwbQjs9WN8KQ3ZSLazG8ZIcM= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Paul Elder Subject: [PATCH v3 02/41] libcamera: delayed_controls: Handle missed pushes Date: Mon, 14 Sep 2026 16:02:15 +0200 Message-ID: <20260914140309.3354666-3-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In unstable systems it can happen, that some sequence numbers are missed in the IPA. Gracefully handle that situation by queuing no-ops. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder --- Changes in v3: - Dropped assert as it breaks the tests and will not be needed after "libcamera: delayed_controls: Ignore double pushes for the same frame number" is applied Changes in v2: - Collected tag --- src/libcamera/delayed_controls.cpp | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/libcamera/delayed_controls.cpp b/src/libcamera/delayed_controls.cpp index 36d1f20a3680..af0befa56f77 100644 --- a/src/libcamera/delayed_controls.cpp +++ b/src/libcamera/delayed_controls.cpp @@ -178,6 +178,13 @@ bool DelayedControls::push(uint32_t sequence, const ControlList &controls) << " current queue index: " << queueCount_; } + while (sequence > queueCount_) { + LOG(DelayedControls, Warning) + << "Missed push for sequence " << queueCount_ + << " Auto queue no-op."; + push(queueCount_, {}); + } + /* Copy state from previous frame. */ for (auto &ctrl : values_) { Info &info = ctrl.second[queueCount_]; From patchwork Mon Sep 14 14:02:16 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28247 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 49D8FC3226 for ; Mon, 14 Sep 2026 14:03:38 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D2384686B0; Mon, 14 Sep 2026 16:03:37 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ssXxO1bo"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A5F78686AF for ; Mon, 14 Sep 2026 16:03:35 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id A72E1512; Mon, 14 Sep 2026 16:01:55 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394515; bh=Yd+RyhUHIp7GkNoW+/BJpZ7CcF6gqziUBzT1iYLjaUc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ssXxO1bo/mKv/lucPaDTNKndNtOUkD9CSmnFYJEy/g/SrCBjjC8oKf/jRDpEdAfg/ 5ILovoJ2g61TtoB09lPdyUzOu+j2enHVMcq7mcHp1akWt7Hb1v+vbaaCbJWCuTrmJu 7VpIO0PG+Zhrbd/dezoVJo1Dh0sVi2x3MwjfTarw= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Paul Elder Subject: [PATCH v3 03/41] libcamera: delayed_controls: Increase log level for dummy pushes Date: Mon, 14 Sep 2026 16:02:16 +0200 Message-ID: <20260914140309.3354666-4-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When an automatic no-op is queued a debug message is printed. Change the debug level to warning, because that situation is actually an indication of something going seriously out of sync. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder --- Changes in v2: - Collected tag --- src/libcamera/delayed_controls.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libcamera/delayed_controls.cpp b/src/libcamera/delayed_controls.cpp index af0befa56f77..87d4c48425c6 100644 --- a/src/libcamera/delayed_controls.cpp +++ b/src/libcamera/delayed_controls.cpp @@ -311,7 +311,7 @@ void DelayedControls::applyControls(uint32_t sequence) writeCount_ = sequence + 1; while (writeCount_ > queueCount_) { - LOG(DelayedControls, Debug) + LOG(DelayedControls, Warning) << "Queue is empty, auto queue no-op."; push(queueCount_, {}); } From patchwork Mon Sep 14 14:02:17 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28248 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 62150C3272 for ; Mon, 14 Sep 2026 14:03:40 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E5B75686A0; Mon, 14 Sep 2026 16:03:39 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="C14LY+Lk"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B1234686A0 for ; Mon, 14 Sep 2026 16:03:37 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id B6EF09A4; Mon, 14 Sep 2026 16:01:57 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394517; bh=dP2/sEDvMmG2rcB4M3932K3yUA6MxTPHJZiiIAuQRCA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=C14LY+LkSpQxXKa0q/rtBT3iKKWaQ3x63Q5gn42tK2Dy4OZx/+ewZfKi9MEv/MhWh FWwEUtS9ltZTAEPVgLg9Q1DZ7OL/9tPNLy/ANwUwu2vTarNE+MfMBvw7yZt6pfae/E xwBBNODBUsFV4FaWbBSJrPPQPjq5TdFBxNTajGcs= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 04/41] libcamera: delayed_controls: Queue noop when needed, not before Date: Mon, 14 Sep 2026 16:02:17 +0200 Message-ID: <20260914140309.3354666-5-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" A no-op is queued when DelayedControls runs out of controls at the end of applyControls(). But these controls are only needed on the next call to applyControls(), so there is still time for a proper push. To fix that, move the no-op push to the beginning of applyContols(). Signed-off-by: Stefan Klug --- Changes in v3: - Refrained dropping the writeCount_ member. It will be reused later Changes in v2: - Improved a log message - Improved commit message - dropped writeCount_ member --- src/libcamera/delayed_controls.cpp | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/src/libcamera/delayed_controls.cpp b/src/libcamera/delayed_controls.cpp index 87d4c48425c6..7e3a72e2cb1d 100644 --- a/src/libcamera/delayed_controls.cpp +++ b/src/libcamera/delayed_controls.cpp @@ -270,6 +270,13 @@ void DelayedControls::applyControls(uint32_t sequence) { LOG(DelayedControls, Debug) << "frame " << sequence << " started"; + while (queueCount_ - 1 < sequence) { + LOG(DelayedControls, Warning) + << "Queue is empty, auto queue no-op for sequence " + << queueCount_; + push(queueCount_, {}); + } + /* * Create control list peeking ahead in the value queue to ensure * values are set in time to satisfy the sensor delay. @@ -278,7 +285,7 @@ void DelayedControls::applyControls(uint32_t sequence) for (auto &ctrl : values_) { const ControlId *id = ctrl.first; unsigned int delayDiff = maxDelay_ - controlParams_[id].delay; - unsigned int index = std::max(0, writeCount_ - delayDiff); + unsigned int index = std::max(0, sequence - delayDiff); Info &info = ctrl.second[index]; if (info.updated) { @@ -308,15 +315,8 @@ void DelayedControls::applyControls(uint32_t sequence) } } - writeCount_ = sequence + 1; - - while (writeCount_ > queueCount_) { - LOG(DelayedControls, Warning) - << "Queue is empty, auto queue no-op."; - push(queueCount_, {}); - } - device_->setControls(&out); + writeCount_ = sequence; } } /* namespace libcamera */ From patchwork Mon Sep 14 14:02:18 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28249 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4FE5AC327D for ; Mon, 14 Sep 2026 14:03:42 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9248F686B4; Mon, 14 Sep 2026 16:03:41 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="Z1Vr1TTF"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 59A5C686B3 for ; Mon, 14 Sep 2026 16:03:40 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 5A6C1929; Mon, 14 Sep 2026 16:02:00 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394520; bh=8sPGJnl0E/oMaJA/vOVQQnyJCC8XZsXHgv+oOgsXwpc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Z1Vr1TTF3lIuuJ0tBsCfUFM9sHvgK+btzOhknndIf6FvzaLZHmnyTuPBePQslhsyr 7juxZ5gdq0U7pP8yQpbJcx0QQOpBy8+lP0V/dBsNnbPWBusEmQQUGm9nX/vSspnrzv ZW6p2KXTbuPr+dTTavvwGxlTyOsxz/e5mXvUSZZI= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Paul Elder Subject: [PATCH v3 05/41] libcamera: delayed_controls: Add maxDelay() function Date: Mon, 14 Sep 2026 16:02:18 +0200 Message-ID: <20260914140309.3354666-6-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a maxDelay() function to be able to query the maximum delay of the sensor. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder --- Changes in v2: - Collected tag --- include/libcamera/internal/delayed_controls.h | 2 ++ src/libcamera/delayed_controls.cpp | 7 +++++++ 2 files changed, 9 insertions(+) diff --git a/include/libcamera/internal/delayed_controls.h b/include/libcamera/internal/delayed_controls.h index c650e672d964..3182ebddd1fa 100644 --- a/include/libcamera/internal/delayed_controls.h +++ b/include/libcamera/internal/delayed_controls.h @@ -35,6 +35,8 @@ public: bool push(uint32_t sequence, const ControlList &controls); ControlList get(uint32_t sequence); + uint32_t maxDelay() const { return maxDelay_; } + void applyControls(uint32_t sequence); private: diff --git a/src/libcamera/delayed_controls.cpp b/src/libcamera/delayed_controls.cpp index 7e3a72e2cb1d..1f4fa6974bde 100644 --- a/src/libcamera/delayed_controls.cpp +++ b/src/libcamera/delayed_controls.cpp @@ -257,6 +257,13 @@ ControlList DelayedControls::get(uint32_t sequence) return out; } +/** + * \fn DelayedControls::maxDelay() + * \brief Get the maximum delay of the sensor + * + * \return The maximum delay of the sensor + */ + /** * \brief Inform DelayedControls of the start of a new frame * \param[in] sequence Sequence number of the frame that started From patchwork Mon Sep 14 14:02:19 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28250 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C8F32C328D for ; Mon, 14 Sep 2026 14:03:45 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 56FFD686B3; Mon, 14 Sep 2026 16:03:45 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="u3PO0Dnl"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4B1FF686A0 for ; Mon, 14 Sep 2026 16:03:43 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 48E501B39; Mon, 14 Sep 2026 16:02:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394523; bh=7e+Pp3RO+yZZoWpy72EW4RwGdT5thwzl8Pe939YVFMw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=u3PO0DnlgRKvHBhTV1/I0Oi7kUiBVfMI3taz6lMVee79PecNuaN8+8mAco4ZpjPfZ bbnL+sjTZW/lIkjj3gzXNex5LSy2W2Wf+TktIGdR9e8WzCuo8XtAK2rk163Suda3lc Oejj5jdGXzGWzT2id6w8qfz5Y9OPtZk7nSXxuSPE= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 06/41] pipeline: rkisp1: Add a frameStart function to handle DelayedControls::applyControls Date: Mon, 14 Sep 2026 16:02:19 +0200 Message-ID: <20260914140309.3354666-7-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Move the call to applyControls into an intermediate function for upcoming modfications. As the frameStart handler checks for an activeCamera we can safely connect the signal in match() where all the other signals get connected. Signed-off-by: Stefan Klug --- Changes in v2: - Moved signal connection into match() function. --- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index 96382c93a427..74b2bb5c2659 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -1472,8 +1472,6 @@ int PipelineHandlerRkISP1::createCamera(MediaEntity *sensor) data->delayedCtrls_ = std::make_unique(data->sensor_->device(), params); - isp_->frameStart.connect(data->delayedCtrls_.get(), - &DelayedControls::applyControls); uint32_t supportedBlocks = kDefaultExtParamsBlocks; @@ -1508,6 +1506,15 @@ int PipelineHandlerRkISP1::createCamera(MediaEntity *sensor) return 0; } +void PipelineHandlerRkISP1::frameStart(uint32_t sequence) +{ + if (!activeCamera_) + return; + + RkISP1CameraData *data = cameraData(activeCamera_); + data->delayedCtrls_->applyControls(sequence); +} + bool PipelineHandlerRkISP1::match(DeviceEnumerator *enumerator) { DeviceMatch dm("rkisp1"); @@ -1550,6 +1557,7 @@ bool PipelineHandlerRkISP1::match(DeviceEnumerator *enumerator) if (hasSelfPath_ && !selfPath_.init(media_)) return false; + isp_->frameStart.connect(this, &PipelineHandlerRkISP1::frameStart); mainPath_.bufferReady().connect(this, &PipelineHandlerRkISP1::imageBufferReady); if (hasSelfPath_) selfPath_.bufferReady().connect(this, &PipelineHandlerRkISP1::imageBufferReady); From patchwork Mon Sep 14 14:02:20 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28251 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 639A8C3200 for ; Mon, 14 Sep 2026 14:03:48 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 02100686B3; Mon, 14 Sep 2026 16:03:48 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="JLoxayKu"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 075E0686B0 for ; Mon, 14 Sep 2026 16:03:46 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 09C13929; Mon, 14 Sep 2026 16:02:06 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394526; bh=nWZLKFrPgzz/XX1ekQon6uyXjRtnk3jOVF3ncr2ssqc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JLoxayKuDDGJj42JUdOl2i3d6zKywQ2+Du9RQiePQBx+I6bglV5pP5W7eecsRM8ep mAFg2v3fd22brYCzVwftEBMST0OENZu8lGqwiYGc8+M+MoJNWllcSWaT8hqf2PxBDr xT28lgQDR09yeDycqRXtDxQ8GYR5jvktqB3rDUBY= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Paul Elder Subject: [PATCH v3 07/41] pipeline: rkisp1: Include frame number when pushing to delayed controls Date: Mon, 14 Sep 2026 16:02:20 +0200 Message-ID: <20260914140309.3354666-8-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Pass the frame number when pushing to delayed controls, to ease further development. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder --- Changes in v2: - Collected tag --- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index 74b2bb5c2659..a21b921f228c 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -500,10 +500,10 @@ void RkISP1CameraData::paramsComputed(unsigned int frame, unsigned int bytesused selfPath_->queueBuffer(info->selfPathBuffer); } -void RkISP1CameraData::setSensorControls([[maybe_unused]] unsigned int frame, +void RkISP1CameraData::setSensorControls(unsigned int frame, const ControlList &sensorControls) { - delayedCtrls_->push(sensorControls); + delayedCtrls_->push(frame, sensorControls); } void RkISP1CameraData::metadataReady(unsigned int frame, const ControlList &metadata) From patchwork Mon Sep 14 14:02:21 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28252 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1D0C8C3352 for ; Mon, 14 Sep 2026 14:03:50 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A6E96686BF; Mon, 14 Sep 2026 16:03:49 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="kXTi5Fl7"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C1DB5686B9 for ; Mon, 14 Sep 2026 16:03:48 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C2A9EC3B; Mon, 14 Sep 2026 16:02:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394528; bh=NFBjekpEEWS3VC084F7dtUFcwKF0GpX7g0vXDqANL+w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kXTi5Fl7Rfp+jMYa1k2lVgDaa61iLzdtg7PaiO6HFAANmjq0yK/MRkvBmXFoBoiox XYbeWB7o09DylGsSW8gWuX5iaQ5j4sjFuzHOz7XpYx2M4PPm2FE96kL+LNBgbyKPxT PEwMn2RLHhK/V7phTdlFltkSu98idmYkB4QXRoqU= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 08/41] libcamera: delayed_controls: Change semantics of sequence numbers Date: Mon, 14 Sep 2026 16:02:21 +0200 Message-ID: <20260914140309.3354666-9-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In the context of per frame controls the semantics of DelayedControls::applyControls() are quite difficult to grasp. The function void frameStart(int s) { delayedCtrls->applyControls(s); } seems intuitively wrong as the frame has already started. I think it is easier to think about what needs to be done for a specific sequence number. So (assuming a max sensor delay of 2) the actions would be: - delayedControls.push(n) stores the controls that shall be active on frame n - delayedControls.get(n) returns these controls again - delayedControls.apply(n) applies the slowest control for frame n and does a look back on other controls. So when a frameStart for frame n occurs, it is time to call delayedControls.apply(n + maxDelay) Changing these semantics on delayed controls doesn't require much code change and has the added benefit that we don't run into clamping for get() on frames < maxDelay. Signed-off-by: Stefan Klug --- Changes in v3: - Fixed unit tests - Added error log when get() is called with an illegal sequence - Added fixup for SimplePipeline Changes in v2: - Applied the semantics change to other pipelines as well. --- src/libcamera/delayed_controls.cpp | 21 +++++--- src/libcamera/pipeline/ipu3/ipu3.cpp | 2 +- src/libcamera/pipeline/mali-c55/mali-c55.cpp | 6 ++- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 3 +- src/libcamera/pipeline/simple/simple.cpp | 15 ++++-- test/delayed_controls.cpp | 57 ++++++++++---------- 6 files changed, 60 insertions(+), 44 deletions(-) diff --git a/src/libcamera/delayed_controls.cpp b/src/libcamera/delayed_controls.cpp index 1f4fa6974bde..90c7046c66f7 100644 --- a/src/libcamera/delayed_controls.cpp +++ b/src/libcamera/delayed_controls.cpp @@ -239,7 +239,13 @@ bool DelayedControls::push(uint32_t sequence, const ControlList &controls) */ ControlList DelayedControls::get(uint32_t sequence) { - unsigned int index = std::max(0, sequence - maxDelay_); + unsigned int index = sequence; + + if (sequence > writeCount_) + LOG(DelayedControls, Error) + << "Get controls for frame " << sequence + << " but only up to frame " << writeCount_ + << " were sent to the device"; ControlList out(device_->controls()); for (const auto &ctrl : values_) { @@ -265,13 +271,14 @@ ControlList DelayedControls::get(uint32_t sequence) */ /** - * \brief Inform DelayedControls of the start of a new frame - * \param[in] sequence Sequence number of the frame that started + * \brief Apply controls for a frame + * \param[in] sequence Sequence number of the frame to apply * - * Inform the state machine that a new frame has started and of its sequence - * number. Any user of these helpers is responsible to inform the helper about - * the start of any frame. This can be connected with ease to the start of a - * exposure (SOE) V4L2 event. + * Apply controls for the frame \a sequence. This applies the controls with the + * largest delay. For controls with a smaller delay it does a look back and + * applies the controls for the previous sequence. So usually this function is + * called in a start of exposure event as applyControls(startedSequence + + * maxDelay) */ void DelayedControls::applyControls(uint32_t sequence) { diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index 14cab9e5559b..1341119f6f66 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -1384,7 +1384,7 @@ void IPU3CameraData::statBufferReady(FrameBuffer *buffer) */ void IPU3CameraData::frameStart(uint32_t sequence) { - delayedCtrls_->applyControls(sequence); + delayedCtrls_->applyControls(sequence + delayedCtrls_->maxDelay()); if (processingRequests_.empty()) return; diff --git a/src/libcamera/pipeline/mali-c55/mali-c55.cpp b/src/libcamera/pipeline/mali-c55/mali-c55.cpp index 73a03373c833..70c751fc7d90 100644 --- a/src/libcamera/pipeline/mali-c55/mali-c55.cpp +++ b/src/libcamera/pipeline/mali-c55/mali-c55.cpp @@ -1849,8 +1849,10 @@ bool PipelineHandlerMaliC55::registerSensorCamera(MediaLink *ispLink) V4L2Subdevice *sensorSubdev = in->sensor_->device(); data->delayedCtrls_ = std::make_unique(sensorSubdev, params); - isp_->frameStart.connect(data->delayedCtrls_.get(), - &DelayedControls::applyControls); + isp_->frameStart.connect(data->delayedCtrls_.get(), [&](uint32_t seq) { + uint32_t lookahead = data->delayedCtrls_->maxDelay(); + data->delayedCtrls_->applyControls(seq + lookahead); + }); /* \todo Init properties. */ diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index a21b921f228c..ffd49157ec67 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -1512,7 +1512,8 @@ void PipelineHandlerRkISP1::frameStart(uint32_t sequence) return; RkISP1CameraData *data = cameraData(activeCamera_); - data->delayedCtrls_->applyControls(sequence); + uint32_t sequenceToApply = sequence + data->delayedCtrls_->maxDelay(); + data->delayedCtrls_->applyControls(sequenceToApply); } bool PipelineHandlerRkISP1::match(DeviceEnumerator *enumerator) diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp index 35c29ceca1bb..572a73307fdb 100644 --- a/src/libcamera/pipeline/simple/simple.cpp +++ b/src/libcamera/pipeline/simple/simple.cpp @@ -361,6 +361,8 @@ public: std::unique_ptr swIsp_; SimpleFrames frameInfo_; + void frameStart(uint32_t sequence); + private: void tryPipeline(unsigned int code, const Size &size); static std::vector routedSourcePads(MediaPad *sink); @@ -1058,6 +1060,11 @@ void SimpleCameraData::setSensorControls(const ControlList &sensorControls) } } +void SimpleCameraData::frameStart(uint32_t sequence) +{ + delayedCtrls_->applyControls(sequence + delayedCtrls_->maxDelay()); +} + /* Retrieve all source pads connected to a sink pad through active routes. */ std::vector SimpleCameraData::routedSourcePads(MediaPad *sink) { @@ -1671,8 +1678,8 @@ int SimplePipelineHandler::start(Camera *camera, [[maybe_unused]] const ControlL stop(camera); return ret; } - frameStartEmitter->frameStart.connect(data->delayedCtrls_.get(), - &DelayedControls::applyControls); + frameStartEmitter->frameStart.connect(data, + &SimpleCameraData::frameStart); } ret = video->streamOn(); @@ -1711,8 +1718,8 @@ void SimplePipelineHandler::stopDevice(Camera *camera) if (frameStartEmitter) { frameStartEmitter->setFrameStartEnabled(false); - frameStartEmitter->frameStart.disconnect(data->delayedCtrls_.get(), - &DelayedControls::applyControls); + frameStartEmitter->frameStart.disconnect(data, + &SimpleCameraData::frameStart); } if (data->useConversion_) { diff --git a/test/delayed_controls.cpp b/test/delayed_controls.cpp index 7bd30e7aead8..5bfef285bf2b 100644 --- a/test/delayed_controls.cpp +++ b/test/delayed_controls.cpp @@ -84,23 +84,20 @@ protected: dev_->setControls(&ctrls); delayed->reset(); - /* Trigger the first frame start event */ - delayed->applyControls(0); - /* Test control without delay are set at once. */ for (unsigned int i = 1; i < 100; i++) { int32_t value = 100 + i; ctrls.set(V4L2_CID_BRIGHTNESS, value); - delayed->push(ctrls); + delayed->push(i, ctrls); delayed->applyControls(i); - ControlList result = delayed->get(i); + ControlList result = delayed->get(i - delayed->maxDelay()); int32_t brightness = result.get(V4L2_CID_BRIGHTNESS).get(); if (brightness != value) { cerr << "Failed single control without delay" - << " frame " << i + << " frame " << i - delayed->maxDelay() << " expected " << value << " got " << brightness << endl; @@ -126,23 +123,19 @@ protected: dev_->setControls(&ctrls); delayed->reset(); - /* Trigger the first frame start event */ - delayed->applyControls(0); - /* Test single control with delay. */ for (unsigned int i = 1; i < 100; i++) { int32_t value = 10 + i; ctrls.set(V4L2_CID_BRIGHTNESS, value); - delayed->push(ctrls); - + delayed->push(i, ctrls); delayed->applyControls(i); - ControlList result = delayed->get(i); + ControlList result = delayed->get(i - delayed->maxDelay()); int32_t brightness = result.get(V4L2_CID_BRIGHTNESS).get(); if (brightness != expected) { cerr << "Failed single control with delay" - << " frame " << i + << " frame " << i - delayed->maxDelay() << " expected " << expected << " got " << brightness << endl; @@ -167,32 +160,38 @@ protected: std::make_unique(dev_.get(), delays); ControlList ctrls; - /* Reset control to value that will be first two frames in test. */ + /* + * Reset control to value that will be first two frames in test. + * We expect the following values: + * Frame 0 1 2 3 4 5 ... + * Brightness 200 11 12 13 14 15 + * Contrast 201 12 13 14 15 16 + */ int32_t expected = 200; ctrls.set(V4L2_CID_BRIGHTNESS, expected); ctrls.set(V4L2_CID_CONTRAST, expected + 1); dev_->setControls(&ctrls); delayed->reset(); - /* Trigger the first frame start event */ - delayed->applyControls(0); - /* Test dual control with delay. */ for (unsigned int i = 1; i < 100; i++) { int32_t value = 10 + i; ctrls.set(V4L2_CID_BRIGHTNESS, value); ctrls.set(V4L2_CID_CONTRAST, value + 1); - delayed->push(ctrls); + delayed->push(i, ctrls); delayed->applyControls(i); - ControlList result = delayed->get(i); + if (i < maxDelay) + continue; + + ControlList result = delayed->get(i - delayed->maxDelay()); int32_t brightness = result.get(V4L2_CID_BRIGHTNESS).get(); int32_t contrast = result.get(V4L2_CID_CONTRAST).get(); if (brightness != expected || contrast != expected + 1) { cerr << "Failed dual controls" - << " frame " << i + << " frame " << i - delayed->maxDelay() << " brightness " << brightness << " contrast " << contrast << " expected " << expected @@ -225,35 +224,35 @@ protected: dev_->setControls(&ctrls); delayed->reset(); - /* Trigger the first frame start event */ - delayed->applyControls(0); - /* * Queue all controls before any fake frame start. Note we * can't queue up more then the delayed controls history size * which is 16. Where one spot is used by the reset control. */ - for (unsigned int i = 0; i < 15; i++) { + for (unsigned int i = 1; i < 15; i++) { int32_t value = 10 + i; ctrls.set(V4L2_CID_BRIGHTNESS, value); ctrls.set(V4L2_CID_CONTRAST, value); - delayed->push(ctrls); + delayed->push(i, ctrls); } /* Process all queued controls. */ - for (unsigned int i = 1; i < 16; i++) { - int32_t value = 10 + i - 1; + for (unsigned int i = 1; i < 15; i++) { + int32_t value = 10 + i; delayed->applyControls(i); - ControlList result = delayed->get(i); + if (i < maxDelay) + continue; + + ControlList result = delayed->get(i - maxDelay); int32_t brightness = result.get(V4L2_CID_BRIGHTNESS).get(); int32_t contrast = result.get(V4L2_CID_CONTRAST).get(); if (brightness != expected || contrast != expected) { cerr << "Failed multi queue" - << " frame " << i + << " frame " << i - maxDelay << " brightness " << brightness << " contrast " << contrast << " expected " << expected From patchwork Mon Sep 14 14:02:22 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28253 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id ECF6DC3226 for ; Mon, 14 Sep 2026 14:03:52 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8DFB9686BC; Mon, 14 Sep 2026 16:03:52 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="vSstf5ie"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 60076686B0 for ; Mon, 14 Sep 2026 16:03:51 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 5F49F9A4; Mon, 14 Sep 2026 16:02:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394531; bh=BEF9XjIpOJrqROwo03ZcUJTqHJg/IM34H063Q1tBce8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vSstf5ietrRSd7VNACizduuqEfT8UzMAhaC8JPj5Z1R3rjFPvlHqRwFVT3Uwg9WLz uPIpwXuHb6bh5eumU5dWzSMxFeU5JLMAvjTpN6Iygd7CTp8ZpOBsVo7oLj3koahBeF TlDLGRGZl0sHpbli0xeLAhjnoU7VkzK8eSX7binU= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Paul Elder Subject: [PATCH v3 09/41] libcamera: delayed_controls: Ignore double pushes for the same frame number Date: Mon, 14 Sep 2026 16:02:22 +0200 Message-ID: <20260914140309.3354666-10-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" For successful PFC a single sequence must only be pushed once to delayed controls. Such a situation can occur if no-ops were pushed in delayed controls due to a buffer underrun. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder --- Changes in v2: - Collated double log message - Collected tag --- src/libcamera/delayed_controls.cpp | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/src/libcamera/delayed_controls.cpp b/src/libcamera/delayed_controls.cpp index 90c7046c66f7..10246d31cbb1 100644 --- a/src/libcamera/delayed_controls.cpp +++ b/src/libcamera/delayed_controls.cpp @@ -174,8 +174,9 @@ bool DelayedControls::push(uint32_t sequence, const ControlList &controls) { if (sequence < queueCount_) { LOG(DelayedControls, Warning) - << "Double push for sequence " << sequence - << " current queue index: " << queueCount_; + << "Ignored double push for sequence " << sequence + << ". Current queue index: " << queueCount_; + return true; } while (sequence > queueCount_) { @@ -282,7 +283,10 @@ ControlList DelayedControls::get(uint32_t sequence) */ void DelayedControls::applyControls(uint32_t sequence) { - LOG(DelayedControls, Debug) << "frame " << sequence << " started"; + LOG(DelayedControls, Debug) + << "Apply controls for: " << sequence + << " (instant controls for frame " + << (sequence - maxDelay_) << ")"; while (queueCount_ - 1 < sequence) { LOG(DelayedControls, Warning) From patchwork Mon Sep 14 14:02:23 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28254 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 378CEC3272 for ; Mon, 14 Sep 2026 14:03:56 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 32CD2686B0; Mon, 14 Sep 2026 16:03:56 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="NZOusl8p"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 65DCB686B0 for ; Mon, 14 Sep 2026 16:03:54 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 6688A1B39; Mon, 14 Sep 2026 16:02:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394534; bh=EtkQjgTmtJNbfQKzGd9nS/igUdSJBUS5yVnj7iChuKA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NZOusl8pmIjLSLE1clB85/1th5ZKmsiSy3Tsj4mZ6TWL2CNAXnCvonuOL3ql5nIAF hH62ZoKa/iegkP4mrQNNwV4LmHZvLSvksMW5Qw1ofdl/vJwe+T26amYeKRaQzmY8ZC l2Yu0HVr+KEX6tMvOhUOA7i+2zDRNby7Z/uVRBRE= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Jacopo Mondi Subject: [PATCH v3 10/41] libcamera: v4l2_videodevice: Do not hide frame drops Date: Mon, 14 Sep 2026 16:02:23 +0200 Message-ID: <20260914140309.3354666-11-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Jacopo Mondi Autocorrecting the sequence number in case it doesn't start with 0 produces difficult to debug problems in cases where the first frame got lost. We need to handle these properly in the pipeline handler (and fix kernel drivers if needed). Signed-off-by: Jacopo Mondi Signed-off-by: Stefan Klug --- Changes in v2: - Rewrote commit message. --- include/libcamera/internal/v4l2_videodevice.h | 1 - src/libcamera/v4l2_videodevice.cpp | 15 --------------- 2 files changed, 16 deletions(-) diff --git a/include/libcamera/internal/v4l2_videodevice.h b/include/libcamera/internal/v4l2_videodevice.h index f63d74879ff2..cc962bf81187 100644 --- a/include/libcamera/internal/v4l2_videodevice.h +++ b/include/libcamera/internal/v4l2_videodevice.h @@ -286,7 +286,6 @@ private: std::unique_ptr fdBufferNotifier_; State state_; - std::optional firstFrame_; Timer watchdog_; utils::Duration watchdogDuration_; diff --git a/src/libcamera/v4l2_videodevice.cpp b/src/libcamera/v4l2_videodevice.cpp index 41dc5d0f65ed..807107f99a03 100644 --- a/src/libcamera/v4l2_videodevice.cpp +++ b/src/libcamera/v4l2_videodevice.cpp @@ -1902,19 +1902,6 @@ FrameBuffer *V4L2VideoDevice::dequeueBuffer() if (V4L2_TYPE_IS_OUTPUT(buf.type)) return buffer; - /* - * Detect kernel drivers which do not reset the sequence number to zero - * on stream start. - */ - if (!firstFrame_.has_value()) { - if (buf.sequence) - LOG(V4L2, Info) - << "Zero sequence expected for first frame (got " - << buf.sequence << ")"; - firstFrame_ = buf.sequence; - } - metadata.sequence -= firstFrame_.value(); - std::span framebufferPlanes = buffer->planes(); unsigned int numV4l2Planes = multiPlanar ? buf.length : 1; @@ -1991,8 +1978,6 @@ int V4L2VideoDevice::streamOn() { int ret; - firstFrame_.reset(); - ret = ioctl(VIDIOC_STREAMON, &bufferType_); if (ret < 0) { LOG(V4L2, Error) From patchwork Mon Sep 14 14:02:24 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28255 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 59466C3354 for ; Mon, 14 Sep 2026 14:03:59 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D0F34686C5; Mon, 14 Sep 2026 16:03:58 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="XV6A5yV0"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 48C72686BC for ; Mon, 14 Sep 2026 16:03:57 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 4B2D79A4; Mon, 14 Sep 2026 16:02:17 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394537; bh=Tv/J5lbXmd1UOS99u1m+cWapEO23gdgM2r36Q5E7xIQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XV6A5yV03x40FHkr715+Vvpzp0qW7z/97ZQTcLZUkV0HpXnOqZPYlZSDW/MmoTy27 a826+rNFluJVXtseaWkwIVpwjp5f8iacmTWITIOtTiwfnhYWQTyfJ82+Up1Wc7qcwG GyO2HSt3/IFG2Ruy6Yc7dZfdwaL66PYxvQ+jSGyg= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Paul Elder Subject: [PATCH v3 11/41] libipa: fc_queue: Rename template argument to FC Date: Mon, 14 Sep 2026 16:02:24 +0200 Message-ID: <20260914140309.3354666-12-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The FCQueue has a template argument called FrameContext which is easily confused with the class FrameContext defined in the same file. Reduce that confusion by renaming the template argument to FC. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder --- Changes in v2: - Collected tag --- src/ipa/libipa/fc_queue.h | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/ipa/libipa/fc_queue.h b/src/ipa/libipa/fc_queue.h index a1d136521107..1f4f84c27fbc 100644 --- a/src/ipa/libipa/fc_queue.h +++ b/src/ipa/libipa/fc_queue.h @@ -28,7 +28,7 @@ private: bool initialised = false; }; -template +template class FCQueue { public: @@ -39,15 +39,15 @@ public: void clear() { - for (FrameContext &ctx : contexts_) { + for (FC &ctx : contexts_) { ctx.initialised = false; ctx.frame = 0; } } - FrameContext &alloc(const uint32_t frame) + FC &alloc(const uint32_t frame) { - FrameContext &frameContext = contexts_[frame % contexts_.size()]; + FC &frameContext = contexts_[frame % contexts_.size()]; /* * Do not re-initialise if a get() call has already fetched this @@ -69,9 +69,9 @@ public: return frameContext; } - FrameContext &get(uint32_t frame) + FC &get(uint32_t frame) { - FrameContext &frameContext = contexts_[frame % contexts_.size()]; + FC &frameContext = contexts_[frame % contexts_.size()]; /* * If the IPA algorithms try to access a frame context slot which @@ -122,14 +122,14 @@ public: } private: - void init(FrameContext &frameContext, const uint32_t frame) + void init(FC &frameContext, const uint32_t frame) { frameContext = {}; frameContext.frame = frame; frameContext.initialised = true; } - std::vector contexts_; + std::vector contexts_; }; } /* namespace ipa */ From patchwork Mon Sep 14 14:02:25 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28256 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id DD13EC327D for ; Mon, 14 Sep 2026 14:04:01 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 91729686C2; Mon, 14 Sep 2026 16:04:01 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="GWG31NgV"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8E95C686BC for ; Mon, 14 Sep 2026 16:03:59 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 923DEC3B; Mon, 14 Sep 2026 16:02:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394539; bh=7KTYxEy2XudBigdIM/OQBv0LjUa9AuVd4TcydHhIsyY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GWG31NgVd+75za5T45/LOfvxPSr1UXOgEBRDwtOEwE+RNU2lNMymzqRZr0Nwv8SvU pKCU91avVr7cQNrxUyGlr2leNX7jixKvRdoouLtQWUUFjttn43fXbJkGtqhe0X7S/Q gPEoNpqzZN9wE3TNRT4vbv43MTzGTtxY63eRFxPU= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Paul Elder Subject: [PATCH v3 12/41] libipa: fc_queue: Add trailing underscore to private members of FrameContext Date: Mon, 14 Sep 2026 16:02:25 +0200 Message-ID: <20260914140309.3354666-13-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" It is not immediately obvious that FCQueue accesses private members of the FrameContext class with the help of the friend declaration. This gets even more confusing when such a member is shadowed by a declaration in the actual IPA FrameContext class. Improve that by accessing the FrameContext members via references to that type. Additionally add an underscore to the variable names like we do on other members and which allows us to create a frame() accessor without a name clash in an upcoming commit. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder --- Changes in v2: - Collected tag --- src/ipa/libipa/fc_queue.cpp | 2 +- src/ipa/libipa/fc_queue.h | 51 ++++++++++++++++++++----------------- 2 files changed, 28 insertions(+), 25 deletions(-) diff --git a/src/ipa/libipa/fc_queue.cpp b/src/ipa/libipa/fc_queue.cpp index 0365e9197748..39222c2ed204 100644 --- a/src/ipa/libipa/fc_queue.cpp +++ b/src/ipa/libipa/fc_queue.cpp @@ -34,7 +34,7 @@ namespace ipa { * update any specific action for this frame, and finally to update the metadata * control lists when the frame is fully completed. * - * \var FrameContext::frame + * \var FrameContext::frame_ * \brief The frame number */ diff --git a/src/ipa/libipa/fc_queue.h b/src/ipa/libipa/fc_queue.h index 1f4f84c27fbc..1128e42f8ca6 100644 --- a/src/ipa/libipa/fc_queue.h +++ b/src/ipa/libipa/fc_queue.h @@ -24,8 +24,8 @@ class FCQueue; struct FrameContext { private: template friend class FCQueue; - uint32_t frame; - bool initialised = false; + uint32_t frame_; + bool initialised_ = false; }; template @@ -40,14 +40,15 @@ public: void clear() { for (FC &ctx : contexts_) { - ctx.initialised = false; - ctx.frame = 0; + ctx.initialised_ = false; + ctx.frame_ = 0; } } FC &alloc(const uint32_t frame) { - FC &frameContext = contexts_[frame % contexts_.size()]; + FC &fc = contexts_[frame % contexts_.size()]; + FrameContext &frameContext = fc; /* * Do not re-initialise if a get() call has already fetched this @@ -60,18 +61,19 @@ public: * time the application has queued a request. Does this deserve * an error condition ? */ - if (frame != 0 && frame <= frameContext.frame) + if (frame != 0 && frame <= frameContext.frame_) LOG(FCQueue, Warning) << "Frame " << frame << " already initialised"; else - init(frameContext, frame); + init(fc, frame); - return frameContext; + return fc; } FC &get(uint32_t frame) { - FC &frameContext = contexts_[frame % contexts_.size()]; + FC &fc = contexts_[frame % contexts_.size()]; + FrameContext &frameContext = fc; /* * If the IPA algorithms try to access a frame context slot which @@ -81,28 +83,28 @@ public: * queueing more requests to the IPA than the frame context * queue size. */ - if (frame < frameContext.frame) + if (frame < frameContext.frame_) LOG(FCQueue, Fatal) << "Frame context for " << frame << " has been overwritten by " - << frameContext.frame; + << frameContext.frame_; - if (frame == 0 && !frameContext.initialised) { + if (frame == 0 && !frameContext.initialised_) { /* * If the IPA calls get() at start() time it will get an * un-intialized FrameContext as the below "frame == - * frameContext.frame" check will return success because - * FrameContexts are zeroed at creation time. + * frameContext.frame_" check will return success + * because FrameContexts are zeroed at creation time. * * Make sure the FrameContext gets initialised if get() * is called before alloc() by the IPA for frame#0. */ - init(frameContext, frame); + init(fc, frame); - return frameContext; + return fc; } - if (frame == frameContext.frame) - return frameContext; + if (frame == frameContext.frame_) + return fc; /* * The frame context has been retrieved before it was @@ -116,17 +118,18 @@ public: LOG(FCQueue, Warning) << "Obtained an uninitialised FrameContext for " << frame; - init(frameContext, frame); + init(fc, frame); - return frameContext; + return fc; } private: - void init(FC &frameContext, const uint32_t frame) + void init(FC &fc, const uint32_t frame) { - frameContext = {}; - frameContext.frame = frame; - frameContext.initialised = true; + fc = {}; + FrameContext &frameContext = fc; + frameContext.frame_ = frame; + frameContext.initialised_ = true; } std::vector contexts_; From patchwork Mon Sep 14 14:02:26 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28257 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9B5D0C3355 for ; Mon, 14 Sep 2026 14:04:04 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2490B686C2; Mon, 14 Sep 2026 16:04:04 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ANMyAzPV"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 55A84686C9 for ; Mon, 14 Sep 2026 16:04:02 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 5D51E512; Mon, 14 Sep 2026 16:02:22 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394542; bh=/GrBvfL83EEIs5aDPpyM8p3Re4Cf6bZwTSgtSEsGulU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ANMyAzPVpUcUReGkhmBNjkeeR+eQQxGHjLrxTUvkOQGzpnINUJ+1WFBwr3FPYRkb9 Xi5d4nmmeLdiqh+lVS8m6WHpBACce/cniMoUD5Au2Vs5IKwyMD7mcSe2lWuujCWIui yPEgVTIFP0OTD76mKodK9cSGNo7Zp0LizX2+E1Gc= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Paul Elder Subject: [PATCH v3 13/41] ipa: rkisp1: Refactor setControls() Date: Mon, 14 Sep 2026 16:02:26 +0200 Message-ID: <20260914140309.3354666-14-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" IPARkISP1::setControls() is called when new sensor controls shall be queued to the pipeline handler. It constructs a list of sensor controls and then emits the setSensorControls signal. To be able to return initial sensor controls from the IPARkISP1::start() function, similar functionality will be needed. Prepare for that by changing the setControls() function to a getSensorControls() that is passed a frame context and by moving the setSensorControls.emit() out of the function. Signed-off-by: Stefan Klug Reviewed-by: Paul Elder --- Changes in v3: - Mergeconflict automatically fixed by meld Changes in v2: - Collected tag --- src/ipa/libipa/fc_queue.cpp | 6 ++++++ src/ipa/libipa/fc_queue.h | 2 ++ src/ipa/rkisp1/rkisp1.cpp | 26 +++++++++++++------------- 3 files changed, 21 insertions(+), 13 deletions(-) diff --git a/src/ipa/libipa/fc_queue.cpp b/src/ipa/libipa/fc_queue.cpp index 39222c2ed204..7ba28ed21611 100644 --- a/src/ipa/libipa/fc_queue.cpp +++ b/src/ipa/libipa/fc_queue.cpp @@ -38,6 +38,12 @@ namespace ipa { * \brief The frame number */ +/** + * \fn FrameContext::frame() + * \brief Get the frame of that frame context + * \return THe frame number + */ + /** * \class FCQueue * \brief A support class for managing FrameContext instances in IPA modules diff --git a/src/ipa/libipa/fc_queue.h b/src/ipa/libipa/fc_queue.h index 1128e42f8ca6..812022c496ed 100644 --- a/src/ipa/libipa/fc_queue.h +++ b/src/ipa/libipa/fc_queue.h @@ -22,6 +22,8 @@ template class FCQueue; struct FrameContext { + uint32_t frame() const { return frame_; } + private: template friend class FCQueue; uint32_t frame_; diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 79ab7338c140..eff1f40d605b 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -72,7 +72,7 @@ protected: private: void updateControls(ControlInfoMap *ipaControls); - void setControls(unsigned int frame); + ControlList getSensorControls(const IPAFrameContext &context); std::map buffers_; std::map mappedBuffers_; @@ -337,7 +337,12 @@ void IPARkISP1::processStats(const uint32_t frame, const uint32_t bufferId, algo->process(context_, frame, frameContext, stats, metadata); } - setControls(frame); + /* + * \todo: Here we should do a lookahead that takes the sensor delays + * into account. + */ + ControlList ctrls = getSensorControls(frameContext); + setSensorControls.emit(frame, ctrls); context_.debugMetadata.moveEntries(metadata); metadataReady.emit(frame, metadata); @@ -351,27 +356,22 @@ void IPARkISP1::updateControls(ControlInfoMap *ipaControls) *ipaControls = ControlInfoMap(std::move(ctrlMap), controls::controls); } -void IPARkISP1::setControls(unsigned int frame) +ControlList IPARkISP1::getSensorControls(const IPAFrameContext &frameContext) { - /* - * \todo The frame number is most likely wrong here, we need to take - * internal sensor delays and other timing parameters into account. - */ - - IPAFrameContext &frameContext = context_.frameContexts.get(frame); uint32_t exposure = frameContext.agc.exposure; uint32_t vblank = frameContext.agc.vblank; LOG(IPARkISP1, Debug) - << "Set controls for frame " << frame << ": exposure " << exposure - << ", gain " << frameContext.agc.gain << ", vblank " << vblank; + << "Set controls for frame " << frameContext.frame() + << ": exposure " << exposure + << ", gain " << frameContext.agc.gain + << ", vblank " << vblank; ControlList ctrls(context_.sensorControls); agc::prepareControls(ctrls, context_.camHelper.get(), exposure, frameContext.agc.gain); ctrls.set(V4L2_CID_VBLANK, static_cast(vblank)); - - setSensorControls.emit(frame, ctrls); + return ctrls; } } /* namespace ipa::rkisp1 */ From patchwork Mon Sep 14 14:02:27 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28258 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 3A70EC3356 for ; Mon, 14 Sep 2026 14:04:07 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id ABC7D686CF; Mon, 14 Sep 2026 16:04:06 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="J3UqkKfD"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4174C686C9 for ; Mon, 14 Sep 2026 16:04:05 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 4C055929; Mon, 14 Sep 2026 16:02:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394545; bh=dn4PZNmJZpW7ZylyG9DrVpbfcC/F4Y2fSphY/XstT7E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=J3UqkKfDMO1QIwj3xSjq8S604KZlEJrVZprPyLthx6uZ36Olp4jHpdNxdKZ612YP1 rUA41KSTOcOGbKBNPe6mSntRvRxICNlu9Hy5IthGzL3CFEjPZaHZCxM0ofaoY0ywK+ 8p17qVNYGT/hgQ7fbEHsw+lA5LnD93tmcMkhuecE= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 14/41] ipa: rkisp1: Move setSensorControls signal to computeParams Date: Mon, 14 Sep 2026 16:02:27 +0200 Message-ID: <20260914140309.3354666-15-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The setSensorControls event is emitted in the processStats() function. On first sight this looks reasonable as in processStats() we got the latest statistics and can therefore calculate the most up to date sensor controls. In the light of per-frame-controls however it produces difficult to solve timing issues: - The frame context in processStats() is the frame context of the frame that produced the stats, not for the frame that should be prepared and sent to the sensor. - To synchronize digital gain applied in the ISP with the analog gain applied in the sensor the set of parameters prepared for sensor and ISP must also be synchronized, which is currently not the case. To fix that, move the calculation and setting of sensor controls into the computeParams(). This way the model is far more easy to understand. We lose a tiny option for optimizations in that (in theory) we could delay the calculation of ISP parameters by another frame (assuming the sensor has a typical 2-frame delay). But all discussions and tests showed that keeping all parameters in sync is more important than that possible optimization for one frame. To ensure setSensorControls() still gets emitted in raw mode, allow computeParams() to be called with a zero bufferId. This strategy is also used for processStats() to ensure that metadata gets filled in raw mode. Then call computeParams() for raw mode also. Signed-off-by: Stefan Klug --- Changes in v3: - Squashed the fix for raw mode into this patch as it is a logical unit Changes in v2: - Collected tag --- src/ipa/rkisp1/rkisp1.cpp | 22 ++++++++++------------ src/libcamera/pipeline/rkisp1/rkisp1.cpp | 6 ++++++ 2 files changed, 16 insertions(+), 12 deletions(-) diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index eff1f40d605b..98fc3244d9a6 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -302,13 +302,18 @@ void IPARkISP1::computeParams(const uint32_t frame, const uint32_t bufferId) { IPAFrameContext &frameContext = context_.frameContexts.get(frame); - RkISP1Params params(context_.configuration.paramFormat, - mappedBuffers_.at(bufferId).planes()[0]); + if (bufferId != 0) { + RkISP1Params params(context_.configuration.paramFormat, + mappedBuffers_.at(bufferId).planes()[0]); - for (const auto &algo : algorithms()) - algo->prepare(context_, frame, frameContext, ¶ms); + for (const auto &algo : algorithms()) + algo->prepare(context_, frame, frameContext, ¶ms); - paramsComputed.emit(frame, params.bytesused()); + paramsComputed.emit(frame, params.bytesused()); + } + + ControlList ctrls = getSensorControls(frameContext); + setSensorControls.emit(frame, ctrls); } void IPARkISP1::processStats(const uint32_t frame, const uint32_t bufferId, @@ -337,13 +342,6 @@ void IPARkISP1::processStats(const uint32_t frame, const uint32_t bufferId, algo->process(context_, frame, frameContext, stats, metadata); } - /* - * \todo: Here we should do a lookahead that takes the sensor delays - * into account. - */ - ControlList ctrls = getSensorControls(frameContext); - setSensorControls.emit(frame, ctrls); - context_.debugMetadata.moveEntries(metadata); metadataReady.emit(frame, metadata); } diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index ffd49157ec67..d3940692c11b 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -1352,6 +1352,12 @@ int PipelineHandlerRkISP1::queueRequestDevice(Camera *camera, Request *request) if (data->selfPath_ && info->selfPathBuffer) data->selfPath_->queueBuffer(info->selfPathBuffer); + + /* + * Call computeParams with an empty param buffer to trigger the + * setSensorControls signal. + */ + data->ipa_->computeParams(data->frame_, 0); } else { data->ipa_->computeParams(data->frame_, info->paramBuffer->cookie()); From patchwork Mon Sep 14 14:02:28 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28259 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7E4C4C328D for ; Mon, 14 Sep 2026 14:04:10 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2F21D686C2; Mon, 14 Sep 2026 16:04:10 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="OGLgigAV"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A7E7C686BC for ; Mon, 14 Sep 2026 16:04:08 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id B0BF6929; Mon, 14 Sep 2026 16:02:28 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394548; bh=pSi4t6jcjWJyp6ik+irfkfIlbFLgOMOQKjO0poXzd5w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OGLgigAVODpIa4PUdBSAOV5YbNkrLR+8Izk2lQA1MBvzPjO4ESKwkbJRQXc8y+IEl GJ4PMrsH/2m5NxNpZytNUNJHr8AkdEnkweTNY5zukpBzZ8G9HbNH3Pwt46ikkVGEWq M+ZggEYLUuiTHVpbWfsQPndZn0GKWRDRBwka4l0Q= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 15/41] ipa: rkisp1: Add initializeFrameContext() function Date: Mon, 14 Sep 2026 16:02:28 +0200 Message-ID: <20260914140309.3354666-16-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In preparation to handling startup controls, split the frame context initialization from queueRequest into a separate function. This patch contains no functional changes. Signed-off-by: Stefan Klug --- src/ipa/rkisp1/rkisp1.cpp | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 98fc3244d9a6..3ab3645800cf 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -64,6 +64,9 @@ public: void queueRequest(const uint32_t frame, const ControlList &controls) override; void computeParams(const uint32_t frame, const uint32_t bufferId) override; + void initializeFrameContext(const uint32_t frame, + IPAFrameContext &frameContext, + const ControlList &controls); void processStats(const uint32_t frame, const uint32_t bufferId, const ControlList &sensorControls) override; @@ -290,6 +293,13 @@ void IPARkISP1::queueRequest(const uint32_t frame, const ControlList &controls) IPAFrameContext &frameContext = context_.frameContexts.alloc(frame); context_.debugMetadata.enableByControl(controls); + initializeFrameContext(frame, frameContext, controls); +} + +void IPARkISP1::initializeFrameContext(const uint32_t frame, + IPAFrameContext &frameContext, + const ControlList &controls) +{ for (const auto &a : algorithms()) { Algorithm *algo = static_cast(a.get()); if (algo->disabled_) From patchwork Mon Sep 14 14:02:29 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28260 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B7E79C3357 for ; Mon, 14 Sep 2026 14:04:12 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0134B686DD; Mon, 14 Sep 2026 16:04:12 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="kiV6xYVA"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id EF96D686BC for ; Mon, 14 Sep 2026 16:04:10 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id F0015512; Mon, 14 Sep 2026 16:02:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394551; bh=C2Of6ZYPO93+YNbpgDlaW+BRualeM+mOgH94CrUAvxE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kiV6xYVAjkzsZZQmZlWu0Cdpdh1Wsgjhr6K/NQ60l8kjyRHUAiL8mhE5hGQ617k3I LvMgpJVoLY8wm0eM2NY7X8TMMflKkOYpOUhb40Nb4MWLlCK/7YpMcLHpgoSl/9Z/g0 clPoSqYxp9lKOuF79/vwtbCdYyKmVaf8rFmjHuA0= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 16/41] pipeline: rkisp1: Apply initial controls Date: Mon, 14 Sep 2026 16:02:29 +0200 Message-ID: <20260914140309.3354666-17-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Controls passed to Camera::start() are not handled at the moment. Implement that by initializing a separate frame context and then returning the controls synchronously to the caller. In the pipeline the controls get passed to the sensor before the call to streamOn() to ensure the controls are applied right away. Passing the controls to the pipeline using the setSensorControls signal is not appropriate because it is asynchronous and would reach the sensor too late (initial controls need to be applied before the sensor starts and before delayed controls initializes its start condition). Signed-off-by: Stefan Klug --- Changes in v3: - reflowed commit message - Dropped control list copy and unnecesary temporary --- include/libcamera/ipa/rkisp1.mojom | 7 ++++++- src/ipa/rkisp1/rkisp1.cpp | 10 ++++++---- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 8 ++++++-- 3 files changed, 18 insertions(+), 7 deletions(-) diff --git a/include/libcamera/ipa/rkisp1.mojom b/include/libcamera/ipa/rkisp1.mojom index 068e898848c4..4c29b53cd7f9 100644 --- a/include/libcamera/ipa/rkisp1.mojom +++ b/include/libcamera/ipa/rkisp1.mojom @@ -14,13 +14,18 @@ struct IPAConfigInfo { uint32 paramFormat; }; +struct StartResult { + libcamera.ControlList controls; + int32 code; +}; + interface IPARkISP1Interface { init(libcamera.IPASettings settings, uint32 hwRevision, uint32 supportedBlocks, libcamera.IPACameraSensorInfo sensorInfo, libcamera.ControlInfoMap sensorControls) => (int32 ret, libcamera.ControlInfoMap ipaControls); - start() => (int32 ret); + start(libcamera.ControlList controls) => (StartResult result); stop(); configure(IPAConfigInfo configInfo, diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 3ab3645800cf..fe1a42af36a3 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -53,7 +53,7 @@ public: const IPACameraSensorInfo &sensorInfo, const ControlInfoMap &sensorControls, ControlInfoMap *ipaControls) override; - int start() override; + void start(const ControlList &controls, StartResult *result) override; void stop() override; int configure(const IPAConfigInfo &ipaConfig, @@ -206,10 +206,12 @@ int IPARkISP1::init(const IPASettings &settings, unsigned int hwRevision, return 0; } -int IPARkISP1::start() +void IPARkISP1::start(const ControlList &controls, StartResult *result) { - /* \todo Properly handle startup controls. */ - return 0; + IPAFrameContext frameContext = {}; + initializeFrameContext(0, frameContext, controls); + result->controls = getSensorControls(frameContext); + result->code = 0; } void IPARkISP1::stop() diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index d3940692c11b..edd9f352524c 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -1240,13 +1240,17 @@ int PipelineHandlerRkISP1::start(Camera *camera, [[maybe_unused]] const ControlL return ret; actions += [&]() { freeBuffers(camera); }; - ret = data->ipa_->start(); - if (ret) { + ipa::rkisp1::StartResult res; + data->ipa_->start(controls ? *controls : ControlList{ controls::controls }, + &res); + if (res.code) { LOG(RkISP1, Error) << "Failed to start IPA " << camera->id(); return ret; } actions += [&]() { data->ipa_->stop(); }; + data->sensor_->setControls(&res.controls); + data->delayedCtrls_->reset(); data->frame_ = 0; From patchwork Mon Sep 14 14:02:30 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28261 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2AA5FC3358 for ; Mon, 14 Sep 2026 14:04:16 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A6258686D4; Mon, 14 Sep 2026 16:04:15 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="cyh9u1/m"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 73BFB686BC for ; Mon, 14 Sep 2026 16:04:13 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 801FE929; Mon, 14 Sep 2026 16:02:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394553; bh=+5KBRUUB7Qee0qL72eq+kOCXGxnCd3hacTvWsTqO5rw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cyh9u1/m4YIv7hUbU5IsSbxlZ2eQfOvqLztYe+oRwdqAd6KK4Dtidr5AzYmNZLfaG 3cvWuxWAcY6+J/q5bLHEuFvP6OXpuLTYAC7cYchrZ25GWBvYl3KMGq0WMj1LDROoq2 E1fOevQ2OGfXH4pMBFeM+neffQvcobUuv5fs7RB0= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 17/41] libipa: agc: Pass agc::Session to prepare() Date: Mon, 14 Sep 2026 16:02:30 +0200 Message-ID: <20260914140309.3354666-18-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In an upcoming change the prepare() function needs to access the session. Prepare for that by passing the session into AgcAlgorithm::prepare() without using it. Update the IPA implementations accordingly. Signed-off-by: Stefan Klug --- src/ipa/ipu3/algorithms/agc.cpp | 2 +- src/ipa/libipa/agc.cpp | 4 +++- src/ipa/libipa/agc.h | 3 ++- src/ipa/mali-c55/algorithms/agc.cpp | 2 +- src/ipa/rkisp1/algorithms/agc.cpp | 2 +- src/ipa/softisp/algorithms/agc.cpp | 2 +- 6 files changed, 9 insertions(+), 6 deletions(-) diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 57cfb325f1fc..67c910049a6d 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -111,7 +111,7 @@ void Agc::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, [[maybe_unused]] ipu3_uapi_params *params) { - agc_.prepare(context.activeState.agc, frameContext.agc); + agc_.prepare(context.configuration.agc, context.activeState.agc, frameContext.agc); } Histogram Agc::parseStatistics(const ipu3_uapi_stats_3a *stats, diff --git a/src/ipa/libipa/agc.cpp b/src/ipa/libipa/agc.cpp index 51b05e3c2acd..77e5f1370e9c 100644 --- a/src/ipa/libipa/agc.cpp +++ b/src/ipa/libipa/agc.cpp @@ -650,6 +650,7 @@ void AgcAlgorithm::queueRequest(const agc::Session &session, agc::ActiveState &s /** * \brief Prepare a frame + * \param[in] session The agc session configuration * \param[in] state The agc active state * \param[in] frameContext The agc frame context * @@ -663,7 +664,8 @@ void AgcAlgorithm::queueRequest(const agc::Session &session, agc::ActiveState &s * * \sa Algorithm::prepare() */ -void AgcAlgorithm::prepare(agc::ActiveState &state, agc::FrameContext &frameContext) +void AgcAlgorithm::prepare([[maybe_unused]] const agc::Session &session, agc::ActiveState &state, + agc::FrameContext &frameContext) { uint32_t activeAutoExposure = state.automatic.exposure; double activeAutoGain = state.automatic.gain; diff --git a/src/ipa/libipa/agc.h b/src/ipa/libipa/agc.h index a5700d26254e..9bdda8c72213 100644 --- a/src/ipa/libipa/agc.h +++ b/src/ipa/libipa/agc.h @@ -146,7 +146,8 @@ public: void queueRequest(const agc::Session &session, agc::ActiveState &state, agc::FrameContext &frameContext, const ControlList &controls); - void prepare(agc::ActiveState &state, agc::FrameContext &frameContext); + void prepare(const agc::Session &session, agc::ActiveState &state, + agc::FrameContext &frameContext); void process(const agc::Session &session, agc::ActiveState &state, agc::FrameContext &frameContext, std::optional &¶ms, diff --git a/src/ipa/mali-c55/algorithms/agc.cpp b/src/ipa/mali-c55/algorithms/agc.cpp index 2c747da89170..5684b4df954c 100644 --- a/src/ipa/mali-c55/algorithms/agc.cpp +++ b/src/ipa/mali-c55/algorithms/agc.cpp @@ -207,7 +207,7 @@ void Agc::fillWeightsArrayBuffer(MaliC55Params *params, const enum MaliC55Blocks void Agc::prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, MaliC55Params *params) { - agc_.prepare(context.activeState.agc, frameContext.agc); + agc_.prepare(context.configuration.agc, context.activeState.agc, frameContext.agc); if (frame > 0) return; diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 1f1d6c96a3c6..34e186d8b68f 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -209,7 +209,7 @@ void Agc::queueRequest(IPAContext &context, void Agc::prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, RkISP1Params *params) { - agc_.prepare(context.activeState.agc, frameContext.agc); + agc_.prepare(context.configuration.agc, context.activeState.agc, frameContext.agc); if (context.configuration.compress.supported) { frameContext.compress.enable = true; diff --git a/src/ipa/softisp/algorithms/agc.cpp b/src/ipa/softisp/algorithms/agc.cpp index ea234bc2ae4b..7d2a53b38576 100644 --- a/src/ipa/softisp/algorithms/agc.cpp +++ b/src/ipa/softisp/algorithms/agc.cpp @@ -74,7 +74,7 @@ void Agc::queueRequest(IPAContext &context, [[maybe_unused]] const uint32_t fram void Agc::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, [[maybe_unused]] DebayerParams *params) { - agc_.prepare(context.activeState.agc, frameContext.agc); + agc_.prepare(context.configuration.agc, context.activeState.agc, frameContext.agc); } void Agc::process(IPAContext &context, From patchwork Mon Sep 14 14:02:31 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28262 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8BFDCC3200 for ; Mon, 14 Sep 2026 14:04:18 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 358EC686D4; Mon, 14 Sep 2026 16:04:18 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="AkJscySh"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6960D686A1 for ; Mon, 14 Sep 2026 16:04:16 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 6DFEA512; Mon, 14 Sep 2026 16:02:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394556; bh=lP52oR/pts54AJEJyGnL8kjz8nhrdRXCt4NL3sfE++k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AkJscyShW15lseSw/yFIcdUXI64i3nddTzjO2VJ/s01KJkdv5HBaijvZLSG/OAxcV emYizOiiWM/pwwEYcSIW/He6mJGacCxwAl5YUiuM4wJZgzEXXxdjzQB9rffzhcJgmv V2xvexcooieEF/4/+n/uqTEi/t3s0+azHRyU3Ync= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Paul Elder Subject: [PATCH v3 18/41] libipa: agc: Process frame duration at the right time Date: Mon, 14 Sep 2026 16:02:31 +0200 Message-ID: <20260914140309.3354666-19-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The frame duration and vblank should not be calculated during process() but within prepare(), where the data for that frame get's computed. In raw mode, process is not called, so also update it in queueRequest(). Signed-off-by: Stefan Klug Reviewed-by: Paul Elder --- Changes in v3: - Rebased on top of libipa agc rework Changes in v2: - Squashed with next patch - Collected tag --- src/ipa/libipa/agc.cpp | 25 ++++++++++++++----------- src/ipa/libipa/agc.h | 3 +-- 2 files changed, 15 insertions(+), 13 deletions(-) diff --git a/src/ipa/libipa/agc.cpp b/src/ipa/libipa/agc.cpp index 77e5f1370e9c..9016e5f68ab7 100644 --- a/src/ipa/libipa/agc.cpp +++ b/src/ipa/libipa/agc.cpp @@ -646,6 +646,9 @@ void AgcAlgorithm::queueRequest(const agc::Session &session, agc::ActiveState &s } frameContext.minFrameDuration = state.minFrameDuration; frameContext.maxFrameDuration = state.maxFrameDuration; + + /* V-blank needs to be valid for the start controls handling. Update it. */ + processFrameDuration(session, frameContext); } /** @@ -664,7 +667,7 @@ void AgcAlgorithm::queueRequest(const agc::Session &session, agc::ActiveState &s * * \sa Algorithm::prepare() */ -void AgcAlgorithm::prepare([[maybe_unused]] const agc::Session &session, agc::ActiveState &state, +void AgcAlgorithm::prepare(const agc::Session &session, agc::ActiveState &state, agc::FrameContext &frameContext) { uint32_t activeAutoExposure = state.automatic.exposure; @@ -696,6 +699,12 @@ void AgcAlgorithm::prepare([[maybe_unused]] const agc::Session &session, agc::Ac } frameContext.yTarget = state.automatic.yTarget; + + /* + * Expand the target frame duration so that we do not run faster than + * the minimum frame duration when we have short exposures. + */ + processFrameDuration(session, frameContext); } /** @@ -726,7 +735,6 @@ void AgcAlgorithm::process(const agc::Session &session, agc::ActiveState &state, ControlList &metadata) { if (!params) { - processFrameDuration(session, frameContext, frameContext.minFrameDuration); fillMetadata(session, frameContext, metadata); return; } @@ -830,13 +838,6 @@ void AgcAlgorithm::process(const agc::Session &session, agc::ActiveState &state, << "quantization-gain: " << state.automatic.quantizationGain << ", " << "digital-gain: " << state.automatic.digitalGain; - /* - * Expand the target frame duration so that we do not run faster than - * the minimum frame duration when we have short exposures. - */ - processFrameDuration(session, frameContext, - std::max(frameContext.minFrameDuration, newExposureTime)); - fillMetadata(session, frameContext, metadata); } @@ -849,10 +850,12 @@ void AgcAlgorithm::process(const agc::Session &session, agc::ActiveState &state, * Compute and populate vblank from the target frame duration. */ void AgcAlgorithm::processFrameDuration(const agc::Session &session, - agc::FrameContext &frameContext, - utils::Duration frameDuration) + agc::FrameContext &frameContext) { const utils::Duration &lineDuration = session.lineDuration; + utils::Duration frameDuration = frameContext.exposure * lineDuration; + + frameDuration = std::max(frameDuration, frameContext.minFrameDuration); frameContext.vblank = (frameDuration / lineDuration) - session.sensor.outputSize.height; diff --git a/src/ipa/libipa/agc.h b/src/ipa/libipa/agc.h index 9bdda8c72213..4914c9cee24b 100644 --- a/src/ipa/libipa/agc.h +++ b/src/ipa/libipa/agc.h @@ -155,8 +155,7 @@ public: private: void processFrameDuration(const agc::Session &session, - agc::FrameContext &frameContext, - utils::Duration frameDuration); + agc::FrameContext &frameContext); void fillMetadata(const agc::Session &session, const agc::FrameContext &frameContext, ControlList &metadata); From patchwork Mon Sep 14 14:02:32 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28263 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B515FC3359 for ; Mon, 14 Sep 2026 14:04:21 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 162EC686D4; Mon, 14 Sep 2026 16:04:21 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="nwbcAESR"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C450D6869A for ; Mon, 14 Sep 2026 16:04:19 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C64F4929; Mon, 14 Sep 2026 16:02:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394559; bh=yQyRn0mTAx67FWBQZf4EQMEDnwk3h9NCS12HIBD+l5k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nwbcAESRIe2o9da3Xk5ccJWDnX2uGMSQQfSwoqrrgcfucsM7R4wYei8/BcMlS33WS cH+wUqSSwESERGVNrCtx4aWFLvTRAQg5Ji94rQnhsvPWdRvUEdpRwkEGqBtBJxCRz7 O9uNDUdbAxwJvqou2eQhTmkyOFB00a+kCR0U3amg= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 19/41] ipa: rkisp1: Allow processStats() to be called without stats buffer Date: Mon, 14 Sep 2026 16:02:32 +0200 Message-ID: <20260914140309.3354666-20-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When there are no stats available for a frame, it still makes sense to call processStats() to fill in the metadata of that frame. This mechanism is already used for the raw path. Allow it's use for non-raw also. The current code never produces buffers with id 0, but it is not enforced. Add a assert to enforce that. Signed-off-by: Stefan Klug --- Changes in v2: - Added an assert to ensure there is no buffer with id 0 --- src/ipa/rkisp1/rkisp1.cpp | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index fe1a42af36a3..08b7dde6b2de 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -263,6 +263,9 @@ int IPARkISP1::configure(const IPAConfigInfo &ipaConfig, void IPARkISP1::mapBuffers(const std::vector &buffers) { for (const IPABuffer &buffer : buffers) { + /* A buffer id of 0 is considered invalid */ + ASSERT(buffer.id != 0); + auto elem = buffers_.emplace(std::piecewise_construct, std::forward_as_tuple(buffer.id), std::forward_as_tuple(buffer.planes)); @@ -338,7 +341,7 @@ void IPARkISP1::processStats(const uint32_t frame, const uint32_t bufferId, * provided. */ const rkisp1_stat_buffer *stats = nullptr; - if (!context_.configuration.raw) + if (bufferId != 0) stats = reinterpret_cast( mappedBuffers_.at(bufferId).planes()[0].data()); From patchwork Mon Sep 14 14:02:33 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28264 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A8FD4C335A for ; Mon, 14 Sep 2026 14:04:23 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E9F83686DF; Mon, 14 Sep 2026 16:04:22 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="iQk1OIO7"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D475A6869A for ; Mon, 14 Sep 2026 16:04:21 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id DDBB4512; Mon, 14 Sep 2026 16:02:41 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394562; bh=bIpiT8bCnAABkkMf6Kxg4is1UDEnWbYRVOroeChEYJQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iQk1OIO7uap43O9GBwfo/A1BaSnAqfCZClqmQUTiZYv4ZLZMsOtS+eHFH+qDtNq9L jDScEcaDDFPfMx6vKwEatoUfVjFnawTus1W3tROFI1INIaKQAfDOZOaCRkTkKBFPY4 PPfmOCshO6xIgbk4msZ5yozq8c17IsW/uztelpIU= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 20/41] libipa: awb: Populate metadata when stats are invalid Date: Mon, 14 Sep 2026 16:02:33 +0200 Message-ID: <20260914140309.3354666-21-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" A pipeline handler can call processStats() on the IPA even in the case that no valid stats are available, just to populate the metadata with the values from the frame context. Add support for that to the AWB algorithm by populating the metadata before checking if stats are valid. Signed-off-by: Stefan Klug --- src/ipa/libipa/awb.cpp | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/ipa/libipa/awb.cpp b/src/ipa/libipa/awb.cpp index da835bec018d..81d616c3a646 100644 --- a/src/ipa/libipa/awb.cpp +++ b/src/ipa/libipa/awb.cpp @@ -372,6 +372,12 @@ void AwbAlgorithmBase::process(awb::ActiveState &state, const AwbStats &stats, unsigned int lux, ControlList &metadata) { + /* Populate metadata. */ + metadata.set(controls::AwbEnable, frameContext.autoEnabled); + metadata.set(controls::ColourGains, { static_cast(frameContext.gains.r()), + static_cast(frameContext.gains.b()) }); + metadata.set(controls::ColourTemperature, frameContext.colourTemperature); + if (!stats.valid()) return; @@ -393,12 +399,6 @@ void AwbAlgorithmBase::process(awb::ActiveState &state, state.automatic.gains = awbResult.gains * speed + state.automatic.gains * (1 - speed); - /* Populate metadata. */ - metadata.set(controls::AwbEnable, frameContext.autoEnabled); - metadata.set(controls::ColourGains, { static_cast(frameContext.gains.r()), - static_cast(frameContext.gains.b()) }); - metadata.set(controls::ColourTemperature, frameContext.colourTemperature); - LOG(Awb, Debug) << std::showpoint << "Means " << stats.rgbMeans() << ", gains " << state.automatic.gains << ", temp " << state.automatic.colourTemperature << "K"; From patchwork Mon Sep 14 14:02:34 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28265 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 26C9FC335B for ; Mon, 14 Sep 2026 14:04:27 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8FFD9686E0; Mon, 14 Sep 2026 16:04:26 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="S79LZzHx"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7F6206869A for ; Mon, 14 Sep 2026 16:04:24 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 8BDBC929; Mon, 14 Sep 2026 16:02:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394564; bh=CqHZMSbuFPqXSDjcwGf1Sq/oLtjBHdFYKPLZUd+H6bU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=S79LZzHxAa1Q+RSMX0FiX1tlkRAAPMTnuXCiNYk09YiaymFK9EyYYmFz/TgfaCm+A mLGesMXO8A6+RGGCsp1P8sjI8RejfQf7gVeTFXP+uERIu0YExURb+AsfPemqYa6ZXp AfXw24FMB7m5PW0tVMIJGTdNJmYHzcGOMvEXbM6A= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 21/41] ipa: rksip1: Call libipa::Awb::process() when stats are null Date: Mon, 14 Sep 2026 16:02:34 +0200 Message-ID: <20260914140309.3354666-22-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Call process() on the algorithm implementation even without a stats buffer to ensure that metadata is populated unconditionally. Print the error only in the case that there was a stats buffer. Having no stats buffer at all is a legal condition. Signed-off-by: Stefan Klug --- src/ipa/rkisp1/algorithms/awb.cpp | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/ipa/rkisp1/algorithms/awb.cpp b/src/ipa/rkisp1/algorithms/awb.cpp index cf8dc14308e2..f39153355a12 100644 --- a/src/ipa/rkisp1/algorithms/awb.cpp +++ b/src/ipa/rkisp1/algorithms/awb.cpp @@ -195,7 +195,10 @@ void Awb::process(IPAContext &context, ControlList &metadata) { if (!stats || !(stats->meas_type & RKISP1_CIF_ISP_STAT_AWB)) { - LOG(RkISP1Awb, Error) << "AWB data is missing in statistics"; + if (stats) + LOG(RkISP1Awb, Error) << "AWB data is missing in statistics"; + awbAlgo_.process(context.activeState.awb, frameContext.awb, RkISP1AwbStats({}), + frameContext.lux.lux, metadata); return; } From patchwork Mon Sep 14 14:02:35 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28266 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id DB160C335C for ; Mon, 14 Sep 2026 14:04:28 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 49AFE686E9; Mon, 14 Sep 2026 16:04:28 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="DEjuW/2T"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id EE9E7686E4 for ; Mon, 14 Sep 2026 16:04:26 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 0055D512; Mon, 14 Sep 2026 16:02:46 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394567; bh=T4CSUUoDZUeDsp32OtDdQSjsm7rtiO8pQp+ARUN9oSw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DEjuW/2TaDmJG3z2qhCuoobzpEwPcZNLRwUXGlUYdiW33gKITdOMstLVqgYh+G7QA yiBzLGcSEEvlfAytO/D9D0BY7g3+Eshx16PaV2imLrTf4PFhoJanbf/1YAyU9HxKWy cJutjJr1g97e9lVlTVvRWdaAb4z/9Dln5eA2O19k= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug , Paul Elder Subject: [PATCH v3 22/41] libipa: agc: Populate frameContext in queueRequest() in auto mode Date: Mon, 14 Sep 2026 16:02:35 +0200 Message-ID: <20260914140309.3354666-23-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In case of a param buffer underrun computeParams() is not called on a frame. At the moment this causes a "ERROR AgcMeanLuminance agc_mean_luminance.cpp:689 Effective exposure value is 0. This is a bug in AGC and must be fixed for proper operation." log message, when processStats() is called on such a frame because frameContext.gain is 0. Fix that by initializing all the values in queueRequest(). Signed-off-by: Stefan Klug Reviewed-by: Paul Elder --- Changes in v3: - Manually rebased as agc was moved to libipa - Changed commit message to better reflect the issue that this fixes Changes in v2: - Collected tag --- src/ipa/libipa/agc.cpp | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/ipa/libipa/agc.cpp b/src/ipa/libipa/agc.cpp index 9016e5f68ab7..a2438cb49d1b 100644 --- a/src/ipa/libipa/agc.cpp +++ b/src/ipa/libipa/agc.cpp @@ -608,12 +608,18 @@ void AgcAlgorithm::queueRequest(const agc::Session &session, agc::ActiveState &s frameContext.autoExposureEnabled = state.autoExposureEnabled; frameContext.autoGainEnabled = state.autoGainEnabled; - if (!frameContext.autoExposureEnabled) + if (frameContext.autoExposureEnabled) + frameContext.exposure = state.automatic.exposure; + else frameContext.exposure = state.manual.exposure; - if (!frameContext.autoGainEnabled) + if (frameContext.autoGainEnabled) + frameContext.gain = state.automatic.gain; + else frameContext.gain = state.manual.gain; - if (!frameContext.autoExposureEnabled && !frameContext.autoGainEnabled) + if (frameContext.autoExposureEnabled || frameContext.autoGainEnabled) + frameContext.quantizationGain = state.automatic.quantizationGain; + else frameContext.quantizationGain = 1.0; const auto &exposureMode = controls.get(controls::AeExposureMode); From patchwork Mon Sep 14 14:02:36 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28267 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E09A0C3352 for ; Mon, 14 Sep 2026 14:04:31 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 44615686F2; Mon, 14 Sep 2026 16:04:31 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ZtfiNPU5"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A8008686E4 for ; Mon, 14 Sep 2026 16:04:29 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id A97279A4; Mon, 14 Sep 2026 16:02:49 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394569; bh=Fy5oVvw0unihSeDZTKVspaE6n2gFyT0lok7vXfBUUYc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZtfiNPU5lDWdtWSvziV19xIjbINvGGb8C5/xjF6t6MX9fvu5YDKd5wVbWOkj7F958 VXj7gTBwIQsbboZWD/hmY4CNJiXww/rIV++7E/+t9gTMsnK70vEf4+8SN4T6qwlIpX XzMD/oNEJNf49YpaayK5Rf/E87u0xxagiujD363g= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 23/41] libipa: awb: Populate frameContext in queueRequest() in auto mode Date: Mon, 14 Sep 2026 16:02:36 +0200 Message-ID: <20260914140309.3354666-24-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In case of a param buffer underrun computeParams() is not called on a frame. At the moment this causes invalid metadata when processStats() is called on that frame. Fix this by initializing all the values in queueRequest(). Signed-off-by: Stefan Klug --- Changes in v3: - Added this commit --- src/ipa/libipa/awb.cpp | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/ipa/libipa/awb.cpp b/src/ipa/libipa/awb.cpp index 81d616c3a646..df3e3b8c283c 100644 --- a/src/ipa/libipa/awb.cpp +++ b/src/ipa/libipa/awb.cpp @@ -304,6 +304,8 @@ void AwbAlgorithmBase::queueRequest(awb::ActiveState &state, } frameContext.autoEnabled = state.autoEnabled; + frameContext.gains = state.automatic.gains; + frameContext.colourTemperature = state.automatic.colourTemperature; if (frameContext.autoEnabled) return; From patchwork Mon Sep 14 14:02:37 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28268 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 83149C335D for ; Mon, 14 Sep 2026 14:04:34 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0479D686EE; Mon, 14 Sep 2026 16:04:34 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="viGxjoMm"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B727F686EC for ; Mon, 14 Sep 2026 16:04:31 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C5C57512; Mon, 14 Sep 2026 16:02:51 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394571; bh=rbq+6rtEALC8VfLUhEWq8LHy4llIg73e8OrM0gKOukQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=viGxjoMm83FHvefF/eHIEdl/d9ichS84f2pu6bdpEVH/cwk7yiEM7GKbMEMkOC+Ut pVrSWXvdstIh4oIZ4r9KVi44hnLegJERsATnd01nxnWG8yHpHi30nbAKl0ebermQ8g nA8k+3CEN8zLavA7Hri0v7u7M1a9X0L7cmtFUoHs= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 24/41] pipeline: rkisp1: Pass bufferId to metadataReady() Date: Mon, 14 Sep 2026 16:02:37 +0200 Message-ID: <20260914140309.3354666-25-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The frame number is not necessarily unique. This should not be the case, but even misbehaving kernel drivers should not be able to negatively influence libcamera. Pass the bufferId, to have a guaranteed unique handle. This patch only introduces the parameter without using it. It is preparatory for the upcoming synchronization rework. Signed-off-by: Stefan Klug --- include/libcamera/ipa/rkisp1.mojom | 4 ++-- src/ipa/rkisp1/rkisp1.cpp | 4 ++-- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 12 ++++++++---- 3 files changed, 12 insertions(+), 8 deletions(-) diff --git a/include/libcamera/ipa/rkisp1.mojom b/include/libcamera/ipa/rkisp1.mojom index 4c29b53cd7f9..04230d0f852e 100644 --- a/include/libcamera/ipa/rkisp1.mojom +++ b/include/libcamera/ipa/rkisp1.mojom @@ -42,7 +42,7 @@ interface IPARkISP1Interface { }; interface IPARkISP1EventInterface { - paramsComputed(uint32 frame, uint32 bytesused); + paramsComputed(uint32 frame, uint32 bufferId, uint32 bytesused); setSensorControls(uint32 frame, libcamera.ControlList sensorControls); - metadataReady(uint32 frame, libcamera.ControlList metadata); + metadataReady(uint32 frame, uint32 bufferId, libcamera.ControlList metadata); }; diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 08b7dde6b2de..abfd3c5ff58f 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -324,7 +324,7 @@ void IPARkISP1::computeParams(const uint32_t frame, const uint32_t bufferId) for (const auto &algo : algorithms()) algo->prepare(context_, frame, frameContext, ¶ms); - paramsComputed.emit(frame, params.bytesused()); + paramsComputed.emit(frame, bufferId, params.bytesused()); } ControlList ctrls = getSensorControls(frameContext); @@ -358,7 +358,7 @@ void IPARkISP1::processStats(const uint32_t frame, const uint32_t bufferId, } context_.debugMetadata.moveEntries(metadata); - metadataReady.emit(frame, metadata); + metadataReady.emit(frame, bufferId, metadata); } void IPARkISP1::updateControls(ControlInfoMap *ipaControls) diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index edd9f352524c..d84044f218b5 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -130,11 +130,11 @@ public: bool usesDewarper_; private: - void paramsComputed(unsigned int frame, unsigned int bytesused); + void paramsComputed(unsigned int frame, unsigned int bufferId, unsigned int bytesused); void setSensorControls(unsigned int frame, const ControlList &sensorControls); - void metadataReady(unsigned int frame, const ControlList &metadata); + void metadataReady(unsigned int frame, unsigned int bufferId, const ControlList &metadata); int loadTuningFile(const std::string &file); }; @@ -475,7 +475,9 @@ int RkISP1CameraData::loadTuningFile(const std::string &path) return 0; } -void RkISP1CameraData::paramsComputed(unsigned int frame, unsigned int bytesused) +void RkISP1CameraData::paramsComputed(unsigned int frame, + [[maybe_unused]] unsigned int bufferId, + unsigned int bytesused) { PipelineHandlerRkISP1 *pipe = RkISP1CameraData::pipe(); RkISP1FrameInfo *info = frameInfo_.find(frame); @@ -506,7 +508,9 @@ void RkISP1CameraData::setSensorControls(unsigned int frame, delayedCtrls_->push(frame, sensorControls); } -void RkISP1CameraData::metadataReady(unsigned int frame, const ControlList &metadata) +void RkISP1CameraData::metadataReady(unsigned int frame, + [[maybe_unused]] unsigned int bufferId, + const ControlList &metadata) { RkISP1FrameInfo *info = frameInfo_.find(frame); if (!info) From patchwork Mon Sep 14 14:02:38 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28269 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1AC39C335E for ; Mon, 14 Sep 2026 14:04:36 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 92AE0686ED; Mon, 14 Sep 2026 16:04:35 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="GQR4tSjw"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D644B686D4 for ; Mon, 14 Sep 2026 16:04:33 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id D0F84929; Mon, 14 Sep 2026 16:02:53 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394574; bh=SAscIwgvEfFc5KQReZg468EVZn7Xw9AdNCinIp+gmcg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GQR4tSjwdipwusLd+h68QZ5u/Eem1M1tE9NwtoTQx98r/QRx91xgxXgkyLkDUhR4M LXtGM9iPoUQ4ib/nqVydyLObQiDwBq0mpQ0nMcE7K0UnjunRLE0RCuD0GwdWzq87GT eEYW8M89FBlhnRREhfTG2rWS7pbnM2sKcHsjEKEQ= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 25/41] ipa: rkisp1: Lazy initialise frame context Date: Mon, 14 Sep 2026 16:02:38 +0200 Message-ID: <20260914140309.3354666-26-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" For per frame control we want to tick the IPA by the sensor frame sequence instead of the request sequence. This has the side effect that the IPA must be able to cope with situations where a frame context is required for a frame that was not queued before (computeParams is called without a corresponding request) or processStats is called for an unexpected sequence number (because a scratch buffer was used on kernel side). With the current FCQueue implementation this is not easy to model, as it has distinct calls for alloc() and get(). Simplify that by passing the FCQueue a callback that it can call to initialize a frame context when needed. This has the added benefit that the FCQueue can collate controls for requests that were queued in too late. This simplifies the logic on the IPA side. As fetching an uninitialized frame context is no error anymore, demote the corresponding warnings to debug messages. Signed-off-by: Stefan Klug --- Changes in v3: - Fixed rebase conflicts Changes in v2: - Rewrote the patch to use a callback based mechanism. --- src/ipa/ipu3/ipu3.cpp | 20 ++++++-- src/ipa/libipa/fc_queue.cpp | 39 +++++++------- src/ipa/libipa/fc_queue.h | 97 +++++++++++++---------------------- src/ipa/mali-c55/mali-c55.cpp | 20 ++++++-- src/ipa/rkisp1/rkisp1.cpp | 25 +++++---- src/ipa/softisp/softisp.cpp | 19 +++++-- 6 files changed, 113 insertions(+), 107 deletions(-) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 30d52cfcf9c6..24bbfa86c255 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -169,6 +169,8 @@ private: void setControls(unsigned int frame); void calculateBdsGrid(const Size &bdsOutputSize); + void initializeFrameContext(IPAFrameContext &frameContext, + const ControlList &controls); std::map buffers_; @@ -181,6 +183,10 @@ private: IPAIPU3::IPAIPU3() : context_(kMaxFrameContexts) { + context_.frameContexts.setInitCallback( + [this](IPAFrameContext &fc, const ControlList &c) { + this->initializeFrameContext(fc, c); + }); } std::string IPAIPU3::logPrefix() const @@ -467,7 +473,7 @@ void IPAIPU3::computeParams(const uint32_t frame, const uint32_t bufferId) */ params->use = {}; - IPAFrameContext &frameContext = context_.frameContexts.get(frame); + IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(frame); for (const auto &algo : algorithms()) algo->prepare(context_, frame, frameContext, params); @@ -500,7 +506,7 @@ void IPAIPU3::processStats(const uint32_t frame, const ipu3_uapi_stats_3a *stats = reinterpret_cast(mem.data()); - IPAFrameContext &frameContext = context_.frameContexts.get(frame); + IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(frame); std::tie(frameContext.sensor.exposure, frameContext.sensor.gain) = agc::extractControls(sensorControls, context_.camHelper.get()); @@ -533,10 +539,14 @@ void IPAIPU3::processStats(const uint32_t frame, */ void IPAIPU3::queueRequest(const uint32_t frame, const ControlList &controls) { - IPAFrameContext &frameContext = context_.frameContexts.alloc(frame); + context_.frameContexts.getOrInitContext(frame, controls); +} +void IPAIPU3::initializeFrameContext(IPAFrameContext &frameContext, + const ControlList &controls) +{ for (const auto &algo : algorithms()) - algo->queueRequest(context_, frame, frameContext, controls); + algo->queueRequest(context_, frameContext.frame(), frameContext, controls); } /** @@ -548,7 +558,7 @@ void IPAIPU3::queueRequest(const uint32_t frame, const ControlList &controls) */ void IPAIPU3::setControls(unsigned int frame) { - IPAFrameContext &frameContext = context_.frameContexts.get(frame); + IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(frame); ControlList ctrls(context_.sensorControls); agc::prepareControls(ctrls, context_.camHelper.get(), diff --git a/src/ipa/libipa/fc_queue.cpp b/src/ipa/libipa/fc_queue.cpp index 7ba28ed21611..7551fffcafd5 100644 --- a/src/ipa/libipa/fc_queue.cpp +++ b/src/ipa/libipa/fc_queue.cpp @@ -101,6 +101,19 @@ namespace ipa { * \param[in] size The number of contexts in the queue */ +/** + * \typedef FCQueue::InitCallback + * \brief The init callback type + */ + +/** + * \fn FCQueue::setInitCallback() + * \brief Set the init callback + * + * The init callback is called when a frame context is allocated and needs to be + * initialized. + */ + /** * \fn FCQueue::clear() * \brief Clear the contexts queue @@ -113,16 +126,17 @@ namespace ipa { */ /** - * \fn FCQueue::alloc(uint32_t frame) - * \brief Allocate and return a FrameContext for the \a frame + * \fn FCQueue::getOrInitContext(uint32_t frame, const ControlList &controls) + * \brief Get or allocate and return a FrameContext for the \a frame * \param[in] frame The frame context sequence number + * \param[in] controls Controls to pass to the init function * - * The first call to obtain a FrameContext from the FCQueue should be handled - * through this function. The FrameContext will be initialised, if not - * initialised already, and returned to the caller. + * If a FrameContext for frame \a frame is not yet initialized, it will be + * initialized and \a controls is passed to the initialization function. * - * If the FrameContext was already initialized for this \a frame, a warning will - * be reported and the previously initialized FrameContext is returned. + * If a FrameContext is already initialized, it is returned to the caller. The + * passed controls are stored and used for the next initialization of a + * FrameContext (the control lists will be merged in order). * * Frame contexts are expected to be initialised when a Request is first passed * to the IPA module in IPAModule::queueRequest(). @@ -130,17 +144,6 @@ namespace ipa { * \return A reference to the FrameContext for sequence \a frame */ -/** - * \fn FCQueue::get(uint32_t frame) - * \brief Obtain the FrameContext for the \a frame - * \param[in] frame The frame context sequence number - * - * If the FrameContext is not correctly initialised for the \a frame, it will be - * initialised. - * - * \return A reference to the FrameContext for sequence \a frame - */ - } /* namespace ipa */ } /* namespace libcamera */ diff --git a/src/ipa/libipa/fc_queue.h b/src/ipa/libipa/fc_queue.h index 812022c496ed..633bf646d88d 100644 --- a/src/ipa/libipa/fc_queue.h +++ b/src/ipa/libipa/fc_queue.h @@ -11,6 +11,7 @@ #include #include +#include namespace libcamera { @@ -27,52 +28,33 @@ struct FrameContext { private: template friend class FCQueue; uint32_t frame_; - bool initialised_ = false; }; template class FCQueue { public: + using InitCallback = std::function; + FCQueue(unsigned int size) : contexts_(size) { } + void setInitCallback(const InitCallback &cb) + { + initCallback_ = cb; + } + void clear() { for (FC &ctx : contexts_) { - ctx.initialised_ = false; ctx.frame_ = 0; } + initialized_ = false; } - FC &alloc(const uint32_t frame) - { - FC &fc = contexts_[frame % contexts_.size()]; - FrameContext &frameContext = fc; - - /* - * Do not re-initialise if a get() call has already fetched this - * frame context to preseve the context. - * - * \todo If the the sequence number of the context to initialise - * is smaller than the sequence number of the queue slot to use, - * it means that we had a serious request underrun and more - * frames than the queue size has been produced since the last - * time the application has queued a request. Does this deserve - * an error condition ? - */ - if (frame != 0 && frame <= frameContext.frame_) - LOG(FCQueue, Warning) - << "Frame " << frame << " already initialised"; - else - init(fc, frame); - - return fc; - } - - FC &get(uint32_t frame) + FC &getOrInitContext(unsigned int frame, const ControlList &controls = {}) { FC &fc = contexts_[frame % contexts_.size()]; FrameContext &frameContext = fc; @@ -90,51 +72,42 @@ public: << " has been overwritten by " << frameContext.frame_; - if (frame == 0 && !frameContext.initialised_) { - /* - * If the IPA calls get() at start() time it will get an - * un-intialized FrameContext as the below "frame == - * frameContext.frame_" check will return success - * because FrameContexts are zeroed at creation time. - * - * Make sure the FrameContext gets initialised if get() - * is called before alloc() by the IPA for frame#0. - */ - init(fc, frame); - + if (initialized_ && frame == frameContext.frame_) { + if (!controls.empty()) { + /* Too late to apply the controls. Store them for later. */ + LOG(FCQueue, Warning) + << "Request underrun. Controls for frame " + << frame << " are delayed "; + controlsToApply_.merge(controls, + ControlList::MergePolicy::OverwriteExisting); + } + LOG(FCQueue, Debug) << "Got " << frame; return fc; } - if (frame == frameContext.frame_) - return fc; + const ControlList *controls2 = &controls; + if (!controlsToApply_.empty()) { + LOG(FCQueue, Debug) << "Applied late controls on frame" << frame; + controlsToApply_.merge(controls, ControlList::MergePolicy::OverwriteExisting); + controls2 = &controlsToApply_; + } - /* - * The frame context has been retrieved before it was - * initialised through the initialise() call. This indicates an - * algorithm attempted to access a Frame context before it was - * queued to the IPA. Controls applied for this request may be - * left unhandled. - * - * \todo Set an error flag for per-frame control errors. - */ - LOG(FCQueue, Warning) - << "Obtained an uninitialised FrameContext for " << frame; + LOG(FCQueue, Debug) << "Init " << frame; - init(fc, frame); + fc = {}; + frameContext.frame_ = frame; + initCallback_(fc, *controls2); + initialized_ = true; + controlsToApply_.clear(); return fc; } private: - void init(FC &fc, const uint32_t frame) - { - fc = {}; - FrameContext &frameContext = fc; - frameContext.frame_ = frame; - frameContext.initialised_ = true; - } - std::vector contexts_; + InitCallback initCallback_; + ControlList controlsToApply_; + bool initialized_; }; } /* namespace ipa */ diff --git a/src/ipa/mali-c55/mali-c55.cpp b/src/ipa/mali-c55/mali-c55.cpp index 9a2918cf2cf1..d3d17bb5ae18 100644 --- a/src/ipa/mali-c55/mali-c55.cpp +++ b/src/ipa/mali-c55/mali-c55.cpp @@ -67,6 +67,8 @@ protected: private: void updateControls(ControlInfoMap *ipaControls); void setControls(const IPAFrameContext &frameContext); + void initializeFrameContext(IPAFrameContext &frameContext, + const ControlList &controls); std::map buffers_; @@ -81,6 +83,10 @@ namespace { IPAMaliC55::IPAMaliC55() : context_(kMaxFrameContexts) { + context_.frameContexts.setInitCallback( + [this](IPAFrameContext &fc, const ControlList &c) { + this->initializeFrameContext(fc, c); + }); } std::string IPAMaliC55::logPrefix() const @@ -219,21 +225,25 @@ void IPAMaliC55::unmapBuffers(const std::vector &buffers) } } -void IPAMaliC55::queueRequest(const uint32_t request, const ControlList &controls) +void IPAMaliC55::queueRequest(const uint32_t frame, const ControlList &controls) { - IPAFrameContext &frameContext = context_.frameContexts.alloc(request); + context_.frameContexts.getOrInitContext(frame, controls); +} +void IPAMaliC55::initializeFrameContext(IPAFrameContext &frameContext, + const ControlList &controls) +{ for (const auto &a : algorithms()) { Algorithm *algo = static_cast(a.get()); - algo->queueRequest(context_, request, frameContext, controls); + algo->queueRequest(context_, frameContext.frame(), frameContext, controls); } } void IPAMaliC55::fillParams(unsigned int request, [[maybe_unused]] uint32_t bufferId) { - IPAFrameContext &frameContext = context_.frameContexts.get(request); + IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(request); MaliC55Params params(buffers_.at(bufferId).planes()[0]); for (const auto &algo : algorithms()) @@ -245,7 +255,7 @@ void IPAMaliC55::fillParams(unsigned int request, void IPAMaliC55::processStats(unsigned int request, unsigned int bufferId, const ControlList &sensorControls) { - IPAFrameContext &frameContext = context_.frameContexts.get(request); + IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(request); const mali_c55_stats_buffer *stats = nullptr; stats = reinterpret_cast( diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index abfd3c5ff58f..3a65dab75cb7 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -6,6 +6,7 @@ */ #include +#include #include #include @@ -64,8 +65,7 @@ public: void queueRequest(const uint32_t frame, const ControlList &controls) override; void computeParams(const uint32_t frame, const uint32_t bufferId) override; - void initializeFrameContext(const uint32_t frame, - IPAFrameContext &frameContext, + void initializeFrameContext(IPAFrameContext &frameContext, const ControlList &controls); void processStats(const uint32_t frame, const uint32_t bufferId, const ControlList &sensorControls) override; @@ -124,6 +124,10 @@ const ControlInfoMap::Map rkisp1Controls{ IPARkISP1::IPARkISP1() : context_(kMaxFrameContexts) { + context_.frameContexts.setInitCallback( + [this](IPAFrameContext &fc, const ControlList &c) { + this->initializeFrameContext(fc, c); + }); } std::string IPARkISP1::logPrefix() const @@ -208,8 +212,7 @@ int IPARkISP1::init(const IPASettings &settings, unsigned int hwRevision, void IPARkISP1::start(const ControlList &controls, StartResult *result) { - IPAFrameContext frameContext = {}; - initializeFrameContext(0, frameContext, controls); + IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(0, controls); result->controls = getSensorControls(frameContext); result->code = 0; } @@ -295,27 +298,23 @@ void IPARkISP1::unmapBuffers(const std::vector &ids) void IPARkISP1::queueRequest(const uint32_t frame, const ControlList &controls) { - IPAFrameContext &frameContext = context_.frameContexts.alloc(frame); context_.debugMetadata.enableByControl(controls); - - initializeFrameContext(frame, frameContext, controls); + context_.frameContexts.getOrInitContext(frame, controls); } -void IPARkISP1::initializeFrameContext(const uint32_t frame, - IPAFrameContext &frameContext, - const ControlList &controls) +void IPARkISP1::initializeFrameContext(IPAFrameContext &fc, const ControlList &controls) { for (const auto &a : algorithms()) { Algorithm *algo = static_cast(a.get()); if (algo->disabled_) continue; - algo->queueRequest(context_, frame, frameContext, controls); + algo->queueRequest(context_, fc.frame(), fc, controls); } } void IPARkISP1::computeParams(const uint32_t frame, const uint32_t bufferId) { - IPAFrameContext &frameContext = context_.frameContexts.get(frame); + IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(frame); if (bufferId != 0) { RkISP1Params params(context_.configuration.paramFormat, @@ -334,7 +333,7 @@ void IPARkISP1::computeParams(const uint32_t frame, const uint32_t bufferId) void IPARkISP1::processStats(const uint32_t frame, const uint32_t bufferId, const ControlList &sensorControls) { - IPAFrameContext &frameContext = context_.frameContexts.get(frame); + IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(frame); /* * In raw capture mode, the ISP is bypassed and no statistics buffer is diff --git a/src/ipa/softisp/softisp.cpp b/src/ipa/softisp/softisp.cpp index 8acfcd1a0b72..c00d36bdff1a 100644 --- a/src/ipa/softisp/softisp.cpp +++ b/src/ipa/softisp/softisp.cpp @@ -48,6 +48,10 @@ public: IPASoftIsp() : context_(kMaxFrameContexts) { + context_.frameContexts.setInitCallback( + [this](IPAFrameContext &fc, const ControlList &c) { + this->initializeFrameContext(fc, c); + }); } ~IPASoftIsp(); @@ -75,6 +79,8 @@ protected: private: void updateExposure(double exposureMSV); + void initializeFrameContext(IPAFrameContext &frameContext, + const ControlList &controls); DebayerParams *params_; SwIspStats *stats_; @@ -216,17 +222,22 @@ void IPASoftIsp::stop() void IPASoftIsp::queueRequest(const uint32_t frame, const ControlList &controls) { - IPAFrameContext &frameContext = context_.frameContexts.alloc(frame); + context_.frameContexts.getOrInitContext(frame, controls); +} + +void IPASoftIsp::initializeFrameContext(IPAFrameContext &frameContext, + const ControlList &controls) +{ for (const auto &algo : algorithms()) - algo->queueRequest(context_, frame, frameContext, controls); + algo->queueRequest(context_, frameContext.frame(), frameContext, controls); } void IPASoftIsp::computeParams(const uint32_t frame) { context_.activeState.combinedMatrix = Matrix::identity(); - IPAFrameContext &frameContext = context_.frameContexts.get(frame); + IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(frame); for (const auto &algo : algorithms()) algo->prepare(context_, frame, frameContext, params_); params_->combinedMatrix = context_.activeState.combinedMatrix; @@ -238,7 +249,7 @@ void IPASoftIsp::processStats(const uint32_t frame, [[maybe_unused]] const uint32_t bufferId, const ControlList &sensorControls) { - IPAFrameContext &frameContext = context_.frameContexts.get(frame); + IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(frame); std::tie(frameContext.sensor.exposure, frameContext.sensor.gain) = agc::extractControls(sensorControls, context_.camHelper.get()); From patchwork Mon Sep 14 14:02:39 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28270 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id EC8A7C3226 for ; Mon, 14 Sep 2026 14:04:37 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 84D2C686FC; Mon, 14 Sep 2026 16:04:37 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="HTSwPGDC"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3EFFF686D4 for ; Mon, 14 Sep 2026 16:04:36 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 3BB77929; Mon, 14 Sep 2026 16:02:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394576; bh=RgJeZuCuLUZXqt4RcR5NXpcplWtH6zhGbt4PDH8M3o4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HTSwPGDC90mjOd7IKMHkeJD5QDAiVIN25jt0WRG1ktNzZ81d90JkQTaLnGyMm8Qbz 4jivW+eySmKJNSgEBtzFH0UB22usgms1OAsOCdsdjwmQrQKNDmcwoGDq9b/dQTUSsB g3qkyT2svdCmtJ+UO+L9Y9hmJKSFitfy9CX+CMwk= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 26/41] libcamera: internal: Add SequenceSyncHelper class Date: Mon, 14 Sep 2026 16:02:39 +0200 Message-ID: <20260914140309.3354666-27-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" On a V4L2 buffer the assigned sequence is not known until the buffer is dequeued. But for per frame controls we have to prepare other data like sensor controls and ISP params in advance. So we try to anticipate the sequence number a given buffer will be. In a perfect world this works well as long as the initial sequence is assigned correctly. But it breaks as soon as things like running out of buffers or incomplete images happen. To make things even more complicated, in most cases more than one buffer is queued to the kernel at a time. So as soon as a sequence number doesn't match the expected one after dequeuing, most likely all the already queued buffers will be dequeued with the same error. It is not sufficient to simply add the correction after dequeuing because the error on all queued frames would accumulate and the whole system starts to oscillate. To work around that add a SequenceSyncHelper class that tracks the expected error and allows to easily query the necessary correction when queuing new buffers. Signed-off-by: Stefan Klug --- Changes in v3: - Moved implementation into cpp - Renamed some functions - Added class documentation Changes in v2: - Moved files to man src dir, to be able to reuse it in other pipelines - Added cancel() function. --- include/libcamera/internal/meson.build | 1 + .../libcamera/internal/sequence_sync_helper.h | 31 +++++ src/libcamera/meson.build | 1 + src/libcamera/sequence_sync_helper.cpp | 113 ++++++++++++++++++ 4 files changed, 146 insertions(+) create mode 100644 include/libcamera/internal/sequence_sync_helper.h create mode 100644 src/libcamera/sequence_sync_helper.cpp diff --git a/include/libcamera/internal/meson.build b/include/libcamera/internal/meson.build index fd375134a5c4..73635e31bad9 100644 --- a/include/libcamera/internal/meson.build +++ b/include/libcamera/internal/meson.build @@ -40,6 +40,7 @@ libcamera_internal_headers = files([ 'process.h', 'pub_key.h', 'request.h', + 'sequence_sync_helper.h', 'shared_mem_object.h', 'source_paths.h', 'sysfs.h', diff --git a/include/libcamera/internal/sequence_sync_helper.h b/include/libcamera/internal/sequence_sync_helper.h new file mode 100644 index 000000000000..d8dcfe2ae81d --- /dev/null +++ b/include/libcamera/internal/sequence_sync_helper.h @@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2025, Ideas on Board + * + * Sequence sync helper + */ + +#pragma once + +#include + +#include + +namespace libcamera { + +class SequenceSyncHelper +{ +public: + int receivedFrame(size_t expectedSequence, size_t actualSequence); + void cancelFrame(); + int correction(); + void pushCorrection(int correction); + void reset(); + +private: + std::queue corrections_; + int correctionToApply_ = 0; + int expectedOffset_ = 0; +}; + +} /* namespace libcamera */ diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build index 17c1b2cb3479..038d2dbf12cf 100644 --- a/src/libcamera/meson.build +++ b/src/libcamera/meson.build @@ -49,6 +49,7 @@ libcamera_internal_sources = files([ 'pipeline_handler.cpp', 'process.cpp', 'pub_key.cpp', + 'sequence_sync_helper.cpp', 'shared_mem_object.cpp', 'source_paths.cpp', 'sysfs.cpp', diff --git a/src/libcamera/sequence_sync_helper.cpp b/src/libcamera/sequence_sync_helper.cpp new file mode 100644 index 000000000000..7c4e2b423666 --- /dev/null +++ b/src/libcamera/sequence_sync_helper.cpp @@ -0,0 +1,113 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2026, Ideas on Board. + * + * Helper to synchronize buffer sequences + */ + +#include "libcamera/internal/sequence_sync_helper.h" + +#include + +namespace libcamera { + +LOG_DEFINE_CATEGORY(SequenceSyncHelper) + +/** + * \file sequence_sync_helper.h + * \class SequenceSyncHelper + * \brief Helper to synchronize buffer sequences + * + * On a V4L2 buffers the sequence is not known until the buffer was dequeued. To + * pre plan regulation it is however necessary to know the sequence of a buffer + * before queueing the buffer (or multiple buffers). By the time an offset is + * detected in most cases more than one buffers are already queued in and all of + * them carry the same error. Simple adding the perceived difference on dequeue + * therefore doesn't help and leads to oscillations. This class tracks the + * sequence corrections over time and helps in keeping the sequence numbers in + * sync. + */ + +/** + * \brief Tell the sync helper that a frame was received + * \param expectedSequence The sequence that was expected for that frame + * \param actualSequence The actual sequence of the frame + * + * This function needs to be called when a frame was dequeued. The sync helper + * calculates necessary corrections and keeps track of corrections already + * applied. + */ +int SequenceSyncHelper::receivedFrame(size_t expectedSequence, + size_t actualSequence) +{ + ASSERT(!corrections_.empty()); + int diff = actualSequence - expectedSequence; + int corr = corrections_.front(); + corrections_.pop(); + expectedOffset_ -= corr; + int necessaryCorrection = diff - expectedOffset_; + correctionToApply_ += necessaryCorrection; + + LOG(SequenceSyncHelper, Debug) + << "Sync frame " + << "expected: " << expectedSequence + << " actual: " << actualSequence + << " correction: " << corr + << " expectedOffset: " << expectedOffset_ + << " correctionToApply " << correctionToApply_; + + expectedOffset_ += necessaryCorrection; + return necessaryCorrection; +} + +/** + * \brief Tell the sync helper that a frame was cancelled + * + * This function needs to be called when a frame was cancelled. + */ +void SequenceSyncHelper::cancelFrame() +{ + int corr = corrections_.front(); + corrections_.pop(); + expectedOffset_ -= corr; +} + +/** + * \brief Get the necessary correction + * + * Get the correction that must be applied to the sequence numbers to + * synchronize. + * + * \return The correction to apply + */ +int SequenceSyncHelper::correction() +{ + return correctionToApply_; +} + +/** + * \brief Tell the sync helper that a correction was pushed + * + * This must be called for every frame that gets pushed into the queue. If + * no correction was applied, it must be called with a correction of 0. + */ +void SequenceSyncHelper::pushCorrection(int correction) +{ + corrections_.push(correction); + correctionToApply_ -= correction; + LOG(SequenceSyncHelper, Debug) + << "Push correction " << correction + << " correctionToApply " << correctionToApply_; +} + +/** + * \brief Reset the sync helper + */ +void SequenceSyncHelper::reset() +{ + corrections_ = {}; + correctionToApply_ = 0; + expectedOffset_ = 0; +} + +} /* namespace libcamera */ From patchwork Mon Sep 14 14:02:40 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28271 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 27918C335F for ; Mon, 14 Sep 2026 14:04:41 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 44A00686ED; Mon, 14 Sep 2026 16:04:41 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="YWqTsYQv"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B2C1C686F2 for ; Mon, 14 Sep 2026 16:04:38 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 9C344929; Mon, 14 Sep 2026 16:02:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394578; bh=zj+66DagDd5/7MPDtoqkIGoNrPAruaemSYRriuLjlLs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YWqTsYQvEAXQ/1jHZLWxGe0/RbrcZ1cNaxcgXe0P9Jn1vAv7i9TowWKNjPI8ecr/m TkAnvR/3Dsfm2Qe/jrEftB5SlWWnzNrEKmQ4ph0nWKrKek4LPPelA1IBv3gsNTawZP bDQq4MZ/YEoIMmLcVstIA4N3D7Aj1qn5cx58LZBs= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 27/41] libcamera: internal: Add a BufferQueue class to handle buffer queues Date: Mon, 14 Sep 2026 16:02:40 +0200 Message-ID: <20260914140309.3354666-28-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a class that encapsulates a queue of v4l2 buffers and the typical use-cases. This simplifies manual queue management and helps in cases where a pre or postprocessing stage is needed. Signed-off-by: Stefan Klug --- Changes in v3: - Fixed incomplete reset in BufferQueue::releaseBuffers() - Added missing disconnect in ~BufferQueueDelegate - Added ASSERT in releaseBuffers() - Updated state diagram to show that cancelled buffers jump the postprocessing stage - Improved documentation Changes in v2: - Added this patch --- include/libcamera/internal/buffer_queue.h | 136 ++++++ include/libcamera/internal/meson.build | 1 + src/libcamera/buffer_queue.cpp | 541 ++++++++++++++++++++++ src/libcamera/meson.build | 1 + 4 files changed, 679 insertions(+) create mode 100644 include/libcamera/internal/buffer_queue.h create mode 100644 src/libcamera/buffer_queue.cpp diff --git a/include/libcamera/internal/buffer_queue.h b/include/libcamera/internal/buffer_queue.h new file mode 100644 index 000000000000..83a6a1f45be7 --- /dev/null +++ b/include/libcamera/internal/buffer_queue.h @@ -0,0 +1,136 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2025, Ideas on Board + * + * Sequence sync helper + */ + +#pragma once + +#include +#include + +#include +#include + +#include + +#include "sequence_sync_helper.h" + +namespace libcamera { + +LOG_DECLARE_CATEGORY(RkISP1Schedule) + +struct BufferQueueDelegateBase { + virtual ~BufferQueueDelegateBase() = default; + virtual int allocateBuffers(unsigned int count, + std::vector> *buffers) = 0; + virtual int importBuffers(unsigned int count) = 0; + virtual int releaseBuffers() = 0; + + virtual int queueBuffer(FrameBuffer *buffer) = 0; + + Signal bufferReady; +}; + +template +struct BufferQueueDelegate : public BufferQueueDelegateBase { + BufferQueueDelegate(T *target) : target_(target) + { + target_->bufferReady.connect(this, [this](FrameBuffer *buffer) { + this->bufferReady.emit(buffer); + }); + } + + ~BufferQueueDelegate() + { + target_->bufferReady.disconnect(this); + } + + int allocateBuffers(unsigned int count, + std::vector> *buffers) override + { + return target_->allocateBuffers(count, buffers); + } + + int importBuffers(unsigned int count) override + { + return target_->importBuffers(count); + } + + int queueBuffer(FrameBuffer *buffer) override + { + return target_->queueBuffer(buffer); + } + + int releaseBuffers() override + { + return target_->releaseBuffers(); + } + +private: + T *target_; +}; + +class BufferQueue +{ +public: + enum State { + Idle = 0, + Preparing, + Capturing, + Postprocessing + }; + + enum Flags { + PrepareStage = 1, + PostprocessStage = 2 + }; + + BufferQueue(std::unique_ptr &&delegate, int flags = 0, std::string name = {}); + + int allocateBuffers(unsigned int count); + int importBuffers(unsigned int count); + int releaseBuffers(); + + int sequenceCorrection(); + uint32_t nextSequence(); + + int prepareBuffer(uint32_t *sequence = nullptr); + int prepareBuffer(FrameBuffer *buffer, uint32_t *sequence = nullptr); + int preparedBuffer(); + + int queueBuffer(uint32_t *sequence = nullptr); + int queueBuffer(FrameBuffer *buffer, uint32_t *sequence = nullptr); + + void postprocessedBuffer(); + + bool empty(State state); + + FrameBuffer *front(State state); + + unsigned int expectedSequence(FrameBuffer *buffer) const; + const std::vector> &buffers() const; + + Signal bufferReady; + +private: + void onBufferReady(FrameBuffer *buffer); + + int internalPrepareBuffer(FrameBuffer *buffer, uint32_t *sequence = nullptr); + int internalPreparedBuffer(); + void internalPostprocessedBuffer(); + + std::map> bufferState_; + std::map expectedSequence_; + std::vector> buffers_; + SequenceSyncHelper syncHelper_; + uint32_t nextSequence_; + std::string name_; + bool ownsBuffers_; + bool hasBuffers_; + int flags_; + std::unique_ptr delegate_; +}; + +} /* namespace libcamera */ diff --git a/include/libcamera/internal/meson.build b/include/libcamera/internal/meson.build index 73635e31bad9..c44c57949382 100644 --- a/include/libcamera/internal/meson.build +++ b/include/libcamera/internal/meson.build @@ -4,6 +4,7 @@ subdir('tracepoints') libcamera_internal_headers = files([ 'bayer_format.h', + 'buffer_queue.h', 'byte_stream_buffer.h', 'camera.h', 'camera_controls.h', diff --git a/src/libcamera/buffer_queue.cpp b/src/libcamera/buffer_queue.cpp new file mode 100644 index 000000000000..b92b633ca04d --- /dev/null +++ b/src/libcamera/buffer_queue.cpp @@ -0,0 +1,541 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2025, Ideas on Board + * + * BufferQueue implementation + */ + +#include "libcamera/internal/buffer_queue.h" + +#include + +namespace libcamera { + +LOG_DEFINE_CATEGORY(BufferQueue) + +/** + * \struct BufferQueueDelegateBase + * \brief Abstract class that defines the delegate interface for the BufferQueue + * + * The BufferQueue needs to have access to the buffer handling functions of the + * object that wraps the underlying device. This is implemented by means of a + * delegate that is passed into the buffer queue at construction time. In most + * cases the \a BufferQueueDelegate template is sufficient. + * + * \fn BufferQueueDelegateBase::allocateBuffers + * \copydoc V4L2VideoDevice::allocateBuffers + * + * \fn BufferQueueDelegateBase::importBuffers + * \copydoc V4L2VideoDevice::importBuffers + * + * \fn BufferQueueDelegateBase::releaseBuffers + * \copydoc V4L2VideoDevice::releaseBuffers + * + * \fn BufferQueueDelegateBase::queueBuffer + * \brief Queue a buffer to the device + * \param[in] buffer The buffer to be queued + * Call queueBuffer on the underlying device. + * + * \see V4L2VideoDevice::queueBuffer + * + * \var BufferQueueDelegateBase::bufferReady + * \copydoc V4L2VideoDevice::bufferReady + * + * \struct BufferQueueDelegate + * \brief Template implementation of the BufferQueueDelegateBase interface + * + * This class implements the BufferQueueDelegateBase interface and forwards it + * to an object that provides the same functions (like V4L2VideoDevice). + * + * A pointer to the target object is passed in at contruction time. The user is + * responsible to ensure that the target object exists for the whole lifetime of + * the delegate. + * + * \fn BufferQueueDelegate::BufferQueueDelegate + * \param target The target to delegate to + * + * \class BufferQueue + * \brief Helper class to handle buffer queues + * + * Handling buffer queues is a common task when dealing with V4L2 video device. + * Depending on the specific use case, additional functionalities are required: + * - Supporting internally allocated and imported buffers + * - Estimate the sequence number that a buffer will have after dequeuing + * - Correct for errors in that sequence scheme due to frames beeing dropped in + * the kernel. + * - Optionally adding a preparation stage for a buffer before it get's queued + * to the device + * - Optionally adding a postprocessing stage after dequeueing + * + * This class encapsulates these functionalities in one component. To support + * arbitrary V4L2VideoDevice like classes, the actual access to the buffer + * related functions is handled through the BufferQueueDelegateBase interface + * with BufferQueueDelegate providing a default implementation that works with + * the V4L2VideDevice class. + * + * On construction time, it must be specified if a preparation stage and/or a + * postprocessing stage shall be added. + * + * Internally there are 4 queues, a buffer can be in: + * Idle, Preparing, Capturing, Postprocessing. + * + * After creation of the BufferQueue it is important to route all buffer related + * actions through the queue. That is: + * 1. Allocation/import of buffers + * 2. Queueing buffers + * 3. Releasing buffers + * 4. Handling the bufferReady signal. + * + * The callable functions depend on the flags passed to the constructor and are + * shown on the following state diagram. + * + * *------* + * | Idle | + * *------* + * | +----------------+ + * +-- prepareBuffer() ---->| Preparing | + * | +----------------+ + * | | + * (no prepare stage) preparedBuffer() + * | | + * | +----------------+ + * \-- queueBuffer() ------>| Capturing | + * +----------------+ + * /--- buffer was cancelled ------/ | + * | | + * | +----------------+ + * | | Postprocessing | --> bufferReady signal + * | +----------------+ + * |/--(no postprocess stage)------/ | + * | | postprocessedBuffer() + * | /--------------------------------/ + * *------* + * | Idle | + * *------* + * + * Notes: + * - If buffers are not allocated by the queue but imported they never end up in + * the idle queue but are passed in by prepareBuffer()/queueBuffer() and leave + * the Queue after postprocessing. + * - If a preparing stage is used, queueBuffer can not be called. + * - If the postprocessing stage is disabled, it will still be used while + * emitting the bufferReady signal but the transition to idle happens + * automatically afterwards. + */ + +/** + * \enum BufferQueue::State + * \brief The states a buffer can be in + * + * \var BufferQueue::Idle + * \brief Buffer is not queued + * + * \var BufferQueue::Preparing + * \brief Buffer is beeing prepared for queueing + * + * \var BufferQueue::Capturing + * \brief Buffer is queued in + * + * \var BufferQueue::Postprocessing + * \brief Buffer beeing postprocessed + */ + +/** + * \enum BufferQueue::Flags + * \brief Flags for a BufferQueue + * + * \var BufferQueue::PrepareStage + * \brief The queue has a prepare stage + * + * \var BufferQueue::PostprocessStage + * \brief The queue has a postprocess stage + */ + +/** + * \brief Construct a BufferQueue + * \param[in] delegate The delegate + * \param[in] flags Optional flags + * \param[in] name Optional name + * + * Construct a buffer queue using the given delegate to forward the buffer + * handling to. The default queue has only Idle and queued states. This can be + * changed using the \a flags parameter, to either add a prepare stage, a + * postprocessing stage or both. + */ +BufferQueue::BufferQueue(std::unique_ptr &&delegate, + int flags, std::string name) + : nextSequence_(0), name_(std::move(name)), ownsBuffers_(false), + hasBuffers_(false), flags_(flags), delegate_(std::move(delegate)) +{ + delegate_->bufferReady.connect(this, &BufferQueue::onBufferReady); +} + +/** + * \brief Allocate buffers + * \param[in] count The number of buffers to allocate + * + * This function allocates \a count buffers by calling allocateBuffers() on the + * delegate and forwarding the return value. A non negative return code is + * treated as success. The buffers are owned by the BufferQueue. + * + * \return The value returned by the BufferQueueDelegateBase::allocateBuffers() + */ +int BufferQueue::allocateBuffers(unsigned int count) +{ + ASSERT(!hasBuffers_); + buffers_.clear(); + int ret = delegate_->allocateBuffers(count, &buffers_); + if (ret < 0) + return ret; + + for (const auto &buffer : buffers_) + bufferState_[Idle].push_back(buffer.get()); + + hasBuffers_ = true; + ownsBuffers_ = true; + return ret; +} + +/** + * \brief Import buffers + * \param[in] count The number of buffers to import + * + * This function imports \a count buffers by calling importBuffers() on the + * delegate and forwarding the return value. A non negative return code is + * treated as success. + * + * \return The value returned by the BufferQueueDelegateBase::importBuffers() + */ +int BufferQueue::importBuffers(unsigned int count) +{ + int ret = delegate_->importBuffers(count); + if (ret < 0) + return ret; + + hasBuffers_ = true; + ownsBuffers_ = false; + return 0; +} + +/** + * \brief Get the necessary correction + * + * \return The offset to add to get in sync again + */ +int BufferQueue::sequenceCorrection() +{ + return syncHelper_.correction(); +} + +/** + * \brief Get the seqence of the next buffer + * + * \return The sequence including necessary corrections + */ +uint32_t BufferQueue::nextSequence() +{ + return nextSequence_ + syncHelper_.correction(); +} + +/** + * \brief Move the next buffer to prepare state + * \param[out] sequence The expected sequence of the buffer + * + * This function moves the front buffer from the idle queue to prepare queue. If + * \a sequence is provided it is set to the expected sequence number of that + * buffer. + * + * \note This function must only be called if the queue has a prepare state and + * owns the buffers. + * + * \return 0 on success, a negative error code otherwise + */ +int BufferQueue::prepareBuffer(uint32_t *sequence) +{ + ASSERT(hasBuffers_); + ASSERT(ownsBuffers_); + ASSERT(flags_ & PrepareStage); + ASSERT(!bufferState_[Idle].empty()); + + FrameBuffer *buffer = bufferState_[Idle].front(); + return prepareBuffer(buffer, sequence); +} + +/** + * \brief Move a buffer to prepare state + * \param[in] buffer The buffer + * \param[out] sequence The expected sequence of the buffer + * + * This function moves \a buffer to prepare queue. If + * \a sequence is provided it is set to the expected sequence number of that + * buffer. + * + * \note This function must only be called if the queue has a prepare state. If + * the queue owns the buffer, \a buffer must point to the front buffer of the + * idle queue. + * + * \return 0 on success, a negative error code otherwise + */ +int BufferQueue::prepareBuffer(FrameBuffer *buffer, uint32_t *sequence) +{ + ASSERT(flags_ & PrepareStage); + + return internalPrepareBuffer(buffer, sequence); +} + +/** + * \brief Exit prepare state + * + * This function pops the frontmost buffer from the prepare queue and queues it + * on the underlying device by calling queueBuffer() on the delegate. + * + * \note This function must only be called if the queue has a prepare state. + * + * \return 0 on success, a negative error code otherwise + */ +int BufferQueue::preparedBuffer() +{ + ASSERT(flags_ & PrepareStage); + + return internalPreparedBuffer(); +} + +/** + * \brief Queue the next buffer + * \param[out] sequence The expected sequence of the buffer + * + * This function queues the front buffer from the idle queue to the underlying + * device ba calling queueBuffer() on the delegate. If \a sequence is provided + * it is set to the expected sequence number of that buffer. + * + * \note This function must only be called if the queue does not have a prepare + * state and owns the buffers. + * + * \return 0 on success, a negative error code otherwise + */ +int BufferQueue::queueBuffer(uint32_t *sequence) +{ + ASSERT(hasBuffers_); + ASSERT(ownsBuffers_); + ASSERT(!bufferState_[Idle].empty()); + + FrameBuffer *buffer = bufferState_[Idle].front(); + return queueBuffer(buffer, sequence); +} + +/** + * \brief Queue a buffer + * \param[in] buffer The buffer + * \param[out] sequence The expected sequence of the buffer + * + * This function queues \a buffer to the underlying device ba calling + * queueBuffer() on the delegate. If \a sequence is provided it is set to the + * expected sequence number of that buffer. + * + * \note This function must only be called if the queue does not have a prepare + * state. If the queue owns the buffers, \a buffer must point to the front + * buffer of the idle queue. + * + * \return 0 on success, a negative error code otherwise + */ +int BufferQueue::queueBuffer(FrameBuffer *buffer, uint32_t *sequence) +{ + ASSERT(!(flags_ & PrepareStage)); + return internalPrepareBuffer(buffer, sequence); +} + +/** + * \brief Exit postprocessed state + * + * This function pops the frontmost buffer from the postprocess queue and puts it + * back to the idle queue in case the buffers are owned by the queue + * + * \note This function must only be called if the queue has a prepare state. + */ +void BufferQueue::postprocessedBuffer() +{ + ASSERT(hasBuffers_); + ASSERT(flags_ & PostprocessStage); + return internalPostprocessedBuffer(); +} + +/** + * \brief Release buffers + * + * This function releases the allocated or imported buffers by calling + * releaseBuffers() on the delegate. + * + * \return 0 on success, a negative error code otherwise + */ +int BufferQueue::releaseBuffers() +{ + ASSERT(bufferState_[BufferQueue::Idle].size() == buffers_.size()); + ASSERT(empty(Preparing) && empty(Capturing) && empty(Postprocessing)); + + bufferState_[BufferQueue::Idle] = {}; + buffers_.clear(); + hasBuffers_ = false; + syncHelper_.reset(); + nextSequence_ = 0; + + return delegate_->releaseBuffers(); +} + +/** + * \brief Check if queue is empty + * \param[in] state The state + * + * \return True if the queue for the given state is empty, false otherwise + */ +bool BufferQueue::empty(BufferQueue::State state) +{ + return bufferState_[state].empty(); +} + +/** + * \brief Get the front buffer of a queue + * \param[in] state The state + * + * \return The front buffer of the queue, or null otherwise + */ +FrameBuffer *BufferQueue::front(BufferQueue::State state) +{ + if (empty(state)) + return nullptr; + return bufferState_[state].front(); +} + +/** + * \brief Get the expected sequence for a buffer + * \param[in] buffer The buffer + * + * \return The expected sequence + */ +unsigned int BufferQueue::expectedSequence(FrameBuffer *buffer) const +{ + auto it = expectedSequence_.find(buffer); + ASSERT(it != expectedSequence_.end()); + return it->second; +} + +/** + * \brief Get the allocated buffers + * + * \return The buffers owned by the queue + */ +const std::vector> &BufferQueue::buffers() const +{ + return buffers_; +} + +/** + * \var BufferQueue::bufferReady + * \brief A Signal emitted when a framebuffer completes + * + * When this signal is emitted the buffer will be in Postprocessing state. If + * the queue was constructed without a postprocessing stage, the buffer will + * automatically move to the idle state after the signal was emitted. Otherwise + * it will stay in postprocessing state until postprocessedBuffer() is called. + */ + +void BufferQueue::onBufferReady(FrameBuffer *buffer) +{ + ASSERT(!empty(Capturing)); + + auto &meta = buffer->metadata(); + auto &queue = bufferState_[Capturing]; + + /* + * V4L2 does not guarantee that buffers are dequeued in order. We expect + * drivers to usually do so, and therefore warn, if a buffer is returned + * out of order. After streamoff V4L2VideoDevice returns the buffers in + * arbitrary order so there is no warning needed in that case. + */ + auto it = std::find(queue.begin(), queue.end(), buffer); + ASSERT(it != queue.end()); + + if (it != queue.begin() && + meta.status != FrameMetadata::FrameCancelled) + LOG(BufferQueue, Warning) << name_ << ": Dequeued buffer out of order " << buffer; + + queue.erase(it); + if (meta.status == FrameMetadata::FrameCancelled) { + syncHelper_.cancelFrame(); + if (ownsBuffers_) + bufferState_[Idle].push_back(buffer); + } else { + syncHelper_.receivedFrame(expectedSequence_[buffer], meta.sequence); + bufferState_[Postprocessing].push_back(buffer); + } + + bufferReady.emit(buffer); + + if (!(flags_ & PostprocessStage) && + meta.status != FrameMetadata::FrameCancelled) + internalPostprocessedBuffer(); +} + +int BufferQueue::internalPrepareBuffer(FrameBuffer *buffer, uint32_t *sequence) +{ + ASSERT(hasBuffers_); + + if (ownsBuffers_) { + ASSERT(!bufferState_[Idle].empty()); + ASSERT(bufferState_[Idle].front() == buffer); + } + + LOG(BufferQueue, Debug) << name_ << ":Buffer prepare: " + << buffer; + int correction = syncHelper_.correction(); + nextSequence_ += correction; + expectedSequence_[buffer] = nextSequence_; + if (ownsBuffers_) + bufferState_[Idle].pop_front(); + bufferState_[Preparing].push_back(buffer); + syncHelper_.pushCorrection(correction); + + if (sequence) + *sequence = nextSequence_; + + nextSequence_++; + + if (!(flags_ & PrepareStage)) + return internalPreparedBuffer(); + + return 0; +} + +int BufferQueue::internalPreparedBuffer() +{ + ASSERT(!bufferState_[Preparing].empty()); + + auto &srcQueue = bufferState_[Preparing]; + FrameBuffer *buffer = srcQueue.front(); + + srcQueue.pop_front(); + int ret = delegate_->queueBuffer(buffer); + if (ret < 0) { + LOG(BufferQueue, Error) << "Failed to queue buffer: " + << strerror(-ret); + if (ownsBuffers_) + bufferState_[Idle].push_back(buffer); + return ret; + } + + LOG(BufferQueue, Debug) << name_ << " Queued buffer: " << buffer; + + bufferState_[Capturing].push_back(buffer); + return 0; +} + +void BufferQueue::internalPostprocessedBuffer() +{ + ASSERT(!empty(Postprocessing)); + + FrameBuffer *buffer = bufferState_[Postprocessing].front(); + bufferState_[Postprocessing].pop_front(); + if (ownsBuffers_) + bufferState_[Idle].push_back(buffer); +} + +} /* namespace libcamera */ diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build index 038d2dbf12cf..1ea97ae997bf 100644 --- a/src/libcamera/meson.build +++ b/src/libcamera/meson.build @@ -18,6 +18,7 @@ libcamera_public_sources = files([ libcamera_internal_sources = files([ 'bayer_format.cpp', + 'buffer_queue.cpp', 'byte_stream_buffer.cpp', 'camera_controls.cpp', 'camera_lens.cpp', From patchwork Mon Sep 14 14:02:41 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28272 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 78155C3360 for ; Mon, 14 Sep 2026 14:04:43 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E5F5C686FB; Mon, 14 Sep 2026 16:04:42 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="TPn7rpR9"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6DF47686F7 for ; Mon, 14 Sep 2026 16:04:41 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 6FD33512; Mon, 14 Sep 2026 16:03:01 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394581; bh=t0/5bYlq3JLuEzL8xDuU2tytwuQt3BrBw9i1iuoG2VM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TPn7rpR9rwGkyJFMQ8FOfDfT79JWGymuxIoDBkz6NEwtsnFOXjn4R3K+IhJRBNNJx L2iy2VeF9kNam2RjcRKLF/Xuo/yaIOl/8T1lgp3NL0k3eeFVXnoZ03Gp8eJJ8HiQge QcZ2bn0ySle6PrlNPgvyNwuBg40ahjzpTBL3N/lE= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 28/41] pipeline: rkisp1: Decouple image, stats and param buffers Date: Mon, 14 Sep 2026 16:02:41 +0200 Message-ID: <20260914140309.3354666-29-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The current code creates FrameInfo objects that tie together params, stats and image buffers and expect that these always stay in these groupings. However there are cases where these sequences get out of sync (e.g. a scratch buffer is used for an image or a params buffer was too late and therefore gets applied one frame later). As these situations are timing related and cpu dependent it is impossible to guarantee the initial grouping of buffers. Split the buffers into separate queues for stats, images and params. Resynchronize on image buffers as soon as they are dequeued from V4L2 as these are the only buffers tied to the libcamera requests coming from the user application. Now handle the other buffer types according to their specific properties: Stats buffers only need to be tracked after dequeueing and can be tied to the corresponding request after the corresponding image buffer was dequeued. If params buffers get out of sync we can either inject the same set of parameters twice or skip one set of params. If image buffers get out of sync we need to update the expected sensor sequence, so that the next request is assigned the correct sensor sequence. Signed-off-by: Stefan Klug --- Changes in v3: - Removed repeated lookups on sensorFrameInfos_[sequence] - Improved a log message - Added fixed from v1 review by Paul - Prevent unnecessary calls to process() just to fill the metadata by marking them as lost in sensorFrameInfos_ and only call process() on demand - Fixed a crash that happened after queuing a buffer to the dewarper failed - Improved complete request log message Changes in v2: - Mostly cosmetic changes Changes in v1: - Moved variables resets in start() further to the top Changes in v0.6: - Fixed multiple assertions on start/stop and a few corner cases - Fixed crash in dewarpRequestReady on stop Changes in v0.5 - Fixed possible use-after-free in RequestInfo --- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 766 +++++++++++++++-------- 1 file changed, 492 insertions(+), 274 deletions(-) diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index d84044f218b5..45082659c931 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -6,6 +6,8 @@ */ #include +#include +#include #include #include #include @@ -44,6 +46,7 @@ #include "libcamera/internal/media_pipeline.h" #include "libcamera/internal/pipeline_handler.h" #include "libcamera/internal/request.h" +#include "libcamera/internal/sequence_sync_helper.h" #include "libcamera/internal/v4l2_subdevice.h" #include "libcamera/internal/v4l2_videodevice.h" #include "libcamera/internal/yaml_parser.h" @@ -53,50 +56,19 @@ namespace libcamera { LOG_DEFINE_CATEGORY(RkISP1) +LOG_DEFINE_CATEGORY(RkISP1Schedule) class PipelineHandlerRkISP1; class RkISP1CameraData; -struct RkISP1FrameInfo { - unsigned int frame; - Request *request; - FrameBuffer *paramBuffer; - FrameBuffer *statBuffer; - FrameBuffer *mainPathBuffer; - FrameBuffer *selfPathBuffer; - - bool paramDequeued; - bool metadataProcessed; -}; - -class RkISP1Frames -{ -public: - RkISP1Frames(PipelineHandler *pipe); - - RkISP1FrameInfo *create(const RkISP1CameraData *data, Request *request, - bool isRaw); - int destroy(unsigned int frame); - void clear(); - - RkISP1FrameInfo *find(unsigned int frame); - RkISP1FrameInfo *find(FrameBuffer *buffer); - RkISP1FrameInfo *find(Request *request); - -private: - void recycleBuffers(const RkISP1FrameInfo &info); - - PipelineHandlerRkISP1 *pipe_; - std::map frameInfo_; -}; class RkISP1CameraData : public Camera::Private { public: RkISP1CameraData(PipelineHandler *pipe, RkISP1MainPath *mainPath, RkISP1SelfPath *selfPath) - : Camera::Private(pipe), frame_(0), frameInfo_(pipe), + : Camera::Private(pipe), frame_(0), mainPath_(mainPath), selfPath_(selfPath), canUseDewarper_(false), usesDewarper_(false) { @@ -110,9 +82,12 @@ public: Stream selfPathStream_; std::unique_ptr sensor_; std::unique_ptr delayedCtrls_; + /* + * The sensor frame sequence of the last request queued to the pipeline + * handler. + */ unsigned int frame_; std::vector ipaBuffers_; - RkISP1Frames frameInfo_; RkISP1MainPath *mainPath_; RkISP1SelfPath *selfPath_; @@ -163,21 +138,55 @@ private: Transform combinedTransform_; }; +struct SensorFrameInfo { + Request *request = nullptr; + FrameBuffer *statsBuffer = nullptr; + ControlList metadata; + bool metadataProcessed = false; + bool statsLost = false; +}; + +struct RequestInfo { + Request *request = nullptr; + /* + * The estimated sensor sequence for this request. Only reliable when + * sequenceValid is true + */ + size_t sequence = 0; + bool sequenceValid = false; +}; + +struct ParamBufferInfo { + FrameBuffer *buffer = nullptr; + size_t expectedSequence = 0; +}; + +struct DewarpBufferInfo { + FrameBuffer *inputBuffer; + FrameBuffer *outputBuffer; +}; + namespace { /* - * Maximum number of requests that shall be queued into the pipeline to keep - * the regulation fast. - * \todo This needs revisiting as soon as buffers got decoupled from requests - * and/or a fast path for controls was implemented. + * This many buffers ensures that the pipeline runs smoothly, without frame + * drops. */ -static constexpr unsigned int kRkISP1MaxQueuedRequests = 4; +static constexpr unsigned int kRkISP1MinBufferCount = 6; /* - * This many internal buffers (or rather parameter and statistics buffer - * pairs) ensures that the pipeline runs smoothly, without frame drops. + * This many internal buffers (params and stats) are needed for smooth operation + * \todo In high framerate or high cpu load situations it might be necessary to + * increase this number. \todo: This also relates to max sensor delay and must + * always be >= maxSensor delay */ -static constexpr unsigned int kRkISP1MinBufferCount = 4; +static constexpr unsigned int kRkISP1InternalBufferCount = 4; + +/* + * This many internal image buffers between ISP and dewarper are needed for + * smooth operation. + */ +static constexpr unsigned int kRkISP1DewarpImageBufferCount = 4; } /* namespace */ @@ -210,18 +219,20 @@ private: friend RkISP1CameraData; friend RkISP1CameraConfiguration; - friend RkISP1Frames; int initLinks(Camera *camera, const RkISP1CameraConfiguration &config); int createCamera(MediaEntity *sensor); - void tryCompleteRequest(RkISP1FrameInfo *info); - void cancelDewarpRequest(RkISP1FrameInfo *info); + void tryCompleteRequests(); + void cancelDewarpRequest(Request *request); void imageBufferReady(FrameBuffer *buffer); void paramBufferReady(FrameBuffer *buffer); void statBufferReady(FrameBuffer *buffer); void dewarpBufferReady(FrameBuffer *buffer); void frameStart(uint32_t sequence); + void queueInternalBuffers(); + void computeParamBuffers(uint32_t maxSequence); + int allocateBuffers(Camera *camera); int freeBuffers(Camera *camera); @@ -244,144 +255,30 @@ private: std::vector> mainPathBuffers_; std::queue availableMainPathBuffers_; + bool running_ = false; + std::vector> paramBuffers_; std::vector> statBuffers_; std::queue availableParamBuffers_; std::queue availableStatBuffers_; + std::deque queuedRequests_; + + std::map sensorFrameInfos_; + + std::deque queuedDewarpBuffers_; + SequenceSyncHelper paramsSyncHelper_; + SequenceSyncHelper imageSyncHelper_; + + std::queue computingParamBuffers_; + std::queue queuedParamBuffers_; + + uint32_t nextParamsSequence_; + uint32_t nextStatsToProcess_; + Camera *activeCamera_; }; -RkISP1Frames::RkISP1Frames(PipelineHandler *pipe) - : pipe_(static_cast(pipe)) -{ -} - -RkISP1FrameInfo *RkISP1Frames::create(const RkISP1CameraData *data, Request *request, - bool isRaw) -{ - unsigned int frame = data->frame_; - - FrameBuffer *paramBuffer = nullptr; - FrameBuffer *statBuffer = nullptr; - FrameBuffer *mainPathBuffer = nullptr; - FrameBuffer *selfPathBuffer = nullptr; - - if (!isRaw) { - if (pipe_->availableParamBuffers_.empty()) { - LOG(RkISP1, Error) << "Parameters buffer underrun"; - return nullptr; - } - - if (pipe_->availableStatBuffers_.empty()) { - LOG(RkISP1, Error) << "Statistic buffer underrun"; - return nullptr; - } - - paramBuffer = pipe_->availableParamBuffers_.front(); - pipe_->availableParamBuffers_.pop(); - - statBuffer = pipe_->availableStatBuffers_.front(); - pipe_->availableStatBuffers_.pop(); - - if (data->usesDewarper_) { - mainPathBuffer = pipe_->availableMainPathBuffers_.front(); - pipe_->availableMainPathBuffers_.pop(); - } - } - - if (!mainPathBuffer) - mainPathBuffer = request->findBuffer(&data->mainPathStream_); - selfPathBuffer = request->findBuffer(&data->selfPathStream_); - - auto [it, inserted] = frameInfo_.try_emplace(frame); - ASSERT(inserted); - - auto &info = it->second; - - info.frame = frame; - info.request = request; - info.paramBuffer = paramBuffer; - info.mainPathBuffer = mainPathBuffer; - info.selfPathBuffer = selfPathBuffer; - info.statBuffer = statBuffer; - info.paramDequeued = false; - info.metadataProcessed = false; - - return &info; -} - -int RkISP1Frames::destroy(unsigned int frame) -{ - auto it = frameInfo_.find(frame); - if (it == frameInfo_.end()) - return -ENOENT; - - recycleBuffers(it->second); - frameInfo_.erase(it); - - return 0; -} - -void RkISP1Frames::clear() -{ - for (const auto &[frame, info] : frameInfo_) - recycleBuffers(info); - - frameInfo_.clear(); -} - -void RkISP1Frames::recycleBuffers(const RkISP1FrameInfo &info) -{ - if (info.paramBuffer) - pipe_->availableParamBuffers_.push(info.paramBuffer); - - if (info.statBuffer) - pipe_->availableStatBuffers_.push(info.statBuffer); - - if (info.mainPathBuffer && pipe_->cameraData(pipe_->activeCamera_)->usesDewarper_) - pipe_->availableMainPathBuffers_.push(info.mainPathBuffer); -} - -RkISP1FrameInfo *RkISP1Frames::find(unsigned int frame) -{ - auto itInfo = frameInfo_.find(frame); - - if (itInfo != frameInfo_.end()) - return &itInfo->second; - - LOG(RkISP1, Fatal) << "Can't locate info from frame"; - - return nullptr; -} - -RkISP1FrameInfo *RkISP1Frames::find(FrameBuffer *buffer) -{ - for (auto &[frame, info] : frameInfo_) { - if (info.paramBuffer == buffer || - info.statBuffer == buffer || - info.mainPathBuffer == buffer || - info.selfPathBuffer == buffer) - return &info; - } - - LOG(RkISP1, Fatal) << "Can't locate info from buffer"; - - return nullptr; -} - -RkISP1FrameInfo *RkISP1Frames::find(Request *request) -{ - for (auto &[frame, info] : frameInfo_) { - if (info.request == request) - return &info; - } - - LOG(RkISP1, Fatal) << "Can't locate info from request"; - - return nullptr; -} - PipelineHandlerRkISP1 *RkISP1CameraData::pipe() { return static_cast(Camera::Private::pipe()); @@ -480,31 +377,34 @@ void RkISP1CameraData::paramsComputed(unsigned int frame, unsigned int bytesused) { PipelineHandlerRkISP1 *pipe = RkISP1CameraData::pipe(); - RkISP1FrameInfo *info = frameInfo_.find(frame); - if (!info) - return; + ParamBufferInfo &pInfo = pipe->computingParamBuffers_.front(); + pipe->computingParamBuffers_.pop(); - info->paramBuffer->_d()->metadata().planes()[0].bytesused = bytesused; + ASSERT(pInfo.expectedSequence == frame); + FrameBuffer *buffer = pInfo.buffer; - int ret = pipe->param_->queueBuffer(info->paramBuffer); + LOG(RkISP1Schedule, Debug) << "Queue params for " << frame << " " << buffer; + + buffer->_d()->metadata().planes()[0].bytesused = bytesused; + int ret = pipe->param_->queueBuffer(buffer); if (ret < 0) { LOG(RkISP1, Error) << "Failed to queue parameter buffer: " << strerror(-ret); + pipe->availableParamBuffers_.push(buffer); return; } - pipe->stat_->queueBuffer(info->statBuffer); - - if (info->mainPathBuffer) - mainPath_->queueBuffer(info->mainPathBuffer); - - if (selfPath_ && info->selfPathBuffer) - selfPath_->queueBuffer(info->selfPathBuffer); + pipe->queuedParamBuffers_.push({ buffer, frame }); } void RkISP1CameraData::setSensorControls(unsigned int frame, const ControlList &sensorControls) { + /* We know delayed controls is prewarmed for frame 0 */ + if (frame == 0) + return; + + LOG(RkISP1Schedule, Debug) << "DelayedControls push " << frame; delayedCtrls_->push(frame, sensorControls); } @@ -512,14 +412,63 @@ void RkISP1CameraData::metadataReady(unsigned int frame, [[maybe_unused]] unsigned int bufferId, const ControlList &metadata) { - RkISP1FrameInfo *info = frameInfo_.find(frame); - if (!info) - return; + PipelineHandlerRkISP1 *pipe = RkISP1CameraData::pipe(); - info->request->_d()->metadata().merge(metadata); - info->metadataProcessed = true; + LOG(RkISP1Schedule, Debug) << " metadataReady " << frame; - pipe()->tryCompleteRequest(info); + auto &info = pipe->sensorFrameInfos_[frame]; + + /* + * We don't necessarily know the request for that sequence number, + * as the dequeue of the image buffer might not have happened yet. + * So we check all known requests and store the metadata otherwise. + */ + for (auto &reqInfo : pipe->queuedRequests_) { + if (!reqInfo.sequenceValid) { + LOG(RkISP1Schedule, Debug) + << "Need to store metadata for later " << frame; + info.metadata = metadata; + break; + } + + if (frame > reqInfo.sequence) { + /* + * We will never get stats for that request. Log an + * error and return it. + */ + LOG(RkISP1, Warning) + << "Stats for frame " << reqInfo.sequence + << " got lost"; + auto &info2 = pipe->sensorFrameInfos_[reqInfo.sequence]; + info2.metadataProcessed = true; + ASSERT(info2.statsBuffer == nullptr); + continue; + } + + if (frame == reqInfo.sequence) { + reqInfo.request->_d()->metadata().merge(metadata); + break; + } + + /* We should never end up here */ + LOG(RkISP1, Error) << "Request for sequence " << frame + << " is already handled. Metadata was too late"; + + break; + } + + info.metadataProcessed = true; + /* + * info.statsBuffer can be null, if ipa->processStats() was called + * without a buffer to just fill the metadata. + */ + if (info.statsBuffer) + pipe->availableStatBuffers_.push(info.statsBuffer); + + info.statsBuffer = nullptr; + + pipe->tryCompleteRequests(); + pipe->queueInternalBuffers(); } /* ----------------------------------------------------------------------------- @@ -786,7 +735,7 @@ CameraConfiguration::Status RkISP1CameraConfiguration::validate() */ PipelineHandlerRkISP1::PipelineHandlerRkISP1(CameraManager *manager) - : PipelineHandler(manager, kRkISP1MaxQueuedRequests), hasSelfPath_(true) + : PipelineHandler(manager), hasSelfPath_(true) { } @@ -1158,18 +1107,19 @@ int PipelineHandlerRkISP1::allocateBuffers(Camera *camera) } }; if (!isRaw_) { - ret = param_->allocateBuffers(kRkISP1MinBufferCount, ¶mBuffers_); + ret = param_->allocateBuffers(kRkISP1InternalBufferCount, ¶mBuffers_); if (ret < 0) return ret; - ret = stat_->allocateBuffers(kRkISP1MinBufferCount, &statBuffers_); + ret = stat_->allocateBuffers(kRkISP1InternalBufferCount, &statBuffers_); if (ret < 0) return ret; } /* If the dewarper is being used, allocate internal buffers for ISP. */ if (data->usesDewarper_) { - ret = mainPath_.exportBuffers(kRkISP1MinBufferCount, &mainPathBuffers_); + ret = mainPath_.exportBuffers(kRkISP1DewarpImageBufferCount, + &mainPathBuffers_); if (ret < 0) return ret; @@ -1244,6 +1194,12 @@ int PipelineHandlerRkISP1::start(Camera *camera, [[maybe_unused]] const ControlL return ret; actions += [&]() { freeBuffers(camera); }; + paramsSyncHelper_.reset(); + imageSyncHelper_.reset(); + nextParamsSequence_ = 0; + nextStatsToProcess_ = 0; + data->frame_ = 0; + ipa::rkisp1::StartResult res; data->ipa_->start(controls ? *controls : ControlList{ controls::controls }, &res); @@ -1256,8 +1212,6 @@ int PipelineHandlerRkISP1::start(Camera *camera, [[maybe_unused]] const ControlL data->sensor_->setControls(&res.controls); data->delayedCtrls_->reset(); - data->frame_ = 0; - if (!isRaw_) { ret = param_->streamOn(); if (ret) { @@ -1304,6 +1258,9 @@ int PipelineHandlerRkISP1::start(Camera *camera, [[maybe_unused]] const ControlL isp_->setFrameStartEnabled(true); activeCamera_ = camera; + running_ = true; + + queueInternalBuffers(); actions.release(); return 0; @@ -1313,6 +1270,9 @@ void PipelineHandlerRkISP1::stopDevice(Camera *camera) { RkISP1CameraData *data = cameraData(camera); int ret; + running_ = false; + + LOG(RkISP1Schedule, Debug) << "Stop device"; isp_->setFrameStartEnabled(false); @@ -1333,46 +1293,156 @@ void PipelineHandlerRkISP1::stopDevice(Camera *camera) LOG(RkISP1, Warning) << "Failed to stop parameters for " << camera->id(); + /* + * The param buffers are not returned in order, so the queue + * becomes useless. + */ + queuedParamBuffers_ = {}; + if (data->usesDewarper_) dewarper_->stop(); } - ASSERT(data->queuedRequests_.empty()); - data->frameInfo_.clear(); + tryCompleteRequests(); + + /* + * There can still be requests that are either waiting for metadata or + * that contain buffers which were not yet queued at all. + */ + while (!queuedRequests_.empty()) { + RequestInfo &reqInfo = queuedRequests_.front(); + cancelRequest(reqInfo.request); + queuedRequests_.pop_front(); + } + sensorFrameInfos_.clear(); + + ASSERT(queuedDewarpBuffers_.empty()); + ASSERT(queuedParamBuffers_.empty()); + ASSERT(computingParamBuffers_.empty()); freeBuffers(camera); activeCamera_ = nullptr; } -int PipelineHandlerRkISP1::queueRequestDevice(Camera *camera, Request *request) +void PipelineHandlerRkISP1::queueInternalBuffers() { - RkISP1CameraData *data = cameraData(camera); + if (!running_) + return; - RkISP1FrameInfo *info = data->frameInfo_.create(data, request, isRaw_); - if (!info) - return -ENOENT; + RkISP1CameraData *data = cameraData(activeCamera_); + + while (!availableStatBuffers_.empty()) { + FrameBuffer *buf = availableStatBuffers_.front(); + availableStatBuffers_.pop(); + data->pipe()->stat_->queueBuffer(buf); + } + + /* + * In case of the dewarper, there is a seperate buffer loop for the main + * path + */ + while (!availableMainPathBuffers_.empty()) { + FrameBuffer *buf = availableMainPathBuffers_.front(); + availableMainPathBuffers_.pop(); + + LOG(RkISP1Schedule, Debug) << "Queue mainPath " << buf; + data->mainPath_->queueBuffer(buf); + } +} + +void PipelineHandlerRkISP1::computeParamBuffers(uint32_t maxSequence) +{ + RkISP1CameraData *data = cameraData(activeCamera_); - data->ipa_->queueRequest(data->frame_, request->controls()); if (isRaw_) { - if (info->mainPathBuffer) - data->mainPath_->queueBuffer(info->mainPathBuffer); - - if (data->selfPath_ && info->selfPathBuffer) - data->selfPath_->queueBuffer(info->selfPathBuffer); - /* * Call computeParams with an empty param buffer to trigger the * setSensorControls signal. */ - data->ipa_->computeParams(data->frame_, 0); - } else { - data->ipa_->computeParams(data->frame_, - info->paramBuffer->cookie()); + data->ipa_->computeParams(maxSequence, 0); + return; } + while (nextParamsSequence_ <= maxSequence) { + if (availableParamBuffers_.empty()) { + LOG(RkISP1Schedule, Warning) + << "Ran out of parameter buffers"; + return; + } + + int correction = paramsSyncHelper_.correction(); + if (correction != 0) + LOG(RkISP1Schedule, Warning) + << "Correcting params sequence " + << correction; + + uint32_t paramsSequence; + if (correction >= 0) { + nextParamsSequence_ += correction; + paramsSyncHelper_.pushCorrection(correction); + paramsSequence = nextParamsSequence_++; + } else { + /* + * Inject the same sequence multiple times, to correct + * for the offset. + */ + paramsSyncHelper_.pushCorrection(-1); + paramsSequence = nextParamsSequence_; + } + + FrameBuffer *buf = availableParamBuffers_.front(); + availableParamBuffers_.pop(); + computingParamBuffers_.push({ buf, paramsSequence }); + LOG(RkISP1Schedule, Debug) << "Request params for " << paramsSequence; + data->ipa_->computeParams(paramsSequence, buf->cookie()); + } +} + +int PipelineHandlerRkISP1::queueRequestDevice(Camera *camera, Request *request) +{ + RkISP1CameraData *data = cameraData(camera); + + RequestInfo info; + info.request = request; + + int correction = imageSyncHelper_.correction(); + if (correction != 0) + LOG(RkISP1Schedule, Debug) + << "Correcting image sequence " + << data->frame_ << " to " << data->frame_ + correction; + data->frame_ += correction; + imageSyncHelper_.pushCorrection(correction); + info.sequence = data->frame_; data->frame_++; + LOG(RkISP1Schedule, Debug) << "Queue request. Request sequence: " + << request->sequence() + << " estimated sensor frame sequence: " << info.sequence + << " queue size: " << (queuedRequests_.size() + 1); + + data->ipa_->queueRequest(info.sequence, request->controls()); + + /* + * When the dewarper is used, the request buffers will be queued in + * imageBufferReady() + */ + if (!data->usesDewarper_) { + FrameBuffer *mainPathBuffer = request->findBuffer(&data->mainPathStream_); + FrameBuffer *selfPathBuffer = request->findBuffer(&data->selfPathStream_); + if (mainPathBuffer) + data->mainPath_->queueBuffer(mainPathBuffer); + + if (data->selfPath_ && selfPathBuffer) + data->selfPath_->queueBuffer(selfPathBuffer); + } + + queuedRequests_.push_back(info); + + /* Kickstart computation of parameters. */ + if (info.sequence < kRkISP1InternalBufferCount) + computeParamBuffers(info.sequence); + return 0; } @@ -1526,8 +1596,11 @@ void PipelineHandlerRkISP1::frameStart(uint32_t sequence) return; RkISP1CameraData *data = cameraData(activeCamera_); + LOG(RkISP1Schedule, Debug) << "frameStart " << sequence; uint32_t sequenceToApply = sequence + data->delayedCtrls_->maxDelay(); data->delayedCtrls_->applyControls(sequenceToApply); + + computeParamBuffers(sequenceToApply + 1); } bool PipelineHandlerRkISP1::match(DeviceEnumerator *enumerator) @@ -1605,29 +1678,52 @@ bool PipelineHandlerRkISP1::match(DeviceEnumerator *enumerator) * Buffer Handling */ -void PipelineHandlerRkISP1::tryCompleteRequest(RkISP1FrameInfo *info) +void PipelineHandlerRkISP1::tryCompleteRequests() { - RkISP1CameraData *data = cameraData(activeCamera_); - Request *request = info->request; + std::optional lastDeletedSequence; - if (request->hasPendingBuffers()) + /* Complete finished requests */ + while (!queuedRequests_.empty()) { + RequestInfo info = queuedRequests_.front(); + + if (info.request->hasPendingBuffers()) + break; + + if (!info.sequenceValid) + break; + + if (!sensorFrameInfos_[info.sequence].metadataProcessed) + break; + + queuedRequests_.pop_front(); + + LOG(RkISP1Schedule, Debug) << "Complete request " << info.request->sequence() + << " frame " << info.sequence; + completeRequest(info.request); + + sensorFrameInfos_[info.sequence].request = nullptr; + lastDeletedSequence = info.sequence; + } + + if (!lastDeletedSequence.has_value()) return; - if (!info->metadataProcessed) - return; + /* Drop all outdated sensor frame infos. */ + while (!sensorFrameInfos_.empty()) { + auto iter = sensorFrameInfos_.begin(); + if (iter->first > lastDeletedSequence.value()) + break; - if (!isRaw_ && !info->paramDequeued) - return; + ASSERT(iter->second.request == nullptr); + ASSERT(iter->second.statsBuffer == nullptr); - data->frameInfo_.destroy(info->frame); - - completeRequest(request); + sensorFrameInfos_.erase(iter); + } } -void PipelineHandlerRkISP1::cancelDewarpRequest(RkISP1FrameInfo *info) +void PipelineHandlerRkISP1::cancelDewarpRequest(Request *request) { RkISP1CameraData *data = cameraData(activeCamera_); - Request *request = info->request; /* * i.MX8MP is the only known platform with dewarper. It has * no self path. Hence, only main path buffer completion is @@ -1646,99 +1742,201 @@ void PipelineHandlerRkISP1::cancelDewarpRequest(RkISP1FrameInfo *info) } } - tryCompleteRequest(info); + tryCompleteRequests(); } void PipelineHandlerRkISP1::imageBufferReady(FrameBuffer *buffer) { ASSERT(activeCamera_); RkISP1CameraData *data = cameraData(activeCamera_); - - RkISP1FrameInfo *info = data->frameInfo_.find(buffer); - if (!info) - return; - const FrameMetadata &metadata = buffer->metadata(); - Request *request = info->request; + RequestInfo *reqInfo = nullptr; + + /* + * When the dewarper is used, the buffer is not yet tied to a request, + * so find the first request without a valid sequence. Otherwise find + * the request for that buffer. This is not necessarily the same, + * because after streamoff the buffers are returned in arbitrary order. + */ + for (auto &info : queuedRequests_) { + if (data->usesDewarper_) { + if (!info.sequenceValid) { + reqInfo = &info; + break; + } + } else { + if (info.request == buffer->request()) { + reqInfo = &info; + break; + } + } + } + + if (!reqInfo && data->usesDewarper_) { + LOG(RkISP1Schedule, Info) + << "Image buffer ready, but no corresponding request"; + availableMainPathBuffers_.push(buffer); + return; + } + + /* In the non-dewarper case, there must be a valid request */ + ASSERT(reqInfo); + + Request *request = reqInfo->request; + + LOG(RkISP1Schedule, Debug) << "Image buffer ready: " << buffer + << " Expected sequence: " << reqInfo->sequence + << " got: " << metadata.sequence; + + uint32_t sequence = metadata.sequence; + + /* + * If the frame was cancelled, the metadata sequnce is usually wrong and + * we assume that our guess was right. + */ + if (metadata.status == FrameMetadata::FrameCancelled) + sequence = reqInfo->sequence; + + /* We now know the buffer sequence that belongs to this request */ + int droppedFrames = imageSyncHelper_.receivedFrame(reqInfo->sequence, sequence); + if (droppedFrames != 0) + LOG(RkISP1Schedule, Warning) + << "Frame " << reqInfo->sequence << ": Dropped " + << droppedFrames << " frames"; + + reqInfo->sequence = sequence; + reqInfo->sequenceValid = true; + + auto &frameInfo = sensorFrameInfos_[sequence]; + if (frameInfo.metadataProcessed) { + LOG(RkISP1Schedule, Debug) + << "Apply stored metadata " << sequence; + request->_d()->metadata().merge(frameInfo.metadata); + } + + if (frameInfo.statsLost) { + LOG(RkISP1Schedule, Warning) << "Send empty stats to fill metadata for " + << sequence; + data->ipa_->processStats(nextStatsToProcess_, 0, + data->delayedCtrls_->get(nextStatsToProcess_)); + } if (metadata.status != FrameMetadata::FrameCancelled) { /* * Record the sensor's timestamp in the request metadata. * - * \todo The sensor timestamp should be better estimated by connecting - * to the V4L2Device::frameStart signal. + * \todo The sensor timestamp should be better estimated by + * connecting to the V4L2Device::frameStart signal. */ request->_d()->metadata().set(controls::SensorTimestamp, metadata.timestamp); + /* In raw mode call processStats() to fill the metadata */ if (isRaw_) { const ControlList &ctrls = - data->delayedCtrls_->get(metadata.sequence); - data->ipa_->processStats(info->frame, 0, ctrls); + data->delayedCtrls_->get(sequence); + data->ipa_->processStats(sequence, 0, ctrls); } } else { - if (isRaw_) - info->metadataProcessed = true; + /* No need to block waiting for metadata on a cancelled frame. */ + frameInfo.metadataProcessed = true; } if (!data->usesDewarper_) { - completeBuffer(request, buffer); - tryCompleteRequest(info); + completeBuffer(reqInfo->request, buffer); + tryCompleteRequests(); return; } - /* Do not queue cancelled frames to dewarper. */ + /* Do not queue cancelled frames to the dewarper. */ if (metadata.status == FrameMetadata::FrameCancelled) { - cancelDewarpRequest(info); + cancelDewarpRequest(reqInfo->request); + availableMainPathBuffers_.push(buffer); return; } dewarper_->setControls(&data->mainPathStream_, request->controls()); /* - * Queue input and output buffers to the dewarper. The output - * buffers for the dewarper are the buffers of the request, supplied - * by the application. + * Queue input and output buffers to the dewarper. The output buffers + * for the dewarper are the buffers of the request, supplied by the + * application. */ + DewarpBufferInfo dewarpInfo{ buffer, reqInfo->request->findBuffer(&data->mainPathStream_) }; + LOG(RkISP1Schedule, Debug) << "Queue dewarper " << dewarpInfo.inputBuffer + << " " << dewarpInfo.outputBuffer; + int ret = dewarper_->queueBuffers(buffer, request->buffers()); if (ret < 0) { LOG(RkISP1, Error) << "Failed to queue buffers to dewarper: -" << strerror(-ret); - cancelDewarpRequest(info); - + cancelDewarpRequest(reqInfo->request); + availableMainPathBuffers_.push(buffer); return; } + queuedDewarpBuffers_.push_back(dewarpInfo); dewarper_->populateMetadata(&data->mainPathStream_, request->_d()->metadata()); } void PipelineHandlerRkISP1::dewarpBufferReady(FrameBuffer *buffer) { - ASSERT(activeCamera_); - RkISP1CameraData *data = cameraData(activeCamera_); Request *request = buffer->request(); + const FrameMetadata &metadata = buffer->metadata(); - RkISP1FrameInfo *info = data->frameInfo_.find(buffer->request()); - if (!info) - return; + /* + * After stopping the dewarper, the buffers are returned out of order. + * Search the list for the corresponding info and handle it. In regular + * operation it will always be the first entry. + */ + for (DewarpBufferInfo &dwInfo : queuedDewarpBuffers_) { + if (dwInfo.outputBuffer != buffer) + continue; - completeBuffer(request, buffer); - tryCompleteRequest(info); + availableMainPathBuffers_.push(dwInfo.inputBuffer); + dwInfo.inputBuffer = nullptr; + dwInfo.outputBuffer = nullptr; + + if (metadata.status == FrameMetadata::FrameCancelled) + buffer->_d()->cancel(); + + completeBuffer(request, buffer); + } + + while (!queuedDewarpBuffers_.empty() && + queuedDewarpBuffers_.front().inputBuffer == nullptr) + queuedDewarpBuffers_.pop_front(); + + tryCompleteRequests(); + queueInternalBuffers(); } void PipelineHandlerRkISP1::paramBufferReady(FrameBuffer *buffer) { - ASSERT(activeCamera_); - RkISP1CameraData *data = cameraData(activeCamera_); + LOG(RkISP1Schedule, Debug) << "Param buffer ready " << buffer; - RkISP1FrameInfo *info = data->frameInfo_.find(buffer); - if (!info) + /* + * After stream off, the buffers are returned out of order, so we don't + * care about the rest. + */ + if (!running_) { + availableParamBuffers_.push(buffer); return; + } - info->paramDequeued = true; - tryCompleteRequest(info); + ParamBufferInfo pInfo = queuedParamBuffers_.front(); + queuedParamBuffers_.pop(); + + ASSERT(pInfo.buffer == buffer); + + size_t metaSequence = buffer->metadata().sequence; + LOG(RkISP1Schedule, Debug) << "Params buffer ready " + << " Expected: " << pInfo.expectedSequence + << " got: " << metaSequence; + paramsSyncHelper_.receivedFrame(pInfo.expectedSequence, metaSequence); + availableParamBuffers_.push(buffer); } void PipelineHandlerRkISP1::statBufferReady(FrameBuffer *buffer) @@ -1746,21 +1944,41 @@ void PipelineHandlerRkISP1::statBufferReady(FrameBuffer *buffer) ASSERT(activeCamera_); RkISP1CameraData *data = cameraData(activeCamera_); - RkISP1FrameInfo *info = data->frameInfo_.find(buffer); - if (!info) - return; + size_t sequence = buffer->metadata().sequence; if (buffer->metadata().status == FrameMetadata::FrameCancelled) { - info->metadataProcessed = true; - tryCompleteRequest(info); + LOG(RkISP1Schedule, Warning) << "Stats cancelled " << sequence; + /* + * We can't assume that the sequence of the stat buffer is valid, + * so there is nothing left to do. + */ + availableStatBuffers_.push(buffer); return; } - if (data->frame_ <= buffer->metadata().sequence) - data->frame_ = buffer->metadata().sequence + 1; + LOG(RkISP1Schedule, Debug) << "Stats ready " << sequence; - data->ipa_->processStats(info->frame, info->statBuffer->cookie(), - data->delayedCtrls_->get(buffer->metadata().sequence)); + if (nextStatsToProcess_ != sequence) + LOG(RkISP1Schedule, Warning) << "Stats sequence out of sync." + << " Expected: " << nextStatsToProcess_ + << " got: " << sequence; + + if (nextStatsToProcess_ > sequence) { + LOG(RkISP1Schedule, Warning) << "Stats were too late. Ignored"; + availableStatBuffers_.push(buffer); + return; + } + + while (nextStatsToProcess_ < sequence) + sensorFrameInfos_[nextStatsToProcess_++].statsLost = true; + + sensorFrameInfos_[sequence].statsBuffer = buffer; + + LOG(RkISP1Schedule, Debug) << "Process stats " << sequence; + data->ipa_->processStats(sequence, buffer->cookie(), + data->delayedCtrls_->get(sequence)); + + nextStatsToProcess_++; } REGISTER_PIPELINE_HANDLER(PipelineHandlerRkISP1, "rkisp1") From patchwork Mon Sep 14 14:02:42 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28273 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A3BC2C3361 for ; Mon, 14 Sep 2026 14:04:45 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2697768705; Mon, 14 Sep 2026 16:04:45 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="Aaf5BqZd"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8D74368704 for ; Mon, 14 Sep 2026 16:04:43 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 8D14A9A4; Mon, 14 Sep 2026 16:03:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394583; bh=MZ6ApmSD82YbKyfCR94H1TakDv6al0EAMDlQ2K+uH/Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Aaf5BqZdxSwFsiZok8ssBqFuLNeEpsywyDEsdsTkFAIMmI8tuyrfbL2xmzjDNhoIv A9R9f7fAi8zz7rSxbFw9ky4GLELaGDOK4oR3zLn7tcdTn317iGRzgZxTsnbHN4Heou HzeF0Rh2RcYE1cRX8jmqPm1LA5afHhk1hMqSVhpU= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 29/41] pipeline: rkisp1: Reinstantiate maxQueuedRequestsDevice limit Date: Mon, 14 Sep 2026 16:02:42 +0200 Message-ID: <20260914140309.3354666-30-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" With the pipeline rework, the maxQueuedRequestsDevice should not be necessary anymore, as prepare() and therefore the calculation for the ISP regulation is called only as late as possible when a params buffer was dequeued. However with unlimited maxQueuedRequestsDevice all the incoming requests get immediately queued to the ipa with the sensor sequence number that was anticipated for that request at queueRequestDevice time. Now when the correction tries to mitigate dropped sequence numbers, it will call computeParams() with sensor frame numbers that were not anticipated for the requests queued to the IPA. There might still be a better solution to this, but reinstantiating the limit reduces the effect. Signed-off-by: Stefan Klug --- Changes in v2: - Added this patch --- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index 45082659c931..832f58e95d37 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -735,7 +735,7 @@ CameraConfiguration::Status RkISP1CameraConfiguration::validate() */ PipelineHandlerRkISP1::PipelineHandlerRkISP1(CameraManager *manager) - : PipelineHandler(manager), hasSelfPath_(true) + : PipelineHandler(manager, kRkISP1MinBufferCount), hasSelfPath_(true) { } From patchwork Mon Sep 14 14:02:43 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28274 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2763DC3272 for ; Mon, 14 Sep 2026 14:04:47 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D3C50686F2; Mon, 14 Sep 2026 16:04:46 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="NAV84mGN"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9199F686F2 for ; Mon, 14 Sep 2026 16:04:45 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 9389B9A4; Mon, 14 Sep 2026 16:03:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394585; bh=kiKlq85KJtU0lcsJeo7KzKJVQVlty/L7aIWxMyr6LrI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NAV84mGNcnfTsXR14JTGOhaUGLVjP40hib5ozIXvCkBuWRU8ltYbibU7H4CprpdzB bWFS1RDw5Jx8AAhTg3ewlHBDoItI1GHu6dTaPQYCDDD7fmMkmwhQMvgTjsOdtPiuR5 aICnPf7u/J/xbjTrStefTTuM5qfujtcitM1y9bQU= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 30/41] pipeline: rkisp1: Correctly handle params buffer for frame 0 Date: Mon, 14 Sep 2026 16:02:43 +0200 Message-ID: <20260914140309.3354666-31-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The parameters for frame 0 are only active on frame 0 if the corresponding parameter buffer is queued before STREAMON is called on the ISP. Therefore the normal mechanics of calling ipa->computeParams() do not work, as it gets called after the first request is queued and therefore the ISP is already started. To fix that, handle the parameter buffer the same way the initial sensor controls are handled by passing it to the ipa->start() function and then queuing the params buffer before starting the isp. Signed-off-by: Stefan Klug --- Changes in v3: - Moved bufferId== 0 check out of computeParamsInternal for better readability - setSensorControls is now emitted after paramsComputed but that must not be a problem - Dropped rby tag due to bigger changes Changes in v2: - Small cleanup - Fixed crash in raw case where a null buffer is passed on start - Collected tag --- include/libcamera/ipa/rkisp1.mojom | 5 +++- src/ipa/rkisp1/rkisp1.cpp | 35 ++++++++++++++++++------ src/libcamera/pipeline/rkisp1/rkisp1.cpp | 18 +++++++++++- 3 files changed, 47 insertions(+), 11 deletions(-) diff --git a/include/libcamera/ipa/rkisp1.mojom b/include/libcamera/ipa/rkisp1.mojom index 04230d0f852e..6571a6d096f0 100644 --- a/include/libcamera/ipa/rkisp1.mojom +++ b/include/libcamera/ipa/rkisp1.mojom @@ -17,6 +17,7 @@ struct IPAConfigInfo { struct StartResult { libcamera.ControlList controls; int32 code; + uint32 paramBufferBytesUsed; }; interface IPARkISP1Interface { @@ -25,7 +26,9 @@ interface IPARkISP1Interface { libcamera.IPACameraSensorInfo sensorInfo, libcamera.ControlInfoMap sensorControls) => (int32 ret, libcamera.ControlInfoMap ipaControls); - start(libcamera.ControlList controls) => (StartResult result); + start(libcamera.ControlList controls, + uint32 paramBufferId) + => (StartResult result); stop(); configure(IPAConfigInfo configInfo, diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 3a65dab75cb7..b8b4a79830c8 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -54,7 +54,8 @@ public: const IPACameraSensorInfo &sensorInfo, const ControlInfoMap &sensorControls, ControlInfoMap *ipaControls) override; - void start(const ControlList &controls, StartResult *result) override; + void start(const ControlList &controls, const uint32_t paramBufferId, + StartResult *result) override; void stop() override; int configure(const IPAConfigInfo &ipaConfig, @@ -74,6 +75,8 @@ protected: std::string logPrefix() const override; private: + uint32_t computeParamsInternal(IPAFrameContext &frameContext, const uint32_t bufferId); + void updateControls(ControlInfoMap *ipaControls); ControlList getSensorControls(const IPAFrameContext &context); @@ -210,9 +213,17 @@ int IPARkISP1::init(const IPASettings &settings, unsigned int hwRevision, return 0; } -void IPARkISP1::start(const ControlList &controls, StartResult *result) +void IPARkISP1::start(const ControlList &controls, const uint32_t paramBufferId, + StartResult *result) { IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(0, controls); + + if (paramBufferId != 0) + result->paramBufferBytesUsed = computeParamsInternal(frameContext, + paramBufferId); + else + result->paramBufferBytesUsed = 0; + result->controls = getSensorControls(frameContext); result->code = 0; } @@ -312,18 +323,24 @@ void IPARkISP1::initializeFrameContext(IPAFrameContext &fc, const ControlList &c } } +uint32_t IPARkISP1::computeParamsInternal(IPAFrameContext &frameContext, const uint32_t bufferId) +{ + RkISP1Params params(context_.configuration.paramFormat, + mappedBuffers_.at(bufferId).planes()[0]); + + for (const auto &algo : algorithms()) + algo->prepare(context_, frameContext.frame(), frameContext, ¶ms); + + return params.bytesused(); +} + void IPARkISP1::computeParams(const uint32_t frame, const uint32_t bufferId) { IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(frame); if (bufferId != 0) { - RkISP1Params params(context_.configuration.paramFormat, - mappedBuffers_.at(bufferId).planes()[0]); - - for (const auto &algo : algorithms()) - algo->prepare(context_, frame, frameContext, ¶ms); - - paramsComputed.emit(frame, bufferId, params.bytesused()); + uint32_t size = computeParamsInternal(frameContext, bufferId); + paramsComputed.emit(frame, bufferId, size); } ControlList ctrls = getSensorControls(frameContext); diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index 832f58e95d37..b051dd21fb4c 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -104,8 +104,8 @@ public: bool canUseDewarper_; bool usesDewarper_; -private: void paramsComputed(unsigned int frame, unsigned int bufferId, unsigned int bytesused); +private: void setSensorControls(unsigned int frame, const ControlList &sensorControls); @@ -1200,14 +1200,30 @@ int PipelineHandlerRkISP1::start(Camera *camera, [[maybe_unused]] const ControlL nextStatsToProcess_ = 0; data->frame_ = 0; + uint32_t paramBufferId = 0; + FrameBuffer *paramBuffer = nullptr; + if (!isRaw_) { + paramBuffer = availableParamBuffers_.front(); + paramBufferId = paramBuffer->cookie(); + } + ipa::rkisp1::StartResult res; data->ipa_->start(controls ? *controls : ControlList{ controls::controls }, + paramBufferId, &res); if (res.code) { LOG(RkISP1, Error) << "Failed to start IPA " << camera->id(); return ret; } + + if (paramBuffer) { + availableParamBuffers_.pop(); + computingParamBuffers_.push({ paramBuffer, nextParamsSequence_++ }); + paramsSyncHelper_.pushCorrection(0); + data->paramsComputed(0, paramBufferId, res.paramBufferBytesUsed); + } + actions += [&]() { data->ipa_->stop(); }; data->sensor_->setControls(&res.controls); data->delayedCtrls_->reset(); From patchwork Mon Sep 14 14:02:44 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28275 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 48804BDCB7 for ; Mon, 14 Sep 2026 14:04:51 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 7AF7368700; Mon, 14 Sep 2026 16:04:50 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="PDcXiHbI"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A5466686FF for ; Mon, 14 Sep 2026 16:04:48 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id A52E4C14; Mon, 14 Sep 2026 16:03:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394588; bh=HyVpHVQjQoA4KLAiVBsGN2o+3XnJhzx+Yqpgd/tR8BE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PDcXiHbIRa9bOwtkLZXtP2Av8QLbLc65Coud+WPDG5mqH9PheNhzIVSWVLQbVOabO +YLsDa9ayO9EpzTlvcekC82sU5kqG2WyHh3qK2a75q8P6mDA9TTIdmmslhYhh0s1+O +SaWA37wDFoabGEx538LdhsFsoiEDINrOLEurDkw= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 31/41] pipeline: rkisp1: Fix buffer metadata when using the dewarper Date: Mon, 14 Sep 2026 16:02:44 +0200 Message-ID: <20260914140309.3354666-32-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When the dewarper is part of the pipeline, the output buffers shall still carry status (in case of FrameError), timestamp and sequence from the corresponding image buffer. Timestamp is automatically copied over by the m2m device. Manually transfer status and sequence. This change fixes an issue where frames with error status were marked as successful after running through the dewarper. Signed-off-by: Stefan Klug --- Changes in v2: - Added this patch --- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index b051dd21fb4c..ca88f9a521d2 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -163,6 +163,10 @@ struct ParamBufferInfo { struct DewarpBufferInfo { FrameBuffer *inputBuffer; + struct { + FrameMetadata::Status status; + unsigned int sequence; + } inputMeta; FrameBuffer *outputBuffer; }; @@ -1879,7 +1883,11 @@ void PipelineHandlerRkISP1::imageBufferReady(FrameBuffer *buffer) * for the dewarper are the buffers of the request, supplied by the * application. */ - DewarpBufferInfo dewarpInfo{ buffer, reqInfo->request->findBuffer(&data->mainPathStream_) }; + DewarpBufferInfo dewarpInfo{ + buffer, + { metadata.status, metadata.sequence }, + reqInfo->request->findBuffer(&data->mainPathStream_) + }; LOG(RkISP1Schedule, Debug) << "Queue dewarper " << dewarpInfo.inputBuffer << " " << dewarpInfo.outputBuffer; @@ -1900,7 +1908,6 @@ void PipelineHandlerRkISP1::imageBufferReady(FrameBuffer *buffer) void PipelineHandlerRkISP1::dewarpBufferReady(FrameBuffer *buffer) { Request *request = buffer->request(); - const FrameMetadata &metadata = buffer->metadata(); /* * After stopping the dewarper, the buffers are returned out of order. @@ -1911,13 +1918,18 @@ void PipelineHandlerRkISP1::dewarpBufferReady(FrameBuffer *buffer) if (dwInfo.outputBuffer != buffer) continue; + FrameMetadata &outputMeta = buffer->_d()->metadata(); + + if (outputMeta.status != FrameMetadata::FrameCancelled && + dwInfo.inputMeta.status == FrameMetadata::FrameError) + outputMeta.status = FrameMetadata::FrameError; + + outputMeta.sequence = dwInfo.inputMeta.sequence; + availableMainPathBuffers_.push(dwInfo.inputBuffer); dwInfo.inputBuffer = nullptr; dwInfo.outputBuffer = nullptr; - if (metadata.status == FrameMetadata::FrameCancelled) - buffer->_d()->cancel(); - completeBuffer(request, buffer); } From patchwork Mon Sep 14 14:02:45 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28276 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 34839C3363 for ; Mon, 14 Sep 2026 14:04:54 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 1B18968709; Mon, 14 Sep 2026 16:04:53 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="jBDoRspV"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 787A1686ED for ; Mon, 14 Sep 2026 16:04:51 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 79CBF9A4; Mon, 14 Sep 2026 16:03:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394591; bh=0niw2Uq+kNxZ2bjz/Vf+AG/812OXhoxrE730wPXkIxk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jBDoRspVvwCTtUPeo2X5kUqCCgLb3ZTSdlwBO2QdKjahk9B7kXdn2ihfjRdVsZlAo Ik1c8YjGiZ67Ln1x4NxPo6dLSHLo92UKwtvYMTsU0vg2TkY5sABqugFrQLqjFjhpq0 v4RGt+KJsgBuKbc83XcDPDB/7nSWBU7g2rMcjNP0= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 32/41] pipeline: rkisp1: rkisp1_path: Modify interface to be compatible with BufferQueue Date: Mon, 14 Sep 2026 16:02:45 +0200 Message-ID: <20260914140309.3354666-33-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The BufferQueue delegate expects an interface equal to the one provided by the V4L2VideoDevice. Modify the RkISP1Path class to support that interface to prepare for the migration to BufferQueue. Signed-off-by: Stefan Klug --- Changes in v2: - Added this patch --- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 44 ++++++++++++++----- src/libcamera/pipeline/rkisp1/rkisp1_path.cpp | 42 ++---------------- src/libcamera/pipeline/rkisp1/rkisp1_path.h | 23 ++++++++-- 3 files changed, 56 insertions(+), 53 deletions(-) diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index ca88f9a521d2..e3d188526ca5 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -1101,14 +1101,15 @@ int PipelineHandlerRkISP1::exportFrameBuffers([[maybe_unused]] Camera *camera, S int PipelineHandlerRkISP1::allocateBuffers(Camera *camera) { RkISP1CameraData *data = cameraData(camera); + utils::ScopeExitActions actions; unsigned int ipaBufferId = 1; int ret; - auto errorCleanup = utils::scope_exit{ [&]() { + actions += [&]() { paramBuffers_.clear(); statBuffers_.clear(); mainPathBuffers_.clear(); - } }; + }; if (!isRaw_) { ret = param_->allocateBuffers(kRkISP1InternalBufferCount, ¶mBuffers_); @@ -1129,6 +1130,22 @@ int PipelineHandlerRkISP1::allocateBuffers(Camera *camera) for (std::unique_ptr &buffer : mainPathBuffers_) availableMainPathBuffers_.push(buffer.get()); + } else if (data->mainPath_->isEnabled()) { + ret = mainPath_.importBuffers(data->mainPathStream_.configuration().bufferCount); + + if (ret < 0) + return ret; + + actions += [&]() { mainPath_.releaseBuffers(); }; + } + + if (hasSelfPath_ && data->selfPath_->isEnabled()) { + ret = selfPath_.importBuffers(data->selfPathStream_.configuration().bufferCount); + + if (ret < 0) + return ret; + + actions += [&]() { selfPath_.releaseBuffers(); }; } auto pushBuffers = [&](const std::vector> &buffers, @@ -1149,7 +1166,7 @@ int PipelineHandlerRkISP1::allocateBuffers(Camera *camera) data->ipa_->mapBuffers(data->ipaBuffers_); - errorCleanup.release(); + actions.release(); return 0; } @@ -1183,6 +1200,12 @@ int PipelineHandlerRkISP1::freeBuffers(Camera *camera) if (stat_->releaseBuffers()) LOG(RkISP1, Error) << "Failed to release stat buffers"; + if (mainPath_.releaseBuffers()) + LOG(RkISP1, Error) << "Failed to release main path buffers"; + + if (hasSelfPath_ && selfPath_.releaseBuffers()) + LOG(RkISP1, Error) << "Failed to release self path buffers"; + return 0; } @@ -1263,16 +1286,17 @@ int PipelineHandlerRkISP1::start(Camera *camera, [[maybe_unused]] const ControlL } if (data->mainPath_->isEnabled()) { - ret = mainPath_.start(data->mainPathStream_.configuration().bufferCount); + ret = mainPath_.streamOn(); if (ret) return ret; - actions += [&]() { mainPath_.stop(); }; + actions += [&]() { mainPath_.streamOff(); }; } if (hasSelfPath_ && data->selfPath_->isEnabled()) { - ret = selfPath_.start(data->selfPathStream_.configuration().bufferCount); + ret = selfPath_.streamOn(); if (ret) return ret; + actions += [&]() { selfPath_.streamOff(); }; } isp_->setFrameStartEnabled(true); @@ -1299,8 +1323,8 @@ void PipelineHandlerRkISP1::stopDevice(Camera *camera) data->ipa_->stop(); if (hasSelfPath_) - selfPath_.stop(); - mainPath_.stop(); + selfPath_.streamOff(); + mainPath_.streamOff(); if (!isRaw_) { ret = stat_->streamOff(); @@ -1666,9 +1690,9 @@ bool PipelineHandlerRkISP1::match(DeviceEnumerator *enumerator) return false; isp_->frameStart.connect(this, &PipelineHandlerRkISP1::frameStart); - mainPath_.bufferReady().connect(this, &PipelineHandlerRkISP1::imageBufferReady); + mainPath_.bufferReady.connect(this, &PipelineHandlerRkISP1::imageBufferReady); if (hasSelfPath_) - selfPath_.bufferReady().connect(this, &PipelineHandlerRkISP1::imageBufferReady); + selfPath_.bufferReady.connect(this, &PipelineHandlerRkISP1::imageBufferReady); stat_->bufferReady.connect(this, &PipelineHandlerRkISP1::statBufferReady); param_->bufferReady.connect(this, &PipelineHandlerRkISP1::paramBufferReady); diff --git a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp index 20157337022e..dc41ecb68c1d 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1_path.cpp @@ -8,6 +8,7 @@ #include "rkisp1_path.h" #include +#include #include @@ -58,7 +59,7 @@ const std::map formatToMediaBus = { RkISP1Path::RkISP1Path(const char *name, std::span formats, const Size &minResolution, const Size &maxResolution) - : name_(name), running_(false), formats_(formats), + : name_(name), formats_(formats), minResolution_(minResolution), maxResolution_(maxResolution), link_(nullptr) { @@ -77,6 +78,7 @@ bool RkISP1Path::init(std::shared_ptr media) if (video_->open() < 0) return false; + video_->bufferReady.connect(this, [this](FrameBuffer *buffer) { this->bufferReady.emit(buffer); }); populateFormats(); link_ = media->link("rkisp1_isp", 2, resizer, 0); @@ -480,44 +482,6 @@ int RkISP1Path::configure(const StreamConfiguration &config, return 0; } -int RkISP1Path::start(unsigned int bufferCount) -{ - int ret; - - if (running_) - return -EBUSY; - - ret = video_->importBuffers(bufferCount); - if (ret) - return ret; - - ret = video_->streamOn(); - if (ret) { - LOG(RkISP1, Error) - << "Failed to start " << name_ << " path"; - - video_->releaseBuffers(); - return ret; - } - - running_ = true; - - return 0; -} - -void RkISP1Path::stop() -{ - if (!running_) - return; - - if (video_->streamOff()) - LOG(RkISP1, Warning) << "Failed to stop " << name_ << " path"; - - video_->releaseBuffers(); - - running_ = false; -} - /* * \todo Remove the hardcoded resolutions and formats once kernels older than * v6.4 will stop receiving LTS support (scheduled for December 2027 for v6.1). diff --git a/src/libcamera/pipeline/rkisp1/rkisp1_path.h b/src/libcamera/pipeline/rkisp1/rkisp1_path.h index 16e6890352fc..0bb06d7672ed 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1_path.h +++ b/src/libcamera/pipeline/rkisp1/rkisp1_path.h @@ -58,19 +58,34 @@ public: return video_->exportBuffers(bufferCount, buffers); } - int start(unsigned int bufferCount); - void stop(); + int allocateBuffers(unsigned int bufferCount, + std::vector> *buffers) + { + return video_->allocateBuffers(bufferCount, buffers); + } + + int importBuffers(unsigned int count) + { + return video_->importBuffers(count); + } + + int releaseBuffers() + { + return video_->releaseBuffers(); + } + + int streamOn() { return video_->streamOn(); } + int streamOff() { return video_->streamOff(); } int queueBuffer(FrameBuffer *buffer) { return video_->queueBuffer(buffer); } - Signal &bufferReady() { return video_->bufferReady; } const Size &maxResolution() const { return maxResolution_; } + Signal bufferReady; private: void populateFormats(); Size filterSensorResolution(const CameraSensor *sensor); const char *name_; - bool running_; const std::span formats_; std::set streamFormats_; From patchwork Mon Sep 14 14:02:46 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28277 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9C0CFC3364 for ; Mon, 14 Sep 2026 14:04:56 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E2BCE686FF; Mon, 14 Sep 2026 16:04:55 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="uvcpC0PK"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 5E40F686ED for ; Mon, 14 Sep 2026 16:04:54 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 5861C9A4; Mon, 14 Sep 2026 16:03:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394594; bh=KpGq7koja11lrM6y4eK2Uc1CDwGAMeTHmIbVEYMH/XY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uvcpC0PKPLaeZJ4Sb+5qwI1BZhv3Qat52cTg8A5Gd41TdneWDt7uDlXyZoVLpdMdl n3y29zijkTKq0/DMf/jtESutEcLVjT+rWRAO1kRSsyeg16mrh4HwPQNlwESwJpf1J+ xvvO+wImP0uw0MZ1FpLiI5nD4gF1jGwSLpzZMZ+0= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 33/41] pipeline: rkisp1: Use BufferQueue for buffer handling Date: Mon, 14 Sep 2026 16:02:46 +0200 Message-ID: <20260914140309.3354666-34-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Migrate the pipeline handler to use the BufferQueue helper. This simplifies the code and makes it easier to understand. As a nice side effect, we can get rid of the statsBuffer member in SensorFrameInfo Signed-off-by: Stefan Klug --- Changes in v3: - Added assert in statsBufferReady - Dropped SensorFrameInfo::statsBuffer - Don't call postprocessedBuffers() on cancelled buffers if no request is queued in Changes in v2: - Added this patch --- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 240 ++++++++--------------- 1 file changed, 84 insertions(+), 156 deletions(-) diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index e3d188526ca5..fa530769bb04 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -12,7 +12,6 @@ #include #include #include -#include #include #include @@ -35,6 +34,7 @@ #include #include +#include "libcamera/internal/buffer_queue.h" #include "libcamera/internal/camera.h" #include "libcamera/internal/camera_sensor.h" #include "libcamera/internal/camera_sensor_properties.h" @@ -88,6 +88,7 @@ public: */ unsigned int frame_; std::vector ipaBuffers_; + std::vector ipaFrameBuffers_; RkISP1MainPath *mainPath_; RkISP1SelfPath *selfPath_; @@ -140,7 +141,6 @@ private: struct SensorFrameInfo { Request *request = nullptr; - FrameBuffer *statsBuffer = nullptr; ControlList metadata; bool metadataProcessed = false; bool statsLost = false; @@ -156,11 +156,6 @@ struct RequestInfo { bool sequenceValid = false; }; -struct ParamBufferInfo { - FrameBuffer *buffer = nullptr; - size_t expectedSequence = 0; -}; - struct DewarpBufferInfo { FrameBuffer *inputBuffer; struct { @@ -247,6 +242,9 @@ private: std::unique_ptr param_; std::unique_ptr stat_; + std::unique_ptr paramQueue_; + std::unique_ptr statQueue_; + bool hasSelfPath_; bool isRaw_; @@ -256,28 +254,18 @@ private: std::unique_ptr dewarper_; /* Internal buffers used when dewarper is being used */ - std::vector> mainPathBuffers_; - std::queue availableMainPathBuffers_; + std::unique_ptr mainPathQueue_; bool running_ = false; - std::vector> paramBuffers_; - std::vector> statBuffers_; - std::queue availableParamBuffers_; - std::queue availableStatBuffers_; - std::deque queuedRequests_; std::map sensorFrameInfos_; std::deque queuedDewarpBuffers_; - SequenceSyncHelper paramsSyncHelper_; + SequenceSyncHelper imageSyncHelper_; - std::queue computingParamBuffers_; - std::queue queuedParamBuffers_; - - uint32_t nextParamsSequence_; uint32_t nextStatsToProcess_; Camera *activeCamera_; @@ -376,29 +364,23 @@ int RkISP1CameraData::loadTuningFile(const std::string &path) return 0; } -void RkISP1CameraData::paramsComputed(unsigned int frame, - [[maybe_unused]] unsigned int bufferId, +void RkISP1CameraData::paramsComputed(unsigned int frame, unsigned int bufferId, unsigned int bytesused) { PipelineHandlerRkISP1 *pipe = RkISP1CameraData::pipe(); - ParamBufferInfo &pInfo = pipe->computingParamBuffers_.front(); - pipe->computingParamBuffers_.pop(); - ASSERT(pInfo.expectedSequence == frame); - FrameBuffer *buffer = pInfo.buffer; + FrameBuffer *buffer = pipe->paramQueue_->front(BufferQueue::Preparing); + + ASSERT(buffer->cookie() == bufferId); LOG(RkISP1Schedule, Debug) << "Queue params for " << frame << " " << buffer; buffer->_d()->metadata().planes()[0].bytesused = bytesused; - int ret = pipe->param_->queueBuffer(buffer); - if (ret < 0) { + + int ret = pipe->paramQueue_->preparedBuffer(); + if (ret < 0) LOG(RkISP1, Error) << "Failed to queue parameter buffer: " << strerror(-ret); - pipe->availableParamBuffers_.push(buffer); - return; - } - - pipe->queuedParamBuffers_.push({ buffer, frame }); } void RkISP1CameraData::setSensorControls(unsigned int frame, @@ -413,7 +395,7 @@ void RkISP1CameraData::setSensorControls(unsigned int frame, } void RkISP1CameraData::metadataReady(unsigned int frame, - [[maybe_unused]] unsigned int bufferId, + unsigned int bufferId, const ControlList &metadata) { PipelineHandlerRkISP1 *pipe = RkISP1CameraData::pipe(); @@ -445,7 +427,6 @@ void RkISP1CameraData::metadataReady(unsigned int frame, << " got lost"; auto &info2 = pipe->sensorFrameInfos_[reqInfo.sequence]; info2.metadataProcessed = true; - ASSERT(info2.statsBuffer == nullptr); continue; } @@ -463,13 +444,11 @@ void RkISP1CameraData::metadataReady(unsigned int frame, info.metadataProcessed = true; /* - * info.statsBuffer can be null, if ipa->processStats() was called + * bufferId can be 0, if ipa->processStats() was called * without a buffer to just fill the metadata. */ - if (info.statsBuffer) - pipe->availableStatBuffers_.push(info.statsBuffer); - - info.statsBuffer = nullptr; + if (bufferId) + pipe->statQueue_->postprocessedBuffer(); pipe->tryCompleteRequests(); pipe->queueInternalBuffers(); @@ -884,6 +863,15 @@ int PipelineHandlerRkISP1::configure(Camera *camera, CameraConfiguration *c) isRaw_ = info.colourEncoding == PixelFormatInfo::ColourEncodingRAW; data->usesDewarper_ = data->canUseDewarper_ && !isRaw_; + auto delegate = std::make_unique>(&mainPath_); + if (data->usesDewarper_) { + mainPathQueue_ = std::make_unique(std::move(delegate), BufferQueue::PostprocessStage, "MainPath"); + } else { + mainPathQueue_ = std::make_unique(std::move(delegate), 0, "MainPath"); + } + + mainPathQueue_->bufferReady.connect(this, &PipelineHandlerRkISP1::imageBufferReady); + Transform transform = config->combinedTransform(); bool transposeAfterIsp = false; if (data->usesDewarper_) { @@ -1090,9 +1078,9 @@ int PipelineHandlerRkISP1::exportFrameBuffers([[maybe_unused]] Camera *camera, S if (data->usesDewarper_) return dewarper_->exportBuffers(&data->mainPathStream_, count, buffers); else - return mainPath_.exportBuffers(count, buffers); + return data->mainPath_->exportBuffers(count, buffers); } else if (hasSelfPath_ && stream == &data->selfPathStream_) { - return selfPath_.exportBuffers(count, buffers); + return data->selfPath_->exportBuffers(count, buffers); } return -EINVAL; @@ -1104,39 +1092,35 @@ int PipelineHandlerRkISP1::allocateBuffers(Camera *camera) utils::ScopeExitActions actions; unsigned int ipaBufferId = 1; int ret; + /* Start at index 1 */ + data->ipaFrameBuffers_.resize(1); actions += [&]() { - paramBuffers_.clear(); - statBuffers_.clear(); - mainPathBuffers_.clear(); + paramQueue_->releaseBuffers(); + statQueue_->releaseBuffers(); + mainPathQueue_->releaseBuffers(); }; if (!isRaw_) { - ret = param_->allocateBuffers(kRkISP1InternalBufferCount, ¶mBuffers_); + ret = paramQueue_->allocateBuffers(kRkISP1InternalBufferCount); if (ret < 0) return ret; - ret = stat_->allocateBuffers(kRkISP1InternalBufferCount, &statBuffers_); + ret = statQueue_->allocateBuffers(kRkISP1InternalBufferCount); if (ret < 0) return ret; } /* If the dewarper is being used, allocate internal buffers for ISP. */ if (data->usesDewarper_) { - ret = mainPath_.exportBuffers(kRkISP1DewarpImageBufferCount, - &mainPathBuffers_); + ret = mainPathQueue_->allocateBuffers(kRkISP1DewarpImageBufferCount); if (ret < 0) return ret; - - for (std::unique_ptr &buffer : mainPathBuffers_) - availableMainPathBuffers_.push(buffer.get()); } else if (data->mainPath_->isEnabled()) { - ret = mainPath_.importBuffers(data->mainPathStream_.configuration().bufferCount); + ret = mainPathQueue_->importBuffers(data->mainPathStream_.configuration().bufferCount); if (ret < 0) return ret; - - actions += [&]() { mainPath_.releaseBuffers(); }; } if (hasSelfPath_ && data->selfPath_->isEnabled()) { @@ -1148,8 +1132,7 @@ int PipelineHandlerRkISP1::allocateBuffers(Camera *camera) actions += [&]() { selfPath_.releaseBuffers(); }; } - auto pushBuffers = [&](const std::vector> &buffers, - std::queue &queue) { + auto pushBuffers = [&](const std::vector> &buffers) { for (const std::unique_ptr &buffer : buffers) { std::span planes = buffer->planes(); @@ -1157,12 +1140,12 @@ int PipelineHandlerRkISP1::allocateBuffers(Camera *camera) data->ipaBuffers_.emplace_back(buffer->cookie(), std::vector{ planes.begin(), planes.end() }); - queue.push(buffer.get()); + data->ipaFrameBuffers_.push_back(buffer.get()); } }; - pushBuffers(paramBuffers_, availableParamBuffers_); - pushBuffers(statBuffers_, availableStatBuffers_); + pushBuffers(paramQueue_->buffers()); + pushBuffers(statQueue_->buffers()); data->ipa_->mapBuffers(data->ipaBuffers_); @@ -1174,19 +1157,6 @@ int PipelineHandlerRkISP1::freeBuffers(Camera *camera) { RkISP1CameraData *data = cameraData(camera); - while (!availableStatBuffers_.empty()) - availableStatBuffers_.pop(); - - while (!availableParamBuffers_.empty()) - availableParamBuffers_.pop(); - - while (!availableMainPathBuffers_.empty()) - availableMainPathBuffers_.pop(); - - paramBuffers_.clear(); - statBuffers_.clear(); - mainPathBuffers_.clear(); - std::vector ids; for (IPABuffer &ipabuf : data->ipaBuffers_) ids.push_back(ipabuf.id); @@ -1194,14 +1164,9 @@ int PipelineHandlerRkISP1::freeBuffers(Camera *camera) data->ipa_->unmapBuffers(ids); data->ipaBuffers_.clear(); - if (param_->releaseBuffers()) - LOG(RkISP1, Error) << "Failed to release parameters buffers"; - - if (stat_->releaseBuffers()) - LOG(RkISP1, Error) << "Failed to release stat buffers"; - - if (mainPath_.releaseBuffers()) - LOG(RkISP1, Error) << "Failed to release main path buffers"; + paramQueue_->releaseBuffers(); + statQueue_->releaseBuffers(); + mainPathQueue_->releaseBuffers(); if (hasSelfPath_ && selfPath_.releaseBuffers()) LOG(RkISP1, Error) << "Failed to release self path buffers"; @@ -1221,16 +1186,14 @@ int PipelineHandlerRkISP1::start(Camera *camera, [[maybe_unused]] const ControlL return ret; actions += [&]() { freeBuffers(camera); }; - paramsSyncHelper_.reset(); imageSyncHelper_.reset(); - nextParamsSequence_ = 0; nextStatsToProcess_ = 0; data->frame_ = 0; uint32_t paramBufferId = 0; FrameBuffer *paramBuffer = nullptr; if (!isRaw_) { - paramBuffer = availableParamBuffers_.front(); + paramBuffer = paramQueue_->front(BufferQueue::Idle); paramBufferId = paramBuffer->cookie(); } @@ -1245,10 +1208,9 @@ int PipelineHandlerRkISP1::start(Camera *camera, [[maybe_unused]] const ControlL } if (paramBuffer) { - availableParamBuffers_.pop(); - computingParamBuffers_.push({ paramBuffer, nextParamsSequence_++ }); - paramsSyncHelper_.pushCorrection(0); - data->paramsComputed(0, paramBufferId, res.paramBufferBytesUsed); + uint32_t seq; + paramQueue_->prepareBuffer(&seq); + data->paramsComputed(seq, paramBufferId, res.paramBufferBytesUsed); } actions += [&]() { data->ipa_->stop(); }; @@ -1337,12 +1299,6 @@ void PipelineHandlerRkISP1::stopDevice(Camera *camera) LOG(RkISP1, Warning) << "Failed to stop parameters for " << camera->id(); - /* - * The param buffers are not returned in order, so the queue - * becomes useless. - */ - queuedParamBuffers_ = {}; - if (data->usesDewarper_) dewarper_->stop(); } @@ -1361,8 +1317,6 @@ void PipelineHandlerRkISP1::stopDevice(Camera *camera) sensorFrameInfos_.clear(); ASSERT(queuedDewarpBuffers_.empty()); - ASSERT(queuedParamBuffers_.empty()); - ASSERT(computingParamBuffers_.empty()); freeBuffers(camera); @@ -1376,29 +1330,30 @@ void PipelineHandlerRkISP1::queueInternalBuffers() RkISP1CameraData *data = cameraData(activeCamera_); - while (!availableStatBuffers_.empty()) { - FrameBuffer *buf = availableStatBuffers_.front(); - availableStatBuffers_.pop(); - data->pipe()->stat_->queueBuffer(buf); + while (!statQueue_->empty(BufferQueue::Idle)) { + int ret = statQueue_->queueBuffer(); + if (ret) + break; } /* * In case of the dewarper, there is a seperate buffer loop for the main * path */ - while (!availableMainPathBuffers_.empty()) { - FrameBuffer *buf = availableMainPathBuffers_.front(); - availableMainPathBuffers_.pop(); + if (!data->usesDewarper_) + return; - LOG(RkISP1Schedule, Debug) << "Queue mainPath " << buf; - data->mainPath_->queueBuffer(buf); + while (!mainPathQueue_->empty(BufferQueue::Idle)) { + LOG(RkISP1Schedule, Debug) << "Queue mainPath " << mainPathQueue_->front(BufferQueue::Idle); + int ret = mainPathQueue_->queueBuffer(); + if (ret) + break; } } void PipelineHandlerRkISP1::computeParamBuffers(uint32_t maxSequence) { RkISP1CameraData *data = cameraData(activeCamera_); - if (isRaw_) { /* * Call computeParams with an empty param buffer to trigger the @@ -1408,38 +1363,24 @@ void PipelineHandlerRkISP1::computeParamBuffers(uint32_t maxSequence) return; } - while (nextParamsSequence_ <= maxSequence) { - if (availableParamBuffers_.empty()) { + while (paramQueue_->nextSequence() <= maxSequence) { + if (paramQueue_->empty(BufferQueue::Idle)) { LOG(RkISP1Schedule, Warning) << "Ran out of parameter buffers"; return; } - int correction = paramsSyncHelper_.correction(); + int correction = paramQueue_->sequenceCorrection(); if (correction != 0) LOG(RkISP1Schedule, Warning) << "Correcting params sequence " << correction; uint32_t paramsSequence; - if (correction >= 0) { - nextParamsSequence_ += correction; - paramsSyncHelper_.pushCorrection(correction); - paramsSequence = nextParamsSequence_++; - } else { - /* - * Inject the same sequence multiple times, to correct - * for the offset. - */ - paramsSyncHelper_.pushCorrection(-1); - paramsSequence = nextParamsSequence_; - } - - FrameBuffer *buf = availableParamBuffers_.front(); - availableParamBuffers_.pop(); - computingParamBuffers_.push({ buf, paramsSequence }); + FrameBuffer *buffer = paramQueue_->front(BufferQueue::Idle); + paramQueue_->prepareBuffer(¶msSequence); LOG(RkISP1Schedule, Debug) << "Request params for " << paramsSequence; - data->ipa_->computeParams(paramsSequence, buf->cookie()); + data->ipa_->computeParams(paramsSequence, buffer->cookie()); } } @@ -1475,7 +1416,7 @@ int PipelineHandlerRkISP1::queueRequestDevice(Camera *camera, Request *request) FrameBuffer *mainPathBuffer = request->findBuffer(&data->mainPathStream_); FrameBuffer *selfPathBuffer = request->findBuffer(&data->selfPathStream_); if (mainPathBuffer) - data->mainPath_->queueBuffer(mainPathBuffer); + mainPathQueue_->queueBuffer(mainPathBuffer); if (data->selfPath_ && selfPathBuffer) data->selfPath_->queueBuffer(selfPathBuffer); @@ -1678,10 +1619,18 @@ bool PipelineHandlerRkISP1::match(DeviceEnumerator *enumerator) if (stat_->open() < 0) return false; + statQueue_ = std::make_unique(std::make_unique>(stat_.get()), + BufferQueue::PostprocessStage, + "Stat"); + param_ = V4L2VideoDevice::fromEntityName(media_.get(), "rkisp1_params"); if (param_->open() < 0) return false; + paramQueue_ = std::make_unique(std::make_unique>(param_.get()), + BufferQueue::PrepareStage, + "Params"); + /* Locate and open the ISP main and self paths. */ if (!mainPath_.init(media_)) return false; @@ -1690,7 +1639,6 @@ bool PipelineHandlerRkISP1::match(DeviceEnumerator *enumerator) return false; isp_->frameStart.connect(this, &PipelineHandlerRkISP1::frameStart); - mainPath_.bufferReady.connect(this, &PipelineHandlerRkISP1::imageBufferReady); if (hasSelfPath_) selfPath_.bufferReady.connect(this, &PipelineHandlerRkISP1::imageBufferReady); stat_->bufferReady.connect(this, &PipelineHandlerRkISP1::statBufferReady); @@ -1759,7 +1707,6 @@ void PipelineHandlerRkISP1::tryCompleteRequests() break; ASSERT(iter->second.request == nullptr); - ASSERT(iter->second.statsBuffer == nullptr); sensorFrameInfos_.erase(iter); } @@ -1819,7 +1766,8 @@ void PipelineHandlerRkISP1::imageBufferReady(FrameBuffer *buffer) if (!reqInfo && data->usesDewarper_) { LOG(RkISP1Schedule, Info) << "Image buffer ready, but no corresponding request"; - availableMainPathBuffers_.push(buffer); + if (metadata.status != FrameMetadata::FrameCancelled) + mainPathQueue_->postprocessedBuffer(); return; } @@ -1896,7 +1844,6 @@ void PipelineHandlerRkISP1::imageBufferReady(FrameBuffer *buffer) /* Do not queue cancelled frames to the dewarper. */ if (metadata.status == FrameMetadata::FrameCancelled) { cancelDewarpRequest(reqInfo->request); - availableMainPathBuffers_.push(buffer); return; } @@ -1921,7 +1868,7 @@ void PipelineHandlerRkISP1::imageBufferReady(FrameBuffer *buffer) << strerror(-ret); cancelDewarpRequest(reqInfo->request); - availableMainPathBuffers_.push(buffer); + mainPathQueue_->postprocessedBuffer(); return; } queuedDewarpBuffers_.push_back(dewarpInfo); @@ -1950,7 +1897,7 @@ void PipelineHandlerRkISP1::dewarpBufferReady(FrameBuffer *buffer) outputMeta.sequence = dwInfo.inputMeta.sequence; - availableMainPathBuffers_.push(dwInfo.inputBuffer); + mainPathQueue_->postprocessedBuffer(); dwInfo.inputBuffer = nullptr; dwInfo.outputBuffer = nullptr; @@ -1969,26 +1916,10 @@ void PipelineHandlerRkISP1::paramBufferReady(FrameBuffer *buffer) { LOG(RkISP1Schedule, Debug) << "Param buffer ready " << buffer; - /* - * After stream off, the buffers are returned out of order, so we don't - * care about the rest. - */ - if (!running_) { - availableParamBuffers_.push(buffer); - return; - } - - ParamBufferInfo pInfo = queuedParamBuffers_.front(); - queuedParamBuffers_.pop(); - - ASSERT(pInfo.buffer == buffer); - size_t metaSequence = buffer->metadata().sequence; LOG(RkISP1Schedule, Debug) << "Params buffer ready " - << " Expected: " << pInfo.expectedSequence + << " Expected: " << paramQueue_->expectedSequence(buffer) << " got: " << metaSequence; - paramsSyncHelper_.receivedFrame(pInfo.expectedSequence, metaSequence); - availableParamBuffers_.push(buffer); } void PipelineHandlerRkISP1::statBufferReady(FrameBuffer *buffer) @@ -1999,12 +1930,11 @@ void PipelineHandlerRkISP1::statBufferReady(FrameBuffer *buffer) size_t sequence = buffer->metadata().sequence; if (buffer->metadata().status == FrameMetadata::FrameCancelled) { - LOG(RkISP1Schedule, Warning) << "Stats cancelled " << sequence; /* - * We can't assume that the sequence of the stat buffer is valid, - * so there is nothing left to do. + * Frame cancelled only happens after the ipa was stopped, so + * there must be no more buffers in postprocessing stage */ - availableStatBuffers_.push(buffer); + ASSERT(statQueue_->empty(BufferQueue::Postprocessing)); return; } @@ -2017,15 +1947,13 @@ void PipelineHandlerRkISP1::statBufferReady(FrameBuffer *buffer) if (nextStatsToProcess_ > sequence) { LOG(RkISP1Schedule, Warning) << "Stats were too late. Ignored"; - availableStatBuffers_.push(buffer); + statQueue_->postprocessedBuffer(); return; } while (nextStatsToProcess_ < sequence) sensorFrameInfos_[nextStatsToProcess_++].statsLost = true; - sensorFrameInfos_[sequence].statsBuffer = buffer; - LOG(RkISP1Schedule, Debug) << "Process stats " << sequence; data->ipa_->processStats(sequence, buffer->cookie(), data->delayedCtrls_->get(sequence)); From patchwork Mon Sep 14 14:02:47 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28278 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 02825C3354 for ; Mon, 14 Sep 2026 14:05:00 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A493A6870C; Mon, 14 Sep 2026 16:05:00 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="FsJUET+a"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4F6EF6870A for ; Mon, 14 Sep 2026 16:04:56 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 534999A4; Mon, 14 Sep 2026 16:03:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394596; bh=Q9pFjEBzqrZG6vMEJRlIZErNKzUJ8ocd3965jLakZ34=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FsJUET+a+xjqF+LQWH6Q0Icl0xw2YeZUkEMocY+vJ0x9MvC6Q8/wYnt5HXkMghM2A pTQBT3jnti4f2nMty6pZuh1hD+9Uk9DYT0Np3G0zLfXNkZkrIxVz1yx8QsrvPh2rq5 RXFLS2a9Fcy90iy5GAsdEKeOkN1VEGnxbOuAJ7uo= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 34/41] libipa: algorithm: Update documentation Date: Mon, 14 Sep 2026 16:02:47 +0200 Message-ID: <20260914140309.3354666-35-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Update the algorithm documentation to reflect the changed timing model. Signed-off-by: Stefan Klug --- Changes in v3: - Be more specific on the requirements on prepare() Changes in v2: - Added more documentation --- src/ipa/libipa/algorithm.cpp | 46 ++++++++++++++++++++++++------------ 1 file changed, 31 insertions(+), 15 deletions(-) diff --git a/src/ipa/libipa/algorithm.cpp b/src/ipa/libipa/algorithm.cpp index 757ce3519652..e1869dc1fe78 100644 --- a/src/ipa/libipa/algorithm.cpp +++ b/src/ipa/libipa/algorithm.cpp @@ -76,11 +76,20 @@ namespace ipa { * * This function is called for each request queued to the camera. It provides * the controls stored in the request to the algorithm. The \a frame number - * is the Request sequence number and identifies the desired corresponding + * is the sensor sequence number and identifies the desired corresponding * frame to target for the controls to take effect. * * Algorithms shall read the applicable controls and store their value for later * use during frame processing. + * + * Care shall be taken to ensure that all values in \a frameContext that get + * accessed from within process() are initialized. Examples for this are + * exposure and gain. These should be initialized even if they + * get updated again in prepare(). + * There are two reasons for this. In RAW mode prepare() is not called at all. + * The other case is when a resynchronization happens. In that case it is not + * guaranteed that prepare() is called for every frame but process() should + * still be able to update the active state. */ /** @@ -92,20 +101,29 @@ namespace ipa { * \param[out] params The ISP specific parameters * * This function is called for every frame when the camera is running before it - * is processed by the ISP to prepare the ISP processing parameters for that - * frame. + * is processed by the ISP to prepare the ISP processing parameters and the + * sensor parameters for that frame. * * Algorithms shall fill in the parameter structure fields appropriately to * configure the ISP processing blocks that they are responsible for. This * includes setting fields and flags that enable those processing blocks. + * + * Additionally \a frameContext shall be updated with the most up to date values + * necessary to configure the sensor. After prepare() the \a frameContext for + * this frame shall be treated read only. + * + * \todo: For offline ISPs there might be use cases where it is beneficial to + * separate the calculation of sensor parameters from the calculation of ISP + * paremeters. This is currently not supported. */ /** * \fn Algorithm::process() * \brief Process ISP statistics, and run algorithm operations * \param[in] context The shared IPA context - * \param[in] frame The frame context sequence number - * \param[in] frameContext The current frame's context + * \param[in] frame The frame sequence number that produces the stats + * \param[in] frameContext The frame context for the frame that produced the + * stats * \param[in] stats The IPA statistics and ISP results * \param[out] metadata Metadata for the frame, to be filled by the algorithm * @@ -118,19 +136,17 @@ namespace ipa { * computationally expensive calculations or operations must be handled * asynchronously in a separate thread. * - * Algorithms can store state in their respective IPAFrameContext structures, - * and reference state from the IPAFrameContext of other algorithms. - * - * \todo Historical data may be required as part of the processing. - * Either the previous frame, or the IPAFrameContext state of the frame - * that generated the statistics for this operation may be required for - * some advanced algorithms to prevent oscillations or support control - * loops correctly. Only a single IPAFrameContext is available currently, - * and so any data stored may represent the results of the previously - * completed operations. + * Care must be taken to ensure that the frameContext is only updated in cases + * where the frame was not processed yet. This usually differs between offline + * and inline ISPs. In an inline ISP the stats are received after processing the + * frame. In this case the frame context *must not* be updated. Algorithms + * typically update the active state which is then picked up in prepare(). * * Care shall be taken to ensure the ordering of access to the information * such that the algorithms use up to date state as required. + * + * The \a stats parameter can be null in which case only the frame metadata + * shall be filled with the data from frameContext. */ /** From patchwork Mon Sep 14 14:02:48 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28279 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id BD907C3365 for ; Mon, 14 Sep 2026 14:05:01 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 650AF68713; Mon, 14 Sep 2026 16:05:01 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="wJlMOo6i"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A08C3686ED for ; Mon, 14 Sep 2026 16:04:58 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id AD6AF512; Mon, 14 Sep 2026 16:03:18 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394598; bh=CGXKU/APkeXXTWLUMT/Ytnyz/+s2rblwYRyNuuEUxmA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=wJlMOo6i2BnMKle7fEQxqShX7XsiJl01L++eQHwKL/ZMOz/n2jXPzKpt8vf5USWOo XyTYom5NM7HqNpODaB/3tmnVROpSOou+CfDSa9TFY/H/PB6lhReSwP9J5d4p4dFcmQ EHaXRn2JjERXcdcxtbc8WlRI3KnSBt0MP9t549Tk= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 35/41] libipa: fc_queue: Early out on first context Date: Mon, 14 Sep 2026 16:02:48 +0200 Message-ID: <20260914140309.3354666-36-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Handle the special case of an uninitialized FCQueue separately, to make the following logic easier to follow. This is a preparatory patch and no functional changes are intended. Signed-off-by: Stefan Klug --- src/ipa/libipa/fc_queue.h | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/src/ipa/libipa/fc_queue.h b/src/ipa/libipa/fc_queue.h index 633bf646d88d..796ab656cf40 100644 --- a/src/ipa/libipa/fc_queue.h +++ b/src/ipa/libipa/fc_queue.h @@ -59,6 +59,14 @@ public: FC &fc = contexts_[frame % contexts_.size()]; FrameContext &frameContext = fc; + if (!initialized_) { + fc = {}; + frameContext.frame_ = frame; + initCallback_(fc, controls); + initialized_ = true; + return fc; + } + /* * If the IPA algorithms try to access a frame context slot which * has been already overwritten by a newer context, it means the @@ -72,7 +80,7 @@ public: << " has been overwritten by " << frameContext.frame_; - if (initialized_ && frame == frameContext.frame_) { + if (frame == frameContext.frame_) { if (!controls.empty()) { /* Too late to apply the controls. Store them for later. */ LOG(FCQueue, Warning) @@ -97,7 +105,6 @@ public: fc = {}; frameContext.frame_ = frame; initCallback_(fc, *controls2); - initialized_ = true; controlsToApply_.clear(); return fc; From patchwork Mon Sep 14 14:02:49 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28280 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id AAD23C3366 for ; Mon, 14 Sep 2026 14:05:03 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 1518868718; Mon, 14 Sep 2026 16:05:03 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="UGYnMyBI"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1B3576870A for ; Mon, 14 Sep 2026 16:05:02 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 20CA6929; Mon, 14 Sep 2026 16:03:22 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394602; bh=UDA4ImMqVdvbHPbmNsWuTv7RDLMVk6T2Y29NC9rq8Ik=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UGYnMyBI2pQuOtOf/ceZobiRZOdCc9m06RdoR4aVVV82o5D/SqMscTzfm6y3SNOqB RWHKV6VB0YQjQb07l1U62WDZmyXeXETLfNh/NeIe2oahLwOgrweoaCJb6CQr5lcpcu /zrtFtoAq+Ei+QMM+eVxpVKEXO/46VhWUQGm6/ts= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 36/41] libipa: fc_queue: Return nullptr instead of crashing Date: Mon, 14 Sep 2026 16:02:49 +0200 Message-ID: <20260914140309.3354666-37-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" If a requested frame context in the FCQueue was overwritten, libcamera crashes with a fatal log message. This condition occurs in rare cases with very high system load or in multi camera setups where the main camera thread is blocked by e.g. starting a seond camera for a considerable amount of time. Still we sould not crash in these cases. To be able to handle the situation in upcoming patches, modify getOrInitContext() to return a pointer instead of a reference and return a nullptr instead of the fatal log. Add an ASSERT() to all callers so that we still crash in these cases. More fine grained handling needs to be implemented later. Signed-off-by: Stefan Klug --- src/ipa/ipu3/ipu3.cpp | 17 ++++++++++------- src/ipa/libipa/fc_queue.h | 32 ++++++++++++++------------------ src/ipa/mali-c55/mali-c55.cpp | 16 ++++++++++------ src/ipa/rkisp1/rkisp1.cpp | 21 ++++++++++++--------- src/ipa/softisp/softisp.cpp | 15 +++++++++------ 5 files changed, 55 insertions(+), 46 deletions(-) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 24bbfa86c255..ed873784c730 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -473,10 +473,11 @@ void IPAIPU3::computeParams(const uint32_t frame, const uint32_t bufferId) */ params->use = {}; - IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(frame); + IPAFrameContext *frameContext = context_.frameContexts.getOrInitContext(frame); + ASSERT(frameContext); for (const auto &algo : algorithms()) - algo->prepare(context_, frame, frameContext, params); + algo->prepare(context_, frame, *frameContext, params); paramsComputed.emit(frame); } @@ -506,15 +507,16 @@ void IPAIPU3::processStats(const uint32_t frame, const ipu3_uapi_stats_3a *stats = reinterpret_cast(mem.data()); - IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(frame); + IPAFrameContext *frameContext = context_.frameContexts.getOrInitContext(frame); + ASSERT(frameContext); - std::tie(frameContext.sensor.exposure, frameContext.sensor.gain) = + std::tie(frameContext->sensor.exposure, frameContext->sensor.gain) = agc::extractControls(sensorControls, context_.camHelper.get()); ControlList metadata(controls::controls); for (const auto &algo : algorithms()) - algo->process(context_, frame, frameContext, stats, metadata); + algo->process(context_, frame, *frameContext, stats, metadata); setControls(frame); @@ -558,11 +560,12 @@ void IPAIPU3::initializeFrameContext(IPAFrameContext &frameContext, */ void IPAIPU3::setControls(unsigned int frame) { - IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(frame); + IPAFrameContext *frameContext = context_.frameContexts.getOrInitContext(frame); + ASSERT(frameContext); ControlList ctrls(context_.sensorControls); agc::prepareControls(ctrls, context_.camHelper.get(), - frameContext.agc.exposure, frameContext.agc.gain); + frameContext->agc.exposure, frameContext->agc.gain); ControlList lensCtrls(lensCtrls_); lensCtrls.set(V4L2_CID_FOCUS_ABSOLUTE, diff --git a/src/ipa/libipa/fc_queue.h b/src/ipa/libipa/fc_queue.h index 796ab656cf40..1dd04ad1ae2c 100644 --- a/src/ipa/libipa/fc_queue.h +++ b/src/ipa/libipa/fc_queue.h @@ -54,7 +54,7 @@ public: initialized_ = false; } - FC &getOrInitContext(unsigned int frame, const ControlList &controls = {}) + FC *getOrInitContext(unsigned int frame, const ControlList &controls = {}) { FC &fc = contexts_[frame % contexts_.size()]; FrameContext &frameContext = fc; @@ -64,23 +64,10 @@ public: frameContext.frame_ = frame; initCallback_(fc, controls); initialized_ = true; - return fc; + return &fc; } - /* - * If the IPA algorithms try to access a frame context slot which - * has been already overwritten by a newer context, it means the - * frame context queue has overflowed and the desired context - * has been forever lost. The pipeline handler shall avoid - * queueing more requests to the IPA than the frame context - * queue size. - */ - if (frame < frameContext.frame_) - LOG(FCQueue, Fatal) << "Frame context for " << frame - << " has been overwritten by " - << frameContext.frame_; - - if (frame == frameContext.frame_) { + if (frame <= frameContext.frame_) { if (!controls.empty()) { /* Too late to apply the controls. Store them for later. */ LOG(FCQueue, Warning) @@ -89,8 +76,17 @@ public: controlsToApply_.merge(controls, ControlList::MergePolicy::OverwriteExisting); } + + if (frame < frameContext.frame_) { + LOG(FCQueue, Warning) + << "Frame context for " << frame + << " is already overwritten by " + << frameContext.frame_; + return nullptr; + } + LOG(FCQueue, Debug) << "Got " << frame; - return fc; + return &fc; } const ControlList *controls2 = &controls; @@ -107,7 +103,7 @@ public: initCallback_(fc, *controls2); controlsToApply_.clear(); - return fc; + return &fc; } private: diff --git a/src/ipa/mali-c55/mali-c55.cpp b/src/ipa/mali-c55/mali-c55.cpp index d3d17bb5ae18..7ee4a9349af1 100644 --- a/src/ipa/mali-c55/mali-c55.cpp +++ b/src/ipa/mali-c55/mali-c55.cpp @@ -243,11 +243,13 @@ void IPAMaliC55::initializeFrameContext(IPAFrameContext &frameContext, void IPAMaliC55::fillParams(unsigned int request, [[maybe_unused]] uint32_t bufferId) { - IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(request); + IPAFrameContext *frameContext = context_.frameContexts.getOrInitContext(request); MaliC55Params params(buffers_.at(bufferId).planes()[0]); + ASSERT(frameContext); + for (const auto &algo : algorithms()) - algo->prepare(context_, request, frameContext, ¶ms); + algo->prepare(context_, request, *frameContext, ¶ms); paramsComputed.emit(request, params.bytesused()); } @@ -255,13 +257,15 @@ void IPAMaliC55::fillParams(unsigned int request, void IPAMaliC55::processStats(unsigned int request, unsigned int bufferId, const ControlList &sensorControls) { - IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(request); + IPAFrameContext *frameContext = context_.frameContexts.getOrInitContext(request); const mali_c55_stats_buffer *stats = nullptr; + ASSERT(frameContext); + stats = reinterpret_cast( buffers_.at(bufferId).planes()[0].data()); - std::tie(frameContext.sensor.exposure, frameContext.sensor.gain) = + std::tie(frameContext->sensor.exposure, frameContext->sensor.gain) = agc::extractControls(sensorControls, context_.camHelper.get()); ControlList metadata(controls::controls); @@ -269,10 +273,10 @@ void IPAMaliC55::processStats(unsigned int request, unsigned int bufferId, for (const auto &a : algorithms()) { Algorithm *algo = static_cast(a.get()); - algo->process(context_, request, frameContext, stats, metadata); + algo->process(context_, request, *frameContext, stats, metadata); } - setControls(frameContext); + setControls(*frameContext); statsProcessed.emit(request, metadata); } diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index b8b4a79830c8..750c0fa72733 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -216,15 +216,16 @@ int IPARkISP1::init(const IPASettings &settings, unsigned int hwRevision, void IPARkISP1::start(const ControlList &controls, const uint32_t paramBufferId, StartResult *result) { - IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(0, controls); + IPAFrameContext *frameContext = context_.frameContexts.getOrInitContext(0, controls); + ASSERT(frameContext); if (paramBufferId != 0) - result->paramBufferBytesUsed = computeParamsInternal(frameContext, + result->paramBufferBytesUsed = computeParamsInternal(*frameContext, paramBufferId); else result->paramBufferBytesUsed = 0; - result->controls = getSensorControls(frameContext); + result->controls = getSensorControls(*frameContext); result->code = 0; } @@ -336,21 +337,23 @@ uint32_t IPARkISP1::computeParamsInternal(IPAFrameContext &frameContext, const u void IPARkISP1::computeParams(const uint32_t frame, const uint32_t bufferId) { - IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(frame); + IPAFrameContext *frameContext = context_.frameContexts.getOrInitContext(frame); + ASSERT(frameContext); if (bufferId != 0) { - uint32_t size = computeParamsInternal(frameContext, bufferId); + uint32_t size = computeParamsInternal(*frameContext, bufferId); paramsComputed.emit(frame, bufferId, size); } - ControlList ctrls = getSensorControls(frameContext); + ControlList ctrls = getSensorControls(*frameContext); setSensorControls.emit(frame, ctrls); } void IPARkISP1::processStats(const uint32_t frame, const uint32_t bufferId, const ControlList &sensorControls) { - IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(frame); + IPAFrameContext *frameContext = context_.frameContexts.getOrInitContext(frame); + ASSERT(frameContext); /* * In raw capture mode, the ISP is bypassed and no statistics buffer is @@ -361,7 +364,7 @@ void IPARkISP1::processStats(const uint32_t frame, const uint32_t bufferId, stats = reinterpret_cast( mappedBuffers_.at(bufferId).planes()[0].data()); - std::tie(frameContext.sensor.exposure, frameContext.sensor.gain) = + std::tie(frameContext->sensor.exposure, frameContext->sensor.gain) = agc::extractControls(sensorControls, context_.camHelper.get()); ControlList metadata(controls::controls); @@ -370,7 +373,7 @@ void IPARkISP1::processStats(const uint32_t frame, const uint32_t bufferId, Algorithm *algo = static_cast(a.get()); if (algo->disabled_) continue; - algo->process(context_, frame, frameContext, stats, metadata); + algo->process(context_, frame, *frameContext, stats, metadata); } context_.debugMetadata.moveEntries(metadata); diff --git a/src/ipa/softisp/softisp.cpp b/src/ipa/softisp/softisp.cpp index c00d36bdff1a..2a65cd7cebfe 100644 --- a/src/ipa/softisp/softisp.cpp +++ b/src/ipa/softisp/softisp.cpp @@ -237,9 +237,11 @@ void IPASoftIsp::computeParams(const uint32_t frame) { context_.activeState.combinedMatrix = Matrix::identity(); - IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(frame); + IPAFrameContext *frameContext = context_.frameContexts.getOrInitContext(frame); + ASSERT(frameContext); + for (const auto &algo : algorithms()) - algo->prepare(context_, frame, frameContext, params_); + algo->prepare(context_, frame, *frameContext, params_); params_->combinedMatrix = context_.activeState.combinedMatrix; paramsComputed.emit(frame); @@ -249,19 +251,20 @@ void IPASoftIsp::processStats(const uint32_t frame, [[maybe_unused]] const uint32_t bufferId, const ControlList &sensorControls) { - IPAFrameContext &frameContext = context_.frameContexts.getOrInitContext(frame); + IPAFrameContext *frameContext = context_.frameContexts.getOrInitContext(frame); + ASSERT(frameContext); - std::tie(frameContext.sensor.exposure, frameContext.sensor.gain) = + std::tie(frameContext->sensor.exposure, frameContext->sensor.gain) = agc::extractControls(sensorControls, context_.camHelper.get()); ControlList metadata(controls::controls); for (const auto &algo : algorithms()) - algo->process(context_, frame, frameContext, stats_, metadata); + algo->process(context_, frame, *frameContext, stats_, metadata); metadataReady.emit(frame, metadata); ControlList ctrls(context_.sensorControls); agc::prepareControls(ctrls, context_.camHelper.get(), - frameContext.agc.exposure, frameContext.agc.gain); + frameContext->agc.exposure, frameContext->agc.gain); setSensorControls.emit(ctrls); } From patchwork Mon Sep 14 14:02:50 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28281 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 82F8EC3367 for ; Mon, 14 Sep 2026 14:05:07 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9A26A6870A; Mon, 14 Sep 2026 16:05:06 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="PPf2umGD"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 805796870A for ; Mon, 14 Sep 2026 16:05:04 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 666D9C14; Mon, 14 Sep 2026 16:03:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394604; bh=Z+ZxcD9yE9yuzGYZWeab41p5YRyBwnBsfnNzp/xA3Lw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PPf2umGDlYvo8m/4jQx2ZJ7WgnMLv3aNmtImv/F5HXCqR8UPFqK0PDDW8efzj0aY0 XCGTjd90WE3XEPsq17kWWDRMk98kqC56ctx2dEF7BXLZtdH8rnxey7DLH+PoGI6ybs oLveVNm2Y6iHWrtuESCco+HwJVi9augqfo2JPVjU= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 37/41] libipa: algorithm: Add a initialize parameter to prepare Date: Mon, 14 Sep 2026 16:02:50 +0200 Message-ID: <20260914140309.3354666-38-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In prepare() the paremeters for the ISP block are assembled. Usually there is an initial step involved when enabling the ISP block for the first time which is often handled by checking if frame == 0. In cases of a FCQueue overflow we basically have the same condition as we don't know the last parameters that were applied to the ISP block. Add a initialize flag to the prepare function() that can be used to trigger a full initialization of an ISP block. For now it is only set in case of frame == 0. Signed-off-by: Stefan Klug --- src/ipa/ipu3/algorithms/af.cpp | 3 ++- src/ipa/ipu3/algorithms/af.h | 3 ++- src/ipa/ipu3/algorithms/agc.cpp | 3 ++- src/ipa/ipu3/algorithms/agc.h | 3 ++- src/ipa/ipu3/algorithms/awb.cpp | 3 ++- src/ipa/ipu3/algorithms/awb.h | 3 ++- src/ipa/ipu3/algorithms/blc.cpp | 4 +++- src/ipa/ipu3/algorithms/blc.h | 3 ++- src/ipa/ipu3/algorithms/tone_mapping.cpp | 4 +++- src/ipa/ipu3/algorithms/tone_mapping.h | 3 ++- src/ipa/ipu3/ipu3.cpp | 2 +- src/ipa/libipa/algorithm.cpp | 6 ++++++ src/ipa/libipa/algorithm.h | 3 ++- src/ipa/mali-c55/algorithms/agc.cpp | 7 ++++--- src/ipa/mali-c55/algorithms/agc.h | 3 ++- src/ipa/mali-c55/algorithms/awb.cpp | 7 ++++--- src/ipa/mali-c55/algorithms/awb.h | 3 ++- src/ipa/mali-c55/algorithms/blc.cpp | 6 +++--- src/ipa/mali-c55/algorithms/blc.h | 3 ++- src/ipa/mali-c55/algorithms/ccm.cpp | 5 +++-- src/ipa/mali-c55/algorithms/ccm.h | 3 ++- src/ipa/mali-c55/algorithms/lsc.cpp | 4 ++-- src/ipa/mali-c55/algorithms/lsc.h | 3 ++- src/ipa/mali-c55/mali-c55.cpp | 3 ++- src/ipa/rkisp1/algorithms/agc.cpp | 7 ++++--- src/ipa/rkisp1/algorithms/agc.h | 3 ++- src/ipa/rkisp1/algorithms/awb.cpp | 7 ++++--- src/ipa/rkisp1/algorithms/awb.h | 3 ++- src/ipa/rkisp1/algorithms/blc.cpp | 7 ++++--- src/ipa/rkisp1/algorithms/blc.h | 3 ++- src/ipa/rkisp1/algorithms/ccm.cpp | 3 ++- src/ipa/rkisp1/algorithms/ccm.h | 3 ++- src/ipa/rkisp1/algorithms/compress.cpp | 3 ++- src/ipa/rkisp1/algorithms/compress.h | 3 ++- src/ipa/rkisp1/algorithms/cproc.cpp | 5 +++-- src/ipa/rkisp1/algorithms/cproc.h | 3 ++- src/ipa/rkisp1/algorithms/dpcc.cpp | 7 ++++--- src/ipa/rkisp1/algorithms/dpcc.h | 3 ++- src/ipa/rkisp1/algorithms/dpf.cpp | 7 ++++--- src/ipa/rkisp1/algorithms/dpf.h | 3 ++- src/ipa/rkisp1/algorithms/filter.cpp | 5 +++-- src/ipa/rkisp1/algorithms/filter.h | 3 ++- src/ipa/rkisp1/algorithms/goc.cpp | 5 +++-- src/ipa/rkisp1/algorithms/goc.h | 3 ++- src/ipa/rkisp1/algorithms/gsl.cpp | 7 ++++--- src/ipa/rkisp1/algorithms/gsl.h | 3 ++- src/ipa/rkisp1/algorithms/lsc.cpp | 5 +++-- src/ipa/rkisp1/algorithms/lsc.h | 3 ++- src/ipa/rkisp1/algorithms/lux.cpp | 3 ++- src/ipa/rkisp1/algorithms/lux.h | 3 ++- src/ipa/rkisp1/algorithms/wdr.cpp | 3 ++- src/ipa/rkisp1/algorithms/wdr.h | 3 ++- src/ipa/rkisp1/rkisp1.cpp | 4 +++- src/ipa/softisp/algorithms/adjust.cpp | 3 ++- src/ipa/softisp/algorithms/adjust.h | 3 ++- src/ipa/softisp/algorithms/agc.cpp | 4 +++- src/ipa/softisp/algorithms/agc.h | 3 ++- src/ipa/softisp/algorithms/awb.cpp | 3 ++- src/ipa/softisp/algorithms/awb.h | 3 ++- src/ipa/softisp/algorithms/blc.cpp | 3 ++- src/ipa/softisp/algorithms/blc.h | 3 ++- src/ipa/softisp/algorithms/ccm.cpp | 4 +++- src/ipa/softisp/algorithms/ccm.h | 3 ++- src/ipa/softisp/softisp.cpp | 2 +- 64 files changed, 157 insertions(+), 87 deletions(-) diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp index e796b06a2071..9437f45f9932 100644 --- a/src/ipa/ipu3/algorithms/af.cpp +++ b/src/ipa/ipu3/algorithms/af.cpp @@ -184,7 +184,8 @@ int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo) void Af::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, - ipu3_uapi_params *params) + ipu3_uapi_params *params, + [[maybe_unused]] bool initialize) { const struct ipu3_uapi_grid_config &grid = context.configuration.af.afGrid; params->acc_param.af.grid_cfg = grid; diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h index 320b674962ad..d616cd45a210 100644 --- a/src/ipa/ipu3/algorithms/af.h +++ b/src/ipa/ipu3/algorithms/af.h @@ -33,7 +33,8 @@ public: int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - ipu3_uapi_params *params) override; + ipu3_uapi_params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const ipu3_uapi_stats_3a *stats, diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp index 67c910049a6d..de5f47f7f587 100644 --- a/src/ipa/ipu3/algorithms/agc.cpp +++ b/src/ipa/ipu3/algorithms/agc.cpp @@ -109,7 +109,8 @@ void Agc::queueRequest(IPAContext &context, [[maybe_unused]] const uint32_t fram */ void Agc::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, - [[maybe_unused]] ipu3_uapi_params *params) + [[maybe_unused]] ipu3_uapi_params *params, + [[maybe_unused]] bool initialize) { agc_.prepare(context.configuration.agc, context.activeState.agc, frameContext.agc); } diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h index 634b1f8a7a8f..e8034ce14105 100644 --- a/src/ipa/ipu3/algorithms/agc.h +++ b/src/ipa/ipu3/algorithms/agc.h @@ -36,7 +36,8 @@ public: IPAFrameContext &frameContext, const ControlList &controls) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - ipu3_uapi_params *params) override; + ipu3_uapi_params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const ipu3_uapi_stats_3a *stats, diff --git a/src/ipa/ipu3/algorithms/awb.cpp b/src/ipa/ipu3/algorithms/awb.cpp index 55de05d9e39f..d5b20f890a9f 100644 --- a/src/ipa/ipu3/algorithms/awb.cpp +++ b/src/ipa/ipu3/algorithms/awb.cpp @@ -249,7 +249,8 @@ constexpr uint16_t Awb::gainValue(double gain) void Awb::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, - ipu3_uapi_params *params) + ipu3_uapi_params *params, + [[maybe_unused]] bool initialize) { /* * Green saturation thresholds are reduced because we are using the diff --git a/src/ipa/ipu3/algorithms/awb.h b/src/ipa/ipu3/algorithms/awb.h index dbf69c9073a1..0b26a669a034 100644 --- a/src/ipa/ipu3/algorithms/awb.h +++ b/src/ipa/ipu3/algorithms/awb.h @@ -43,7 +43,8 @@ public: int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - ipu3_uapi_params *params) override; + ipu3_uapi_params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const ipu3_uapi_stats_3a *stats, diff --git a/src/ipa/ipu3/algorithms/blc.cpp b/src/ipa/ipu3/algorithms/blc.cpp index 35748fb2ce85..6164738256dd 100644 --- a/src/ipa/ipu3/algorithms/blc.cpp +++ b/src/ipa/ipu3/algorithms/blc.cpp @@ -40,6 +40,7 @@ BlackLevelCorrection::BlackLevelCorrection() * \param[in] frame The frame context sequence number * \param[in] frameContext The FrameContext for this frame * \param[out] params The IPU3 parameters + * \param[in] initialize True if the ISP module should be reinitialzed * * Populate the IPU3 parameter structure with the correction values for each * channel and enable the corresponding ImgU block processing. @@ -47,7 +48,8 @@ BlackLevelCorrection::BlackLevelCorrection() void BlackLevelCorrection::prepare([[maybe_unused]] IPAContext &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, - ipu3_uapi_params *params) + ipu3_uapi_params *params, + [[maybe_unused]] bool initialize) { /* * The Optical Black Level correction values diff --git a/src/ipa/ipu3/algorithms/blc.h b/src/ipa/ipu3/algorithms/blc.h index 6274804548bb..65ce3cbae0f2 100644 --- a/src/ipa/ipu3/algorithms/blc.h +++ b/src/ipa/ipu3/algorithms/blc.h @@ -20,7 +20,8 @@ public: void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - ipu3_uapi_params *params) override; + ipu3_uapi_params *params, + bool initialize) override; }; } /* namespace ipa::ipu3::algorithms */ diff --git a/src/ipa/ipu3/algorithms/tone_mapping.cpp b/src/ipa/ipu3/algorithms/tone_mapping.cpp index 160338c13944..74a30b729547 100644 --- a/src/ipa/ipu3/algorithms/tone_mapping.cpp +++ b/src/ipa/ipu3/algorithms/tone_mapping.cpp @@ -53,6 +53,7 @@ int ToneMapping::configure(IPAContext &context, * \param[in] frame The frame context sequence number * \param[in] frameContext The FrameContext for this frame * \param[out] params The IPU3 parameters + * \param[in] initialize True if the ISP module should be reinitialzed * * Populate the IPU3 parameter structure with our tone mapping look up table and * enable the gamma control module in the processing blocks. @@ -60,7 +61,8 @@ int ToneMapping::configure(IPAContext &context, void ToneMapping::prepare([[maybe_unused]] IPAContext &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, - ipu3_uapi_params *params) + ipu3_uapi_params *params, + [[maybe_unused]] bool initialize) { /* Copy the calculated LUT into the parameters buffer. */ memcpy(params->acc_param.gamma.gc_lut.lut, diff --git a/src/ipa/ipu3/algorithms/tone_mapping.h b/src/ipa/ipu3/algorithms/tone_mapping.h index b2b380108e01..f9615c6bec88 100644 --- a/src/ipa/ipu3/algorithms/tone_mapping.h +++ b/src/ipa/ipu3/algorithms/tone_mapping.h @@ -20,7 +20,8 @@ public: int configure(IPAContext &context, const IPAConfigInfo &configInfo) override; void prepare(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameContext, ipu3_uapi_params *params) override; + IPAFrameContext &frameContext, ipu3_uapi_params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const ipu3_uapi_stats_3a *stats, diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index ed873784c730..0705cecc23b2 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -477,7 +477,7 @@ void IPAIPU3::computeParams(const uint32_t frame, const uint32_t bufferId) ASSERT(frameContext); for (const auto &algo : algorithms()) - algo->prepare(context_, frame, *frameContext, params); + algo->prepare(context_, frame, *frameContext, params, frame == 0); paramsComputed.emit(frame); } diff --git a/src/ipa/libipa/algorithm.cpp b/src/ipa/libipa/algorithm.cpp index e1869dc1fe78..72b47a42a890 100644 --- a/src/ipa/libipa/algorithm.cpp +++ b/src/ipa/libipa/algorithm.cpp @@ -99,6 +99,7 @@ namespace ipa { * \param[in] frame The frame context sequence number * \param[in] frameContext The FrameContext for this frame * \param[out] params The ISP specific parameters + * \param[in] initialize True if the ISP module should be reinitialzed * * This function is called for every frame when the camera is running before it * is processed by the ISP to prepare the ISP processing parameters and the @@ -112,6 +113,11 @@ namespace ipa { * necessary to configure the sensor. After prepare() the \a frameContext for * this frame shall be treated read only. * + * The \a initialize parameter indicates if the ISP module should be fully + * initialized. This is always set on the first frame and in rare cases when + * frame contexts were overwritten and the ISP module has to be reinitialized to + * guarantee a defined state. + * * \todo: For offline ISPs there might be use cases where it is beneficial to * separate the calculation of sensor parameters from the calculation of ISP * paremeters. This is currently not supported. diff --git a/src/ipa/libipa/algorithm.h b/src/ipa/libipa/algorithm.h index 4ddb16ef3920..140148c81db5 100644 --- a/src/ipa/libipa/algorithm.h +++ b/src/ipa/libipa/algorithm.h @@ -48,7 +48,8 @@ public: virtual void prepare([[maybe_unused]] typename Module::Context &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] typename Module::FrameContext &frameContext, - [[maybe_unused]] typename Module::Params *params) + [[maybe_unused]] typename Module::Params *params, + [[maybe_unused]] bool initialize) { } diff --git a/src/ipa/mali-c55/algorithms/agc.cpp b/src/ipa/mali-c55/algorithms/agc.cpp index 5684b4df954c..b5c86c44dd28 100644 --- a/src/ipa/mali-c55/algorithms/agc.cpp +++ b/src/ipa/mali-c55/algorithms/agc.cpp @@ -204,12 +204,13 @@ void Agc::fillWeightsArrayBuffer(MaliC55Params *params, const enum MaliC55Blocks std::fill(weights.begin(), weights.end(), 1); } -void Agc::prepare(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameContext, MaliC55Params *params) +void Agc::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, MaliC55Params *params, + bool initialize) { agc_.prepare(context.configuration.agc, context.activeState.agc, frameContext.agc); - if (frame > 0) + if (!initialize) return; fillParamsBuffer(params, MaliC55Blocks::AexpHist); diff --git a/src/ipa/mali-c55/algorithms/agc.h b/src/ipa/mali-c55/algorithms/agc.h index 809a57c53acc..5c7312804e3b 100644 --- a/src/ipa/mali-c55/algorithms/agc.h +++ b/src/ipa/mali-c55/algorithms/agc.h @@ -57,7 +57,8 @@ public: const ControlList &controls) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - MaliC55Params *params) override; + MaliC55Params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const mali_c55_stats_buffer *stats, diff --git a/src/ipa/mali-c55/algorithms/awb.cpp b/src/ipa/mali-c55/algorithms/awb.cpp index 353a9f0d3f97..1d2dc5d0d981 100644 --- a/src/ipa/mali-c55/algorithms/awb.cpp +++ b/src/ipa/mali-c55/algorithms/awb.cpp @@ -151,8 +151,9 @@ void Awb::fillConfigParamBlock(MaliC55Params *params) /** * \copydoc libcamera::ipa::Algorithm::prepare */ -void Awb::prepare(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameContext, MaliC55Params *params) +void Awb::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, MaliC55Params *params, + bool initialize) { awbAlgo_.prepare(context.activeState.awb, frameContext.awb); @@ -176,7 +177,7 @@ void Awb::prepare(IPAContext &context, const uint32_t frame, block->gain11 = UQ<4, 8>(static_cast(frameContext.awb.gains.b())) .quantized(); - if (frame > 0) + if (!initialize) return; fillConfigParamBlock(params); diff --git a/src/ipa/mali-c55/algorithms/awb.h b/src/ipa/mali-c55/algorithms/awb.h index 4be7a5917ac5..c41f63daa4f1 100644 --- a/src/ipa/mali-c55/algorithms/awb.h +++ b/src/ipa/mali-c55/algorithms/awb.h @@ -39,7 +39,8 @@ public: const ControlList &controls) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - MaliC55Params *params) override; + MaliC55Params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const mali_c55_stats_buffer *stats, diff --git a/src/ipa/mali-c55/algorithms/blc.cpp b/src/ipa/mali-c55/algorithms/blc.cpp index 591a5eaf2dc8..2e6ff7f36be7 100644 --- a/src/ipa/mali-c55/algorithms/blc.cpp +++ b/src/ipa/mali-c55/algorithms/blc.cpp @@ -83,11 +83,11 @@ int BlackLevelCorrection::configure(IPAContext &context, * \copydoc libcamera::ipa::Algorithm::prepare */ void BlackLevelCorrection::prepare([[maybe_unused]] IPAContext &context, - const uint32_t frame, + [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, - MaliC55Params *params) + MaliC55Params *params, bool initialize) { - if (frame > 0) + if (!initialize) return; if (!tuningParameters_) diff --git a/src/ipa/mali-c55/algorithms/blc.h b/src/ipa/mali-c55/algorithms/blc.h index bce343e20c55..053a88e3b7b0 100644 --- a/src/ipa/mali-c55/algorithms/blc.h +++ b/src/ipa/mali-c55/algorithms/blc.h @@ -23,7 +23,8 @@ public: const IPACameraSensorInfo &configInfo) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - MaliC55Params *params) override; + MaliC55Params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const mali_c55_stats_buffer *stats, diff --git a/src/ipa/mali-c55/algorithms/ccm.cpp b/src/ipa/mali-c55/algorithms/ccm.cpp index b5196a7ee85b..994fa5bf3572 100644 --- a/src/ipa/mali-c55/algorithms/ccm.cpp +++ b/src/ipa/mali-c55/algorithms/ccm.cpp @@ -128,7 +128,8 @@ void Ccm::setParameters(MaliC55Params *params, const IPAFrameContext &frameConte * \copydoc libcamera::ipa::Algorithm::prepare */ void Ccm::prepare(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameContext, MaliC55Params *params) + IPAFrameContext &frameContext, MaliC55Params *params, + bool initialize) { if (!frameContext.awb.autoEnabled) { setParameters(params, frameContext); @@ -141,7 +142,7 @@ void Ccm::prepare(IPAContext &context, const uint32_t frame, * changes of a certain amount. */ float ct = frameContext.awb.colourTemperature * 1.0f; - if (frame > 0 && (ct < lastCt_ * 1.2 && ct > lastCt_ * 0.8)) { + if (!initialize && (ct < lastCt_ * 1.2 && ct > lastCt_ * 0.8)) { frameContext.ccm.ccm = context.activeState.ccm.automatic.ccm; frameContext.ccm.offsets = context.activeState.ccm.automatic.offsets; lastCt_ = ct; diff --git a/src/ipa/mali-c55/algorithms/ccm.h b/src/ipa/mali-c55/algorithms/ccm.h index 73649204a7ee..4916f260b827 100644 --- a/src/ipa/mali-c55/algorithms/ccm.h +++ b/src/ipa/mali-c55/algorithms/ccm.h @@ -39,7 +39,8 @@ public: const ControlList &controls) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - MaliC55Params *params) override; + MaliC55Params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const mali_c55_stats_buffer *stats, diff --git a/src/ipa/mali-c55/algorithms/lsc.cpp b/src/ipa/mali-c55/algorithms/lsc.cpp index fff0dc7d0e64..a39089bdd429 100644 --- a/src/ipa/mali-c55/algorithms/lsc.cpp +++ b/src/ipa/mali-c55/algorithms/lsc.cpp @@ -183,7 +183,7 @@ std::tuple Lsc::findBankAndAlpha(uint32_t ct) const void Lsc::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, - MaliC55Params *params) + MaliC55Params *params, bool initialize) { /* * For each frame we assess the colour temperature of the **last** frame @@ -208,7 +208,7 @@ void Lsc::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, fillSelectionParamsBlock(params, bank, alpha); - if (frame > 0) + if (!initialize) return; /* diff --git a/src/ipa/mali-c55/algorithms/lsc.h b/src/ipa/mali-c55/algorithms/lsc.h index 5f752eebaef5..12069850a6e0 100644 --- a/src/ipa/mali-c55/algorithms/lsc.h +++ b/src/ipa/mali-c55/algorithms/lsc.h @@ -36,7 +36,8 @@ public: const ControlList &controls) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - MaliC55Params *params) override; + MaliC55Params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const mali_c55_stats_buffer *stats, diff --git a/src/ipa/mali-c55/mali-c55.cpp b/src/ipa/mali-c55/mali-c55.cpp index 7ee4a9349af1..8a93dec2f66c 100644 --- a/src/ipa/mali-c55/mali-c55.cpp +++ b/src/ipa/mali-c55/mali-c55.cpp @@ -249,7 +249,8 @@ void IPAMaliC55::fillParams(unsigned int request, ASSERT(frameContext); for (const auto &algo : algorithms()) - algo->prepare(context_, request, *frameContext, ¶ms); + algo->prepare(context_, request, *frameContext, ¶ms, + frameContext->frame() == 0); paramsComputed.emit(request, params.bytesused()); } diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 34e186d8b68f..41680e5c4ffd 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -206,8 +206,9 @@ void Agc::queueRequest(IPAContext &context, /** * \copydoc libcamera::ipa::Algorithm::prepare */ -void Agc::prepare(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameContext, RkISP1Params *params) +void Agc::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, RkISP1Params *params, + bool initialize) { agc_.prepare(context.configuration.agc, context.activeState.agc, frameContext.agc); @@ -216,7 +217,7 @@ void Agc::prepare(IPAContext &context, const uint32_t frame, frameContext.compress.gain = frameContext.agc.quantizationGain; } - if (frame > 0 && !frameContext.agc.updateMetering) + if (!initialize && !frameContext.agc.updateMetering) return; /* diff --git a/src/ipa/rkisp1/algorithms/agc.h b/src/ipa/rkisp1/algorithms/agc.h index b8d59352e70d..5704d162919b 100644 --- a/src/ipa/rkisp1/algorithms/agc.h +++ b/src/ipa/rkisp1/algorithms/agc.h @@ -35,7 +35,8 @@ public: const ControlList &controls) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) override; + RkISP1Params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const rkisp1_stat_buffer *stats, diff --git a/src/ipa/rkisp1/algorithms/awb.cpp b/src/ipa/rkisp1/algorithms/awb.cpp index f39153355a12..bc4ace26624a 100644 --- a/src/ipa/rkisp1/algorithms/awb.cpp +++ b/src/ipa/rkisp1/algorithms/awb.cpp @@ -122,8 +122,9 @@ void Awb::queueRequest(IPAContext &context, const uint32_t frame, /** * \copydoc libcamera::ipa::Algorithm::prepare */ -void Awb::prepare(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameContext, RkISP1Params *params) +void Awb::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, RkISP1Params *params, + bool initialize) { awbAlgo_.prepare(context.activeState.awb, frameContext.awb); @@ -136,7 +137,7 @@ void Awb::prepare(IPAContext &context, const uint32_t frame, gainConfig->gain_green_r = std::clamp(256 * frameContext.awb.gains.g(), 0, 0x3ff); /* If we have already set the AWB measurement parameters, return. */ - if (frame > 0) + if (!initialize) return; auto awbConfig = params->block(); diff --git a/src/ipa/rkisp1/algorithms/awb.h b/src/ipa/rkisp1/algorithms/awb.h index 89a3e37b20f9..f4d1b35eea03 100644 --- a/src/ipa/rkisp1/algorithms/awb.h +++ b/src/ipa/rkisp1/algorithms/awb.h @@ -39,7 +39,8 @@ public: const ControlList &controls) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) override; + RkISP1Params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const rkisp1_stat_buffer *stats, diff --git a/src/ipa/rkisp1/algorithms/blc.cpp b/src/ipa/rkisp1/algorithms/blc.cpp index 1ed700a205c8..b7c05c6d4892 100644 --- a/src/ipa/rkisp1/algorithms/blc.cpp +++ b/src/ipa/rkisp1/algorithms/blc.cpp @@ -127,14 +127,15 @@ int BlackLevelCorrection::configure(IPAContext &context, * \copydoc libcamera::ipa::Algorithm::prepare */ void BlackLevelCorrection::prepare(IPAContext &context, - const uint32_t frame, + [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, - RkISP1Params *params) + RkISP1Params *params, + bool initialize) { if (context.configuration.raw) return; - if (frame > 0) + if (!initialize) return; if (!supported_) diff --git a/src/ipa/rkisp1/algorithms/blc.h b/src/ipa/rkisp1/algorithms/blc.h index 3b2b0ce6e2a8..6d0e9c2c7523 100644 --- a/src/ipa/rkisp1/algorithms/blc.h +++ b/src/ipa/rkisp1/algorithms/blc.h @@ -24,7 +24,8 @@ public: const IPACameraSensorInfo &configInfo) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) override; + RkISP1Params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const rkisp1_stat_buffer *stats, diff --git a/src/ipa/rkisp1/algorithms/ccm.cpp b/src/ipa/rkisp1/algorithms/ccm.cpp index 0e54c987104d..9c79c5f7eb18 100644 --- a/src/ipa/rkisp1/algorithms/ccm.cpp +++ b/src/ipa/rkisp1/algorithms/ccm.cpp @@ -92,7 +92,8 @@ void Ccm::setParameters(RkISP1Params *params, IPAFrameContext &context) * \copydoc libcamera::ipa::Algorithm::prepare */ void Ccm::prepare(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameContext, RkISP1Params *params) + IPAFrameContext &frameContext, RkISP1Params *params, + [[maybe_unused]] bool initialize) { if (frameContext.awb.autoEnabled) ccmAlgo_.prepare(context.activeState.ccm, frameContext.ccm, diff --git a/src/ipa/rkisp1/algorithms/ccm.h b/src/ipa/rkisp1/algorithms/ccm.h index 6689c42092f3..fc855793ddfc 100644 --- a/src/ipa/rkisp1/algorithms/ccm.h +++ b/src/ipa/rkisp1/algorithms/ccm.h @@ -39,7 +39,8 @@ public: const ControlList &controls) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) override; + RkISP1Params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const rkisp1_stat_buffer *stats, diff --git a/src/ipa/rkisp1/algorithms/compress.cpp b/src/ipa/rkisp1/algorithms/compress.cpp index 2afa449608b7..9c9eb6acbfba 100644 --- a/src/ipa/rkisp1/algorithms/compress.cpp +++ b/src/ipa/rkisp1/algorithms/compress.cpp @@ -67,7 +67,8 @@ int Compress::configure(IPAContext &context, void Compress::prepare([[maybe_unused]] IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) + RkISP1Params *params, + [[maybe_unused]] bool initialize) { if (!context.configuration.compress.supported) return; diff --git a/src/ipa/rkisp1/algorithms/compress.h b/src/ipa/rkisp1/algorithms/compress.h index 87797b8ebcc5..758b82108a2b 100644 --- a/src/ipa/rkisp1/algorithms/compress.h +++ b/src/ipa/rkisp1/algorithms/compress.h @@ -23,7 +23,8 @@ public: const IPACameraSensorInfo &configInfo) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) override; + RkISP1Params *params, + bool initialize) override; }; } /* namespace ipa::rkisp1::algorithms */ diff --git a/src/ipa/rkisp1/algorithms/cproc.cpp b/src/ipa/rkisp1/algorithms/cproc.cpp index fae7510de07b..86fe29ac606f 100644 --- a/src/ipa/rkisp1/algorithms/cproc.cpp +++ b/src/ipa/rkisp1/algorithms/cproc.cpp @@ -198,10 +198,11 @@ void ColorProcessing::queueRequest(IPAContext &context, void ColorProcessing::prepare([[maybe_unused]] IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) + RkISP1Params *params, + bool initialize) { /* Check if the algorithm configuration has been updated. */ - if (!frameContext.cproc.update) + if (!frameContext.cproc.update && !initialize) return; auto config = params->block(); diff --git a/src/ipa/rkisp1/algorithms/cproc.h b/src/ipa/rkisp1/algorithms/cproc.h index 1387d4565d3f..cd01da57977c 100644 --- a/src/ipa/rkisp1/algorithms/cproc.h +++ b/src/ipa/rkisp1/algorithms/cproc.h @@ -29,7 +29,8 @@ public: const ControlList &controls) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) override; + RkISP1Params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const rkisp1_stat_buffer *stats, diff --git a/src/ipa/rkisp1/algorithms/dpcc.cpp b/src/ipa/rkisp1/algorithms/dpcc.cpp index 20b35d29cbcd..806c20e82b09 100644 --- a/src/ipa/rkisp1/algorithms/dpcc.cpp +++ b/src/ipa/rkisp1/algorithms/dpcc.cpp @@ -230,11 +230,12 @@ int DefectPixelClusterCorrection::init([[maybe_unused]] IPAContext &context, * \copydoc libcamera::ipa::Algorithm::prepare */ void DefectPixelClusterCorrection::prepare([[maybe_unused]] IPAContext &context, - const uint32_t frame, + [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, - RkISP1Params *params) + RkISP1Params *params, + bool initialize) { - if (frame > 0) + if (!initialize) return; auto config = params->block(); diff --git a/src/ipa/rkisp1/algorithms/dpcc.h b/src/ipa/rkisp1/algorithms/dpcc.h index 50b62e9bab3f..06c47035c8dd 100644 --- a/src/ipa/rkisp1/algorithms/dpcc.h +++ b/src/ipa/rkisp1/algorithms/dpcc.h @@ -22,7 +22,8 @@ public: int init(IPAContext &context, const ValueNode &tuningData) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) override; + RkISP1Params *params, + bool initialize) override; private: rkisp1_cif_isp_dpcc_config config_; diff --git a/src/ipa/rkisp1/algorithms/dpf.cpp b/src/ipa/rkisp1/algorithms/dpf.cpp index b2c9ec568b97..9ad1fb84a75d 100644 --- a/src/ipa/rkisp1/algorithms/dpf.cpp +++ b/src/ipa/rkisp1/algorithms/dpf.cpp @@ -217,10 +217,11 @@ void Dpf::queueRequest(IPAContext &context, /** * \copydoc libcamera::ipa::Algorithm::prepare */ -void Dpf::prepare(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameContext, RkISP1Params *params) +void Dpf::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, + IPAFrameContext &frameContext, RkISP1Params *params, + bool initialize) { - if (!frameContext.dpf.update && frame > 0) + if (!frameContext.dpf.update && !initialize) return; auto config = params->block(); diff --git a/src/ipa/rkisp1/algorithms/dpf.h b/src/ipa/rkisp1/algorithms/dpf.h index b07067cec0a5..5edb633d5100 100644 --- a/src/ipa/rkisp1/algorithms/dpf.h +++ b/src/ipa/rkisp1/algorithms/dpf.h @@ -27,7 +27,8 @@ public: const ControlList &controls) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) override; + RkISP1Params *params, + bool initialize) override; private: struct rkisp1_cif_isp_dpf_config config_; diff --git a/src/ipa/rkisp1/algorithms/filter.cpp b/src/ipa/rkisp1/algorithms/filter.cpp index 2e9b4e285503..38bacce883ed 100644 --- a/src/ipa/rkisp1/algorithms/filter.cpp +++ b/src/ipa/rkisp1/algorithms/filter.cpp @@ -115,10 +115,11 @@ void Filter::queueRequest(IPAContext &context, */ void Filter::prepare([[maybe_unused]] IPAContext &context, [[maybe_unused]] const uint32_t frame, - IPAFrameContext &frameContext, RkISP1Params *params) + IPAFrameContext &frameContext, RkISP1Params *params, + bool initialize) { /* Check if the algorithm configuration has been updated. */ - if (!frameContext.filter.update) + if (!frameContext.filter.update && !initialize) return; static constexpr uint16_t filt_fac_sh0[] = { diff --git a/src/ipa/rkisp1/algorithms/filter.h b/src/ipa/rkisp1/algorithms/filter.h index 9f0188da7880..3b9eb87896e8 100644 --- a/src/ipa/rkisp1/algorithms/filter.h +++ b/src/ipa/rkisp1/algorithms/filter.h @@ -27,7 +27,8 @@ public: const ControlList &controls) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) override; + RkISP1Params *params, + bool initialize) override; }; } /* namespace ipa::rkisp1::algorithms */ diff --git a/src/ipa/rkisp1/algorithms/goc.cpp b/src/ipa/rkisp1/algorithms/goc.cpp index e8f64bf3d5e0..96e5d1ceb831 100644 --- a/src/ipa/rkisp1/algorithms/goc.cpp +++ b/src/ipa/rkisp1/algorithms/goc.cpp @@ -99,12 +99,13 @@ void GammaOutCorrection::queueRequest(IPAContext &context, const uint32_t frame, void GammaOutCorrection::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) + RkISP1Params *params, + bool initialize) { ASSERT(context.hw.numGammaOutSamples == RKISP1_CIF_ISP_GAMMA_OUT_MAX_SAMPLES_V10); - if (!frameContext.goc.update) + if (!frameContext.goc.update && !initialize) return; /* diff --git a/src/ipa/rkisp1/algorithms/goc.h b/src/ipa/rkisp1/algorithms/goc.h index bd79fe19cc86..0cb419aa89aa 100644 --- a/src/ipa/rkisp1/algorithms/goc.h +++ b/src/ipa/rkisp1/algorithms/goc.h @@ -28,7 +28,8 @@ public: const ControlList &controls) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) override; + RkISP1Params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const rkisp1_stat_buffer *stats, diff --git a/src/ipa/rkisp1/algorithms/gsl.cpp b/src/ipa/rkisp1/algorithms/gsl.cpp index d83aab8506f3..cd505dce73cf 100644 --- a/src/ipa/rkisp1/algorithms/gsl.cpp +++ b/src/ipa/rkisp1/algorithms/gsl.cpp @@ -117,11 +117,12 @@ int GammaSensorLinearization::init([[maybe_unused]] IPAContext &context, * \copydoc libcamera::ipa::Algorithm::prepare */ void GammaSensorLinearization::prepare([[maybe_unused]] IPAContext &context, - const uint32_t frame, + [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, - RkISP1Params *params) + RkISP1Params *params, + bool initialize) { - if (frame > 0) + if (!initialize) return; auto config = params->block(); diff --git a/src/ipa/rkisp1/algorithms/gsl.h b/src/ipa/rkisp1/algorithms/gsl.h index 3ef5630713ab..a9aefe501960 100644 --- a/src/ipa/rkisp1/algorithms/gsl.h +++ b/src/ipa/rkisp1/algorithms/gsl.h @@ -22,7 +22,8 @@ public: int init(IPAContext &context, const ValueNode &tuningData) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) override; + RkISP1Params *params, + bool initialize) override; private: uint32_t gammaDx_[2]; diff --git a/src/ipa/rkisp1/algorithms/lsc.cpp b/src/ipa/rkisp1/algorithms/lsc.cpp index 36f9a344ed3b..e965dfb64239 100644 --- a/src/ipa/rkisp1/algorithms/lsc.cpp +++ b/src/ipa/rkisp1/algorithms/lsc.cpp @@ -206,13 +206,14 @@ void LensShadingCorrection::queueRequest(IPAContext &context, void LensShadingCorrection::prepare([[maybe_unused]] IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) + RkISP1Params *params, + bool initialize) { uint32_t ct = frameContext.awb.colourTemperature; unsigned int quantizedCt = quantize(ct, kColourTemperatureQuantization); /* Check if we can skip the update. */ - if (!frameContext.lsc.update) { + if (!frameContext.lsc.update && !initialize) { if (!frameContext.lsc.enabled) return; diff --git a/src/ipa/rkisp1/algorithms/lsc.h b/src/ipa/rkisp1/algorithms/lsc.h index c1f8904426c0..7ea986ae1ca8 100644 --- a/src/ipa/rkisp1/algorithms/lsc.h +++ b/src/ipa/rkisp1/algorithms/lsc.h @@ -35,7 +35,8 @@ public: const ControlList &controls) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) override; + RkISP1Params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const rkisp1_stat_buffer *stats, diff --git a/src/ipa/rkisp1/algorithms/lux.cpp b/src/ipa/rkisp1/algorithms/lux.cpp index ce6928a55d2b..eabeecdb4785 100644 --- a/src/ipa/rkisp1/algorithms/lux.cpp +++ b/src/ipa/rkisp1/algorithms/lux.cpp @@ -51,7 +51,8 @@ int Lux::init([[maybe_unused]] IPAContext &context, const ValueNode &tuningData) */ void Lux::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, - [[maybe_unused]] RkISP1Params *params) + [[maybe_unused]] RkISP1Params *params, + [[maybe_unused]] bool initialize) { frameContext.lux.lux = context.activeState.lux.lux; } diff --git a/src/ipa/rkisp1/algorithms/lux.h b/src/ipa/rkisp1/algorithms/lux.h index 8cb35cbae20d..6cb4653a1fa1 100644 --- a/src/ipa/rkisp1/algorithms/lux.h +++ b/src/ipa/rkisp1/algorithms/lux.h @@ -25,7 +25,8 @@ public: int init(IPAContext &context, const ValueNode &tuningData) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) override; + RkISP1Params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const rkisp1_stat_buffer *stats, diff --git a/src/ipa/rkisp1/algorithms/wdr.cpp b/src/ipa/rkisp1/algorithms/wdr.cpp index c3d73da2c5b2..046495a2f7b5 100644 --- a/src/ipa/rkisp1/algorithms/wdr.cpp +++ b/src/ipa/rkisp1/algorithms/wdr.cpp @@ -394,7 +394,8 @@ void WideDynamicRange::queueRequest([[maybe_unused]] IPAContext &context, void WideDynamicRange::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) + RkISP1Params *params, + [[maybe_unused]] bool initialize) { if (!params) { LOG(RkISP1Wdr, Warning) << "Params is null"; diff --git a/src/ipa/rkisp1/algorithms/wdr.h b/src/ipa/rkisp1/algorithms/wdr.h index f79de66fe73b..7cfd13c8e8aa 100644 --- a/src/ipa/rkisp1/algorithms/wdr.h +++ b/src/ipa/rkisp1/algorithms/wdr.h @@ -31,7 +31,8 @@ public: const ControlList &controls) override; void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - RkISP1Params *params) override; + RkISP1Params *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const rkisp1_stat_buffer *stats, diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 750c0fa72733..8627b4044d30 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -329,8 +329,10 @@ uint32_t IPARkISP1::computeParamsInternal(IPAFrameContext &frameContext, const u RkISP1Params params(context_.configuration.paramFormat, mappedBuffers_.at(bufferId).planes()[0]); + unsigned int frame = frameContext.frame(); for (const auto &algo : algorithms()) - algo->prepare(context_, frameContext.frame(), frameContext, ¶ms); + algo->prepare(context_, frame, frameContext, + ¶ms, frame == 0); return params.bytesused(); } diff --git a/src/ipa/softisp/algorithms/adjust.cpp b/src/ipa/softisp/algorithms/adjust.cpp index 56e2cf0e8362..ec4698c4b376 100644 --- a/src/ipa/softisp/algorithms/adjust.cpp +++ b/src/ipa/softisp/algorithms/adjust.cpp @@ -95,7 +95,8 @@ void Adjust::applySaturation(Matrix &matrix, float saturation) void Adjust::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, - DebayerParams *params) + DebayerParams *params, + [[maybe_unused]] bool initialize) { frameContext.gamma = context.activeState.knobs.gamma; frameContext.contrast = context.activeState.knobs.contrast; diff --git a/src/ipa/softisp/algorithms/adjust.h b/src/ipa/softisp/algorithms/adjust.h index 1acf7cdf1262..1c37f09035cd 100644 --- a/src/ipa/softisp/algorithms/adjust.h +++ b/src/ipa/softisp/algorithms/adjust.h @@ -35,7 +35,8 @@ public: void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - DebayerParams *params) override; + DebayerParams *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const SwIspStats *stats, diff --git a/src/ipa/softisp/algorithms/agc.cpp b/src/ipa/softisp/algorithms/agc.cpp index 7d2a53b38576..0453b0215a60 100644 --- a/src/ipa/softisp/algorithms/agc.cpp +++ b/src/ipa/softisp/algorithms/agc.cpp @@ -72,7 +72,9 @@ void Agc::queueRequest(IPAContext &context, [[maybe_unused]] const uint32_t fram } void Agc::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, - IPAFrameContext &frameContext, [[maybe_unused]] DebayerParams *params) + IPAFrameContext &frameContext, + [[maybe_unused]] DebayerParams *params, + [[maybe_unused]] bool initialize) { agc_.prepare(context.configuration.agc, context.activeState.agc, frameContext.agc); } diff --git a/src/ipa/softisp/algorithms/agc.h b/src/ipa/softisp/algorithms/agc.h index 10484cb448f8..3337b3a15044 100644 --- a/src/ipa/softisp/algorithms/agc.h +++ b/src/ipa/softisp/algorithms/agc.h @@ -26,7 +26,8 @@ public: IPAFrameContext &frameContext, const ControlList &controls) override; void prepare(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameContext, DebayerParams *params) override; + IPAFrameContext &frameContext, DebayerParams *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, diff --git a/src/ipa/softisp/algorithms/awb.cpp b/src/ipa/softisp/algorithms/awb.cpp index 55fd326fc182..c8bfc16a97a9 100644 --- a/src/ipa/softisp/algorithms/awb.cpp +++ b/src/ipa/softisp/algorithms/awb.cpp @@ -105,7 +105,8 @@ void Awb::queueRequest(IPAContext &context, const uint32_t frame, void Awb::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, IPAFrameContext &frameContext, - DebayerParams *params) + DebayerParams *params, + [[maybe_unused]] bool initialize) { awbAlgo_.prepare(context.activeState.awb, frameContext.awb); diff --git a/src/ipa/softisp/algorithms/awb.h b/src/ipa/softisp/algorithms/awb.h index c92bdbfd5780..1dcd84b7d99a 100644 --- a/src/ipa/softisp/algorithms/awb.h +++ b/src/ipa/softisp/algorithms/awb.h @@ -40,7 +40,8 @@ public: void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - DebayerParams *params) override; + DebayerParams *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, diff --git a/src/ipa/softisp/algorithms/blc.cpp b/src/ipa/softisp/algorithms/blc.cpp index c7714aa34506..734cbcfd132b 100644 --- a/src/ipa/softisp/algorithms/blc.cpp +++ b/src/ipa/softisp/algorithms/blc.cpp @@ -62,7 +62,8 @@ int BlackLevel::configure(IPAContext &context, void BlackLevel::prepare(IPAContext &context, [[maybe_unused]] const uint32_t frame, [[maybe_unused]] IPAFrameContext &frameContext, - DebayerParams *params) + DebayerParams *params, + [[maybe_unused]] bool initialize) { /* Latch the blacklevel gain so GPUISP can apply. */ params->blackLevel = RGB(context.activeState.blc.level / 255.0f); diff --git a/src/ipa/softisp/algorithms/blc.h b/src/ipa/softisp/algorithms/blc.h index 1d602927f9f3..3fae12c350fb 100644 --- a/src/ipa/softisp/algorithms/blc.h +++ b/src/ipa/softisp/algorithms/blc.h @@ -27,7 +27,8 @@ public: void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - DebayerParams *params) override; + DebayerParams *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const SwIspStats *stats, diff --git a/src/ipa/softisp/algorithms/ccm.cpp b/src/ipa/softisp/algorithms/ccm.cpp index 039074494c9d..f965fb5d57c1 100644 --- a/src/ipa/softisp/algorithms/ccm.cpp +++ b/src/ipa/softisp/algorithms/ccm.cpp @@ -53,7 +53,9 @@ void Ccm::queueRequest(IPAContext &context, } void Ccm::prepare(IPAContext &context, const uint32_t frame, - IPAFrameContext &frameContext, [[maybe_unused]] DebayerParams *params) + IPAFrameContext &frameContext, + [[maybe_unused]] DebayerParams *params, + [[maybe_unused]] bool initialize) { if (frameContext.awb.autoEnabled) ccmAlgo_.prepare(context.activeState.ccm, frameContext.ccm, diff --git a/src/ipa/softisp/algorithms/ccm.h b/src/ipa/softisp/algorithms/ccm.h index 2a2030a41915..6eabe5e0cecd 100644 --- a/src/ipa/softisp/algorithms/ccm.h +++ b/src/ipa/softisp/algorithms/ccm.h @@ -35,7 +35,8 @@ public: void prepare(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, - DebayerParams *params) override; + DebayerParams *params, + bool initialize) override; void process(IPAContext &context, const uint32_t frame, IPAFrameContext &frameContext, const SwIspStats *stats, diff --git a/src/ipa/softisp/softisp.cpp b/src/ipa/softisp/softisp.cpp index 2a65cd7cebfe..0d26ad210ebf 100644 --- a/src/ipa/softisp/softisp.cpp +++ b/src/ipa/softisp/softisp.cpp @@ -241,7 +241,7 @@ void IPASoftIsp::computeParams(const uint32_t frame) ASSERT(frameContext); for (const auto &algo : algorithms()) - algo->prepare(context_, frame, *frameContext, params_); + algo->prepare(context_, frame, *frameContext, params_, frame == 0); params_->combinedMatrix = context_.activeState.combinedMatrix; paramsComputed.emit(frame); From patchwork Mon Sep 14 14:02:51 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28282 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E7E01C3368 for ; Mon, 14 Sep 2026 14:05:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 43F5368716; Mon, 14 Sep 2026 16:05:09 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="tHUIGJCd"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1EB9068716 for ; Mon, 14 Sep 2026 16:05:07 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 2BE22512; Mon, 14 Sep 2026 16:03:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394607; bh=TFxhtbHvbIZ9IQ0dgoTOz2berkdVByWH0UVL3YfNy7U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tHUIGJCd0uRroJVkliauHJXS1RG0zKEmhBOGMnEURVmbP+Hm7CSruZkKf+UTV5Zzb ZxZYmx/WJwYBE/m3j+5D5NLIonVoOO6g8p3vJLZHU1bX3V8HNzsPg/4y2u5+xveVS/ nV/IHOaj4zKF2PrSWdx2Zrab8Hgb3I74QM2l080I= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 38/41] ipa: rkisp1: Gracefully handle FCQueue overruns Date: Mon, 14 Sep 2026 16:02:51 +0200 Message-ID: <20260914140309.3354666-39-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Gracefully handle all cases where the FCQueue could overflow. The most interesting case is computeParams(), because in that situation we don't know what we might have missed and therefore need to ensure that the ISP parameters get fully reinitialised on the next successful call to computeParams(). In the other cases we just log an error. Signed-off-by: Stefan Klug --- src/ipa/rkisp1/rkisp1.cpp | 27 ++++++++++++++++++++++----- 1 file changed, 22 insertions(+), 5 deletions(-) diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 8627b4044d30..ca3b0d31d9f6 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -85,6 +85,7 @@ private: /* Local parameter storage */ struct IPAContext context_; + bool initializeParams_; }; namespace { @@ -218,6 +219,7 @@ void IPARkISP1::start(const ControlList &controls, const uint32_t paramBufferId, { IPAFrameContext *frameContext = context_.frameContexts.getOrInitContext(0, controls); ASSERT(frameContext); + initializeParams_ = true; if (paramBufferId != 0) result->paramBufferBytesUsed = computeParamsInternal(*frameContext, @@ -332,7 +334,9 @@ uint32_t IPARkISP1::computeParamsInternal(IPAFrameContext &frameContext, const u unsigned int frame = frameContext.frame(); for (const auto &algo : algorithms()) algo->prepare(context_, frame, frameContext, - ¶ms, frame == 0); + ¶ms, initializeParams_); + + initializeParams_ = false; return params.bytesused(); } @@ -340,7 +344,15 @@ uint32_t IPARkISP1::computeParamsInternal(IPAFrameContext &frameContext, const u void IPARkISP1::computeParams(const uint32_t frame, const uint32_t bufferId) { IPAFrameContext *frameContext = context_.frameContexts.getOrInitContext(frame); - ASSERT(frameContext); + + if (!frameContext) { + LOG(IPARkISP1, Error) << "Failed to compute params for frame: " + << frame; + initializeParams_ = true; + if (bufferId != 0) + paramsComputed.emit(frame, bufferId, 0); + return; + } if (bufferId != 0) { uint32_t size = computeParamsInternal(*frameContext, bufferId); @@ -355,7 +367,14 @@ void IPARkISP1::processStats(const uint32_t frame, const uint32_t bufferId, const ControlList &sensorControls) { IPAFrameContext *frameContext = context_.frameContexts.getOrInitContext(frame); - ASSERT(frameContext); + ControlList metadata(controls::controls); + + if (!frameContext) { + LOG(IPARkISP1, Error) << "Failed to process stats for frame: " + << frame; + metadataReady.emit(frame, bufferId, metadata); + return; + } /* * In raw capture mode, the ISP is bypassed and no statistics buffer is @@ -369,8 +388,6 @@ void IPARkISP1::processStats(const uint32_t frame, const uint32_t bufferId, std::tie(frameContext->sensor.exposure, frameContext->sensor.gain) = agc::extractControls(sensorControls, context_.camHelper.get()); - ControlList metadata(controls::controls); - for (const auto &a : algorithms()) { Algorithm *algo = static_cast(a.get()); if (algo->disabled_) From patchwork Mon Sep 14 14:02:52 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28283 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C0701C3369 for ; Mon, 14 Sep 2026 14:05:11 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4705C68700; Mon, 14 Sep 2026 16:05:11 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="RRbGT7df"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 07673686B5 for ; Mon, 14 Sep 2026 16:05:09 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 1328D512; Mon, 14 Sep 2026 16:03:29 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394609; bh=ZPL5vGE2AHSWC7DNb8XqzqupLruS1w0usmfdgTKjoD4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RRbGT7dfByCBC9nlYT3+5g+0iRi9rzhN9e7UI0+wAAcnVG2aeq2xRZL0lOreyE1nO 9ALD2NkXMcE4Mv5Oxmpr67HxIxWjrduXHBOT33IkbJjFptCI2bRq/QIDGI+809pkSd VC+cRLYniWJSvog/Lc67goOgdbapdk6GEhIbovr4= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 39/41] ipa: rkisp1: Handle frame jumps in calls to computeParams() Date: Mon, 14 Sep 2026 16:02:52 +0200 Message-ID: <20260914140309.3354666-40-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In the situation of a parameter buffer underrun (due to some external event) the calls to computeParams will jump frames. It must be guaranteed that every change in the jumped frames still gets applied to the isp. To do that, call computeParams() on every missed frame or issue a full reinitialization if more than 3 frames were lost. The number of 3 is arbitrarily chosen but fits to the subjective observation that we either lose 1-2 frames due to signal issues or way more than 3 because something else blocked the system. Signed-off-by: Stefan Klug --- src/ipa/rkisp1/rkisp1.cpp | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index ca3b0d31d9f6..7147de4a0f02 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -86,6 +86,7 @@ private: /* Local parameter storage */ struct IPAContext context_; bool initializeParams_; + uint32_t lastParamsComputed_; }; namespace { @@ -220,6 +221,7 @@ void IPARkISP1::start(const ControlList &controls, const uint32_t paramBufferId, IPAFrameContext *frameContext = context_.frameContexts.getOrInitContext(0, controls); ASSERT(frameContext); initializeParams_ = true; + lastParamsComputed_ = 0; if (paramBufferId != 0) result->paramBufferBytesUsed = computeParamsInternal(*frameContext, @@ -332,10 +334,37 @@ uint32_t IPARkISP1::computeParamsInternal(IPAFrameContext &frameContext, const u mappedBuffers_.at(bufferId).planes()[0]); unsigned int frame = frameContext.frame(); + + /* + * In the corner case that the previous params buffer was not computed + * (due to resynchronization), there is a risk that a change was missed + * and not sent to the kernel. This can have quite negative side + * effects, if e.g. a lsc table was not written. To mitigate that, run + * over all missed frames and apply them in turn or do a full reinit if + * too many frames were missed. + */ + while (!initializeParams_ && lastParamsComputed_ + 1 < frame) { + lastParamsComputed_++; + IPAFrameContext *fc = context_.frameContexts.getOrInitContext(lastParamsComputed_); + if (!fc || frame - lastParamsComputed_ > 3) { + LOG(IPARkISP1, Warning) + << "Collect missed params with full reinit"; + initializeParams_ = true; + break; + } + + LOG(IPARkISP1, Warning) << "Collect missed params for frame: " + << lastParamsComputed_; + for (const auto &algo : algorithms()) + algo->prepare(context_, lastParamsComputed_, *fc, + ¶ms, false); + } + for (const auto &algo : algorithms()) algo->prepare(context_, frame, frameContext, ¶ms, initializeParams_); + lastParamsComputed_ = std::max(frame, lastParamsComputed_); initializeParams_ = false; return params.bytesused(); @@ -376,6 +405,11 @@ void IPARkISP1::processStats(const uint32_t frame, const uint32_t bufferId, return; } + if (frame > lastParamsComputed_) { + LOG(IPARkISP1, Debug) << "Process stats on frame " << frame + << " without prior compute params"; + } + /* * In raw capture mode, the ISP is bypassed and no statistics buffer is * provided. From patchwork Mon Sep 14 14:02:53 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28284 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9B94FC336A for ; Mon, 14 Sep 2026 14:05:13 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id ECB9F68722; Mon, 14 Sep 2026 16:05:12 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="donrIXDu"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 373AD686FC for ; Mon, 14 Sep 2026 16:05:11 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 33599929; Mon, 14 Sep 2026 16:03:31 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394611; bh=raQ/c+eS4u2LKeTBRlo0CbtF8LZEQRdQt+IB7gIc3aw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=donrIXDu2aj/zpupFVHJQE24ffk/Moex5nMv++fYxYkckY+6FFGiTh6gS8k771HhE q0/Br7pApq/DcKAh5ZzKI1bDeA11IGhGUccINK1V8kzKQ+cJf/6K1VgA3pz+zHjOZT 6FBqLNBoKN4BkLfQSjqjjP0Qa7mCYIfKz784l6KA= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 40/41] libipa: agc: Make startup frames and regulations speed configurable Date: Mon, 14 Sep 2026 16:02:53 +0200 Message-ID: <20260914140309.3354666-41-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The rkisp1 has now a good synchronization between sensor settings/isp params and stats. It can therefore benefit from a faster regulation. To do that without breaking other IPAs that use AgcMeanLuminance, move regulation speed and startup frames into configuration parameters that can be set at runtime. Signed-off-by: Stefan Klug --- src/ipa/libipa/agc.cpp | 14 ++++++++++++++ src/ipa/libipa/agc.h | 2 ++ src/ipa/libipa/agc_mean_luminance.cpp | 25 +++++++++++++++++++++---- src/ipa/libipa/agc_mean_luminance.h | 3 +++ 4 files changed, 40 insertions(+), 4 deletions(-) diff --git a/src/ipa/libipa/agc.cpp b/src/ipa/libipa/agc.cpp index a2438cb49d1b..5c06ca475926 100644 --- a/src/ipa/libipa/agc.cpp +++ b/src/ipa/libipa/agc.cpp @@ -262,6 +262,15 @@ namespace agc { * control only, without automatic adjustments. In this mode statistics * must not be provided to AgcAlgorithm::process(), and ExposureTimeMode * and AnalogueGainMode will only advertise manual control. + * + * \var AgcAlgorithm::ConfigurationParams::numStartupFrames + * \brief Number of startup frames + * + * During these frame the regulation speed is set to 1.0 to reach faster + * convergence. + * + * \var AgcAlgorithm::ConfigurationParams::regulationSpeed + * \brief The regulation speed */ /** @@ -513,6 +522,11 @@ int AgcAlgorithm::configure(agc::Session &session, agc::ActiveState &state, state.automatic.yTarget = impl.effectiveYTarget(0, 1); impl.configure(session.lineDuration, sensor_); + if (config.numStartupFrames) + impl.numStartupFrames_ = config.numStartupFrames.value(); + + if (config.regulationSpeed) + impl.regulationSpeed_ = config.regulationSpeed.value(); if (!session.autoAllowed) return; diff --git a/src/ipa/libipa/agc.h b/src/ipa/libipa/agc.h index 4914c9cee24b..a9140e7a3e61 100644 --- a/src/ipa/libipa/agc.h +++ b/src/ipa/libipa/agc.h @@ -126,6 +126,8 @@ public: const ControlInfoMap &sensorControls; ControlInfoMap::Map &ctrlMap; bool autoAllowed = true; + std::optional numStartupFrames = std::nullopt; + std::optional regulationSpeed = std::nullopt; }; struct ProcessParams { diff --git a/src/ipa/libipa/agc_mean_luminance.cpp b/src/ipa/libipa/agc_mean_luminance.cpp index 72833247c5fd..3244e631517b 100644 --- a/src/ipa/libipa/agc_mean_luminance.cpp +++ b/src/ipa/libipa/agc_mean_luminance.cpp @@ -37,7 +37,7 @@ namespace ipa { * Number of frames for which to run the algorithm at full speed, before slowing * down to prevent large and jarring changes in exposure from frame to frame. */ -static constexpr uint32_t kNumStartupFrames = 10; +static constexpr uint32_t kDefaultNumStartupFrames = 10; /* * Default relative luminance target @@ -178,7 +178,8 @@ static constexpr unsigned int kDefaultLuxLevel = 500; */ AgcMeanLuminance::AgcMeanLuminance() - : filteredExposure_(0s), luxWarningEnabled_(true), frameCount_(0) + : numStartupFrames_(kDefaultNumStartupFrames), regulationSpeed_(0.2), + filteredExposure_(0s), luxWarningEnabled_(true), frameCount_(0) { } @@ -587,10 +588,10 @@ double AgcMeanLuminance::effectiveYTarget(double lux, double exposureCompensatio */ utils::Duration AgcMeanLuminance::filterExposure(utils::Duration exposureValue) { - double speed = 0.2; + double speed = regulationSpeed_; /* Adapt instantly if we are in startup phase. */ - if (frameCount_ < kNumStartupFrames) + if (frameCount_ < numStartupFrames_) speed = 1.0; /* @@ -701,6 +702,22 @@ AgcMeanLuminance::calculateNewEv(const Params ¶ms) return { exposureModeHelper.splitExposure(newExposureValue), yTarget }; } +/** + * \var AgcMeanLuminance::numStartupFrames_ + * \brief The number of startup frames + * + * During this number of frames after startup, the regulation is very aggressive + * to reach the target value within one or two cycles. + * + * \var AgcMeanLuminance::regulationSpeed_ + * The regulation speed. This controls the speed at which new target values are + * applied. The new target value is calculated as: + * + * \code{.unparsed} + * value = target * speed + oldValue * (1.0 - speed) + * \endcode + */ + } /* namespace ipa */ } /* namespace libcamera */ diff --git a/src/ipa/libipa/agc_mean_luminance.h b/src/ipa/libipa/agc_mean_luminance.h index f799618ffc2c..b7df7ea03702 100644 --- a/src/ipa/libipa/agc_mean_luminance.h +++ b/src/ipa/libipa/agc_mean_luminance.h @@ -79,6 +79,9 @@ public: double effectiveYTarget(double lux, double exposureCompensation) const; + uint32_t numStartupFrames_; + double regulationSpeed_; + private: int parseRelativeLuminanceTarget(const ValueNode &tuningData); int parseConstraint(const ValueNode &modeDict, int32_t id); From patchwork Mon Sep 14 14:02:54 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Klug X-Patchwork-Id: 28285 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id CE189C327D for ; Mon, 14 Sep 2026 14:05:16 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 13C396872B; Mon, 14 Sep 2026 16:05:16 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="i1tC5Q9K"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 980D7686FC for ; Mon, 14 Sep 2026 16:05:14 +0200 (CEST) Received: from ideasonboard.com (unknown [IPv6:2a00:6020:448c:6c00:a279:75fa:1f6c:7f40]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 9D9AEC14; Mon, 14 Sep 2026 16:03:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1789394614; bh=5Fv6TX6SsefuJFcdFIpAt6c+RZ+DeHocFzw/lruY12c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=i1tC5Q9KymjFCfxZRQLB9gENPjWQd53rEfTmwlttV6Hg8D4XjWMsJMxjLeM9o8vL3 /ywJwRAJldAaeRHMFsPpX9qITIf+a/fp3g1Q9N5gl2lxpXWhKUXGhmdAHHRLX+1IN0 Ly1sRWHJjuVYfq1Cs0AKtYDnWz9O9VUDThxRpihQ= From: Stefan Klug To: libcamera-devel@lists.libcamera.org Cc: Stefan Klug Subject: [PATCH v3 41/41] ipa: rkisp1: Increase regulation speed Date: Mon, 14 Sep 2026 16:02:54 +0200 Message-ID: <20260914140309.3354666-42-stefan.klug@ideasonboard.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> References: <20260914140309.3354666-1-stefan.klug@ideasonboard.com> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that the rkisp1 is well synchronized, we can safely increase the regulation speed and reduce the number of startup frames. With current settings, the results based on the stats for frame 0 are active on frame 5. Setting the startup frames to 7 leaves room for one additional lost frame. While at it, format the code in a checkstyle compatible way. Signed-off-by: Stefan Klug --- src/ipa/rkisp1/algorithms/agc.cpp | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/src/ipa/rkisp1/algorithms/agc.cpp b/src/ipa/rkisp1/algorithms/agc.cpp index 41680e5c4ffd..0c1ad3b6c525 100644 --- a/src/ipa/rkisp1/algorithms/agc.cpp +++ b/src/ipa/rkisp1/algorithms/agc.cpp @@ -137,11 +137,13 @@ int Agc::init(IPAContext &context, const ValueNode &tuningData) { int ret; - ret = agc_.init(tuningData, context.camHelper.get(), { - .sensorInfo = context.sensorInfo, - .sensorControls = context.sensorControls, - .ctrlMap = context.ctrlMap, - }); + ret = agc_.init(tuningData, context.camHelper.get(), + { .sensorInfo = context.sensorInfo, + .sensorControls = context.sensorControls, + .ctrlMap = context.ctrlMap, + /* rkisp1 is well synchronized, increase the speed. */ + .numStartupFrames = 7, + .regulationSpeed = 0.6 }); if (ret) return ret;