From patchwork Mon Dec 12 15:02:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 17981 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 63413BDE6B for ; Mon, 12 Dec 2022 15:03:10 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 22D1D63361; Mon, 12 Dec 2022 16:03:10 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1670857390; bh=YWPWwqQkHBcUXVJShx6TFB02q7dRvu1h8sWW3ltEzD0=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=bsVwbRAcswi6FuiBBDZt9VLHyxRfBrXzPNH0SjyPEFrzdW/qFqC7lKHOLqFatwY9f HXLiiRmSINmvMQ+4YPRMv9yJiTSg2wq6/dTMfFnmHFMew9IfE4AblbYn5BaSxn88FE L3zJVcrSfxQLxZdu4GVZczwREawYRnCbMG9UX277hhfLXFwmkqjflPlicIEUjOWu1y 6OgFpVwYHskx1/MH1AOkDYlkmWE3eQxZVSmOwf0naCZuD/MVrWV4ROqyNLGYG0PTp/ hTz9KIEdMquP1VB4NoGIUnYH8Vu8M2rq+TReQYy8UDp8j7280cEjZLnKh3l09/J6nz T2WOLpIPnLqHA== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A22CB63361 for ; Mon, 12 Dec 2022 16:03:08 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="SM9u1sQq"; dkim-atps=neutral Received: from umang.jainideasonboard.com (unknown [IPv6:2401:4900:1f3e:7d24:3f0:3e81:fb16:ab4d]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 38FB46CF; Mon, 12 Dec 2022 16:03:06 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1670857388; bh=YWPWwqQkHBcUXVJShx6TFB02q7dRvu1h8sWW3ltEzD0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SM9u1sQqV3N2l7Pao6Af3949b3MqXPkDtYotvC7WcheOAf7eAkZNeMWtM6zjvFv0Q PU86HKsDJRiz3ghDubkorBfJ6L1XHFFqeAv3SYo794hgyhR+10mQimHN4CADePlSOF PLgoN4j6ReOu7kF4u9tOOts1i8SXp6dLlS+yomhA= To: libcamera-devel@lists.libcamera.org Date: Mon, 12 Dec 2022 20:32:55 +0530 Message-Id: <20221212150256.69004-2-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221212150256.69004-1-umang.jain@ideasonboard.com> References: <20221212150256.69004-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 1/2] ipa: Move kMaxFrameContexts to mojom interface X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Umang Jain via libcamera-devel From: Umang Jain Reply-To: Umang Jain Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Move kMaxFrameContexts to the mojom interface. This will prepare the pipeline handler class to manage queuing of requests to the IPA. Signed-off-by: Umang Jain --- include/libcamera/ipa/ipu3.mojom | 3 +++ include/libcamera/ipa/rkisp1.mojom | 3 +++ src/ipa/ipu3/ipu3.cpp | 3 --- src/ipa/rkisp1/rkisp1.cpp | 3 --- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/include/libcamera/ipa/ipu3.mojom b/include/libcamera/ipa/ipu3.mojom index d1b1c6b8..b1ecd57f 100644 --- a/include/libcamera/ipa/ipu3.mojom +++ b/include/libcamera/ipa/ipu3.mojom @@ -8,6 +8,9 @@ module ipa.ipu3; import "include/libcamera/ipa/core.mojom"; +/* Maximum number of frame contexts to be held */ +const uint32 kMaxFrameContexts = 16; + struct IPAConfigInfo { libcamera.IPACameraSensorInfo sensorInfo; libcamera.ControlInfoMap sensorControls; diff --git a/include/libcamera/ipa/rkisp1.mojom b/include/libcamera/ipa/rkisp1.mojom index 1009e970..62c59fbd 100644 --- a/include/libcamera/ipa/rkisp1.mojom +++ b/include/libcamera/ipa/rkisp1.mojom @@ -8,6 +8,9 @@ module ipa.rkisp1; import "include/libcamera/ipa/core.mojom"; +/* Maximum number of frame contexts to be held */ +const uint32 kMaxFrameContexts = 16; + struct IPAConfigInfo { libcamera.IPACameraSensorInfo sensorInfo; libcamera.ControlInfoMap sensorControls; diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 08ee6eb3..1cfae8fe 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -55,9 +55,6 @@ static constexpr uint32_t kMinCellSizeLog2 = 3; /* log2 of the maximum grid cell width and height, in pixels */ static constexpr uint32_t kMaxCellSizeLog2 = 6; -/* Maximum number of frame contexts to be held */ -static constexpr uint32_t kMaxFrameContexts = 16; - namespace libcamera { LOG_DEFINE_CATEGORY(IPAIPU3) diff --git a/src/ipa/rkisp1/rkisp1.cpp b/src/ipa/rkisp1/rkisp1.cpp index 6544c925..509be2d3 100644 --- a/src/ipa/rkisp1/rkisp1.cpp +++ b/src/ipa/rkisp1/rkisp1.cpp @@ -41,9 +41,6 @@ using namespace std::literals::chrono_literals; namespace ipa::rkisp1 { -/* Maximum number of frame contexts to be held */ -static constexpr uint32_t kMaxFrameContexts = 16; - class IPARkISP1 : public IPARkISP1Interface, public Module { public: From patchwork Mon Dec 12 15:02:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 17982 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C38E0BDE6B for ; Mon, 12 Dec 2022 15:03:12 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8C58763368; Mon, 12 Dec 2022 16:03:12 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1670857392; bh=XcTwOEslkOFI0txG9wzzN/XmWGaYwB5fIdS3RRnzCKI=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=Yu+g8PIX1Xq2DndYyslszFvaofUcwJUr04dpTqafs0UCmgKlPKIwbMLCax+yHlIzG lAdvORBhhslDA8RPLpqjVdG2FegoGcGetQkf7Innrtx9hIuqhqiEHRMGS8lgZyhFwl IJuGAXwq1qhY8QZ66Of6oprxFfs3MtkVpQKmWpiRf+2boIED3VRmWfv2Sg5qzmVSkF 8tX0eoE9yNgFW5S0cHIfUv+ggL4CDW5Z7FlsfgKvtY87AxKXM85njRZGhkGSWY838s 447uYceFZzUxg9jSor96jGbKlADjUJIZeVHtkA3Sp6k9MgjbzwkgN+Seo2fHQNfL0r 09Cj+YXSICPKQ== Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3AB276333F for ; Mon, 12 Dec 2022 16:03:11 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="DkFZdc83"; dkim-atps=neutral Received: from umang.jainideasonboard.com (unknown [IPv6:2401:4900:1f3e:7d24:3f0:3e81:fb16:ab4d]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C182B6CF; Mon, 12 Dec 2022 16:03:09 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1670857391; bh=XcTwOEslkOFI0txG9wzzN/XmWGaYwB5fIdS3RRnzCKI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DkFZdc83c5CCiheDk7KacCD41p3ERey/DGcV/eRbpNOXdktbEQwDLNz3fIk+5aLRG 7qDNyyccY1hvjPwyXkdMGrvX2vfR8sDoD12FJtAIJMkIsLvzggRM3MtsMBScsg20HC ElXqjNzx5hf1Y89FiluIZc+jGgAnmUn/gl3LXcdU= To: libcamera-devel@lists.libcamera.org Date: Mon, 12 Dec 2022 20:32:56 +0530 Message-Id: <20221212150256.69004-3-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221212150256.69004-1-umang.jain@ideasonboard.com> References: <20221212150256.69004-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 2/2] libcamera: pipeline-handler: Consider max in-flight requests constraint X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Umang Jain via libcamera-devel From: Umang Jain Reply-To: Umang Jain Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This patch introduces a new constraint in the pipeline handler base class which allows various derived pipeline handlers to set and restrict the number of maximum in-flight requests that can be queued to the underlying components (for e.g. the IPA). The pipeline handler is now equipped with the responsibility of not to over queue the requests to the underlying layers. The derived pipeline handler (or even IPA) can also have various kind of requests queue(arrays, queues or ring-buffer) hence, it might be an issue where the application queues the requests at a rate where these kind of queues can over-flow. The patch ensures that the base PipelineHandler will never let the requests overflow to underlying layers, once the derived pipeline handler sets its maximum capacity of handling in-flight requests using PipelineHandler::setMaxQueueRequests(). The queue request management introduced in the pipeline handler base class is now used by the IPU3 and RkISP1 pipeline handlers. This will prevent over-writing of frame contexts (i.e. FCQueue) in these two pipeline handlers. Signed-off-by: Umang Jain --- include/libcamera/internal/pipeline_handler.h | 4 ++ src/libcamera/pipeline/ipu3/ipu3.cpp | 1 + src/libcamera/pipeline/rkisp1/rkisp1.cpp | 1 + src/libcamera/pipeline_handler.cpp | 51 ++++++++++++++++++- 4 files changed, 56 insertions(+), 1 deletion(-) diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h index ec4f662d..83f8bd9f 100644 --- a/include/libcamera/internal/pipeline_handler.h +++ b/include/libcamera/internal/pipeline_handler.h @@ -70,6 +70,7 @@ public: protected: void registerCamera(std::shared_ptr camera); void hotplugMediaDevice(MediaDevice *media); + void setMaxQueueRequests(uint32_t maxRequests); virtual int queueRequestDevice(Camera *camera, Request *request) = 0; virtual void stopDevice(Camera *camera) = 0; @@ -97,6 +98,9 @@ private: Mutex lock_; unsigned int useCount_ LIBCAMERA_TSA_GUARDED_BY(lock_); + uint32_t maxQueueRequests_; + uint32_t requestsQueueCounter_; + friend class PipelineHandlerFactoryBase; }; diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index e4d79ea4..d1d42f78 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -422,6 +422,7 @@ CameraConfiguration::Status IPU3CameraConfiguration::validate() PipelineHandlerIPU3::PipelineHandlerIPU3(CameraManager *manager) : PipelineHandler(manager), cio2MediaDev_(nullptr), imguMediaDev_(nullptr) { + setMaxQueueRequests(ipa::ipu3::kMaxFrameContexts); } std::unique_ptr diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index eb9ad65c..a48adba9 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -599,6 +599,7 @@ CameraConfiguration::Status RkISP1CameraConfiguration::validate() PipelineHandlerRkISP1::PipelineHandlerRkISP1(CameraManager *manager) : PipelineHandler(manager), hasSelfPath_(true) { + setMaxQueueRequests(ipa::rkisp1::kMaxFrameContexts); } std::unique_ptr diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp index cfade490..103f9db0 100644 --- a/src/libcamera/pipeline_handler.cpp +++ b/src/libcamera/pipeline_handler.cpp @@ -67,7 +67,8 @@ LOG_DEFINE_CATEGORY(Pipeline) * through the PipelineHandlerFactoryBase::create() function. */ PipelineHandler::PipelineHandler(CameraManager *manager) - : manager_(manager), useCount_(0) + : manager_(manager), useCount_(0), maxQueueRequests_(0), + requestsQueueCounter_(0) { } @@ -428,6 +429,9 @@ void PipelineHandler::doQueueRequest(Request *request) Camera::Private *data = camera->_d(); data->queuedRequests_.push_back(request); + if (maxQueueRequests_) + requestsQueueCounter_++; + request->_d()->sequence_ = data->requestSequence_++; if (request->_d()->cancelled_) { @@ -455,6 +459,10 @@ void PipelineHandler::doQueueRequests() if (!request->_d()->prepared_) break; + if (maxQueueRequests_ && + requestsQueueCounter_ >= maxQueueRequests_) + break; + doQueueRequest(request); waitingRequests_.pop(); } @@ -531,6 +539,9 @@ void PipelineHandler::completeRequest(Request *request) ASSERT(!req->hasPendingBuffers()); data->queuedRequests_.pop_front(); camera->requestComplete(req); + + if (maxQueueRequests_) + requestsQueueCounter_--; } } @@ -647,6 +658,44 @@ void PipelineHandler::disconnect() * constant for the whole lifetime of the pipeline handler. */ +/** + * \var PipelineHandler::maxQueueRequests_ + * \brief Maximum number of in-flight requests that can be queued + * + * A hardware can handle a certain number of maximum requests at a given + * point. If such a constraint exists, set maxQueueRequests_ via + * \a setMaxQueueRequests() in the derived pipeline handler. + * + * The derived pipeline handler can choose not to define such constraint as + * well. In that case, the derived pipeline handler can avoid setting + * \a setMaxQueueReqeuests(), hence \a maxQueueRequests_ and + * \a requestsQueueCounter_ will be 0. + */ + +/** + * \var PipelineHandler::requestsQueueCounter_ + * \brief Number of requests queued to the underlying hardware + * + * If \a setMaxQueueRequests() is set by the derived pipeline handler, + * requestsQueueCounter_ reflects the number of requests queued + * to the underlying hardware by the pipeline handler. + */ + +/** + * \brief Sets the maximum number of requests that can be queued + * \param[in] maxRequests Maximum number of in-flight requests + * + * A hardware can handle a certain number of requests at a given point. + * This function sets the maximum number of in-flight requests that can + * be queued to the hardware by the pipeline handler. Each derived pipeline + * handler should set the maximum number of in-flight requests it can handle + * at a given point using this function, if at all such a constraint exists. + */ +void PipelineHandler::setMaxQueueRequests(uint32_t maxRequests) +{ + maxQueueRequests_ = maxRequests; +} + /** * \fn PipelineHandler::name() * \brief Retrieve the pipeline handler name