{"id":17313,"url":"https://patchwork.libcamera.org/api/patches/17313/?format=json","web_url":"https://patchwork.libcamera.org/patch/17313/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20220908014200.28728-5-laurent.pinchart@ideasonboard.com>","date":"2022-09-08T01:41:32","name":"[libcamera-devel,v4,04/32] ipa: libipa: Introduce FrameContextQueue","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"c2c4339ec2548a2148e43ee72c70ae19bee48b93","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/?format=json","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/17313/mbox/","series":[{"id":3476,"url":"https://patchwork.libcamera.org/api/series/3476/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=3476","date":"2022-09-08T01:41:28","name":"ipa: Frame context queue, IPU3 & RkISP consolidation, and RkISP1 improvements","version":4,"mbox":"https://patchwork.libcamera.org/series/3476/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/17313/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/17313/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 64E1EC3272\n\tfor ;\n\tThu, 8 Sep 2022 01:42:25 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 14DD26209E;\n\tThu, 8 Sep 2022 03:42:25 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 6B9766209F\n\tfor ;\n\tThu, 8 Sep 2022 03:42:23 +0200 (CEST)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id CB60A6CC;\n\tThu, 8 Sep 2022 03:42:22 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1662601345;\n\tbh=Lfvv4mWBWOcWKc2TlDW7eM3J7wXgI98zO/kEion9OG0=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=sQK/HcbBS/c46no/Yreju6heygCAj/5HkzzWrOoi+D3UFmk4glgO1IlSDDVwRrPBx\n\tdAor8OOS4plYigoGNUoOToiPiy2o0tUfYrMwMAKr3nsRJl4+xdsqDm0ntpBxfmJJ2e\n\tshL0jezivC7Z1d84aXF/GWPF8HnMF3VLIpURdmHK4dM7lOextI5RpKu5FlKsCkv3Q8\n\tLcnuooTjH+ep3Yuwv6mpJ/9a+mrw6lnMRIcEFZmXk+sfQ1Tvb/EzWqKpNJDAPAoEuy\n\tybJ/7zD7nB8Ate+giuDIxOTIGiL11ZVDOMYTpGv8LymoRYlE30M7OTYbTENmXB+7Ew\n\tY8YRsT16xNOag==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1662601343;\n\tbh=Lfvv4mWBWOcWKc2TlDW7eM3J7wXgI98zO/kEion9OG0=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=Wm5DGkOUgNShYLRsOOWzs68dm0hDxrVnszvABqzESKPFTAayz8dMeXKpIaPlpaz4b\n\tVFYiSTiAF5aqc2JD2LwOPG9SRA+3qe1bLl36tqSoOOpxZI1mWpxJGPOJ9fSk9zMN+/\n\tT73UAKJaACLf2IVOdW2fRUa6UzlzHDfY3y6BvACU="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"Wm5DGkOU\"; dkim-atps=neutral","To":"libcamera-devel@lists.libcamera.org","Date":"Thu, 8 Sep 2022 04:41:32 +0300","Message-Id":"<20220908014200.28728-5-laurent.pinchart@ideasonboard.com>","X-Mailer":"git-send-email 2.35.1","In-Reply-To":"<20220908014200.28728-1-laurent.pinchart@ideasonboard.com>","References":"<20220908014200.28728-1-laurent.pinchart@ideasonboard.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH v4 04/32] ipa: libipa: Introduce\n\tFrameContextQueue","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Laurent Pinchart via libcamera-devel\n\t","Reply-To":"Laurent Pinchart ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"From: Umang Jain \n\nIntroduce a common implementation in libipa to represent the queue of\nframe contexts.\n\nSigned-off-by: Umang Jain \nSigned-off-by: Kieran Bingham \nSigned-off-by: Jacopo Mondi \n---\nChanges since v3:\n\n- Split the IPU3 changes to a separate patch\n- Use composition instead of inheritance\n- Use vector instead of array for backend storage\n- Make the queue size dynamic\n- Rename initialise() to init()\n---\n src/ipa/libipa/fc_queue.cpp | 117 ++++++++++++++++++++++++++++++++++++\n src/ipa/libipa/fc_queue.h | 108 +++++++++++++++++++++++++++++++++\n src/ipa/libipa/meson.build | 2 +\n 3 files changed, 227 insertions(+)\n create mode 100644 src/ipa/libipa/fc_queue.cpp\n create mode 100644 src/ipa/libipa/fc_queue.h","diff":"diff --git a/src/ipa/libipa/fc_queue.cpp b/src/ipa/libipa/fc_queue.cpp\nnew file mode 100644\nindex 000000000000..b81d497e255a\n--- /dev/null\n+++ b/src/ipa/libipa/fc_queue.cpp\n@@ -0,0 +1,117 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2022, Google Inc.\n+ *\n+ * fc_queue.cpp - IPA Frame context queue\n+ */\n+\n+#include \"fc_queue.h\"\n+\n+#include \n+\n+namespace libcamera {\n+\n+LOG_DEFINE_CATEGORY(FCQueue)\n+\n+namespace ipa {\n+\n+/**\n+ * \\file fc_queue.h\n+ * \\brief Queue to access per-frame Context\n+ */\n+\n+/**\n+ * \\class FCQueue\n+ * \\brief A support class for queueing FrameContext instances through the IPA\n+ * \\tparam FrameContext The IPA specific FrameContext derived class type\n+ *\n+ * The frame context queue provides a simple circular buffer implementation to\n+ * store IPA specific context for each frame through its lifetime within the\n+ * IPA.\n+ *\n+ * FrameContext instances are expected to be referenced by a monotonically\n+ * increasing sequence count corresponding to a frame sequence number.\n+ *\n+ * A FrameContext is initialised for a given frame when the corresponding\n+ * Request is first queued into the IPA. From that point on the FrameContext can\n+ * be obtained by the IPA and its algorithms by referencing it from the frame\n+ * sequence number.\n+ *\n+ * A frame sequence number from the image source must correspond to the request\n+ * sequence number for this implementation to be supported in an IPA. It is\n+ * expected that the same sequence number will be used to reference the context\n+ * of the frame from the point of queueing the request, specifying controls for\n+ * a given frame, and processing of any ISP related controls and statistics for\n+ * the same corresponding image.\n+ *\n+ * IPA specific frame context implementations shall inherit from the\n+ * IPAFrameContext base class to support the minimum required features for a\n+ * FrameContext.\n+ *\n+ * FrameContexts are overwritten when they are recycled and re-initialised by\n+ * the first access made on them by either init(frame) or get(frame). It is\n+ * required that the number of available slots in the frame context queue is\n+ * larger or equal to the maximum number of in-flight requests a pipeline can\n+ * handle to avoid overwriting frame context instances that still have to be\n+ * processed.\n+ *\n+ * In the case an application does not queue requests to the camera fast\n+ * enough, frames can be produced and processed by the IPA without a\n+ * corresponding Request being queued. In this case the IPA algorithm\n+ * will try to access the FrameContext with a call to get() before it\n+ * had been initialized at queueRequest() time. In this case, there\n+ * is no way the controls associated with the late Request could be\n+ * applied in time, as the frame as already been processed by the IPA.\n+ *\n+ * \\todo Set an error flag for per-frame control errors.\n+ */\n+\n+/**\n+ * \\fn FCQueue::FCQueue(unsigned int size)\n+ * \\brief Construct a frame context queue of a specified size\n+ * \\param[in] size The number of contexts in the queue\n+ */\n+\n+/**\n+ * \\fn FCQueue::clear()\n+ * \\brief Clear the context queue\n+ *\n+ * Reset the queue and ensure all FrameContext slots are re-initialised.\n+ * IPA modules are expected to clear the frame context queue at the beginning of\n+ * a new streaming session, in IPAModule::start().\n+ */\n+\n+/**\n+ * \\fn FCQueue::init(uint32_t frame)\n+ * \\brief Initialize and return the Frame Context at slot specified by \\a frame\n+ * \\param[in] frame The frame context sequence number\n+ *\n+ * The first call to obtain a FrameContext from the FCQueue should be handled\n+ * through this call. The FrameContext will be initialised for the frame and\n+ * returned to the caller if it was not already initialised.\n+ *\n+ * If the FrameContext was already initialized for this sequence, a warning will\n+ * be reported and the previously initialized FrameContext is returned.\n+ *\n+ * Frame contexts are expected to be initialised when a Request is first passed\n+ * to the IPA module in IPAModule::queueRequest().\n+ *\n+ * \\return A reference to the FrameContext for sequence \\a frame\n+ */\n+\n+/**\n+ * \\fn FCQueue::get()\n+ * \\brief Obtain the Frame Context at slot specified by \\a frame\n+ * \\param[in] frame The frame context sequence number\n+ *\n+ * Obtains an existing FrameContext from the queue and returns it to the caller.\n+ *\n+ * If the FrameContext is not correctly initialised for the \\a frame, it will be\n+ * initialised.\n+ *\n+ * \\return A reference to the FrameContext for sequence \\a frame\n+ */\n+\n+} /* namespace ipa */\n+\n+} /* namespace libcamera */\ndiff --git a/src/ipa/libipa/fc_queue.h b/src/ipa/libipa/fc_queue.h\nnew file mode 100644\nindex 000000000000..0f3af0f3a189\n--- /dev/null\n+++ b/src/ipa/libipa/fc_queue.h\n@@ -0,0 +1,108 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2022, Google Inc.\n+ *\n+ * fc_queue.h - IPA Frame context queue\n+ */\n+\n+#pragma once\n+\n+#include \n+#include \n+\n+#include \n+\n+namespace libcamera {\n+\n+LOG_DECLARE_CATEGORY(FCQueue)\n+\n+namespace ipa {\n+\n+template\n+class FCQueue\n+{\n+public:\n+\tFCQueue(unsigned int size)\n+\t\t: contexts_(size)\n+\t{\n+\t}\n+\n+\tvoid clear()\n+\t{\n+\t\tstd::fill(contexts_.begin(), contexts_.end(), FrameContext{});\n+\t}\n+\n+\tFrameContext &init(const uint32_t frame)\n+\t{\n+\t\tFrameContext &frameContext = contexts_[frame % contexts_.size()];\n+\n+\t\t/*\n+\t\t * Do not re-initialise if a get() call has already fetched this\n+\t\t * frame context to preseve the context.\n+\t\t *\n+\t\t * \\todo If the the sequence number of the context to initialise\n+\t\t * is smaller than the sequence number of the queue slot to use,\n+\t\t * it means that we had a serious request underrun and more\n+\t\t * frames than the queue size has been produced since the last\n+\t\t * time the application has queued a request. Does this deserve\n+\t\t * an error condition ?\n+\t\t */\n+\t\tif (frame != 0 && frame <= frameContext.frame)\n+\t\t\tLOG(FCQueue, Warning)\n+\t\t\t\t<< \"Frame \" << frame << \" already initialised\";\n+\t\telse\n+\t\t\tinit(frameContext, frame);\n+\n+\t\treturn frameContext;\n+\t}\n+\n+\tFrameContext &get(uint32_t frame)\n+\t{\n+\t\tFrameContext &frameContext = contexts_[frame % contexts_.size()];\n+\n+\t\t/*\n+\t\t * If the IPA algorithms try to access a frame context slot which\n+\t\t * has been already overwritten by a newer context, it means the\n+\t\t * frame context queue has overflowed and the desired context\n+\t\t * has been forever lost. The pipeline handler shall avoid\n+\t\t * queueing more requests to the IPA than the frame context\n+\t\t * queue size.\n+\t\t */\n+\t\tif (frame < frameContext.frame)\n+\t\t\tLOG(FCQueue, Fatal) << \"Frame Context for \" << frame\n+\t\t\t\t\t << \" has been overwritten by \"\n+\t\t\t\t\t << frameContext.frame;\n+\n+\t\tif (frame == frameContext.frame)\n+\t\t\treturn frameContext;\n+\n+\t\t/*\n+\t\t * The frame context has been retrieved before it was\n+\t\t * initialised through the initialise() call. This indicates an\n+\t\t * algorithm attempted to access a Frame context before it was\n+\t\t * queued to the IPA. Controls applied for this request may be\n+\t\t * left unhandled.\n+\t\t *\n+\t\t * \\todo Set an error flag for per-frame control errors.\n+\t\t */\n+\t\tLOG(FCQueue, Warning)\n+\t\t\t<< \"Obtained an uninitialised FrameContext for \" << frame;\n+\n+\t\tinit(frameContext, frame);\n+\n+\t\treturn frameContext;\n+\t}\n+\n+private:\n+\tvoid init(FrameContext &frameContext, const uint32_t frame)\n+\t{\n+\t\tframeContext = {};\n+\t\tframeContext.frame = frame;\n+\t}\n+\n+\tstd::vector contexts_;\n+};\n+\n+} /* namespace ipa */\n+\n+} /* namespace libcamera */\ndiff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build\nindex fb894bc614af..016b8e0ec9be 100644\n--- a/src/ipa/libipa/meson.build\n+++ b/src/ipa/libipa/meson.build\n@@ -3,6 +3,7 @@\n libipa_headers = files([\n 'algorithm.h',\n 'camera_sensor_helper.h',\n+ 'fc_queue.h',\n 'histogram.h',\n 'module.h',\n ])\n@@ -10,6 +11,7 @@ libipa_headers = files([\n libipa_sources = files([\n 'algorithm.cpp',\n 'camera_sensor_helper.cpp',\n+ 'fc_queue.cpp',\n 'histogram.cpp',\n 'module.cpp',\n ])\n","prefixes":["libcamera-devel","v4","04/32"]}