{"id":17415,"url":"https://patchwork.libcamera.org/api/patches/17415/?format=json","web_url":"https://patchwork.libcamera.org/patch/17415/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20220927023642.12341-5-laurent.pinchart@ideasonboard.com>","date":"2022-09-27T02:36:13","name":"[libcamera-devel,v5,04/33] ipa: libipa: Introduce FrameContextQueue","commit_ref":null,"pull_url":null,"state":"accepted","archived":false,"hash":"5592d410aa93133b690c7b4fec8789bb328c2758","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/?format=json","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/17415/mbox/","series":[{"id":3506,"url":"https://patchwork.libcamera.org/api/series/3506/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=3506","date":"2022-09-27T02:36:09","name":"ipa: Frame context queue, IPU3 & RkISP consolidation, and RkISP1 improvements","version":5,"mbox":"https://patchwork.libcamera.org/series/3506/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/17415/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/17415/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 5E800C327E\n\tfor ;\n\tTue, 27 Sep 2022 02:37:08 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 09E6B6226F;\n\tTue, 27 Sep 2022 04:37:08 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C26046225F\n\tfor ;\n\tTue, 27 Sep 2022 04:37:05 +0200 (CEST)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 2AB22E5;\n\tTue, 27 Sep 2022 04:37:05 +0200 (CEST)"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org;\n\ts=mail; t=1664246228;\n\tbh=KfSOcY/vBJimasK2A6AaFUzNycSQYT3y1QvbpFZkxnk=;\n\th=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:\n\tList-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:\n\tFrom;\n\tb=MiCf914zmNofFd9pW3EPsL8zv0H49g4zACTatJn+jZFNahS798Z1j7b8coWDLm3my\n\txHkWQbFNIQQXY2vHLCGQpjtgWSfNo7o5M+qM2xytVHadpXEoq+OUzqqPnzX9kyHdrn\n\tL0nfGjxwtFYiBgtTG9rO+s5VhEP2T1u2rVflG4ptj4IIAngEObyH/842zHrEdP6bRq\n\tEezdYYQPTi7fWzezoDrBQ4kV2KiUxupZ85Gaz67M54rETYAlfPyavDaOMnlLgp0CYE\n\tv6FkCR6at7rY0QQr8om5bjZeNqwFpXbNqfQ2SoPqG6FPi/8zZs4l+Gc9GDjt8phMtD\n\tI4w0bN+CZKJ+Q==","v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1664246225;\n\tbh=KfSOcY/vBJimasK2A6AaFUzNycSQYT3y1QvbpFZkxnk=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=AesojHfztaijrQGefMr0YrZIeTlq4KYNsplTrp4v6Zm49FPa91IMlZo9P7KC1jR3n\n\tczc8Xsh+JCj4lxck+zyYROstUbNPG0JZIwfEcDVunmE5QsEIwYG4NRjaPlvAlehZVF\n\t1Sgw2mZUB2lb3/4b2QSmWIRIrcD3Cv2kIAjZ/Cz8="],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key; \n\tunprotected) header.d=ideasonboard.com\n\theader.i=@ideasonboard.com\n\theader.b=\"AesojHfz\"; dkim-atps=neutral","To":"libcamera-devel@lists.libcamera.org","Date":"Tue, 27 Sep 2022 05:36:13 +0300","Message-Id":"<20220927023642.12341-5-laurent.pinchart@ideasonboard.com>","X-Mailer":"git-send-email 2.35.1","In-Reply-To":"<20220927023642.12341-1-laurent.pinchart@ideasonboard.com>","References":"<20220927023642.12341-1-laurent.pinchart@ideasonboard.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH v5 04/33] ipa: libipa: Introduce\n\tFrameContextQueue","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","From":"Laurent Pinchart via libcamera-devel\n\t","Reply-To":"Laurent Pinchart ","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"From: Umang Jain \n\nIntroduce a common implementation in libipa to represent the queue of\nframe contexts.\n\nSigned-off-by: Umang Jain \nSigned-off-by: Kieran Bingham \nSigned-off-by: Jacopo Mondi \nReviewed-by: Kieran Bingham \n---\nChanges since v4:\n\n- Documentation improvements\n- White space fixes\n- Rework class documentation\n- Rename FCQueue::init() to FCQueue::alloc()\n- Only clear frame number in clear()\n- Add more \\todo comments\n\nChanges since v3:\n\n- Split the IPU3 changes to a separate patch\n- Use composition instead of inheritance\n- Use vector instead of array for backend storage\n- Make the queue size dynamic\n- Rename initialise() to init()\n---\n src/ipa/libipa/fc_queue.cpp | 118 ++++++++++++++++++++++++++++++++++++\n src/ipa/libipa/fc_queue.h | 108 +++++++++++++++++++++++++++++++++\n src/ipa/libipa/meson.build | 2 +\n 3 files changed, 228 insertions(+)\n create mode 100644 src/ipa/libipa/fc_queue.cpp\n create mode 100644 src/ipa/libipa/fc_queue.h","diff":"diff --git a/src/ipa/libipa/fc_queue.cpp b/src/ipa/libipa/fc_queue.cpp\nnew file mode 100644\nindex 000000000000..57a369512554\n--- /dev/null\n+++ b/src/ipa/libipa/fc_queue.cpp\n@@ -0,0 +1,118 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2022, Google Inc.\n+ *\n+ * fc_queue.cpp - IPA Frame context queue\n+ */\n+\n+#include \"fc_queue.h\"\n+\n+#include \n+\n+namespace libcamera {\n+\n+LOG_DEFINE_CATEGORY(FCQueue)\n+\n+namespace ipa {\n+\n+/**\n+ * \\file fc_queue.h\n+ * \\brief Queue of per-frame contexts\n+ */\n+\n+/**\n+ * \\class FCQueue\n+ * \\brief A support class for managing FrameContext instances in IPA modules\n+ * \\tparam FrameContext The IPA module-specific FrameContext derived class type\n+ *\n+ * Along with the Module and Algorithm classes, the frame context queue is a\n+ * core component of the libipa infrastructure. It stores per-frame contexts\n+ * used by the Algorithm operations. By centralizing the lifetime management of\n+ * the contexts and implementing safeguards against underflows and overflows, it\n+ * simplifies IPA modules and improves their reliability.\n+ *\n+ * The queue references frame contexts by a monotonically increasing sequence\n+ * number. The FCQueue design assumes that this number matches both the sequence\n+ * number of the corresponding frame, as generated by the camera sensor, and the\n+ * sequence number of the request. This allows IPA modules to obtain the frame\n+ * context from any location where a request or a frame is available.\n+ *\n+ * A frame context normally begins its lifetime when the corresponding request\n+ * is queued, way before the frame is captured by the camera sensor. IPA modules\n+ * allocate the context from the queue at that point, calling alloc() using the\n+ * request number. The queue initializes the context, and the IPA module then\n+ * populates it with data from the request. The context can be later retrieved\n+ * with a call to get(), typically when the IPA module is requested to provide\n+ * sensor or ISP parameters or receives statistics for a frame. The frame number\n+ * is used at that point to identify the context.\n+ *\n+ * If an application fails to queue requests to the camera fast enough, frames\n+ * may be produced by the camera sensor and processed by the IPA module without\n+ * a corresponding request having been queued to the IPA module. This creates an\n+ * underrun condition, where the IPA module will try to get a frame context that\n+ * hasn't been allocated. In this case, the get() function will allocate and\n+ * initialize a context for the frame, and log a message. Algorithms will not\n+ * apply the controls associated with the late request, but should otherwise\n+ * behave correctly.\n+ *\n+ * \\todo Mark the frame context with a per-frame control error flag in case of\n+ * underrun, and research how algorithms should handle this.\n+ *\n+ * At its core, the queue uses a circular buffer to avoid dynamic memory\n+ * allocation at runtime. The buffer is pre-allocated with a maximum number of\n+ * entries when the FCQueue instance is constructed. Entries are initialized on\n+ * first use by alloc() or, in underrun conditions, get(). The queue is not\n+ * allowed to overflow, which must be ensured by pipeline handlers never\n+ * queuing more in-flight requests to the IPA module than the queue size. If an\n+ * overflow condition is detected, the queue will log a fatal error.\n+ */\n+\n+/**\n+ * \\fn FCQueue::FCQueue(unsigned int size)\n+ * \\brief Construct a frame contexts queue of a specified size\n+ * \\param[in] size The number of contexts in the queue\n+ */\n+\n+/**\n+ * \\fn FCQueue::clear()\n+ * \\brief Clear the contexts queue\n+ *\n+ * IPA modules must clear the frame context queue at the beginning of a new\n+ * streaming session, in IPAModule::start().\n+ *\n+ * \\todo Fix any issue this may cause with requests queued before the camera is\n+ * started.\n+ */\n+\n+/**\n+ * \\fn FCQueue::alloc(uint32_t frame)\n+ * \\brief Allocate and return a FrameContext for the \\a frame\n+ * \\param[in] frame The frame context sequence number\n+ *\n+ * The first call to obtain a FrameContext from the FCQueue should be handled\n+ * through this function. The FrameContext will be initialised, if not\n+ * initialised already, and returned to the caller.\n+ *\n+ * If the FrameContext was already initialized for this \\a frame, a warning will\n+ * be reported and the previously initialized FrameContext is returned.\n+ *\n+ * Frame contexts are expected to be initialised when a Request is first passed\n+ * to the IPA module in IPAModule::queueRequest().\n+ *\n+ * \\return A reference to the FrameContext for sequence \\a frame\n+ */\n+\n+/**\n+ * \\fn FCQueue::get(uint32_t frame)\n+ * \\brief Obtain the FrameContext for the \\a frame\n+ * \\param[in] frame The frame context sequence number\n+ *\n+ * If the FrameContext is not correctly initialised for the \\a frame, it will be\n+ * initialised.\n+ *\n+ * \\return A reference to the FrameContext for sequence \\a frame\n+ */\n+\n+} /* namespace ipa */\n+\n+} /* namespace libcamera */\ndiff --git a/src/ipa/libipa/fc_queue.h b/src/ipa/libipa/fc_queue.h\nnew file mode 100644\nindex 000000000000..4f5cb5d35253\n--- /dev/null\n+++ b/src/ipa/libipa/fc_queue.h\n@@ -0,0 +1,108 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2022, Google Inc.\n+ *\n+ * fc_queue.h - IPA Frame context queue\n+ */\n+\n+#pragma once\n+\n+#include \n+\n+#include \n+\n+namespace libcamera {\n+\n+LOG_DECLARE_CATEGORY(FCQueue)\n+\n+namespace ipa {\n+\n+template\n+class FCQueue\n+{\n+public:\n+\tFCQueue(unsigned int size)\n+\t\t: contexts_(size)\n+\t{\n+\t}\n+\n+\tvoid clear()\n+\t{\n+\t\tfor (FrameContext &ctx : contexts_)\n+\t\t\tctx.frame = 0;\n+\t}\n+\n+\tFrameContext &alloc(const uint32_t frame)\n+\t{\n+\t\tFrameContext &frameContext = contexts_[frame % contexts_.size()];\n+\n+\t\t/*\n+\t\t * Do not re-initialise if a get() call has already fetched this\n+\t\t * frame context to preseve the context.\n+\t\t *\n+\t\t * \\todo If the the sequence number of the context to initialise\n+\t\t * is smaller than the sequence number of the queue slot to use,\n+\t\t * it means that we had a serious request underrun and more\n+\t\t * frames than the queue size has been produced since the last\n+\t\t * time the application has queued a request. Does this deserve\n+\t\t * an error condition ?\n+\t\t */\n+\t\tif (frame != 0 && frame <= frameContext.frame)\n+\t\t\tLOG(FCQueue, Warning)\n+\t\t\t\t<< \"Frame \" << frame << \" already initialised\";\n+\t\telse\n+\t\t\tinit(frameContext, frame);\n+\n+\t\treturn frameContext;\n+\t}\n+\n+\tFrameContext &get(uint32_t frame)\n+\t{\n+\t\tFrameContext &frameContext = contexts_[frame % contexts_.size()];\n+\n+\t\t/*\n+\t\t * If the IPA algorithms try to access a frame context slot which\n+\t\t * has been already overwritten by a newer context, it means the\n+\t\t * frame context queue has overflowed and the desired context\n+\t\t * has been forever lost. The pipeline handler shall avoid\n+\t\t * queueing more requests to the IPA than the frame context\n+\t\t * queue size.\n+\t\t */\n+\t\tif (frame < frameContext.frame)\n+\t\t\tLOG(FCQueue, Fatal) << \"Frame context for \" << frame\n+\t\t\t\t\t << \" has been overwritten by \"\n+\t\t\t\t\t << frameContext.frame;\n+\n+\t\tif (frame == frameContext.frame)\n+\t\t\treturn frameContext;\n+\n+\t\t/*\n+\t\t * The frame context has been retrieved before it was\n+\t\t * initialised through the initialise() call. This indicates an\n+\t\t * algorithm attempted to access a Frame context before it was\n+\t\t * queued to the IPA. Controls applied for this request may be\n+\t\t * left unhandled.\n+\t\t *\n+\t\t * \\todo Set an error flag for per-frame control errors.\n+\t\t */\n+\t\tLOG(FCQueue, Warning)\n+\t\t\t<< \"Obtained an uninitialised FrameContext for \" << frame;\n+\n+\t\tinit(frameContext, frame);\n+\n+\t\treturn frameContext;\n+\t}\n+\n+private:\n+\tvoid init(FrameContext &frameContext, const uint32_t frame)\n+\t{\n+\t\tframeContext = {};\n+\t\tframeContext.frame = frame;\n+\t}\n+\n+\tstd::vector contexts_;\n+};\n+\n+} /* namespace ipa */\n+\n+} /* namespace libcamera */\ndiff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build\nindex fb894bc614af..016b8e0ec9be 100644\n--- a/src/ipa/libipa/meson.build\n+++ b/src/ipa/libipa/meson.build\n@@ -3,6 +3,7 @@\n libipa_headers = files([\n 'algorithm.h',\n 'camera_sensor_helper.h',\n+ 'fc_queue.h',\n 'histogram.h',\n 'module.h',\n ])\n@@ -10,6 +11,7 @@ libipa_headers = files([\n libipa_sources = files([\n 'algorithm.cpp',\n 'camera_sensor_helper.cpp',\n+ 'fc_queue.cpp',\n 'histogram.cpp',\n 'module.cpp',\n ])\n","prefixes":["libcamera-devel","v5","04/33"]}