{"id":2145,"url":"https://patchwork.libcamera.org/api/patches/2145/?format=json","web_url":"https://patchwork.libcamera.org/patch/2145/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20191008004534.1585142-8-niklas.soderlund@ragnatech.se>","date":"2019-10-08T00:45:31","name":"[libcamera-devel,v5,07/10] libcamera: ipa: Extend to support IPA interactions","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"f2c2782d4756a19a1bac10da36ab4b37e939711c","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/?format=json","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/2145/mbox/","series":[{"id":527,"url":"https://patchwork.libcamera.org/api/series/527/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=527","date":"2019-10-08T00:45:24","name":"libcamera: ipa: Add basic IPA support","version":5,"mbox":"https://patchwork.libcamera.org/series/527/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/2145/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/2145/checks/","tags":{},"headers":{"Return-Path":"","Received":["from bin-mail-out-05.binero.net (bin-mail-out-05.binero.net\n\t[195.74.38.228])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 3D1CA6196C\n\tfor ;\n\tTue, 8 Oct 2019 02:46:47 +0200 (CEST)","from bismarck.berto.se (unknown [84.172.88.101])\n\tby bin-vsp-out-03.atm.binero.net (Halon) with ESMTPA\n\tid f6430d9e-e964-11e9-837a-0050569116f7;\n\tTue, 08 Oct 2019 02:45:45 +0200 (CEST)"],"X-Halon-ID":"f6430d9e-e964-11e9-837a-0050569116f7","Authorized-sender":"niklas@soderlund.pp.se","From":"=?utf-8?q?Niklas_S=C3=B6derlund?= ","To":"libcamera-devel@lists.libcamera.org","Date":"Tue, 8 Oct 2019 02:45:31 +0200","Message-Id":"<20191008004534.1585142-8-niklas.soderlund@ragnatech.se>","X-Mailer":"git-send-email 2.23.0","In-Reply-To":"<20191008004534.1585142-1-niklas.soderlund@ragnatech.se>","References":"<20191008004534.1585142-1-niklas.soderlund@ragnatech.se>","MIME-Version":"1.0","Content-Type":"text/plain; charset=UTF-8","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [PATCH v5 07/10] libcamera: ipa: Extend to\n\tsupport IPA interactions","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","X-List-Received-Date":"Tue, 08 Oct 2019 00:46:47 -0000"},"content":"The IPA interface needs to support interactions with the pipeline; add\ninterfaces to control the sensor and handling of request ISP parameters\nand statistics.\n\nSigned-off-by: Niklas Söderlund \n---\n include/ipa/ipa_interface.h | 38 +++++\n src/ipa/ipa_dummy.cpp | 7 +-\n src/libcamera/ipa_interface.cpp | 195 ++++++++++++++++++++++++\n src/libcamera/proxy/ipa_proxy_linux.cpp | 14 +-\n 4 files changed, 245 insertions(+), 9 deletions(-)","diff":"diff --git a/include/ipa/ipa_interface.h b/include/ipa/ipa_interface.h\nindex 2c5eb1fd524311cb..499340dc3a8c67c2 100644\n--- a/include/ipa/ipa_interface.h\n+++ b/include/ipa/ipa_interface.h\n@@ -7,14 +7,52 @@\n #ifndef __LIBCAMERA_IPA_INTERFACE_H__\n #define __LIBCAMERA_IPA_INTERFACE_H__\n \n+#include \n+#include \n+\n+#include \n+#include \n+#include \n+#include \n+\n+#include \"v4l2_controls.h\"\n+\n namespace libcamera {\n \n+struct IPAStream {\n+\tunsigned int pixelFormat;\n+\tSize size;\n+};\n+\n+struct IPABuffer {\n+\tunsigned int type;\n+\tunsigned int id;\n+\tBufferMemory buffer;\n+};\n+\n+struct IPAOperationData {\n+\tunsigned int operation;\n+\tstd::vector data;\n+\tstd::vector controls;\n+\tstd::vector v4l2controls;\n+\t/* \\todo: Add more vectors for data types used in pipa<->IPA interactions. */\n+};\n+\n class IPAInterface\n {\n public:\n \tvirtual ~IPAInterface() {}\n \n \tvirtual int init() = 0;\n+\n+\tvirtual void configure(const std::map &streamConfig,\n+\t\t\t const std::map &entityControls) = 0;\n+\n+\tvirtual void mapBuffers(const std::vector &buffers) = 0;\n+\tvirtual void unmapBuffers(const std::vector &buffers) = 0;\n+\n+\tvirtual void processEvent(const IPAOperationData &event) = 0;\n+\tSignal queueFrameAction;\n };\n \n } /* namespace libcamera */\ndiff --git a/src/ipa/ipa_dummy.cpp b/src/ipa/ipa_dummy.cpp\nindex 9d0cbdc8b1ad5565..08f74c4952f3923a 100644\n--- a/src/ipa/ipa_dummy.cpp\n+++ b/src/ipa/ipa_dummy.cpp\n@@ -15,7 +15,12 @@ namespace libcamera {\n class IPADummy : public IPAInterface\n {\n public:\n-\tint init();\n+\tint init() override;\n+\tvoid configure(const std::map &streamConfig,\n+\t\t const std::map &entityControls) override {}\n+\tvoid mapBuffers(const std::vector &buffers) override {}\n+\tvoid unmapBuffers(const std::vector &buffers) override {}\n+\tvoid processEvent(const IPAOperationData &event) override {}\n };\n \n int IPADummy::init()\ndiff --git a/src/libcamera/ipa_interface.cpp b/src/libcamera/ipa_interface.cpp\nindex d7d8ca8881efcf66..c365d14c53669c21 100644\n--- a/src/libcamera/ipa_interface.cpp\n+++ b/src/libcamera/ipa_interface.cpp\n@@ -14,9 +14,136 @@\n \n namespace libcamera {\n \n+/**\n+ * \\struct IPAStream\n+ * \\brief Stream configuration for the IPA protocol\n+ *\n+ * The StreamConfiguration class contains too much information to be suitable\n+ * for IPA protocols. The IPAStream structure mirrors the parameters from\n+ * StreamConfiguration which are useful for the IPA.\n+ */\n+\n+/**\n+ * \\var IPAStream::pixelFormat\n+ * \\brief The pixel format\n+ */\n+\n+/**\n+ * \\var IPAStream::size\n+ * \\brief The size\n+ */\n+\n+/**\n+ * \\struct IPABuffer\n+ * \\brief Buffer information for the IPA protocol\n+ *\n+ * Buffers allocated by a pipeline handler might be needed inside the IPA. This\n+ * structure allows the pipeline handler to associate a buffer with numerical\n+ * ids that uniquely identify the buffer that can be used in the IPA protocol.\n+ *\n+ * The numerical identifiers are \\a type and \\a id. Type describes which buffer\n+ * pool \\a buffer belongs to and id unique identifies the buffer within that\n+ * pool.\n+ *\n+ * \\sa IPAInterface::mapBuffers()\n+ * \\sa IPAInterface::unmapBuffers()\n+ */\n+\n+/**\n+ * \\var IPABuffer::type\n+ * \\brief Type of the buffer\n+ */\n+\n+/**\n+ * \\var IPABuffer::id\n+ * \\brief ID of the buffer\n+ */\n+\n+/**\n+ * \\var IPABuffer::buffer\n+ * \\brief Hardware description of the buffer\n+ *\n+ * The description allows the IPA to map and unmap the buffer memory to access\n+ * its content.\n+ */\n+\n+/**\n+ * \\struct IPAOperationData\n+ * \\brief Parameters for IPA operations\n+ *\n+ * A pipeline defines its IPA protocol by describing how an IPAOperationData\n+ * structure shall be filled out for each of its operations. When the structure\n+ * is populated it can be sent to the other part of the protocol and decoded.\n+ *\n+ * The \\a operation describes which operation the receiver shall perform\n+ * and implicitly describes which of the other fields in the structure are\n+ * populated and how they shall be interpreted. All this is the responsibility\n+ * of the pipeline implementation.\n+ *\n+ * \\sa IPAInterface::processEvent()\n+ * \\sa IPAInterface::queueFrameAction\n+ */\n+\n+/**\n+ * \\var IPAOperationData::operation\n+ * \\brief Operation in the pipeline-to-IPA protocol\n+ *\n+ * The allocation of numerical values to operations are left to the\n+ * implementation of the pipeline protocol.\n+ */\n+\n+/**\n+ * \\var IPAOperationData::data\n+ * \\brief Array of unsigned int values\n+ *\n+ * The interpretation and position of different values in the array are left\n+ * to the implementation of the pipeline protocol.\n+ */\n+\n+/**\n+ * \\var IPAOperationData::controls\n+ * \\brief Array of ControlList\n+ *\n+ * The interpretation and position of different values in the array are left\n+ * to the implementation of the pipeline protocol.\n+ */\n+\n+/**\n+ * \\var IPAOperationData::v4l2controls\n+ * \\brief Array of V4L2ControlList\n+ *\n+ * The interpretation and position of different values in the array are left\n+ * to the implementation of the pipeline protocol.\n+ */\n+\n /**\n * \\class IPAInterface\n * \\brief Interface for IPA implementation\n+ *\n+ * Every pipeline implementation in libcamera may attach all or some of its\n+ * cameras to an Image Processing Algorithm (IPA) plugin. An IPA plugin is\n+ * developed for a specific pipeline and each pipeline might have multiple\n+ * competing IPA implementations, both open and closed source.\n+ *\n+ * To allow for multiple and competing IPA plugins for the same pipeline, an\n+ * interface for the pipeline and IPA communication is needed; IPAInterface\n+ * is this interface. The interface defines how and what type of data can be\n+ * exchanged, not how the data exchanged makes up a protocol. Each pipeline's\n+ * design is unique and so is its IPA protocol. It is the pipeline's\n+ * responsibility to define and document a protocol on top of the IPAInterface\n+ * which can be used by multiple IPA plugins.\n+ *\n+ * The IPAInterface provides some methods to help the protocol design. As the\n+ * requirements of the IPA plugins become more clear it is expected more\n+ * common protocol operations will be identified and added.\n+ * \\todo Add more common operations if possible.\n+ *\n+ * The pipeline shall use the IPAManager to locate an IPAInterface compatible\n+ * with the pipeline. The interface may then be used in the pipeline to interact\n+ * with the IPA plugin to make up the protocol.\n+ * \\todo Add reference to how pipelines shall document their protocol.\n+ *\n+ * \\sa IPAManager\n */\n \n /**\n@@ -24,4 +151,72 @@ namespace libcamera {\n * \\brief Initialise the IPAInterface\n */\n \n+/**\n+ * \\fn IPAInterface::configure()\n+ * \\brief Configure the IPA stream and sensor settings\n+ * \\param[in] streamConfig List of stream configuration descriptions\n+ * \\param[in] entityControls List of controls provided by the pipeline entities\n+ *\n+ * This operation shall be called when the camera is started to inform the IPA\n+ * of the camera's streams and the sensor settings.\n+ */\n+\n+/**\n+ * \\fn IPAInterface::mapBuffers()\n+ * \\brief Map the buffers shared by the pipeline to the IPA\n+ * \\param[in] buffers List of buffers to map\n+ *\n+ * This operation is used to inform the IPA module of the memory buffers set up\n+ * by the pipeline handler and associate to each of them a numerical \\a type and\n+ * \\a id. All buffers that the pipeline wishes to share with an IPA must be\n+ * mapped in the IPA with this operation.\n+ *\n+ * After the buffers have been initialized a specific buffer can be referenced\n+ * using the numerical \\a type and \\a id provided when the buffers were mapped.\n+ *\n+ * The numerical values for type and id used to describe a buffer have no\n+ * meaning in the core libcamera code and the interface is pipeline handler\n+ * specific.\n+ *\n+ * \\sa unmapBuffers()\n+ * \\todo Can we make this a generic function?\n+ */\n+\n+/**\n+ * \\fn IPAInterface::unmapBuffers()\n+ * \\brief Unmap the buffers shared by the pipeline to the IPA\n+ * \\param[in] buffers List of buffers to unmap\n+ *\n+ * This operation removes the mapping set up with mapBuffers().\n+ *\n+ * \\sa mapBuffers()\n+ * \\todo Can we make this a generic function?\n+ */\n+\n+/**\n+ * \\fn IPAInterface::processEvent()\n+ * \\brief Process an event from the pipeline handler\n+ * \\param[in] event Event to process\n+ *\n+ * This operation is used by pipeline handlers to inform the IPA module of\n+ * events that occurred during the on-going capture operation.\n+ *\n+ * Each \\a event notified by the pipeline handler with this method is handled\n+ * by the IPA which interprets the operation parameters in an implementation\n+ * specific way, which needs to be separately documented.\n+ */\n+\n+/**\n+ * \\fn IPAInterface::queueFrameAction()\n+ * \\brief Queue an action associated with a frame to the pipeline handler\n+ * \\param[in] frame The frame number for the action\n+ * \\param[in] controls List of controls associated with the action\n+ *\n+ * This signal is emitted when the IPA wishes to queue a FrameAction on the\n+ * pipeline. The pipeline is still responsible for the scheduling of the action\n+ * on its timeline.\n+ *\n+ * The IPA operation describing the frame action is passed as a parameter.\n+ */\n+\n } /* namespace libcamera */\ndiff --git a/src/libcamera/proxy/ipa_proxy_linux.cpp b/src/libcamera/proxy/ipa_proxy_linux.cpp\nindex 62fcb529e1c7e4ff..14e8bb6067623fc6 100644\n--- a/src/libcamera/proxy/ipa_proxy_linux.cpp\n+++ b/src/libcamera/proxy/ipa_proxy_linux.cpp\n@@ -26,7 +26,12 @@ public:\n \tIPAProxyLinux(IPAModule *ipam);\n \t~IPAProxyLinux();\n \n-\tint init();\n+\tint init() override { return 0; }\n+\tvoid configure(const std::map &streamConfig,\n+\t\t const std::map &entityControls) override {}\n+\tvoid mapBuffers(const std::vector &buffers) override {}\n+\tvoid unmapBuffers(const std::vector &buffers) override {}\n+\tvoid processEvent(const IPAOperationData &event) override {}\n \n private:\n \tvoid readyRead(IPCUnixSocket *ipc);\n@@ -36,13 +41,6 @@ private:\n \tIPCUnixSocket *socket_;\n };\n \n-int IPAProxyLinux::init()\n-{\n-\tLOG(IPAProxy, Debug) << \"initializing IPA via dummy proxy!\";\n-\n-\treturn 0;\n-}\n-\n IPAProxyLinux::IPAProxyLinux(IPAModule *ipam)\n \t: proc_(nullptr), socket_(nullptr)\n {\n","prefixes":["libcamera-devel","v5","07/10"]}