{"id":3342,"url":"https://patchwork.libcamera.org/api/patches/3342/?format=json","web_url":"https://patchwork.libcamera.org/patch/3342/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20200326160819.4088361-8-niklas.soderlund@ragnatech.se>","date":"2020-03-26T16:08:19","name":"[libcamera-devel,RFCv2,7/7] libcamera: ipa_manager: Proxy open-source IPAs to a thread","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"bcdeffc3d296813f68751fab0eced7fe4689d3c1","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/?format=json","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/3342/mbox/","series":[{"id":783,"url":"https://patchwork.libcamera.org/api/series/783/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=783","date":"2020-03-26T16:08:12","name":"libcamera: ipa_manager: Proxy open-source IPAs to a thread","version":1,"mbox":"https://patchwork.libcamera.org/series/783/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/3342/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/3342/checks/","tags":{},"headers":{"Return-Path":"","Received":["from bin-mail-out-06.binero.net (bin-mail-out-06.binero.net\n\t[195.74.38.229])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id E349E605D2\n\tfor ;\n\tThu, 26 Mar 2020 17:10:35 +0100 (CET)","from bismarck.berto.se (p4fca2392.dip0.t-ipconnect.de\n\t[79.202.35.146]) by bin-vsp-out-03.atm.binero.net (Halon) with ESMTPA\n\tid 4e0f3aa4-6f7c-11ea-89d0-0050569116f7;\n\tThu, 26 Mar 2020 17:10:26 +0100 (CET)"],"X-Halon-ID":"4e0f3aa4-6f7c-11ea-89d0-0050569116f7","Authorized-sender":"niklas@soderlund.pp.se","From":"=?utf-8?q?Niklas_S=C3=B6derlund?= ","To":"libcamera-devel@lists.libcamera.org","Date":"Thu, 26 Mar 2020 17:08:19 +0100","Message-Id":"<20200326160819.4088361-8-niklas.soderlund@ragnatech.se>","X-Mailer":"git-send-email 2.25.1","In-Reply-To":"<20200326160819.4088361-1-niklas.soderlund@ragnatech.se>","References":"<20200326160819.4088361-1-niklas.soderlund@ragnatech.se>","MIME-Version":"1.0","Content-Type":"text/plain; charset=UTF-8","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [RFCv2 7/7] libcamera: ipa_manager: Proxy\n\topen-source IPAs to a thread","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","X-List-Received-Date":"Thu, 26 Mar 2020 16:10:37 -0000"},"content":"From: Laurent Pinchart \n\nWhile closed-source IPA modules will always be sandboxed, open-source\nIPA modules may be run in the main libcamera process or be sandboxed,\ndepending on platform configuration. These two models exhibit very\ndifferent timings, which require extensive testing with both\nconfigurations.\n\nWhen run into the main libcamera process, IPA modules are executed in\nthe pipeline handler thread (which is currently a global CameraManager\nthread). Time-consuming operations in the IPA may thus slow down the\npipeline handler and compromise real-time behaviour. At least some\npipeline handlers will thus likely spawn a thread to isolate the IPA,\nleading to code duplication in pipeline handlers.\n\nSolve both issues by always proxying IPA modules. For open-source IPA\nmodules that run in the libcamera process, a new IPAProxyThread class is\nadded to run the IPA in a separate thread.\n\nSigned-off-by: Laurent Pinchart \n[Niklas: Move thread start/stop of thread into start()/stop()]\nSigned-off-by: Niklas Söderlund \n---\n src/libcamera/ipa_manager.cpp | 48 ++++----\n src/libcamera/proxy/ipa_proxy_thread.cpp | 144 +++++++++++++++++++++++\n src/libcamera/proxy/meson.build | 1 +\n 3 files changed, 166 insertions(+), 27 deletions(-)\n create mode 100644 src/libcamera/proxy/ipa_proxy_thread.cpp","diff":"diff --git a/src/libcamera/ipa_manager.cpp b/src/libcamera/ipa_manager.cpp\nindex bcaae3564ea14e07..6d23f470506561b8 100644\n--- a/src/libcamera/ipa_manager.cpp\n+++ b/src/libcamera/ipa_manager.cpp\n@@ -12,7 +12,6 @@\n #include \n #include \n \n-#include \"ipa_context_wrapper.h\"\n #include \"ipa_module.h\"\n #include \"ipa_proxy.h\"\n #include \"log.h\"\n@@ -271,40 +270,35 @@ std::unique_ptr IPAManager::createIPA(PipelineHandler *pipe,\n \tif (!m)\n \t\treturn nullptr;\n \n-\tif (!m->isOpenSource()) {\n-\t\tIPAProxyFactory *pf = nullptr;\n-\t\tstd::vector &factories = IPAProxyFactory::factories();\n+\t/*\n+\t * Load and run the IPA module in a thread if it is open-source, or\n+\t * isolate it in a separate process otherwise.\n+\t *\n+\t * \\todo Implement a better proxy selection\n+\t */\n+\tconst char *proxyName = m->isOpenSource()\n+\t\t\t ? \"IPAProxyThread\" : \"IPAProxyLinux\";\n+\tIPAProxyFactory *pf = nullptr;\n \n-\t\tfor (IPAProxyFactory *factory : factories) {\n-\t\t\t/* TODO: Better matching */\n-\t\t\tif (!strcmp(factory->name().c_str(), \"IPAProxyLinux\")) {\n-\t\t\t\tpf = factory;\n-\t\t\t\tbreak;\n-\t\t\t}\n+\tfor (IPAProxyFactory *factory : IPAProxyFactory::factories()) {\n+\t\tif (!strcmp(factory->name().c_str(), proxyName)) {\n+\t\t\tpf = factory;\n+\t\t\tbreak;\n \t\t}\n-\n-\t\tif (!pf) {\n-\t\t\tLOG(IPAManager, Error) << \"Failed to get proxy factory\";\n-\t\t\treturn nullptr;\n-\t\t}\n-\n-\t\tstd::unique_ptr proxy = pf->create(m);\n-\t\tif (!proxy->isValid()) {\n-\t\t\tLOG(IPAManager, Error) << \"Failed to load proxy\";\n-\t\t\treturn nullptr;\n-\t\t}\n-\n-\t\treturn proxy;\n \t}\n \n-\tif (!m->load())\n+\tif (!pf) {\n+\t\tLOG(IPAManager, Error) << \"Failed to get proxy factory\";\n \t\treturn nullptr;\n+\t}\n \n-\tstruct ipa_context *ctx = m->createContext();\n-\tif (!ctx)\n+\tstd::unique_ptr proxy = pf->create(m);\n+\tif (!proxy->isValid()) {\n+\t\tLOG(IPAManager, Error) << \"Failed to load proxy\";\n \t\treturn nullptr;\n+\t}\n \n-\treturn std::make_unique(ctx);\n+\treturn proxy;\n }\n \n } /* namespace libcamera */\ndiff --git a/src/libcamera/proxy/ipa_proxy_thread.cpp b/src/libcamera/proxy/ipa_proxy_thread.cpp\nnew file mode 100644\nindex 0000000000000000..6719bcdda43f3647\n--- /dev/null\n+++ b/src/libcamera/proxy/ipa_proxy_thread.cpp\n@@ -0,0 +1,144 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2020, Google Inc.\n+ *\n+ * ipa_proxy_thread.cpp - Proxy running an Image Processing Algorithm in a thread\n+ */\n+\n+#include \n+\n+#include \n+#include \n+\n+#include \"ipa_context_wrapper.h\"\n+#include \"ipa_module.h\"\n+#include \"ipa_proxy.h\"\n+#include \"log.h\"\n+#include \"thread.h\"\n+\n+namespace libcamera {\n+\n+LOG_DECLARE_CATEGORY(IPAProxy)\n+\n+class IPAProxyThread : public IPAProxy, public Object\n+{\n+public:\n+\tIPAProxyThread(IPAModule *ipam);\n+\n+\tint init() override;\n+\tint start() override;\n+\tvoid stop() override;\n+\n+\tvoid configure(const std::map &streamConfig,\n+\t\t const std::map &entityControls) override;\n+\tvoid mapBuffers(const std::vector &buffers) override;\n+\tvoid unmapBuffers(const std::vector &ids) override;\n+\tvoid processEvent(const IPAOperationData &event) override;\n+\n+private:\n+\tvoid queueFrameAction(unsigned int frame, const IPAOperationData &data);\n+\n+\t/* Helper class to invoke processEvent() in another thread. */\n+\tclass ThreadProxy : public Object\n+\t{\n+\tpublic:\n+\t\tvoid setIPA(IPAInterface *ipa)\n+\t\t{\n+\t\t\tipa_ = ipa;\n+\t\t}\n+\n+\t\tvoid stop()\n+\t\t{\n+\t\t\tipa_->stop();\n+\t\t}\n+\n+\t\tvoid processEvent(const IPAOperationData &event)\n+\t\t{\n+\t\t\tipa_->processEvent(event);\n+\t\t}\n+\n+\tprivate:\n+\t\tIPAInterface *ipa_;\n+\t};\n+\n+\tThread thread_;\n+\tThreadProxy proxy_;\n+\tstd::unique_ptr ipa_;\n+};\n+\n+IPAProxyThread::IPAProxyThread(IPAModule *ipam)\n+{\n+\tif (!ipam->load())\n+\t\treturn;\n+\n+\tstruct ipa_context *ctx = ipam->createContext();\n+\tif (!ctx) {\n+\t\tLOG(IPAProxy, Error)\n+\t\t\t<< \"Failed to create IPA context for \" << ipam->path();\n+\t\treturn;\n+\t}\n+\n+\tipa_ = std::make_unique(ctx);\n+\tproxy_.setIPA(ipa_.get());\n+\n+\t/*\n+\t * Proxy the queueFrameAction signal to dispatch it in the caller's\n+\t * thread.\n+\t */\n+\tipa_->queueFrameAction.connect(this, &IPAProxyThread::queueFrameAction);\n+\n+\tvalid_ = true;\n+}\n+\n+int IPAProxyThread::init()\n+{\n+\treturn ipa_->init();\n+}\n+\n+int IPAProxyThread::start()\n+{\n+\tthread_.start();\n+\tproxy_.moveToThread(&thread_);\n+\n+\treturn ipa_->start();\n+}\n+\n+void IPAProxyThread::stop()\n+{\n+\tproxy_.invokeMethod(&ThreadProxy::stop, ConnectionTypeBlocking);\n+\n+\tthread_.exit();\n+\tthread_.wait();\n+}\n+\n+void IPAProxyThread::configure(const std::map &streamConfig,\n+\t\t\t const std::map &entityControls)\n+{\n+\tipa_->configure(streamConfig, entityControls);\n+}\n+\n+void IPAProxyThread::mapBuffers(const std::vector &buffers)\n+{\n+\tipa_->mapBuffers(buffers);\n+}\n+\n+void IPAProxyThread::unmapBuffers(const std::vector &ids)\n+{\n+\tipa_->unmapBuffers(ids);\n+}\n+\n+void IPAProxyThread::processEvent(const IPAOperationData &event)\n+{\n+\t/* Dispatch the processEvent() call to the thread. */\n+\tproxy_.invokeMethod(&ThreadProxy::processEvent, ConnectionTypeQueued,\n+\t\t\t event);\n+}\n+\n+void IPAProxyThread::queueFrameAction(unsigned int frame, const IPAOperationData &data)\n+{\n+\tIPAInterface::queueFrameAction.emit(frame, data);\n+}\n+\n+REGISTER_IPA_PROXY(IPAProxyThread)\n+\n+} /* namespace libcamera */\ndiff --git a/src/libcamera/proxy/meson.build b/src/libcamera/proxy/meson.build\nindex efc1132302176f63..6c00d5f30ad2e5f0 100644\n--- a/src/libcamera/proxy/meson.build\n+++ b/src/libcamera/proxy/meson.build\n@@ -1,3 +1,4 @@\n libcamera_sources += files([\n 'ipa_proxy_linux.cpp',\n+ 'ipa_proxy_thread.cpp',\n ])\n","prefixes":["libcamera-devel","RFCv2","7/7"]}