From patchwork Sat Apr 11 00:19:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Niklas_S=C3=B6derlund?= X-Patchwork-Id: 3428 Return-Path: Received: from bin-mail-out-05.binero.net (bin-mail-out-05.binero.net [195.74.38.228]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id F26C662826 for ; Sat, 11 Apr 2020 02:19:26 +0200 (CEST) X-Halon-ID: 15eb8ac9-7b8a-11ea-aeed-005056917f90 Authorized-sender: niklas@soderlund.pp.se Received: from bismarck.berto.se (p4fca2392.dip0.t-ipconnect.de [79.202.35.146]) by bin-vsp-out-02.atm.binero.net (Halon) with ESMTPA id 15eb8ac9-7b8a-11ea-aeed-005056917f90; Sat, 11 Apr 2020 02:19:19 +0200 (CEST) From: =?utf-8?q?Niklas_S=C3=B6derlund?= To: libcamera-devel@lists.libcamera.org Date: Sat, 11 Apr 2020 02:19:11 +0200 Message-Id: <20200411001911.1143155-8-niklas.soderlund@ragnatech.se> X-Mailer: git-send-email 2.26.0 In-Reply-To: <20200411001911.1143155-1-niklas.soderlund@ragnatech.se> References: <20200411001911.1143155-1-niklas.soderlund@ragnatech.se> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 7/7] libcamera: ipa_manager: Proxy open-source IPAs to a thread X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2020 00:19:27 -0000 From: Laurent Pinchart While closed-source IPA modules will always be sandboxed, open-source IPA modules may be run in the main libcamera process or be sandboxed, depending on platform configuration. These two models exhibit very different timings, which require extensive testing with both configurations. When run into the main libcamera process, IPA modules are executed in the pipeline handler thread (which is currently a global CameraManager thread). Time-consuming operations in the IPA may thus slow down the pipeline handler and compromise real-time behaviour. At least some pipeline handlers will thus likely spawn a thread to isolate the IPA, leading to code duplication in pipeline handlers. Solve both issues by always proxying IPA modules. For open-source IPA modules that run in the libcamera process, a new IPAProxyThread class is added to run the IPA in a separate thread. Signed-off-by: Laurent Pinchart [Niklas: Move thread start/stop of thread into start()/stop()] Signed-off-by: Niklas Söderlund --- * Changes since v2 - Move the thread at init() instead of start() - Call start() in the target thread. - Block frame actions in IPAProxyThread instead of leaving it up to each pipeline handler. --- src/libcamera/ipa_interface.cpp | 3 + src/libcamera/ipa_manager.cpp | 48 +++---- src/libcamera/proxy/ipa_proxy_thread.cpp | 162 +++++++++++++++++++++++ src/libcamera/proxy/meson.build | 1 + 4 files changed, 187 insertions(+), 27 deletions(-) create mode 100644 src/libcamera/proxy/ipa_proxy_thread.cpp diff --git a/src/libcamera/ipa_interface.cpp b/src/libcamera/ipa_interface.cpp index d8d57fdb2ae59d2e..8eaa1480e6dfdc9c 100644 --- a/src/libcamera/ipa_interface.cpp +++ b/src/libcamera/ipa_interface.cpp @@ -528,6 +528,9 @@ namespace libcamera { * \a data.operation field, as defined by the IPA protocol, and the rest of the * \a data is interpreted accordingly. The pipeline handler shall queue the * action and execute it as appropriate. + * + * The signal is only emitted when the IPA is running, that is after start() and + * before stop() have been called. */ } /* namespace libcamera */ diff --git a/src/libcamera/ipa_manager.cpp b/src/libcamera/ipa_manager.cpp index bcaae3564ea14e07..6d23f470506561b8 100644 --- a/src/libcamera/ipa_manager.cpp +++ b/src/libcamera/ipa_manager.cpp @@ -12,7 +12,6 @@ #include #include -#include "ipa_context_wrapper.h" #include "ipa_module.h" #include "ipa_proxy.h" #include "log.h" @@ -271,40 +270,35 @@ std::unique_ptr IPAManager::createIPA(PipelineHandler *pipe, if (!m) return nullptr; - if (!m->isOpenSource()) { - IPAProxyFactory *pf = nullptr; - std::vector &factories = IPAProxyFactory::factories(); + /* + * Load and run the IPA module in a thread if it is open-source, or + * isolate it in a separate process otherwise. + * + * \todo Implement a better proxy selection + */ + const char *proxyName = m->isOpenSource() + ? "IPAProxyThread" : "IPAProxyLinux"; + IPAProxyFactory *pf = nullptr; - for (IPAProxyFactory *factory : factories) { - /* TODO: Better matching */ - if (!strcmp(factory->name().c_str(), "IPAProxyLinux")) { - pf = factory; - break; - } + for (IPAProxyFactory *factory : IPAProxyFactory::factories()) { + if (!strcmp(factory->name().c_str(), proxyName)) { + pf = factory; + break; } - - if (!pf) { - LOG(IPAManager, Error) << "Failed to get proxy factory"; - return nullptr; - } - - std::unique_ptr proxy = pf->create(m); - if (!proxy->isValid()) { - LOG(IPAManager, Error) << "Failed to load proxy"; - return nullptr; - } - - return proxy; } - if (!m->load()) + if (!pf) { + LOG(IPAManager, Error) << "Failed to get proxy factory"; return nullptr; + } - struct ipa_context *ctx = m->createContext(); - if (!ctx) + std::unique_ptr proxy = pf->create(m); + if (!proxy->isValid()) { + LOG(IPAManager, Error) << "Failed to load proxy"; return nullptr; + } - return std::make_unique(ctx); + return proxy; } } /* namespace libcamera */ diff --git a/src/libcamera/proxy/ipa_proxy_thread.cpp b/src/libcamera/proxy/ipa_proxy_thread.cpp new file mode 100644 index 0000000000000000..368ccca1cf60b8ec --- /dev/null +++ b/src/libcamera/proxy/ipa_proxy_thread.cpp @@ -0,0 +1,162 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * ipa_proxy_thread.cpp - Proxy running an Image Processing Algorithm in a thread + */ + +#include + +#include +#include + +#include "ipa_context_wrapper.h" +#include "ipa_module.h" +#include "ipa_proxy.h" +#include "log.h" +#include "thread.h" + +namespace libcamera { + +LOG_DECLARE_CATEGORY(IPAProxy) + +class IPAProxyThread : public IPAProxy, public Object +{ +public: + IPAProxyThread(IPAModule *ipam); + + int init() override; + int start() override; + void stop() override; + + void configure(const std::map &streamConfig, + const std::map &entityControls) override; + void mapBuffers(const std::vector &buffers) override; + void unmapBuffers(const std::vector &ids) override; + void processEvent(const IPAOperationData &event) override; + +private: + void queueFrameAction(unsigned int frame, const IPAOperationData &data); + + /* Helper class to invoke processEvent() in another thread. */ + class ThreadProxy : public Object + { + public: + void setIPA(IPAInterface *ipa) + { + ipa_ = ipa; + } + + int start() + { + return ipa_->start(); + } + + void stop() + { + ipa_->stop(); + } + + void processEvent(const IPAOperationData &event) + { + ipa_->processEvent(event); + } + + private: + IPAInterface *ipa_; + }; + + bool running_; + Thread thread_; + ThreadProxy proxy_; + std::unique_ptr ipa_; +}; + +IPAProxyThread::IPAProxyThread(IPAModule *ipam) + : running_(false) +{ + if (!ipam->load()) + return; + + struct ipa_context *ctx = ipam->createContext(); + if (!ctx) { + LOG(IPAProxy, Error) + << "Failed to create IPA context for " << ipam->path(); + return; + } + + ipa_ = std::make_unique(ctx); + proxy_.setIPA(ipa_.get()); + + /* + * Proxy the queueFrameAction signal to dispatch it in the caller's + * thread. + */ + ipa_->queueFrameAction.connect(this, &IPAProxyThread::queueFrameAction); + + valid_ = true; +} + +int IPAProxyThread::init() +{ + int ret = ipa_->init(); + if (ret) + return ret; + + proxy_.moveToThread(&thread_); + + return 0; +} + +int IPAProxyThread::start() +{ + running_ = true; + thread_.start(); + + return proxy_.invokeMethod(&ThreadProxy::start, ConnectionTypeBlocking); +} + +void IPAProxyThread::stop() +{ + running_ = false; + + proxy_.invokeMethod(&ThreadProxy::stop, ConnectionTypeBlocking); + + thread_.exit(); + thread_.wait(); +} + +void IPAProxyThread::configure(const std::map &streamConfig, + const std::map &entityControls) +{ + ipa_->configure(streamConfig, entityControls); +} + +void IPAProxyThread::mapBuffers(const std::vector &buffers) +{ + ipa_->mapBuffers(buffers); +} + +void IPAProxyThread::unmapBuffers(const std::vector &ids) +{ + ipa_->unmapBuffers(ids); +} + +void IPAProxyThread::processEvent(const IPAOperationData &event) +{ + if (!running_) + return; + + /* Dispatch the processEvent() call to the thread. */ + proxy_.invokeMethod(&ThreadProxy::processEvent, ConnectionTypeQueued, + event); +} + +void IPAProxyThread::queueFrameAction(unsigned int frame, const IPAOperationData &data) +{ + IPAInterface::queueFrameAction.emit(frame, data); +} + +REGISTER_IPA_PROXY(IPAProxyThread) + +} /* namespace libcamera */ diff --git a/src/libcamera/proxy/meson.build b/src/libcamera/proxy/meson.build index efc1132302176f63..6c00d5f30ad2e5f0 100644 --- a/src/libcamera/proxy/meson.build +++ b/src/libcamera/proxy/meson.build @@ -1,3 +1,4 @@ libcamera_sources += files([ 'ipa_proxy_linux.cpp', + 'ipa_proxy_thread.cpp', ])