From patchwork Fri Aug 12 09:08:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 17088 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0D0EBC3272 for ; Fri, 12 Aug 2022 09:08:53 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C469863331; Fri, 12 Aug 2022 11:08:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660295332; bh=VO/fxVcJWGHYLOdEQrn7s1ICjuFYL6BRUvHx/cqFiwg=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=rc7tUzTywQPgrRt0wAJ3U5LP0tP6udR1518NrQMemJF3yMm2dsji43JUjoXAk4Ynq ZNzaXJb5nPV3JTWb42p9o9YE/RiEViyQ6SooyCu7nQ9wcxOpMLlNI8mRP9flO+7X5Z tQClMydIzpq3XPhQjW3B0LVpsq2pUicLIMUi5VScpli27XakAoERxP+CrP/mWGDHB7 qdF/z/piY82EV0UIvCfe78m/ltHpxMTdmAuUlBEQVhdnRnReoJQOf9uLtwS+cnTFfq oQIFy82nweVtj4LS+rn9IHG1e0hMO55koIOJ117PV8mbhhwiF7jmIFM71oc4H1XD+v JpmizUV0HwYmw== Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D32E16332D for ; Fri, 12 Aug 2022 11:08:51 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="QbzabP55"; dkim-atps=neutral Received: by mail-pg1-x52c.google.com with SMTP id bh13so328666pgb.4 for ; Fri, 12 Aug 2022 02:08:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=R85XYqoVWEGvT6M8Agx+9ORKuOA7agTT19FBIQSY1xk=; b=QbzabP55WJiV6FkPPgUG7F7cMEBgxRX9eFeHUNN3Hi54veBHzdDrwyPPqWynhz9dvK MNPoq7YQWBDeRX25VmJAQNR/0mQPHd3KSoel3y4I+tx/7MaCHUEjsibHvHjuQgltAA4m 3K2jJBuesxZBdOSR7C9CQ6vbmj5q4IF1QIZZ0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=R85XYqoVWEGvT6M8Agx+9ORKuOA7agTT19FBIQSY1xk=; b=MGv8+W89dek0NfwS+P6qMwaHzaNoN5d9epF1mhkEHphnCgV0r3rGrRTzSAARZu4deW JwxyeocNQQCPU+AUzHlFdWShuY3o58M3ZdPUfw2OOA0CGvXvasiDnBC/c6ZPAzptxlCj 3EQ/ebR6uN0F6XG+nGFjR3gsSwq5sJArKJqyoRhI246ZYciuoCsEB2D26t2KT2OiUKMQ PXrJSSQGIRiRQd7QthYHp3RBTsXoud+eR8lif7pNtjKVMNmTld6Y3ioWa4QzDKnFxBjc BnnryAub0Yzg5AETNrc3frunzDJ8F+mhd2gFNs0g8I9YXGhbwaV0uBE4ODU5cq+45oDd 62eA== X-Gm-Message-State: ACgBeo0FUinRCvMi+ncLsInjQ5AVMJlx47IaV8PXNk4VCjshYHEuLadi TM/OMSu4MBAg6Jq+Ng7c3x2D+cvimE99rA== X-Google-Smtp-Source: AA6agR7i74sDED9cbQYg5vIMwnuQNf/QGGblGRtgjX5G85nWe5hGrBt6VQ1eDHwSGLch/OHfwDqE/A== X-Received: by 2002:a05:6a00:88f:b0:530:dec:81fd with SMTP id q15-20020a056a00088f00b005300dec81fdmr3081455pfj.64.1660295330122; Fri, 12 Aug 2022 02:08:50 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:1705:d284:d114:2e24]) by smtp.gmail.com with UTF8SMTPSA id ij1-20020a170902ab4100b0016f1ef2cd44sm1164281plb.154.2022.08.12.02.08.49 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 12 Aug 2022 02:08:49 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Fri, 12 Aug 2022 17:08:33 +0800 Message-Id: <20220812090838.1784703-3-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog In-Reply-To: <20220812090838.1784703-1-hanlinchen@chromium.org> References: <20220812090838.1784703-1-hanlinchen@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 2/7] libcamera: Camera: Add signals for completion of metadata and partial result X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Allows pipelien handler to signal partial result completion by adding the following signals to Camera: Signal metadataCompleted; Signal partialResultCompleted; Together with the bufferCompleted signal, the pipeline handler is allowed to return buffers, partial metadata and combination of both at any stage of processing. Signed-off-by: Han-Lin Chen --- include/libcamera/camera.h | 2 + include/libcamera/internal/pipeline_handler.h | 4 + include/libcamera/request.h | 40 +++++ src/libcamera/camera.cpp | 10 ++ src/libcamera/pipeline_handler.cpp | 121 ++++++++++++- src/libcamera/request.cpp | 170 +++++++++++++++++- 6 files changed, 332 insertions(+), 15 deletions(-) diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h index 5aa4bf69..187e01d8 100644 --- a/include/libcamera/camera.h +++ b/include/libcamera/camera.h @@ -99,7 +99,9 @@ public: const std::string &id() const; + Signal partialResultCompleted; Signal bufferCompleted; + Signal metadataCompleted; Signal requestCompleted; Signal<> disconnected; diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h index c3e4c258..08519fb3 100644 --- a/include/libcamera/internal/pipeline_handler.h +++ b/include/libcamera/internal/pipeline_handler.h @@ -33,6 +33,7 @@ class FrameBuffer; class MediaDevice; class PipelineHandler; class Request; +class Result; class PipelineHandler : public std::enable_shared_from_this, public Object @@ -62,7 +63,9 @@ public: void registerRequest(Request *request); void queueRequest(Request *request); + void completeMetadata(Request *request, const ControlList &metadata); bool completeBuffer(Request *request, FrameBuffer *buffer); + void completePartialResult(Request *request, Result &&result); void completeRequest(Request *request); const char *name() const { return name_; } @@ -80,6 +83,7 @@ private: void mediaDeviceDisconnected(MediaDevice *media); virtual void disconnect(); + void cancelRequest(Request *request); void doQueueRequest(Request *request); void doQueueRequests(); diff --git a/include/libcamera/request.h b/include/libcamera/request.h index dffde153..ac520226 100644 --- a/include/libcamera/request.h +++ b/include/libcamera/request.h @@ -25,8 +25,43 @@ namespace libcamera { class Camera; class CameraControlValidator; class FrameBuffer; +class Request; class Stream; +class Result +{ +public: + Result(Request *request); + Result(Result &&result); + ~Result(); + + Request *request() const { return request_; } + const ControlList &metadata() const { return metadata_; } + + const std::vector &buffers() const { return buffers_; } + int addBuffer(FrameBuffer *buffer); + + template + void set(const Control &ctrl, const V &value) + { + return metadata_.set(ctrl, value); + } + + void set(unsigned int id, const ControlValue &value); + void merge(const ControlList &source); + + std::string toString() const; + +private: + LIBCAMERA_DISABLE_COPY(Result) + + Request *request_; + ControlList metadata_; + std::vector buffers_; +}; + +std::ostream &operator<<(std::ostream &out, const Result &r); + class Request : public Extensible { LIBCAMERA_DECLARE_PRIVATE() @@ -44,6 +79,7 @@ public: }; using BufferMap = std::map; + using ResultList = std::vector; Request(Camera *camera, uint64_t cookie = 0); ~Request(); @@ -53,9 +89,12 @@ public: ControlList &controls() { return *controls_; } ControlList &metadata() { return *metadata_; } const BufferMap &buffers() const { return bufferMap_; } + ResultList &resultList() { return results_; } + Result *addResult(Result &&result); int addBuffer(const Stream *stream, FrameBuffer *buffer, std::unique_ptr fence = nullptr); FrameBuffer *findBuffer(const Stream *stream) const; + const Stream *findStream(const FrameBuffer *buffer) const; uint32_t sequence() const; uint64_t cookie() const { return cookie_; } @@ -71,6 +110,7 @@ private: ControlList *controls_; ControlList *metadata_; BufferMap bufferMap_; + ResultList results_; const uint64_t cookie_; Status status_; diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp index 4d3523e2..a5fb9b33 100644 --- a/src/libcamera/camera.cpp +++ b/src/libcamera/camera.cpp @@ -765,6 +765,16 @@ const std::string &Camera::id() const * completed */ +/** + * \var Camera::metadataCompleted + * \brief Signal emitted when a partial metadata for a request is completed + */ + +/** + * \var Camera::partialResultCompleted + * \brief Signal emitted when a partial result for a request is completed + */ + /** * \var Camera::requestCompleted * \brief Signal emitted when a request queued to the camera has completed diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp index 7c180a8e..6608c210 100644 --- a/src/libcamera/pipeline_handler.cpp +++ b/src/libcamera/pipeline_handler.cpp @@ -305,8 +305,9 @@ void PipelineHandler::stop(Camera *camera) Request *request = waitingRequests_.front(); waitingRequests_.pop(); - request->_d()->cancel(); - completeRequest(request); + cancelRequest(request); + request->_d()->complete(); + camera->requestCompleted.emit(request); } /* Make sure no requests are pending. */ @@ -402,17 +403,30 @@ void PipelineHandler::doQueueRequest(Request *request) request->_d()->sequence_ = data->requestSequence_++; if (request->_d()->cancelled_) { + cancelRequest(request); completeRequest(request); return; } int ret = queueRequestDevice(camera, request); if (ret) { - request->_d()->cancel(); + cancelRequest(request); completeRequest(request); } } +/** + * \brief Cancel buffers of a request and complete the all buffers + */ +void PipelineHandler::cancelRequest(Request *request) +{ + request->_d()->cancel(); + for (auto it : request->buffers()) { + FrameBuffer *buffer = it.second; + completeBuffer(request, buffer); + } +} + /** * \brief Queue prepared requests to the device * @@ -453,9 +467,10 @@ void PipelineHandler::doQueueRequests() * \param[in] request The request the buffer belongs to * \param[in] buffer The buffer that has completed * - * This function shall be called by pipeline handlers to signal completion of + * This function could be called by pipeline handlers to signal completion of * the \a buffer part of the \a request. It notifies applications of buffer - * completion and updates the request's internal buffer tracking. The request + * completion and updates the request's internal buffer tracking. The function + * notify completion of a partial result including the buffer. The request * is not completed automatically when the last buffer completes to give * pipeline handlers a chance to perform any operation that may still be * needed. They shall complete requests explicitly with completeRequest(). @@ -466,10 +481,75 @@ void PipelineHandler::doQueueRequests() * otherwise */ bool PipelineHandler::completeBuffer(Request *request, FrameBuffer *buffer) +{ + Result result(request); + result.addBuffer(buffer); + completePartialResult(request, std::move(result)); + + return request->_d()->hasPendingBuffers(); +} + +/** + * \brief Complete part of metadata for a request + * \param[in] request The request the buffer belongs to + * \param[in] metadata The partial metadata that has completed + * + * This function could be called by pipeline handlers to signal completion of + * the \a metadata part of the \a request. It notifies applications of metadata + * completion. The function notify completion of a partial result including the + * metadata. The request is not completed automatically when the last metadata + * completes to give pipeline handlers a chance to perform any operation that + * may still be needed. They shall complete requests explicitly with + * completeRequest(). + * + * \context This function shall be called from the CameraManager thread. + */ +void PipelineHandler::completeMetadata(Request *request, const ControlList &metadata) +{ + Result result = Result(request); + result.merge(metadata); + completePartialResult(request, std::move(result)); +} + +/** + * \brief Complete part of metadata and buffer for a request + * \param[in] request The request the buffer belongs to + * \param[in] result The partial result that has completed + * + * This function could be called by pipeline handlers to signal completion of + * the \a result part of the \a request. It notifies applications of partial + * completion. The function notify completion of buffers and metadata included + * in the result. The request is not completed automatically when the last + * result completes to give pipeline handlers a chance to perform any operation + * that may still be needed. They shall complete requests explicitly with + * completeRequest(). The function only accepts rvalue of a Result type and + * its interval content will be moved to the internal store to avoid copying + * big metadata. + * + * \context This function shall be called from the CameraManager thread. + */ +void PipelineHandler::completePartialResult(Request *request, Result &&result) { Camera *camera = request->_d()->camera(); - camera->bufferCompleted.emit(request, buffer); - return request->_d()->completeBuffer(buffer); + Result *movedResult = request->addResult(std::move(result)); + + ASSERT(movedResult->request() == request); + ASSERT(!movedResult->buffers().empty() || + !movedResult->metadata().empty()); + + for (auto buffer : movedResult->buffers()) { + request->_d()->completeBuffer(buffer); + camera->bufferCompleted.emit(request, buffer); + } + + if (!movedResult->metadata().empty()) { + request->metadata().merge(movedResult->metadata()); + camera->metadataCompleted.emit( + request, + &const_cast(movedResult->metadata())); + } + + camera->partialResultCompleted.emit(request, movedResult); } /** @@ -494,6 +574,33 @@ void PipelineHandler::completeRequest(Request *request) Camera::Private *data = camera->_d(); + /* + * Collect metadata which is not yet completed by the Camera, and + * create one partial result to cover the missing metadata before + * completing the whole request. This guarantees the aggregation of + * metadata in completed partial results equals to the global metadata + * in the request. + * + * \todo: Forbid merging metadata into request.metadata() directly and + * force calling completeMetadata() and completePartialResult() to + * report metadata. + */ + std::unordered_set completedMetadata; + for (auto &result : request->resultList()) { + for (auto &[id, _] : result.metadata()) + completedMetadata.insert(id); + } + + ControlList &requestMetadata = request->metadata(); + if (requestMetadata.size() > completedMetadata.size()) { + Result result(request); + for (auto &[id, value] : requestMetadata) + if (!completedMetadata.count(id)) + result.set(id, value); + + completePartialResult(request, std::move(result)); + } + auto iter = data->queuedRequests_.begin(); while (iter != data->queuedRequests_.end()) { if ((*iter)->status() != Request::RequestPending) { diff --git a/src/libcamera/request.cpp b/src/libcamera/request.cpp index d2af1d22..239c29cf 100644 --- a/src/libcamera/request.cpp +++ b/src/libcamera/request.cpp @@ -126,15 +126,10 @@ void Request::Private::complete() void Request::Private::doCancelRequest() { - Request *request = _o(); - - for (FrameBuffer *buffer : pending_) { + for (FrameBuffer *buffer : pending_) buffer->_d()->cancel(); - camera_->bufferCompleted.emit(request, buffer); - } cancelled_ = true; - pending_.clear(); notifiers_.clear(); timer_.reset(); } @@ -144,8 +139,8 @@ void Request::Private::doCancelRequest() * * Mark the request and its associated buffers as cancelled and complete it. * - * Set each pending buffer in error state and emit the buffer completion signal - * before completing the Request. + * Set each pending buffer in error state. The pipeline handler shall complete + * the cancelled buffers to notice the application. */ void Request::Private::cancel() { @@ -323,6 +318,11 @@ void Request::Private::timeout() * \brief A map of Stream to FrameBuffer pointers */ +/** + * \typedef Request::ResultList + * \brief A list partial results associated with a request + */ + /** * \class Request * \brief A frame capture request @@ -394,6 +394,7 @@ void Request::reuse(ReuseFlag flags) status_ = RequestPending; + results_.clear(); controls_->clear(); metadata_->clear(); } @@ -424,6 +425,12 @@ void Request::reuse(ReuseFlag flags) * \return The map of Stream to FrameBuffer */ +/** + * \fn Request::resultList() + * \brief Retrieve the request's partial results + * \return A reference to the list of results that associates with the request + */ + /** * \brief Add a FrameBuffer with its associated Stream to the Request * \param[in] stream The stream the buffer belongs to @@ -489,6 +496,20 @@ int Request::addBuffer(const Stream *stream, FrameBuffer *buffer, return 0; } +/** + * \brief Add result into the internal result list + * \param[in] result The result to add into the request + * + * The function only accepts rvalue and moves its content into the request + * + * \return The result moved into the request + */ +Result *Request::addResult(Result &&result) +{ + results_.emplace_back(std::move(result)); + return &results_.back(); +} + /** * \var Request::bufferMap_ * \brief Mapping of streams to buffers for this request @@ -513,6 +534,21 @@ FrameBuffer *Request::findBuffer(const Stream *stream) const return it->second; } +/** + * \brief Return the stream associated with a buffer + * \param[in] buffer The buffer the stream is associated to + * \return The stream associated with the buffer, or nullptr if the buffer is + * not part of this request + */ +const Stream *Request::findStream(const FrameBuffer *buffer) const +{ + for (auto &[key, value] : bufferMap_) + if (buffer == value) + return key; + + return nullptr; +} + /** * \fn Request::metadata() * \brief Retrieve the request's metadata @@ -606,4 +642,122 @@ std::ostream &operator<<(std::ostream &out, const Request &r) return out; } +/** + * \class Result + * \brief A partial result of a frame capture request + * + * A Result allows pipeline handler to report partial results to the application + */ + +/** + * \brief Create a partial result for a capture request + * \param[in] request The request the result associated to + */ +Result::Result(Request *request) + : request_(request) +{ +} + +/** + * \brief Move constructor of a Result + * \param[in] result The other result + */ +Result::Result(Result &&result) = default; + +Result::~Result() = default; + +/** + * \fn Result::request() + * \brief Retrieve the result's associated request + * \return The Request pointer associated with the result + */ + +/** + * \fn Result::metadata() + * \brief Retrieve the result's metadata + * \return The metadata contained in the result + */ + +/** + * \fn Result::buffers() + * \brief Retrieve the result's buffers + * \return The buffers contained in the result + */ + +/** + * \fn Result::set(const Control &ctrl, const V &value) + * \brief Set the control \a ctrl value to \a value into metadata + * \param[in] ctrl The control + * \param[in] value The control value + */ + +/** + * \brief Add a FrameBuffer with its associated Stream to the Result + * \param[in] buffer The FrameBuffer to add to the result + * + * \return 0 on success or a negative error code otherwise + * \retval -EINVAL The buffer does not reference a valid Stream in the request + */ +int Result::addBuffer(FrameBuffer *buffer) +{ + if (!buffer || !request_->findStream(buffer)) { + LOG(Request, Error) << "Invalid buffer reference"; + return -EINVAL; + } + + buffers_.emplace_back(buffer); + return 0; +} + +/** + * \brief Set the control \a id to \a value + * \param[in] id The control id + * \param[in] value The control value + */ +void Result::set(unsigned int id, const ControlValue &value) +{ + metadata_.set(id, value); +} + +/** + * \brief Merge the \a source into the metadata of the result + * \param[in] source The ControlList to merge into this metadata + */ +void Result::merge(const ControlList &source) +{ + metadata_.merge(source); +} + +/** + * \brief Generate a string representation of the Result internals + * + * This function facilitates debugging of Result state while it is used + * internally within libcamera. + * + * \return A string representing the current state of the result + */ +std::string Result::toString() const +{ + std::stringstream ss; + ss << *this; + + return ss.str(); +} + +/** + * \brief Insert a text representation of a Result into an output stream + * \param[in] out The output stream + * \param[in] r The Result + * \return The output stream \a out + */ +std::ostream &operator<<(std::ostream &out, const Result &r) +{ + /* Example Output: Result(55:1/2) */ + out << "Result(" << r.request()->sequence() << ":" + << r.buffers().size() << "/" + << const_cast(r).metadata().size() << ")"; + + return out; +} + } /* namespace libcamera */