From patchwork Tue Jul 26 18:24:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 16803 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A0A7FBE173 for ; Tue, 26 Jul 2022 18:25:32 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5DC4763318; Tue, 26 Jul 2022 20:25:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1658859932; bh=/J1uvqY5cxlHrHovY8ssTqJmGQWBuqdFF6AammIt0tk=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=ThvY191oImr5lMI1jjL/lsUUGBj2NXmJrDo9tiZwQsjCjINRTXS2ayfWnqPsgVBC5 aq+h0oqJ7zQpvwLR/r9mwO8qJ2UXYZ5vZ76xXkPzEYnqmz5QSC2mA0m4CjWichUwgf Q2mdxfyNNpej31TJqNGmROkoxtAEqRSmv1lhkCgMylgp5lvQB6K846EaBLNAj7IJeH qEVfsjiK3CYplNAjDTeca4k3cuXUTDz8lMuo7zgK0zpj5Yg1uKU046rfdaDoSftEa1 /ISTKYZBi0iVRwU98lDO6rQr+nYn+04x8lO3InevsXrfJdxk4Eww0FugpP1KTUuotu oT1fdpxNJjDlA== Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3670363317 for ; Tue, 26 Jul 2022 20:25:31 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="XfKOWlei"; dkim-atps=neutral Received: by mail-pj1-x1029.google.com with SMTP id o5-20020a17090a3d4500b001ef76490983so14057906pjf.2 for ; Tue, 26 Jul 2022 11:25:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zl9BEdReWHmY1cqNo3JDLudkoxv111RZ4PNfY4ukwoY=; b=XfKOWlei3hAIpmQh/3s+N7zBJp1dwvvyb4L/1Uctq/HjTaTHAvfrXihaHvIQxQmqgM inD/0kmqnRvHnerGt69PVuCE9BKkjP+/SMagcZVzFRi+kbD3zqlj+80hS/lHsnnclMDQ tAI7uBXa8i7cird8hdu6DQwDkmmkey/O6Qoyc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zl9BEdReWHmY1cqNo3JDLudkoxv111RZ4PNfY4ukwoY=; b=6tNHFK0zZDx/S1u3bfZ+35fW/oYmOEFmD1Bhu4J4VYOtGS9/wihM3W5g3IhyUgpD3Y gWlauDhEhWS1neXBbkMywC3QWq5rWe+Xd9rzVahj2yQ/LW9ydTRkLGZrFMyFDABRtXD7 +LU4VGSMuuka2aJRGY0TbHObtNA/CgviwNB+pc6Vm59dxIpTnM2u0qcjfQeJ4t5GLJDA eXiCaeXTXnPYJVDeNCeXlmF4HNFYhh6eRbN567QXP14q/onmFmXMZnFcaBsa5WSdZAVR Ewc3dt3qsSji4FhPE9MtoU6zDEYWmnEG3nmcdGG4JWcjFqicSYm+EE2PZEYu5Nqn23QI yzrg== X-Gm-Message-State: AJIora844G+T/sGzKb+20W0k0h7Q/ASZYN2aZLMzW1BMJppIw0l5tWYl sFTc5AnjiBA6JAD2J+hWVoqhcKUaTQ6MJQ== X-Google-Smtp-Source: AGRyM1t49SkoqTu1YGVDI55gDC1vSe7cqV+kUqOHANCDKiAMx52X53v+SZwX8dJCkbRGfMm8iqBBvg== X-Received: by 2002:a17:90a:5aa3:b0:1f2:2bb7:aea9 with SMTP id n32-20020a17090a5aa300b001f22bb7aea9mr434836pji.197.1658859929224; Tue, 26 Jul 2022 11:25:29 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:739:6574:4032:5911]) by smtp.gmail.com with UTF8SMTPSA id ip13-20020a17090b314d00b001f2fee30a59sm904883pjb.57.2022.07.26.11.25.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 26 Jul 2022 11:25:29 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Wed, 27 Jul 2022 02:24:56 +0800 Message-Id: <20220726182500.425182-3-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.359.gd136c6c3e2-goog In-Reply-To: <20220726182500.425182-1-hanlinchen@chromium.org> References: <20220726182500.425182-1-hanlinchen@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 2/6] libcamera: Camera: Add signals for completion of metadata and partial result X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Allows pipelien handler to signal partial result completion by adding the following signals to Camera: Signal metadataCompleted; Signal partialResultCompleted; Together with the bufferCompleted signal, the pipeline handler is allowed to return buffers, partial metadata and combination of both at any stage of processing. Signed-off-by: Han-Lin Chen --- include/libcamera/camera.h | 2 + include/libcamera/internal/pipeline_handler.h | 3 + include/libcamera/request.h | 40 +++++ src/libcamera/camera.cpp | 10 ++ src/libcamera/pipeline_handler.cpp | 100 ++++++++++- src/libcamera/request.cpp | 159 ++++++++++++++++++ 6 files changed, 310 insertions(+), 4 deletions(-) diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h index 5aa4bf69..187e01d8 100644 --- a/include/libcamera/camera.h +++ b/include/libcamera/camera.h @@ -99,7 +99,9 @@ public: const std::string &id() const; + Signal partialResultCompleted; Signal bufferCompleted; + Signal metadataCompleted; Signal requestCompleted; Signal<> disconnected; diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h index c3e4c258..03d46367 100644 --- a/include/libcamera/internal/pipeline_handler.h +++ b/include/libcamera/internal/pipeline_handler.h @@ -33,6 +33,7 @@ class FrameBuffer; class MediaDevice; class PipelineHandler; class Request; +class Result; class PipelineHandler : public std::enable_shared_from_this, public Object @@ -62,7 +63,9 @@ public: void registerRequest(Request *request); void queueRequest(Request *request); + void completeMetadata(Request *request, const ControlList &metadata); bool completeBuffer(Request *request, FrameBuffer *buffer); + void completePartialResult(Request *request, Result &&result); void completeRequest(Request *request); const char *name() const { return name_; } diff --git a/include/libcamera/request.h b/include/libcamera/request.h index dffde153..ac520226 100644 --- a/include/libcamera/request.h +++ b/include/libcamera/request.h @@ -25,8 +25,43 @@ namespace libcamera { class Camera; class CameraControlValidator; class FrameBuffer; +class Request; class Stream; +class Result +{ +public: + Result(Request *request); + Result(Result &&result); + ~Result(); + + Request *request() const { return request_; } + const ControlList &metadata() const { return metadata_; } + + const std::vector &buffers() const { return buffers_; } + int addBuffer(FrameBuffer *buffer); + + template + void set(const Control &ctrl, const V &value) + { + return metadata_.set(ctrl, value); + } + + void set(unsigned int id, const ControlValue &value); + void merge(const ControlList &source); + + std::string toString() const; + +private: + LIBCAMERA_DISABLE_COPY(Result) + + Request *request_; + ControlList metadata_; + std::vector buffers_; +}; + +std::ostream &operator<<(std::ostream &out, const Result &r); + class Request : public Extensible { LIBCAMERA_DECLARE_PRIVATE() @@ -44,6 +79,7 @@ public: }; using BufferMap = std::map; + using ResultList = std::vector; Request(Camera *camera, uint64_t cookie = 0); ~Request(); @@ -53,9 +89,12 @@ public: ControlList &controls() { return *controls_; } ControlList &metadata() { return *metadata_; } const BufferMap &buffers() const { return bufferMap_; } + ResultList &resultList() { return results_; } + Result *addResult(Result &&result); int addBuffer(const Stream *stream, FrameBuffer *buffer, std::unique_ptr fence = nullptr); FrameBuffer *findBuffer(const Stream *stream) const; + const Stream *findStream(const FrameBuffer *buffer) const; uint32_t sequence() const; uint64_t cookie() const { return cookie_; } @@ -71,6 +110,7 @@ private: ControlList *controls_; ControlList *metadata_; BufferMap bufferMap_; + ResultList results_; const uint64_t cookie_; Status status_; diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp index 4d3523e2..a5fb9b33 100644 --- a/src/libcamera/camera.cpp +++ b/src/libcamera/camera.cpp @@ -765,6 +765,16 @@ const std::string &Camera::id() const * completed */ +/** + * \var Camera::metadataCompleted + * \brief Signal emitted when a partial metadata for a request is completed + */ + +/** + * \var Camera::partialResultCompleted + * \brief Signal emitted when a partial result for a request is completed + */ + /** * \var Camera::requestCompleted * \brief Signal emitted when a request queued to the camera has completed diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp index 7c180a8e..2102da58 100644 --- a/src/libcamera/pipeline_handler.cpp +++ b/src/libcamera/pipeline_handler.cpp @@ -453,9 +453,10 @@ void PipelineHandler::doQueueRequests() * \param[in] request The request the buffer belongs to * \param[in] buffer The buffer that has completed * - * This function shall be called by pipeline handlers to signal completion of + * This function could be called by pipeline handlers to signal completion of * the \a buffer part of the \a request. It notifies applications of buffer - * completion and updates the request's internal buffer tracking. The request + * completion and updates the request's internal buffer tracking. The function + * notify completion of a partial result including the buffer. The request * is not completed automatically when the last buffer completes to give * pipeline handlers a chance to perform any operation that may still be * needed. They shall complete requests explicitly with completeRequest(). @@ -466,10 +467,74 @@ void PipelineHandler::doQueueRequests() * otherwise */ bool PipelineHandler::completeBuffer(Request *request, FrameBuffer *buffer) +{ + Result result(request); + result.addBuffer(buffer); + completePartialResult(request, std::move(result)); + + return request->_d()->hasPendingBuffers(); +} + +/** + * \brief Complete part of metadata for a request + * \param[in] request The request the buffer belongs to + * \param[in] metadata The partial metadata that has completed + * + * This function could be called by pipeline handlers to signal completion of + * the \a metadata part of the \a request. It notifies applications of metadata + * completion. The function notify completion of a partial result including the + * metadata. The request is not completed automatically when the last metadata + * completes to give pipeline handlers a chance to perform any operation that + * may still be needed. They shall complete requests explicitly with + * completeRequest(). + * + * \context This function shall be called from the CameraManager thread. + */ +void PipelineHandler::completeMetadata(Request *request, const ControlList &metadata) +{ + Result result = Result(request); + result.merge(metadata); + completePartialResult(request, std::move(result)); +} + +/** + * \brief Complete part of metadata and buffer for a request + * \param[in] request The request the buffer belongs to + * \param[in] result The partial result that has completed + * + * This function could be called by pipeline handlers to signal completion of + * the \a result part of the \a request. It notifies applications of partial + * completion. The function notify completion of buffers and metadata included + * in the result. The request is not completed automatically when the last + * result completes to give pipeline handlers a chance to perform any operation + * that may still be needed. They shall complete requests explicitly with + * completeRequest(). The function only accepts rvalue of a Result type and + * its interval content will be moved to the internal store to avoid copying + * big metadata. + * + * \context This function shall be called from the CameraManager thread. + */ +void PipelineHandler::completePartialResult(Request *request, Result &&result) { Camera *camera = request->_d()->camera(); - camera->bufferCompleted.emit(request, buffer); - return request->_d()->completeBuffer(buffer); + Result *movedResult = request->addResult(std::move(result)); + + ASSERT(movedResult->request() == request); + ASSERT(!movedResult->buffers().empty() || !movedResult->metadata().empty()); + + for (auto buffer : movedResult->buffers()) { + request->_d()->completeBuffer(buffer); + camera->bufferCompleted.emit(request, buffer); + } + + if (!movedResult->metadata().empty()) { + request->metadata().merge(movedResult->metadata()); + camera->metadataCompleted.emit( + request, + &const_cast(movedResult->metadata())); + } + + camera->partialResultCompleted.emit(request, movedResult); } /** @@ -494,6 +559,33 @@ void PipelineHandler::completeRequest(Request *request) Camera::Private *data = camera->_d(); + /* + * Collect metadata which is not yet completed by the Camera, and + * create one partial result to cover the missing metadata before + * completing the whole request. This guarantees the aggregation of + * metadata in completed partial results equals to the global metadata + * in the request. + * + * \todo: Forbid merging metadata into request.metadata() directly and + * force calling completeMetadata() and completePartialResult() to + * report metadata. + */ + std::unordered_set completedMetadata; + for (auto &result : request->resultList()) { + for (auto &[id, _] : result.metadata()) + completedMetadata.insert(id); + } + + ControlList &requestMetadata = request->metadata(); + if (requestMetadata.size() > completedMetadata.size()) { + Result result(request); + for (auto &[id, value] : requestMetadata) + if (!completedMetadata.count(id)) + result.set(id, value); + + completePartialResult(request, std::move(result)); + } + auto iter = data->queuedRequests_.begin(); while (iter != data->queuedRequests_.end()) { if ((*iter)->status() != Request::RequestPending) { diff --git a/src/libcamera/request.cpp b/src/libcamera/request.cpp index d2af1d22..275722a4 100644 --- a/src/libcamera/request.cpp +++ b/src/libcamera/request.cpp @@ -323,6 +323,11 @@ void Request::Private::timeout() * \brief A map of Stream to FrameBuffer pointers */ +/** + * \typedef Request::ResultList + * \brief A list partial results associated with a request + */ + /** * \class Request * \brief A frame capture request @@ -394,6 +399,7 @@ void Request::reuse(ReuseFlag flags) status_ = RequestPending; + results_.clear(); controls_->clear(); metadata_->clear(); } @@ -424,6 +430,12 @@ void Request::reuse(ReuseFlag flags) * \return The map of Stream to FrameBuffer */ +/** + * \fn Request::resultList() + * \brief Retrieve the request's partial results + * \return A reference to the list of results that associates with the request + */ + /** * \brief Add a FrameBuffer with its associated Stream to the Request * \param[in] stream The stream the buffer belongs to @@ -489,6 +501,20 @@ int Request::addBuffer(const Stream *stream, FrameBuffer *buffer, return 0; } +/** + * \brief Add result into the internal result list + * \param[in] result The result to add into the request + * + * The function only accepts rvalue and moves its content into the request + * + * \return The result moved into the request + */ +Result *Request::addResult(Result &&result) +{ + results_.emplace_back(std::move(result)); + return &results_.back(); +} + /** * \var Request::bufferMap_ * \brief Mapping of streams to buffers for this request @@ -513,6 +539,21 @@ FrameBuffer *Request::findBuffer(const Stream *stream) const return it->second; } +/** + * \brief Return the stream associated with a buffer + * \param[in] buffer The buffer the stream is associated to + * \return The stream associated with the buffer, or nullptr if the buffer is + * not part of this request + */ +const Stream *Request::findStream(const FrameBuffer *buffer) const +{ + for (auto &[key, value] : bufferMap_) + if (buffer == value) + return key; + + return nullptr; +} + /** * \fn Request::metadata() * \brief Retrieve the request's metadata @@ -606,4 +647,122 @@ std::ostream &operator<<(std::ostream &out, const Request &r) return out; } +/** + * \class Result + * \brief A partial result of a frame capture request + * + * A Result allows pipeline handler to report partial results to the application + */ + +/** + * \brief Create a partial result for a capture request + * \param[in] request The request the result associated to + */ +Result::Result(Request *request) + : request_(request) +{ +} + +/** + * \brief Move constructor of a Result + * \param[in] result The other result + */ +Result::Result(Result &&result) = default; + +Result::~Result() = default; + +/** + * \fn Result::request() + * \brief Retrieve the result's associated request + * \return The Request pointer associated with the result + */ + +/** + * \fn Result::metadata() + * \brief Retrieve the result's metadata + * \return The metadata contained in the result + */ + +/** + * \fn Result::buffers() + * \brief Retrieve the result's buffers + * \return The buffers contained in the result + */ + +/** + * \fn Result::set(const Control &ctrl, const V &value) + * \brief Set the control \a ctrl value to \a value into metadata + * \param[in] ctrl The control + * \param[in] value The control value + */ + +/** + * \brief Add a FrameBuffer with its associated Stream to the Result + * \param[in] buffer The FrameBuffer to add to the result + * + * \return 0 on success or a negative error code otherwise + * \retval -EINVAL The buffer does not reference a valid Stream in the request + */ +int Result::addBuffer(FrameBuffer *buffer) +{ + if (!buffer || !request_->findStream(buffer)) { + LOG(Request, Error) << "Invalid buffer reference"; + return -EINVAL; + } + + buffers_.emplace_back(buffer); + return 0; +} + +/** + * \brief Set the control \a id to \a value + * \param[in] id The control id + * \param[in] value The control value + */ +void Result::set(unsigned int id, const ControlValue &value) +{ + metadata_.set(id, value); +} + +/** + * \brief Merge the \a source into the metadata of the result + * \param[in] source The ControlList to merge into this metadata + */ +void Result::merge(const ControlList &source) +{ + metadata_.merge(source); +} + +/** + * \brief Generate a string representation of the Result internals + * + * This function facilitates debugging of Result state while it is used + * internally within libcamera. + * + * \return A string representing the current state of the result + */ +std::string Result::toString() const +{ + std::stringstream ss; + ss << *this; + + return ss.str(); +} + +/** + * \brief Insert a text representation of a Result into an output stream + * \param[in] out The output stream + * \param[in] r The Result + * \return The output stream \a out + */ +std::ostream &operator<<(std::ostream &out, const Result &r) +{ + /* Example Output: Result(55:1/2) */ + out << "Result(" << r.request()->sequence() << ":" + << r.buffers().size() << "/" + << const_cast(r).metadata().size() << ")"; + + return out; +} + } /* namespace libcamera */