From patchwork Fri Aug 12 09:08:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 17087 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A8700C3272 for ; Fri, 12 Aug 2022 09:08:51 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5C3E463328; Fri, 12 Aug 2022 11:08:51 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660295331; bh=dgT2f5xmNTCegEs3xgGu28WwhPYhRhb8x2LyxCNaXZg=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=wOGs1j1pisSi9G6ildhOGegPCoAnp+dGUJ5qv9d3IUZPVJkyqDf2S4/noPa/FtvDT gHYUh19Sj9rjfn6DaYxxRCEsGWKigshwsYlwmE7XEYNVjRplRuesioIshhX+g6V8O5 SbC2El0DxinESckz42roqNCx5o7xclkKLlMZntfBpurvWx+f+4SwOtx62kFbEBhwag 1gzLZc2qnkkkDjopNYqIF9qkcfEcz8ooGyayShLc0HKW469DFQ7Mu7W4AS0+Oi0W5I bRwcnQrDIICyVxjufRsWwZ0p+mAwV6jHGiG4yEPU54Nlf8OhJMb6FS22pg6vfzlPOV Xm/W3qzKromAw== Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id BC04463326 for ; Fri, 12 Aug 2022 11:08:49 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="a+uPj3Ai"; dkim-atps=neutral Received: by mail-pf1-x42d.google.com with SMTP id f28so443676pfk.1 for ; Fri, 12 Aug 2022 02:08:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=S1Ep1tF4EF/a5ouUHjKr/XM2U9J6mN0Ei5b1dI1AzoM=; b=a+uPj3AiIxIOgcUGGQ1+HDsGMkcVKOjTJDTw22K7a568gsu+BMutTIm/yRw2rd1wW+ nSiVOLp6myXg8PZLROEwnv/jBUy0/N3av4VVFBqpGPnqJSdjWdL88K6BmLPAuvYSWSSH E1i6Q9hkMt3sE1qTY+oTKviAKa3QBfm+iS1BI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=S1Ep1tF4EF/a5ouUHjKr/XM2U9J6mN0Ei5b1dI1AzoM=; b=hUjkyYEC+463khwKJAp+NQXoHbDQ7kF8sZ8q1v+RvQCEc7F0CyA1gy/qZkDcAiMVqW C0hyJyqejto9r1htky+GqTlMVKX0iYp6NsR1t3kuaos1PhTudzjn8sK2S6Vysipjmh7x zVdcXIj15+dVU4PZYHSmdaZpQEuk4LpcH/Pk47XlgxChlNRwvG16/UgAz9iM4MdZSsrH VaQ8JDiKKPfaWIkT5V3tILXbsZ4BYMe2dzH6H76rXARm2cy2i0WzSNuyDoXKRy9Dyjh8 atyogAXsZi6jU+2shqAwn5dmQ977EyLC8T2aJsfHiSxZOnIkVJQcxrE7FNx8fgrRTsvm zp0w== X-Gm-Message-State: ACgBeo3pL4NL3DTEIm4Yg0vGzyfVAAl69uhu97DFHS5ynEU3SxUgdLD3 9pgvlngqZCHTodXazDbvpzT25xdncT7R8g== X-Google-Smtp-Source: AA6agR6vSqlU4fLgd+Uw7toluSwzV3Uvw5zlxg6I+qUKthflswoyYxtEXUPYYUBBr8F7+z+fRPecSw== X-Received: by 2002:a05:6a00:d74:b0:52f:575c:2d69 with SMTP id n52-20020a056a000d7400b0052f575c2d69mr2977935pfv.77.1660295328103; Fri, 12 Aug 2022 02:08:48 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:1705:d284:d114:2e24]) by smtp.gmail.com with UTF8SMTPSA id b72-20020a621b4b000000b0052d3d08cd96sm1100372pfb.67.2022.08.12.02.08.47 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 12 Aug 2022 02:08:47 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Fri, 12 Aug 2022 17:08:32 +0800 Message-Id: <20220812090838.1784703-2-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog In-Reply-To: <20220812090838.1784703-1-hanlinchen@chromium.org> References: <20220812090838.1784703-1-hanlinchen@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 1/7] libcamera: Camera: Add RequestCompletionMode to configure the completion order X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add enum RequestCompletionMode to Camera with two value: InSubmissionOrder and Immediately. The purpose is to allow the application configure the order of signaling requestCompleted. The InSubmissionOrder mode is the default mode which signals according to the request submission order. The Immediately mode allows the pipeline handler to signal as soon as a request is completed. Applications need to reconstruct the order by self. Signed-off-by: Han-Lin Chen --- include/libcamera/camera.h | 8 ++++ include/libcamera/internal/camera.h | 4 ++ src/libcamera/camera.cpp | 68 ++++++++++++++++++++++++++++- src/libcamera/pipeline_handler.cpp | 20 ++++++--- 4 files changed, 92 insertions(+), 8 deletions(-) diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h index 5bb06584..5aa4bf69 100644 --- a/include/libcamera/camera.h +++ b/include/libcamera/camera.h @@ -88,6 +88,11 @@ class Camera final : public Object, public std::enable_shared_from_this, LIBCAMERA_DECLARE_PRIVATE() public: + enum RequestCompletionMode { + InSubmissionOrder, + Immediately, + }; + static std::shared_ptr create(std::unique_ptr d, const std::string &id, const std::set &streams); @@ -101,6 +106,9 @@ public: int acquire(); int release(); + int setRequestCompletionMode(RequestCompletionMode order); + RequestCompletionMode requestCompletionMode() const; + const ControlInfoMap &controls() const; const ControlList &properties() const; diff --git a/include/libcamera/internal/camera.h b/include/libcamera/internal/camera.h index 597426a6..68e8d952 100644 --- a/include/libcamera/internal/camera.h +++ b/include/libcamera/internal/camera.h @@ -33,6 +33,9 @@ public: PipelineHandler *pipe() { return pipe_.get(); } + void setRequestCompletionMode(RequestCompletionMode mode); + RequestCompletionMode requestCompletionMode() const; + std::list queuedRequests_; ControlInfoMap controlInfo_; ControlList properties_; @@ -67,6 +70,7 @@ private: bool disconnected_; std::atomic state_; + RequestCompletionMode requestCompletionMode_; std::unique_ptr validator_; }; diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp index 2a8ef60e..4d3523e2 100644 --- a/src/libcamera/camera.cpp +++ b/src/libcamera/camera.cpp @@ -435,7 +435,8 @@ CameraConfiguration::Status CameraConfiguration::validateColorSpaces(ColorSpaceF */ Camera::Private::Private(PipelineHandler *pipe) : requestSequence_(0), pipe_(pipe->shared_from_this()), - disconnected_(false), state_(CameraAvailable) + disconnected_(false), state_(CameraAvailable), + requestCompletionMode_(Camera::InSubmissionOrder) { } @@ -573,6 +574,28 @@ void Camera::Private::setState(State state) state_.store(state, std::memory_order_release); } +/** + * \brief Set the request completion mode + * \param[in] mode The RequestCompletionMode + * + * This function set the request completion mode. + * InSubmissionOrder is the default mode. + */ +void Camera::Private::setRequestCompletionMode(RequestCompletionMode mode) +{ + requestCompletionMode_ = mode; +} + +/** + * \brief Get the request completion mode + * + * \return The current RequestCompletionMode. + */ +Camera::RequestCompletionMode Camera::Private::requestCompletionMode() const +{ + return requestCompletionMode_; +} + /** * \class Camera * \brief Camera device @@ -702,6 +725,15 @@ std::shared_ptr Camera::create(std::unique_ptr d, return std::shared_ptr(camera, Deleter()); } +/** + * \enum Camera::RequestCompletionMode + * \brief The mode of request completion behavior + * \var Camera::RequestCompletionMode::InSubmissionOrder + * \brief requestCompleted will be emited according to the request submission order + * \var Camera::RequestCompletionMode::Immediately + * \brief requestCompleted will be emited immediately when a request is completed. + */ + /** * \brief Retrieve the ID of the camera * @@ -1237,6 +1269,40 @@ int Camera::stop() return 0; } +/** + * \brief Set the request completion mode + * \param[in] mode The RequestCompletionMode + * + * This function set the request completion mode. + * InSubmissionOrder is the default mode. + * + * \return 0 on success or a negative error code otherwise + * \retval -EACCES The camera is running so can't change the behavior + */ +int Camera::setRequestCompletionMode(RequestCompletionMode mode) +{ + Private *const d = _d(); + + int ret = d->isAccessAllowed(Private::CameraAvailable, + Private::CameraConfigured); + if (ret < 0) + return -EACCES; + + d->setRequestCompletionMode(mode); + + return 0; +} + +/** + * \brief Get the request completion mode + * + * \return The current RequestCompletionMode. + */ +Camera::RequestCompletionMode Camera::requestCompletionMode() const +{ + return _d()->requestCompletionMode(); +} + /** * \brief Handle request completion and notify application * \param[in] request The request that has completed diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp index 67540533..7c180a8e 100644 --- a/src/libcamera/pipeline_handler.cpp +++ b/src/libcamera/pipeline_handler.cpp @@ -494,14 +494,20 @@ void PipelineHandler::completeRequest(Request *request) Camera::Private *data = camera->_d(); - while (!data->queuedRequests_.empty()) { - Request *req = data->queuedRequests_.front(); - if (req->status() == Request::RequestPending) - break; + auto iter = data->queuedRequests_.begin(); + while (iter != data->queuedRequests_.end()) { + if ((*iter)->status() != Request::RequestPending) { + ASSERT(!(*iter)->hasPendingBuffers()); + camera->requestComplete((*iter)); + iter = data->queuedRequests_.erase(iter); + continue; + } - ASSERT(!req->hasPendingBuffers()); - data->queuedRequests_.pop_front(); - camera->requestComplete(req); + /* Break at the pending result to reflect the submission order */ + if (camera->requestCompletionMode() == Camera::InSubmissionOrder) { + break; + } + iter++; } } From patchwork Fri Aug 12 09:08:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 17088 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0D0EBC3272 for ; Fri, 12 Aug 2022 09:08:53 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C469863331; Fri, 12 Aug 2022 11:08:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660295332; bh=VO/fxVcJWGHYLOdEQrn7s1ICjuFYL6BRUvHx/cqFiwg=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=rc7tUzTywQPgrRt0wAJ3U5LP0tP6udR1518NrQMemJF3yMm2dsji43JUjoXAk4Ynq ZNzaXJb5nPV3JTWb42p9o9YE/RiEViyQ6SooyCu7nQ9wcxOpMLlNI8mRP9flO+7X5Z tQClMydIzpq3XPhQjW3B0LVpsq2pUicLIMUi5VScpli27XakAoERxP+CrP/mWGDHB7 qdF/z/piY82EV0UIvCfe78m/ltHpxMTdmAuUlBEQVhdnRnReoJQOf9uLtwS+cnTFfq oQIFy82nweVtj4LS+rn9IHG1e0hMO55koIOJ117PV8mbhhwiF7jmIFM71oc4H1XD+v JpmizUV0HwYmw== Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D32E16332D for ; Fri, 12 Aug 2022 11:08:51 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="QbzabP55"; dkim-atps=neutral Received: by mail-pg1-x52c.google.com with SMTP id bh13so328666pgb.4 for ; Fri, 12 Aug 2022 02:08:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=R85XYqoVWEGvT6M8Agx+9ORKuOA7agTT19FBIQSY1xk=; b=QbzabP55WJiV6FkPPgUG7F7cMEBgxRX9eFeHUNN3Hi54veBHzdDrwyPPqWynhz9dvK MNPoq7YQWBDeRX25VmJAQNR/0mQPHd3KSoel3y4I+tx/7MaCHUEjsibHvHjuQgltAA4m 3K2jJBuesxZBdOSR7C9CQ6vbmj5q4IF1QIZZ0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=R85XYqoVWEGvT6M8Agx+9ORKuOA7agTT19FBIQSY1xk=; b=MGv8+W89dek0NfwS+P6qMwaHzaNoN5d9epF1mhkEHphnCgV0r3rGrRTzSAARZu4deW JwxyeocNQQCPU+AUzHlFdWShuY3o58M3ZdPUfw2OOA0CGvXvasiDnBC/c6ZPAzptxlCj 3EQ/ebR6uN0F6XG+nGFjR3gsSwq5sJArKJqyoRhI246ZYciuoCsEB2D26t2KT2OiUKMQ PXrJSSQGIRiRQd7QthYHp3RBTsXoud+eR8lif7pNtjKVMNmTld6Y3ioWa4QzDKnFxBjc BnnryAub0Yzg5AETNrc3frunzDJ8F+mhd2gFNs0g8I9YXGhbwaV0uBE4ODU5cq+45oDd 62eA== X-Gm-Message-State: ACgBeo0FUinRCvMi+ncLsInjQ5AVMJlx47IaV8PXNk4VCjshYHEuLadi TM/OMSu4MBAg6Jq+Ng7c3x2D+cvimE99rA== X-Google-Smtp-Source: AA6agR7i74sDED9cbQYg5vIMwnuQNf/QGGblGRtgjX5G85nWe5hGrBt6VQ1eDHwSGLch/OHfwDqE/A== X-Received: by 2002:a05:6a00:88f:b0:530:dec:81fd with SMTP id q15-20020a056a00088f00b005300dec81fdmr3081455pfj.64.1660295330122; Fri, 12 Aug 2022 02:08:50 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:1705:d284:d114:2e24]) by smtp.gmail.com with UTF8SMTPSA id ij1-20020a170902ab4100b0016f1ef2cd44sm1164281plb.154.2022.08.12.02.08.49 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 12 Aug 2022 02:08:49 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Fri, 12 Aug 2022 17:08:33 +0800 Message-Id: <20220812090838.1784703-3-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog In-Reply-To: <20220812090838.1784703-1-hanlinchen@chromium.org> References: <20220812090838.1784703-1-hanlinchen@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 2/7] libcamera: Camera: Add signals for completion of metadata and partial result X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Allows pipelien handler to signal partial result completion by adding the following signals to Camera: Signal metadataCompleted; Signal partialResultCompleted; Together with the bufferCompleted signal, the pipeline handler is allowed to return buffers, partial metadata and combination of both at any stage of processing. Signed-off-by: Han-Lin Chen --- include/libcamera/camera.h | 2 + include/libcamera/internal/pipeline_handler.h | 4 + include/libcamera/request.h | 40 +++++ src/libcamera/camera.cpp | 10 ++ src/libcamera/pipeline_handler.cpp | 121 ++++++++++++- src/libcamera/request.cpp | 170 +++++++++++++++++- 6 files changed, 332 insertions(+), 15 deletions(-) diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h index 5aa4bf69..187e01d8 100644 --- a/include/libcamera/camera.h +++ b/include/libcamera/camera.h @@ -99,7 +99,9 @@ public: const std::string &id() const; + Signal partialResultCompleted; Signal bufferCompleted; + Signal metadataCompleted; Signal requestCompleted; Signal<> disconnected; diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h index c3e4c258..08519fb3 100644 --- a/include/libcamera/internal/pipeline_handler.h +++ b/include/libcamera/internal/pipeline_handler.h @@ -33,6 +33,7 @@ class FrameBuffer; class MediaDevice; class PipelineHandler; class Request; +class Result; class PipelineHandler : public std::enable_shared_from_this, public Object @@ -62,7 +63,9 @@ public: void registerRequest(Request *request); void queueRequest(Request *request); + void completeMetadata(Request *request, const ControlList &metadata); bool completeBuffer(Request *request, FrameBuffer *buffer); + void completePartialResult(Request *request, Result &&result); void completeRequest(Request *request); const char *name() const { return name_; } @@ -80,6 +83,7 @@ private: void mediaDeviceDisconnected(MediaDevice *media); virtual void disconnect(); + void cancelRequest(Request *request); void doQueueRequest(Request *request); void doQueueRequests(); diff --git a/include/libcamera/request.h b/include/libcamera/request.h index dffde153..ac520226 100644 --- a/include/libcamera/request.h +++ b/include/libcamera/request.h @@ -25,8 +25,43 @@ namespace libcamera { class Camera; class CameraControlValidator; class FrameBuffer; +class Request; class Stream; +class Result +{ +public: + Result(Request *request); + Result(Result &&result); + ~Result(); + + Request *request() const { return request_; } + const ControlList &metadata() const { return metadata_; } + + const std::vector &buffers() const { return buffers_; } + int addBuffer(FrameBuffer *buffer); + + template + void set(const Control &ctrl, const V &value) + { + return metadata_.set(ctrl, value); + } + + void set(unsigned int id, const ControlValue &value); + void merge(const ControlList &source); + + std::string toString() const; + +private: + LIBCAMERA_DISABLE_COPY(Result) + + Request *request_; + ControlList metadata_; + std::vector buffers_; +}; + +std::ostream &operator<<(std::ostream &out, const Result &r); + class Request : public Extensible { LIBCAMERA_DECLARE_PRIVATE() @@ -44,6 +79,7 @@ public: }; using BufferMap = std::map; + using ResultList = std::vector; Request(Camera *camera, uint64_t cookie = 0); ~Request(); @@ -53,9 +89,12 @@ public: ControlList &controls() { return *controls_; } ControlList &metadata() { return *metadata_; } const BufferMap &buffers() const { return bufferMap_; } + ResultList &resultList() { return results_; } + Result *addResult(Result &&result); int addBuffer(const Stream *stream, FrameBuffer *buffer, std::unique_ptr fence = nullptr); FrameBuffer *findBuffer(const Stream *stream) const; + const Stream *findStream(const FrameBuffer *buffer) const; uint32_t sequence() const; uint64_t cookie() const { return cookie_; } @@ -71,6 +110,7 @@ private: ControlList *controls_; ControlList *metadata_; BufferMap bufferMap_; + ResultList results_; const uint64_t cookie_; Status status_; diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp index 4d3523e2..a5fb9b33 100644 --- a/src/libcamera/camera.cpp +++ b/src/libcamera/camera.cpp @@ -765,6 +765,16 @@ const std::string &Camera::id() const * completed */ +/** + * \var Camera::metadataCompleted + * \brief Signal emitted when a partial metadata for a request is completed + */ + +/** + * \var Camera::partialResultCompleted + * \brief Signal emitted when a partial result for a request is completed + */ + /** * \var Camera::requestCompleted * \brief Signal emitted when a request queued to the camera has completed diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp index 7c180a8e..6608c210 100644 --- a/src/libcamera/pipeline_handler.cpp +++ b/src/libcamera/pipeline_handler.cpp @@ -305,8 +305,9 @@ void PipelineHandler::stop(Camera *camera) Request *request = waitingRequests_.front(); waitingRequests_.pop(); - request->_d()->cancel(); - completeRequest(request); + cancelRequest(request); + request->_d()->complete(); + camera->requestCompleted.emit(request); } /* Make sure no requests are pending. */ @@ -402,17 +403,30 @@ void PipelineHandler::doQueueRequest(Request *request) request->_d()->sequence_ = data->requestSequence_++; if (request->_d()->cancelled_) { + cancelRequest(request); completeRequest(request); return; } int ret = queueRequestDevice(camera, request); if (ret) { - request->_d()->cancel(); + cancelRequest(request); completeRequest(request); } } +/** + * \brief Cancel buffers of a request and complete the all buffers + */ +void PipelineHandler::cancelRequest(Request *request) +{ + request->_d()->cancel(); + for (auto it : request->buffers()) { + FrameBuffer *buffer = it.second; + completeBuffer(request, buffer); + } +} + /** * \brief Queue prepared requests to the device * @@ -453,9 +467,10 @@ void PipelineHandler::doQueueRequests() * \param[in] request The request the buffer belongs to * \param[in] buffer The buffer that has completed * - * This function shall be called by pipeline handlers to signal completion of + * This function could be called by pipeline handlers to signal completion of * the \a buffer part of the \a request. It notifies applications of buffer - * completion and updates the request's internal buffer tracking. The request + * completion and updates the request's internal buffer tracking. The function + * notify completion of a partial result including the buffer. The request * is not completed automatically when the last buffer completes to give * pipeline handlers a chance to perform any operation that may still be * needed. They shall complete requests explicitly with completeRequest(). @@ -466,10 +481,75 @@ void PipelineHandler::doQueueRequests() * otherwise */ bool PipelineHandler::completeBuffer(Request *request, FrameBuffer *buffer) +{ + Result result(request); + result.addBuffer(buffer); + completePartialResult(request, std::move(result)); + + return request->_d()->hasPendingBuffers(); +} + +/** + * \brief Complete part of metadata for a request + * \param[in] request The request the buffer belongs to + * \param[in] metadata The partial metadata that has completed + * + * This function could be called by pipeline handlers to signal completion of + * the \a metadata part of the \a request. It notifies applications of metadata + * completion. The function notify completion of a partial result including the + * metadata. The request is not completed automatically when the last metadata + * completes to give pipeline handlers a chance to perform any operation that + * may still be needed. They shall complete requests explicitly with + * completeRequest(). + * + * \context This function shall be called from the CameraManager thread. + */ +void PipelineHandler::completeMetadata(Request *request, const ControlList &metadata) +{ + Result result = Result(request); + result.merge(metadata); + completePartialResult(request, std::move(result)); +} + +/** + * \brief Complete part of metadata and buffer for a request + * \param[in] request The request the buffer belongs to + * \param[in] result The partial result that has completed + * + * This function could be called by pipeline handlers to signal completion of + * the \a result part of the \a request. It notifies applications of partial + * completion. The function notify completion of buffers and metadata included + * in the result. The request is not completed automatically when the last + * result completes to give pipeline handlers a chance to perform any operation + * that may still be needed. They shall complete requests explicitly with + * completeRequest(). The function only accepts rvalue of a Result type and + * its interval content will be moved to the internal store to avoid copying + * big metadata. + * + * \context This function shall be called from the CameraManager thread. + */ +void PipelineHandler::completePartialResult(Request *request, Result &&result) { Camera *camera = request->_d()->camera(); - camera->bufferCompleted.emit(request, buffer); - return request->_d()->completeBuffer(buffer); + Result *movedResult = request->addResult(std::move(result)); + + ASSERT(movedResult->request() == request); + ASSERT(!movedResult->buffers().empty() || + !movedResult->metadata().empty()); + + for (auto buffer : movedResult->buffers()) { + request->_d()->completeBuffer(buffer); + camera->bufferCompleted.emit(request, buffer); + } + + if (!movedResult->metadata().empty()) { + request->metadata().merge(movedResult->metadata()); + camera->metadataCompleted.emit( + request, + &const_cast(movedResult->metadata())); + } + + camera->partialResultCompleted.emit(request, movedResult); } /** @@ -494,6 +574,33 @@ void PipelineHandler::completeRequest(Request *request) Camera::Private *data = camera->_d(); + /* + * Collect metadata which is not yet completed by the Camera, and + * create one partial result to cover the missing metadata before + * completing the whole request. This guarantees the aggregation of + * metadata in completed partial results equals to the global metadata + * in the request. + * + * \todo: Forbid merging metadata into request.metadata() directly and + * force calling completeMetadata() and completePartialResult() to + * report metadata. + */ + std::unordered_set completedMetadata; + for (auto &result : request->resultList()) { + for (auto &[id, _] : result.metadata()) + completedMetadata.insert(id); + } + + ControlList &requestMetadata = request->metadata(); + if (requestMetadata.size() > completedMetadata.size()) { + Result result(request); + for (auto &[id, value] : requestMetadata) + if (!completedMetadata.count(id)) + result.set(id, value); + + completePartialResult(request, std::move(result)); + } + auto iter = data->queuedRequests_.begin(); while (iter != data->queuedRequests_.end()) { if ((*iter)->status() != Request::RequestPending) { diff --git a/src/libcamera/request.cpp b/src/libcamera/request.cpp index d2af1d22..239c29cf 100644 --- a/src/libcamera/request.cpp +++ b/src/libcamera/request.cpp @@ -126,15 +126,10 @@ void Request::Private::complete() void Request::Private::doCancelRequest() { - Request *request = _o(); - - for (FrameBuffer *buffer : pending_) { + for (FrameBuffer *buffer : pending_) buffer->_d()->cancel(); - camera_->bufferCompleted.emit(request, buffer); - } cancelled_ = true; - pending_.clear(); notifiers_.clear(); timer_.reset(); } @@ -144,8 +139,8 @@ void Request::Private::doCancelRequest() * * Mark the request and its associated buffers as cancelled and complete it. * - * Set each pending buffer in error state and emit the buffer completion signal - * before completing the Request. + * Set each pending buffer in error state. The pipeline handler shall complete + * the cancelled buffers to notice the application. */ void Request::Private::cancel() { @@ -323,6 +318,11 @@ void Request::Private::timeout() * \brief A map of Stream to FrameBuffer pointers */ +/** + * \typedef Request::ResultList + * \brief A list partial results associated with a request + */ + /** * \class Request * \brief A frame capture request @@ -394,6 +394,7 @@ void Request::reuse(ReuseFlag flags) status_ = RequestPending; + results_.clear(); controls_->clear(); metadata_->clear(); } @@ -424,6 +425,12 @@ void Request::reuse(ReuseFlag flags) * \return The map of Stream to FrameBuffer */ +/** + * \fn Request::resultList() + * \brief Retrieve the request's partial results + * \return A reference to the list of results that associates with the request + */ + /** * \brief Add a FrameBuffer with its associated Stream to the Request * \param[in] stream The stream the buffer belongs to @@ -489,6 +496,20 @@ int Request::addBuffer(const Stream *stream, FrameBuffer *buffer, return 0; } +/** + * \brief Add result into the internal result list + * \param[in] result The result to add into the request + * + * The function only accepts rvalue and moves its content into the request + * + * \return The result moved into the request + */ +Result *Request::addResult(Result &&result) +{ + results_.emplace_back(std::move(result)); + return &results_.back(); +} + /** * \var Request::bufferMap_ * \brief Mapping of streams to buffers for this request @@ -513,6 +534,21 @@ FrameBuffer *Request::findBuffer(const Stream *stream) const return it->second; } +/** + * \brief Return the stream associated with a buffer + * \param[in] buffer The buffer the stream is associated to + * \return The stream associated with the buffer, or nullptr if the buffer is + * not part of this request + */ +const Stream *Request::findStream(const FrameBuffer *buffer) const +{ + for (auto &[key, value] : bufferMap_) + if (buffer == value) + return key; + + return nullptr; +} + /** * \fn Request::metadata() * \brief Retrieve the request's metadata @@ -606,4 +642,122 @@ std::ostream &operator<<(std::ostream &out, const Request &r) return out; } +/** + * \class Result + * \brief A partial result of a frame capture request + * + * A Result allows pipeline handler to report partial results to the application + */ + +/** + * \brief Create a partial result for a capture request + * \param[in] request The request the result associated to + */ +Result::Result(Request *request) + : request_(request) +{ +} + +/** + * \brief Move constructor of a Result + * \param[in] result The other result + */ +Result::Result(Result &&result) = default; + +Result::~Result() = default; + +/** + * \fn Result::request() + * \brief Retrieve the result's associated request + * \return The Request pointer associated with the result + */ + +/** + * \fn Result::metadata() + * \brief Retrieve the result's metadata + * \return The metadata contained in the result + */ + +/** + * \fn Result::buffers() + * \brief Retrieve the result's buffers + * \return The buffers contained in the result + */ + +/** + * \fn Result::set(const Control &ctrl, const V &value) + * \brief Set the control \a ctrl value to \a value into metadata + * \param[in] ctrl The control + * \param[in] value The control value + */ + +/** + * \brief Add a FrameBuffer with its associated Stream to the Result + * \param[in] buffer The FrameBuffer to add to the result + * + * \return 0 on success or a negative error code otherwise + * \retval -EINVAL The buffer does not reference a valid Stream in the request + */ +int Result::addBuffer(FrameBuffer *buffer) +{ + if (!buffer || !request_->findStream(buffer)) { + LOG(Request, Error) << "Invalid buffer reference"; + return -EINVAL; + } + + buffers_.emplace_back(buffer); + return 0; +} + +/** + * \brief Set the control \a id to \a value + * \param[in] id The control id + * \param[in] value The control value + */ +void Result::set(unsigned int id, const ControlValue &value) +{ + metadata_.set(id, value); +} + +/** + * \brief Merge the \a source into the metadata of the result + * \param[in] source The ControlList to merge into this metadata + */ +void Result::merge(const ControlList &source) +{ + metadata_.merge(source); +} + +/** + * \brief Generate a string representation of the Result internals + * + * This function facilitates debugging of Result state while it is used + * internally within libcamera. + * + * \return A string representing the current state of the result + */ +std::string Result::toString() const +{ + std::stringstream ss; + ss << *this; + + return ss.str(); +} + +/** + * \brief Insert a text representation of a Result into an output stream + * \param[in] out The output stream + * \param[in] r The Result + * \return The output stream \a out + */ +std::ostream &operator<<(std::ostream &out, const Result &r) +{ + /* Example Output: Result(55:1/2) */ + out << "Result(" << r.request()->sequence() << ":" + << r.buffers().size() << "/" + << const_cast(r).metadata().size() << ")"; + + return out; +} + } /* namespace libcamera */ From patchwork Fri Aug 12 09:08:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 17089 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 68A34C3272 for ; Fri, 12 Aug 2022 09:08:55 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 352EC63334; Fri, 12 Aug 2022 11:08:55 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660295335; bh=ngwFPQ51YQid1jL1yJ0V8rBtQg4g+bDYOVeFeyd7ah8=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=cxi1G8eJiiqvEFldr8O92Xik0mYJkr7U+ca8x3vXwwSRSU97vy5gAGeuKYsTRh/Su SmyxzgrkzHjmUlprxfItOnOsEnfLASRM8SfqDNjw8UFQWakwVtNgWTeqXQx3mLZMFL kKz6gpA8IhLIkDiUZUmzUXn9K6NYc5oUSOIW276TBjANK3vbLeM7UDkC2G6ehgWBLR 5z2zJXibLilTdl/ZW4rY1bQIEzrRc1yoF70fZTTAijq8rH7Tjox6foXnkTFC8RNFK0 CvMUw0cKA8bmRifDhK+s8BFuGw2Gw0cYl5gqJvzqv0JN+kxCrR3ErNfHu9pMzXnhhp 1I2pePg1xJUxA== Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E615563326 for ; Fri, 12 Aug 2022 11:08:53 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="CLimVt+7"; dkim-atps=neutral Received: by mail-pf1-x435.google.com with SMTP id h28so412939pfq.11 for ; Fri, 12 Aug 2022 02:08:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=EVORGy2cSfxQ44qSix18weyzDNUIZ9Zn8Yw/5wddF80=; b=CLimVt+7oQwN24pSh8jx19uI1BV18GhZBRBI+JKXR2wwyPutHIcEeRQtDaP6J66wQJ E6TVRHkdqGAujdgunpy0gStJM5la55gPexM8m3rLHgDf6LzhaxX7PSJLqITnIQii/SuR qtSiLu+RkvLgoIsP1WOylvA19nOP6evzOy4K8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=EVORGy2cSfxQ44qSix18weyzDNUIZ9Zn8Yw/5wddF80=; b=w6j5Ph24T4aP3u9nHjKfFCQPPOLkPIMeJiG4r5vd81SdslWIlFPkCuZgatbhlWgxdX Ty5wudjGK/aEilMhcS9j3ES/koldbdHFBJQVOW4KVY/3GhxH1pUBQYj7roB65PtM5892 1LdwuCNKCfG83OWJZdy6Yjyc0Te92Y2Tue4dhExwXrvWsNfFGqhjroByMCLOq4ncO82k BSy9y8vQgzIetqy2eQReZMn/5vH/zUdqUaCvrx76G9q5cGMpqdkV0suvQm7XxiIX5+Om F5pzL9nqFr2tNENzohNZiXBPqEq7wcabV1wcD5Fzmqb1WllxWVitTY1v5oNfs0BHRfu4 SL5g== X-Gm-Message-State: ACgBeo3tYbdOFmR3AL1APF1Kj9eQxQQqUUoUnpt+JCHNu11Q5kQscGuj 1omd5m/vZIUSpHb4J0DkBpTh3lOd+5U6PQ== X-Google-Smtp-Source: AA6agR4PPhnEvuLriaEutRZj27Ok7j4TDsYyDACLHSZG3M4x+fHwE5/d26VqGj/OR6vDBYh8MaxVEA== X-Received: by 2002:aa7:9637:0:b0:52d:b082:cd28 with SMTP id r23-20020aa79637000000b0052db082cd28mr3011647pfg.43.1660295332179; Fri, 12 Aug 2022 02:08:52 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:1705:d284:d114:2e24]) by smtp.gmail.com with UTF8SMTPSA id r38-20020a632066000000b00419acadde52sm962612pgm.46.2022.08.12.02.08.51 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 12 Aug 2022 02:08:51 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Fri, 12 Aug 2022 17:08:34 +0800 Message-Id: <20220812090838.1784703-4-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog In-Reply-To: <20220812090838.1784703-1-hanlinchen@chromium.org> References: <20220812090838.1784703-1-hanlinchen@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 3/7] libcamera: ipu3: returning partial results X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Return metadata at earlier stage. Signed-off-by: Han-Lin Chen --- src/ipa/ipu3/ipu3.cpp | 2 - src/libcamera/pipeline/ipu3/ipu3.cpp | 65 ++++++++++++++++++---------- 2 files changed, 42 insertions(+), 25 deletions(-) diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp index 2f6bb672..5e921875 100644 --- a/src/ipa/ipu3/ipu3.cpp +++ b/src/ipa/ipu3/ipu3.cpp @@ -594,8 +594,6 @@ void IPAIPU3::processStatsBuffer(const uint32_t frame, ctrls.set(controls::ColourTemperature, context_.activeState.awb.temperatureK); - ctrls.set(controls::ExposureTime, frameContext.sensor.exposure * lineDuration); - /* * \todo The Metadata provides a path to getting extended data * out to the application. Further data such as a simplifed Histogram diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index 4fe52f74..f65db3c8 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -41,6 +41,8 @@ namespace libcamera { LOG_DEFINE_CATEGORY(IPU3) +using namespace std::literals::chrono_literals; + static const ControlInfoMap::Map IPU3Controls = { { &controls::draft::PipelineDepth, ControlInfo(2, 3) }, }; @@ -86,6 +88,8 @@ public: ControlInfoMap ipaControls_; + IPACameraSensorInfo sensorInfo_; + private: void metadataReady(unsigned int id, const ControlList &metadata); void paramsBufferReady(unsigned int id); @@ -993,7 +997,7 @@ int PipelineHandlerIPU3::initControls(IPU3CameraData *data) int PipelineHandlerIPU3::updateControls(IPU3CameraData *data) { CameraSensor *sensor = data->cio2_.sensor(); - IPACameraSensorInfo sensorInfo{}; + IPACameraSensorInfo &sensorInfo = data->sensorInfo_; int ret = sensor->sensorInfo(&sensorInfo); if (ret) @@ -1300,7 +1304,7 @@ void IPU3CameraData::metadataReady(unsigned int id, const ControlList &metadata) return; Request *request = info->request; - request->metadata().merge(metadata); + pipe()->completeMetadata(request, metadata); info->metadataProcessed = true; if (frameInfos_.tryComplete(info)) @@ -1324,16 +1328,8 @@ void IPU3CameraData::imguOutputBufferReady(FrameBuffer *buffer) return; Request *request = info->request; - pipe()->completeBuffer(request, buffer); - request->metadata().set(controls::draft::PipelineDepth, 3); - /* \todo Actually apply the scaler crop region to the ImgU. */ - const auto &scalerCrop = request->controls().get(controls::ScalerCrop); - if (scalerCrop) - cropRegion_ = *scalerCrop; - request->metadata().set(controls::ScalerCrop, cropRegion_); - if (frameInfos_.tryComplete(info)) pipe()->completeRequest(request); } @@ -1372,13 +1368,35 @@ void IPU3CameraData::cio2BufferReady(FrameBuffer *buffer) * \todo The sensor timestamp should be better estimated by connecting * to the V4L2Device::frameStart signal. */ - request->metadata().set(controls::SensorTimestamp, - buffer->metadata().timestamp); - info->effectiveSensorControls = delayedCtrls_->get(buffer->metadata().sequence); + ControlList &sensorCtrl = info->effectiveSensorControls; + sensorCtrl = delayedCtrls_->get(buffer->metadata().sequence); - if (request->findBuffer(&rawStream_)) + ControlList metadata; + metadata.set(controls::draft::PipelineDepth, 3); + metadata.set(controls::SensorTimestamp, buffer->metadata().timestamp); + + /* \todo Actually apply the scaler crop region to the ImgU. */ + const auto &scalerCrop = request->controls().get(controls::ScalerCrop); + if (scalerCrop) + cropRegion_ = *scalerCrop; + metadata.set(controls::ScalerCrop, cropRegion_); + + /* + * \todo Move reporting the raw frame related metadata from the IPA + * to here: AnalogueGain, FrameDuration, etc. + */ + utils::Duration lineDuration = sensorInfo_.lineLength * 1.0s / sensorInfo_.pixelRate; + if (sensorCtrl.contains(V4L2_CID_EXPOSURE)) { + int32_t exposureTime = sensorCtrl.get(V4L2_CID_EXPOSURE).get(); + metadata.set(controls::ExposureTime, exposureTime * lineDuration.get()); + } + + pipe()->completeMetadata(request, metadata); + + if (request->findBuffer(&rawStream_)) { pipe()->completeBuffer(request, buffer); + } ipa_->fillParamsBuffer(info->id, info->paramBuffer->cookie()); } @@ -1455,20 +1473,21 @@ void IPU3CameraData::frameStart(uint32_t sequence) Request *request = processingRequests_.front(); processingRequests_.pop(); - const auto &testPatternMode = request->controls().get(controls::draft::TestPatternMode); - if (!testPatternMode) - return; + int32_t testPatternMode = controls::draft::TestPatternModeOff; + const auto &testPatternControl = request->controls().get(controls::draft::TestPatternMode); + if (testPatternControl) + testPatternMode = *testPatternControl; int ret = cio2_.sensor()->setTestPatternMode( - static_cast(*testPatternMode)); + static_cast(testPatternMode)); if (ret) { - LOG(IPU3, Error) << "Failed to set test pattern mode: " - << ret; - return; + LOG(IPU3, Error) << "Failed to set test pattern mode: " << ret; + testPatternMode = controls::draft::TestPatternModeOff; } - request->metadata().set(controls::draft::TestPatternMode, - *testPatternMode); + ControlList metadata; + metadata.set(controls::draft::TestPatternMode, testPatternMode); + pipe()->completeMetadata(request, metadata); } REGISTER_PIPELINE_HANDLER(PipelineHandlerIPU3) From patchwork Fri Aug 12 09:08:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 17090 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E920BC3272 for ; Fri, 12 Aug 2022 09:08:57 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A509D6332D; Fri, 12 Aug 2022 11:08:57 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660295337; bh=EJj/4KHxZ6GW6aEcXrZhsDR4/la2ePmUWSq9KoN4i8g=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=muvI8lgIhxQOFM/wm+yHnXHPK/zSCXuivDMPH4UGxxVtm85a2+dg8myOX/zWETSWl 3XpUmQ0QkwWtdiAbKugTQckvwzk5WhdgSiaTeQKPKrsm5RpAslz6hhY+P1/7sqqoWV y54Fcc6biL6kkYM+H0JrLgVaEWFqm1eIjLI6iDTQXF4plcG+OCe6BL7eEU7gs5c57q pVrNkhqlc2xaBPdDJ1Y0InBxevhXVpNTFTxlzUp5B38u/cWlrFVjcDoFVePiFLvdtK V4atGiUv7JTV6cmT8y0BFOwnpzosN55XtscST44BxAsUfOQya3z+6rh6+GiUFjFHfw wjKY3G0kdV7Hg== Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id EE95A63326 for ; Fri, 12 Aug 2022 11:08:55 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="DJCKwQI3"; dkim-atps=neutral Received: by mail-pg1-x52c.google.com with SMTP id l64so355479pge.0 for ; Fri, 12 Aug 2022 02:08:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=CWynA5+65XjCH6EeaSoIqpclotts5uUrEOCyvbnaAk0=; b=DJCKwQI3ymSxJJvwPVi1p42NcJQlG1IQzsiE5evRueO0OQuJ9YdBkxkPQQWRETS3Tu r65zx/PJzSwkwMWk3mFzMitXUc4vRJkm6Pfy4VIpbYUfKxjHPQ3dFFYFTi18WzCapsNE fCYmkFKo7gLfbNwEeFt7AVDqYyMGX/SbWdfeA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=CWynA5+65XjCH6EeaSoIqpclotts5uUrEOCyvbnaAk0=; b=JvvChxPHbOmEwvOVjtEgtrFRblEff7vwq+bVwdZvBOW+/tNFuyvx8UWp0f7OLUcWIT BKToXc4DOu7e/dmIUE4uAg/NVIZPDKxH+dKNGncQ9/22MguH9Z5/IadvGdrL7jUnL+Ey 7modZjLnDtCwUOC1tlPxdbXKkwofGoviJNQLNNxgCSBvaam50Fgau9uv+YcIWucCLeos p+4wozDhy5etclvnTwQmGGd9ueXvRlZ48dq7kBM9xSmVhq2po66O17fryAynA2XhBMiA bm8Hm7hmI9KWajyH1KQJkXcMctLjqpEff6ImDJ6nSHX6HP5/sfeX0m9qJsbksTlY22II bNcA== X-Gm-Message-State: ACgBeo1or3RqbXRt7FevqCSuHw2+iBzqcmalPiKh0HCRAib5lTukobBG BLnMLLcMmBkPndQ6rhHdLXFTIsLHtQYo8A== X-Google-Smtp-Source: AA6agR7NqssVxLc0SzXAYAeOmGZEbm+KgCpZTYfF6jjVA1LemV24xMVoa/LiUXYlZt+IhXW1IaaAhw== X-Received: by 2002:a65:6c0c:0:b0:41b:8696:d9d7 with SMTP id y12-20020a656c0c000000b0041b8696d9d7mr2392593pgu.16.1660295334261; Fri, 12 Aug 2022 02:08:54 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:1705:d284:d114:2e24]) by smtp.gmail.com with UTF8SMTPSA id i3-20020aa796e3000000b0052e26b8f610sm1086534pfq.89.2022.08.12.02.08.53 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 12 Aug 2022 02:08:54 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Fri, 12 Aug 2022 17:08:35 +0800 Message-Id: <20220812090838.1784703-5-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog In-Reply-To: <20220812090838.1784703-1-hanlinchen@chromium.org> References: <20220812090838.1784703-1-hanlinchen@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 4/7] android: Move StreamBuffer out of Camera3RequestDescriptor X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Move StreamBuffer out of Camera3RequestDescriptor as a refactory before applying partial result to Android adaptor. For easier adding Camera3ResultDescriptor which references both StreamBuffer and Camera3RequestDescriptor. Signed-off-by: Han-Lin Chen --- src/android/camera_device.cpp | 22 +++++----- src/android/camera_device.h | 7 ++-- src/android/camera_request.cpp | 29 +++++++------ src/android/camera_request.h | 52 ++++++++++++++---------- src/android/camera_stream.cpp | 17 ++++---- src/android/camera_stream.h | 6 +-- src/android/jpeg/post_processor_jpeg.cpp | 2 +- src/android/jpeg/post_processor_jpeg.h | 2 +- src/android/post_processor.h | 4 +- src/android/yuv/post_processor_yuv.cpp | 2 +- src/android/yuv/post_processor_yuv.h | 2 +- 11 files changed, 75 insertions(+), 70 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index b20e389b..0248f146 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -860,7 +860,7 @@ void CameraDevice::abortRequest(Camera3RequestDescriptor *descriptor) const notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_REQUEST); for (auto &buffer : descriptor->buffers_) - buffer.status = Camera3RequestDescriptor::Status::Error; + buffer.status = StreamBuffer::Status::Error; descriptor->status_ = Camera3RequestDescriptor::Status::Error; } @@ -1156,7 +1156,7 @@ void CameraDevice::requestComplete(Request *request) if (fence) buffer.fence = fence->release(); } - buffer.status = Camera3RequestDescriptor::Status::Success; + buffer.status = StreamBuffer::Status::Success; } /* @@ -1219,12 +1219,12 @@ void CameraDevice::requestComplete(Request *request) auto iter = descriptor->pendingStreamsToProcess_.begin(); while (iter != descriptor->pendingStreamsToProcess_.end()) { CameraStream *stream = iter->first; - Camera3RequestDescriptor::StreamBuffer *buffer = iter->second; + StreamBuffer *buffer = iter->second; FrameBuffer *src = request->findBuffer(stream->stream()); if (!src) { LOG(HAL, Error) << "Failed to find a source stream buffer"; - setBufferStatus(*buffer, Camera3RequestDescriptor::Status::Error); + setBufferStatus(*buffer, StreamBuffer::Status::Error); iter = descriptor->pendingStreamsToProcess_.erase(iter); continue; } @@ -1234,7 +1234,7 @@ void CameraDevice::requestComplete(Request *request) ++iter; int ret = stream->process(buffer); if (ret) { - setBufferStatus(*buffer, Camera3RequestDescriptor::Status::Error); + setBufferStatus(*buffer, StreamBuffer::Status::Error); descriptor->pendingStreamsToProcess_.erase(stream); /* @@ -1304,7 +1304,7 @@ void CameraDevice::sendCaptureResults() for (auto &buffer : descriptor->buffers_) { camera3_buffer_status status = CAMERA3_BUFFER_STATUS_ERROR; - if (buffer.status == Camera3RequestDescriptor::Status::Success) + if (buffer.status == StreamBuffer::Status::Success) status = CAMERA3_BUFFER_STATUS_OK; /* @@ -1328,11 +1328,11 @@ void CameraDevice::sendCaptureResults() } } -void CameraDevice::setBufferStatus(Camera3RequestDescriptor::StreamBuffer &streamBuffer, - Camera3RequestDescriptor::Status status) +void CameraDevice::setBufferStatus(StreamBuffer &streamBuffer, + StreamBuffer::Status status) { streamBuffer.status = status; - if (status != Camera3RequestDescriptor::Status::Success) { + if (status != StreamBuffer::Status::Success) { notifyError(streamBuffer.request->frameNumber_, streamBuffer.stream->camera3Stream(), CAMERA3_MSG_ERROR_BUFFER); @@ -1356,8 +1356,8 @@ void CameraDevice::setBufferStatus(Camera3RequestDescriptor::StreamBuffer &strea * be generated from post-processing have been completed. Mark the descriptor as * complete using completeDescriptor() in that case. */ -void CameraDevice::streamProcessingComplete(Camera3RequestDescriptor::StreamBuffer *streamBuffer, - Camera3RequestDescriptor::Status status) +void CameraDevice::streamProcessingComplete(StreamBuffer *streamBuffer, + StreamBuffer::Status status) { setBufferStatus(*streamBuffer, status); diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 64050416..9eb7221b 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -64,8 +64,8 @@ public: int configureStreams(camera3_stream_configuration_t *stream_list); int processCaptureRequest(camera3_capture_request_t *request); void requestComplete(libcamera::Request *request); - void streamProcessingComplete(Camera3RequestDescriptor::StreamBuffer *bufferStream, - Camera3RequestDescriptor::Status status); + void streamProcessingComplete(StreamBuffer *bufferStream, + StreamBuffer::Status status); protected: std::string logPrefix() const override; @@ -96,8 +96,7 @@ private: void completeDescriptor(Camera3RequestDescriptor *descriptor) LIBCAMERA_TSA_EXCLUDES(descriptorsMutex_); void sendCaptureResults() LIBCAMERA_TSA_REQUIRES(descriptorsMutex_); - void setBufferStatus(Camera3RequestDescriptor::StreamBuffer &buffer, - Camera3RequestDescriptor::Status status); + void setBufferStatus(StreamBuffer &buffer, StreamBuffer::Status status); std::unique_ptr getResultMetadata( const Camera3RequestDescriptor &descriptor) const; diff --git a/src/android/camera_request.cpp b/src/android/camera_request.cpp index 6c87adba..f5d4d314 100644 --- a/src/android/camera_request.cpp +++ b/src/android/camera_request.cpp @@ -141,7 +141,7 @@ Camera3RequestDescriptor::Camera3RequestDescriptor( Camera3RequestDescriptor::~Camera3RequestDescriptor() = default; /** - * \struct Camera3RequestDescriptor::StreamBuffer + * \class StreamBuffer * \brief Group information for per-stream buffer of Camera3RequestDescriptor * * A capture request placed to the libcamera HAL can contain multiple streams. @@ -150,35 +150,35 @@ Camera3RequestDescriptor::~Camera3RequestDescriptor() = default; * generation. The generation of the stream will depend on its type (refer to * the CameraStream::Type documentation). * - * \var Camera3RequestDescriptor::StreamBuffer::stream + * \var StreamBuffer::stream * \brief Pointer to the corresponding CameraStream * - * \var Camera3RequestDescriptor::StreamBuffer::camera3Buffer + * \var StreamBuffer::camera3Buffer * \brief Native handle to the buffer * - * \var Camera3RequestDescriptor::StreamBuffer::frameBuffer + * \var StreamBuffer::frameBuffer * \brief Encapsulate the dmabuf handle inside a libcamera::FrameBuffer for * direct streams * - * \var Camera3RequestDescriptor::StreamBuffer::fence + * \var StreamBuffer::fence * \brief Acquire fence of the buffer * - * \var Camera3RequestDescriptor::StreamBuffer::status + * \var StreamBuffer::status * \brief Track the status of the buffer * - * \var Camera3RequestDescriptor::StreamBuffer::internalBuffer + * \var StreamBuffer::internalBuffer * \brief Pointer to a buffer internally handled by CameraStream (if any) * - * \var Camera3RequestDescriptor::StreamBuffer::srcBuffer + * \var StreamBuffer::srcBuffer * \brief Pointer to the source frame buffer used for post-processing * - * \var Camera3RequestDescriptor::StreamBuffer::dstBuffer + * \var StreamBuffer::dstBuffer * \brief Pointer to the destination frame buffer used for post-processing * - * \var Camera3RequestDescriptor::StreamBuffer::request + * \var StreamBuffer::request * \brief Back pointer to the Camera3RequestDescriptor to which the StreamBuffer belongs */ -Camera3RequestDescriptor::StreamBuffer::StreamBuffer( +StreamBuffer::StreamBuffer( CameraStream *cameraStream, const camera3_stream_buffer_t &buffer, Camera3RequestDescriptor *requestDescriptor) : stream(cameraStream), camera3Buffer(buffer.buffer), @@ -186,9 +186,8 @@ Camera3RequestDescriptor::StreamBuffer::StreamBuffer( { } -Camera3RequestDescriptor::StreamBuffer::~StreamBuffer() = default; +StreamBuffer::~StreamBuffer() = default; -Camera3RequestDescriptor::StreamBuffer::StreamBuffer(StreamBuffer &&) = default; +StreamBuffer::StreamBuffer(StreamBuffer &&) = default; -Camera3RequestDescriptor::StreamBuffer & -Camera3RequestDescriptor::StreamBuffer::operator=(Camera3RequestDescriptor::StreamBuffer &&) = default; +StreamBuffer &StreamBuffer::operator=(StreamBuffer &&) = default; diff --git a/src/android/camera_request.h b/src/android/camera_request.h index 37b6ae32..5aa4eea8 100644 --- a/src/android/camera_request.h +++ b/src/android/camera_request.h @@ -24,8 +24,9 @@ class CameraBuffer; class CameraStream; +class Camera3RequestDescriptor; -class Camera3RequestDescriptor +class StreamBuffer { public: enum class Status { @@ -33,27 +34,34 @@ public: Error, }; - struct StreamBuffer { - StreamBuffer(CameraStream *stream, - const camera3_stream_buffer_t &buffer, - Camera3RequestDescriptor *request); - ~StreamBuffer(); - - StreamBuffer(StreamBuffer &&); - StreamBuffer &operator=(StreamBuffer &&); - - CameraStream *stream; - buffer_handle_t *camera3Buffer; - std::unique_ptr frameBuffer; - libcamera::UniqueFD fence; - Status status = Status::Success; - libcamera::FrameBuffer *internalBuffer = nullptr; - const libcamera::FrameBuffer *srcBuffer = nullptr; - std::unique_ptr dstBuffer; - Camera3RequestDescriptor *request; - - private: - LIBCAMERA_DISABLE_COPY(StreamBuffer) + StreamBuffer(CameraStream *stream, + const camera3_stream_buffer_t &buffer, + Camera3RequestDescriptor *request); + ~StreamBuffer(); + + StreamBuffer(StreamBuffer &&); + StreamBuffer &operator=(StreamBuffer &&); + + CameraStream *stream; + buffer_handle_t *camera3Buffer; + std::unique_ptr frameBuffer; + libcamera::UniqueFD fence; + Status status = Status::Success; + libcamera::FrameBuffer *internalBuffer = nullptr; + const libcamera::FrameBuffer *srcBuffer = nullptr; + std::unique_ptr dstBuffer; + Camera3RequestDescriptor *request; + +private: + LIBCAMERA_DISABLE_COPY(StreamBuffer) +}; + +class Camera3RequestDescriptor +{ +public: + enum class Status { + Success, + Error, }; /* Keeps track of streams requiring post-processing. */ diff --git a/src/android/camera_stream.cpp b/src/android/camera_stream.cpp index 045e6006..02dc8922 100644 --- a/src/android/camera_stream.cpp +++ b/src/android/camera_stream.cpp @@ -112,14 +112,14 @@ int CameraStream::configure() worker_ = std::make_unique(postProcessor_.get()); postProcessor_->processComplete.connect( - this, [&](Camera3RequestDescriptor::StreamBuffer *streamBuffer, + this, [&](StreamBuffer *streamBuffer, PostProcessor::Status status) { - Camera3RequestDescriptor::Status bufferStatus; + StreamBuffer::Status bufferStatus; if (status == PostProcessor::Status::Success) - bufferStatus = Camera3RequestDescriptor::Status::Success; + bufferStatus = StreamBuffer::Status::Success; else - bufferStatus = Camera3RequestDescriptor::Status::Error; + bufferStatus = StreamBuffer::Status::Error; cameraDevice_->streamProcessingComplete(streamBuffer, bufferStatus); @@ -165,7 +165,7 @@ int CameraStream::waitFence(int fence) return -errno; } -int CameraStream::process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) +int CameraStream::process(StreamBuffer *streamBuffer) { ASSERT(type_ != Type::Direct); @@ -283,7 +283,7 @@ void CameraStream::PostProcessorWorker::start() Thread::start(); } -void CameraStream::PostProcessorWorker::queueRequest(Camera3RequestDescriptor::StreamBuffer *dest) +void CameraStream::PostProcessorWorker::queueRequest(StreamBuffer *dest) { { MutexLocker lock(mutex_); @@ -306,7 +306,7 @@ void CameraStream::PostProcessorWorker::run() if (state_ != State::Running) break; - Camera3RequestDescriptor::StreamBuffer *streamBuffer = requests_.front(); + StreamBuffer *streamBuffer = requests_.front(); requests_.pop(); locker.unlock(); @@ -316,8 +316,7 @@ void CameraStream::PostProcessorWorker::run() } if (state_ == State::Flushing) { - std::queue requests = - std::move(requests_); + std::queue requests = std::move(requests_); locker.unlock(); while (!requests.empty()) { diff --git a/src/android/camera_stream.h b/src/android/camera_stream.h index 4c5078b2..c3b2e325 100644 --- a/src/android/camera_stream.h +++ b/src/android/camera_stream.h @@ -127,7 +127,7 @@ public: CameraStream *sourceStream() const { return sourceStream_; } int configure(); - int process(Camera3RequestDescriptor::StreamBuffer *streamBuffer); + int process(StreamBuffer *streamBuffer); libcamera::FrameBuffer *getBuffer(); void putBuffer(libcamera::FrameBuffer *buffer); void flush(); @@ -146,7 +146,7 @@ private: ~PostProcessorWorker(); void start(); - void queueRequest(Camera3RequestDescriptor::StreamBuffer *request); + void queueRequest(StreamBuffer *request); void flush(); protected: @@ -158,7 +158,7 @@ private: libcamera::Mutex mutex_; libcamera::ConditionVariable cv_; - std::queue requests_ + std::queue requests_ LIBCAMERA_TSA_GUARDED_BY(mutex_); State state_ LIBCAMERA_TSA_GUARDED_BY(mutex_) = State::Stopped; diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp index d72ebc3c..b9ba38ce 100644 --- a/src/android/jpeg/post_processor_jpeg.cpp +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -98,7 +98,7 @@ void PostProcessorJpeg::generateThumbnail(const FrameBuffer &source, } } -void PostProcessorJpeg::process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) +void PostProcessorJpeg::process(StreamBuffer *streamBuffer) { ASSERT(encoder_); diff --git a/src/android/jpeg/post_processor_jpeg.h b/src/android/jpeg/post_processor_jpeg.h index 98309b01..0ba3e01b 100644 --- a/src/android/jpeg/post_processor_jpeg.h +++ b/src/android/jpeg/post_processor_jpeg.h @@ -22,7 +22,7 @@ public: int configure(const libcamera::StreamConfiguration &incfg, const libcamera::StreamConfiguration &outcfg) override; - void process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) override; + void process(StreamBuffer *streamBuffer) override; private: void generateThumbnail(const libcamera::FrameBuffer &source, diff --git a/src/android/post_processor.h b/src/android/post_processor.h index 1a205b05..ac0182cd 100644 --- a/src/android/post_processor.h +++ b/src/android/post_processor.h @@ -27,7 +27,7 @@ public: virtual int configure(const libcamera::StreamConfiguration &inCfg, const libcamera::StreamConfiguration &outCfg) = 0; - virtual void process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) = 0; + virtual void process(StreamBuffer *streamBuffer) = 0; - libcamera::Signal processComplete; + libcamera::Signal processComplete; }; diff --git a/src/android/yuv/post_processor_yuv.cpp b/src/android/yuv/post_processor_yuv.cpp index ed44e6fe..de49b378 100644 --- a/src/android/yuv/post_processor_yuv.cpp +++ b/src/android/yuv/post_processor_yuv.cpp @@ -49,7 +49,7 @@ int PostProcessorYuv::configure(const StreamConfiguration &inCfg, return 0; } -void PostProcessorYuv::process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) +void PostProcessorYuv::process(StreamBuffer *streamBuffer) { const FrameBuffer &source = *streamBuffer->srcBuffer; CameraBuffer *destination = streamBuffer->dstBuffer.get(); diff --git a/src/android/yuv/post_processor_yuv.h b/src/android/yuv/post_processor_yuv.h index a7ac17c5..27cf4bbc 100644 --- a/src/android/yuv/post_processor_yuv.h +++ b/src/android/yuv/post_processor_yuv.h @@ -18,7 +18,7 @@ public: int configure(const libcamera::StreamConfiguration &incfg, const libcamera::StreamConfiguration &outcfg) override; - void process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) override; + void process(StreamBuffer *streamBuffer) override; private: bool isValidBuffers(const libcamera::FrameBuffer &source, From patchwork Fri Aug 12 09:08:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 17091 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 50F70C3272 for ; Fri, 12 Aug 2022 09:09:00 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 18F0D63337; Fri, 12 Aug 2022 11:09:00 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660295340; bh=XusJXHRCLyfE1WSbCkM9zjB4yGYxgU0pXE2ifq9VIcQ=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=pBvJzsJGmNYSgdKbnQRDoeKprhqU3fQeV875ElZoUEW5qVmbPOTfVuatVL5PCaj0Z sKuzhDtKua7xayvDViZy72gHTBeB+eVhQSHU13+O+7CHiZGbztvnVJS5s/XvufL7sH zwEmjSKKzRsDNCDUi0XoYUx8DA7eEwYs5/cEJzgTVX1Nlg2BofD4vJOO3jB/fQE+GO XDPymn5azHSoii2tTWO6Sb3YM89OmAX73n+hRY9oXebMhMPsarcXQ3TjitKI1udngj rzt4/oKuVp9KD4xFWSmmkS3RG+YCIDDu669fNPrG3bfzeNWQaT5zH5b9qVksd+1znK 1s5EnXTcT7TXw== Received: from mail-pg1-x532.google.com (mail-pg1-x532.google.com [IPv6:2607:f8b0:4864:20::532]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6C99763326 for ; Fri, 12 Aug 2022 11:08:58 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="lKRrpvwF"; dkim-atps=neutral Received: by mail-pg1-x532.google.com with SMTP id d71so304692pgc.13 for ; Fri, 12 Aug 2022 02:08:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=JfN4WG4RHWr9fhXxcCB7SBeaeQU7DHKgKdxX9Hwe+ZY=; b=lKRrpvwFJg154QPlJbJiPsvKsWMUh6S/cNozUmfQWzoVA1d2Um2yx6UKIKf+nU0pNq Q3iaAdjREeQVb3qn5bptkbvnupq44mxce+hCwsyZC8PI4AX4FIriVaPDZt1oJv7uB6S3 Md8n0IzD6sNdgbmo5lhYoEKX3ZSD+D8gLCRn0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=JfN4WG4RHWr9fhXxcCB7SBeaeQU7DHKgKdxX9Hwe+ZY=; b=MGbMYU+QOOjicGPbuJRCtjhN/4FJaATkbnBTtRVxBglFcT8+dI0ktsE87XkxZl/WUN Mb3ejSJtW0dXv1fy8JPdY25b2opGceavh1iVPpIsL0xrLV3QlHll8uf1JZfKJtfpambT /AJrirFMXSQOZVU3tiC7svSJ+yKPDfXswNgk04vXoNqFoPGucIIAuL6yEBV1qDClUApl BCo+w8FJ0q7xA7B0PgxoOlLEFmYOmWKtes90BAjpgBQyrb2D04vNPm8bUeR39jwZmwNt Bq14o9lT78oxC/1MyGjN0Kjhb9UsZ+xNHLNLzfcjk9dAiSSgH17FnkKITS/NCWpqBBUH qdfw== X-Gm-Message-State: ACgBeo1kZBGgLwhDbNQg+wqZYKr7bxtzOyS3MAugSOSsHdMX3TtR3Xw1 TCUKkVb+34UkmOxsF5gqbtcEYw4emleNtA== X-Google-Smtp-Source: AA6agR57M+Tn61CvEHN8SmE/j4WPxCzlXkE3/yNYrK32ZMN/9yoQ1jclKTFnBxfPJhzfkIDcbicHtA== X-Received: by 2002:a65:5941:0:b0:41d:a203:c043 with SMTP id g1-20020a655941000000b0041da203c043mr2315382pgu.483.1660295336747; Fri, 12 Aug 2022 02:08:56 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:1705:d284:d114:2e24]) by smtp.gmail.com with UTF8SMTPSA id x2-20020a17090300c200b0016ee708350bsm1179295plc.14.2022.08.12.02.08.55 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 12 Aug 2022 02:08:56 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Fri, 12 Aug 2022 17:08:36 +0800 Message-Id: <20220812090838.1784703-6-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog In-Reply-To: <20220812090838.1784703-1-hanlinchen@chromium.org> References: <20220812090838.1784703-1-hanlinchen@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 5/7] android: Add JpegExifMetadata to store tags setting into Exif X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" With partial result, some metadata needed to adding into Exif may be sent back to framework earlier before Jpeg post-processing. Add a type JpegExifMetadata associated with StreamBuffer to store the values, so Jpeg post-processing doesn't need to reference to current metadata for the them. Signed-off-by: Han-Lin Chen --- src/android/camera_device.cpp | 27 ++++++++++++++++++++++++ src/android/camera_device.h | 2 ++ src/android/camera_request.h | 6 ++++++ src/android/camera_stream.h | 4 ++++ src/android/jpeg/post_processor_jpeg.cpp | 13 ++++++------ 5 files changed, 46 insertions(+), 6 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index 0248f146..a14d5de9 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -1221,6 +1221,10 @@ void CameraDevice::requestComplete(Request *request) CameraStream *stream = iter->first; StreamBuffer *buffer = iter->second; + if (stream->isJpegStream()) { + generateJpegExifMetadata(descriptor, buffer); + } + FrameBuffer *src = request->findBuffer(stream->stream()); if (!src) { LOG(HAL, Error) << "Failed to find a source stream buffer"; @@ -1410,6 +1414,29 @@ void CameraDevice::notifyError(uint32_t frameNumber, camera3_stream_t *stream, callbacks_->notify(callbacks_, ¬ify); } +/* + * Set jpeg metadata used to generate EXIF in the JPEG post processing. + */ +void CameraDevice::generateJpegExifMetadata(Camera3RequestDescriptor *request, + StreamBuffer *buffer) const +{ + const ControlList &metadata = request->request_->metadata(); + auto &jpegExifMetadata = buffer->jpegExifMetadata; + jpegExifMetadata.emplace(StreamBuffer::JpegExifMetadata()); + + jpegExifMetadata->sensorExposureTime = 0; + if (metadata.contains(controls::ExposureTime)) { + jpegExifMetadata->sensorExposureTime = metadata.get(controls::ExposureTime) * 1000ULL; + } + + /* + * todo: Android Sensitivity = analog gain X digital gain only on sensor. + * Digital gain on ISP shouldn't be included. Calculate sensitivity + * accordingingly when we can differentiate the source of digital gains. + */ + jpegExifMetadata->sensorSensitivityISO = 100; +} + /* * Produce a set of fixed result metadata. */ diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 9eb7221b..7b279895 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -97,6 +97,8 @@ private: LIBCAMERA_TSA_EXCLUDES(descriptorsMutex_); void sendCaptureResults() LIBCAMERA_TSA_REQUIRES(descriptorsMutex_); void setBufferStatus(StreamBuffer &buffer, StreamBuffer::Status status); + void generateJpegExifMetadata(Camera3RequestDescriptor *request, + StreamBuffer *buffer) const; std::unique_ptr getResultMetadata( const Camera3RequestDescriptor &descriptor) const; diff --git a/src/android/camera_request.h b/src/android/camera_request.h index 5aa4eea8..f91de955 100644 --- a/src/android/camera_request.h +++ b/src/android/camera_request.h @@ -42,6 +42,11 @@ public: StreamBuffer(StreamBuffer &&); StreamBuffer &operator=(StreamBuffer &&); + struct JpegExifMetadata { + int64_t sensorExposureTime; + int32_t sensorSensitivityISO; + }; + CameraStream *stream; buffer_handle_t *camera3Buffer; std::unique_ptr frameBuffer; @@ -50,6 +55,7 @@ public: libcamera::FrameBuffer *internalBuffer = nullptr; const libcamera::FrameBuffer *srcBuffer = nullptr; std::unique_ptr dstBuffer; + std::optional jpegExifMetadata; Camera3RequestDescriptor *request; private: diff --git a/src/android/camera_stream.h b/src/android/camera_stream.h index c3b2e325..fc50d412 100644 --- a/src/android/camera_stream.h +++ b/src/android/camera_stream.h @@ -125,6 +125,10 @@ public: const libcamera::StreamConfiguration &configuration() const; libcamera::Stream *stream() const; CameraStream *sourceStream() const { return sourceStream_; } + bool isJpegStream() const + { + return camera3Stream_->format == HAL_PIXEL_FORMAT_BLOB; + } int configure(); int process(StreamBuffer *streamBuffer); diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp index b9ba38ce..7e3713f5 100644 --- a/src/android/jpeg/post_processor_jpeg.cpp +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -104,8 +104,11 @@ void PostProcessorJpeg::process(StreamBuffer *streamBuffer) const FrameBuffer &source = *streamBuffer->srcBuffer; CameraBuffer *destination = streamBuffer->dstBuffer.get(); + const std::optional &jpegExifMetadata = + streamBuffer->jpegExifMetadata; ASSERT(destination->numPlanes() == 1); + ASSERT(jpegExifMetadata.has_value()); const CameraMetadata &requestMetadata = streamBuffer->request->settings_; CameraMetadata *resultMetadata = streamBuffer->request->resultMetadata_.get(); @@ -131,15 +134,13 @@ void PostProcessorJpeg::process(StreamBuffer *streamBuffer) */ exif.setTimestamp(std::time(nullptr), 0ms); - ret = resultMetadata->getEntry(ANDROID_SENSOR_EXPOSURE_TIME, &entry); - exif.setExposureTime(ret ? *entry.data.i64 : 0); + exif.setExposureTime(jpegExifMetadata->sensorExposureTime * 1000); + exif.setISO(jpegExifMetadata->sensorSensitivityISO); + ret = requestMetadata.getEntry(ANDROID_LENS_APERTURE, &entry); if (ret) exif.setAperture(*entry.data.f); - ret = resultMetadata->getEntry(ANDROID_SENSOR_SENSITIVITY, &entry); - exif.setISO(ret ? *entry.data.i32 : 100); - exif.setFlash(Exif::Flash::FlashNotPresent); exif.setWhiteBalance(Exif::WhiteBalance::Auto); @@ -152,6 +153,7 @@ void PostProcessorJpeg::process(StreamBuffer *streamBuffer) *entry.data.i64); } + std::vector thumbnail; ret = requestMetadata.getEntry(ANDROID_JPEG_THUMBNAIL_SIZE, &entry); if (ret) { const int32_t *data = entry.data.i32; @@ -163,7 +165,6 @@ void PostProcessorJpeg::process(StreamBuffer *streamBuffer) resultMetadata->addEntry(ANDROID_JPEG_THUMBNAIL_QUALITY, quality); if (thumbnailSize != Size(0, 0)) { - std::vector thumbnail; generateThumbnail(source, thumbnailSize, quality, &thumbnail); if (!thumbnail.empty()) exif.setThumbnail(thumbnail, Exif::Compression::JPEG); From patchwork Fri Aug 12 09:08:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 17092 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D9362C3272 for ; Fri, 12 Aug 2022 09:09:04 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A001463332; Fri, 12 Aug 2022 11:09:04 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660295344; bh=XUVXIH095CW+orozHNb+edNGl6eV+3R8nVMFXij+Gb8=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=mGZh/sPPLbEQ7pxUxL8JvF9siIFMmND9E4/7+2yH2MkALp/10TxkGjA6rXIJGOsDt Q6Tq6g2TOLwPMxdSlW/DYGkgiyZZrJEZtp+2m+Gwz92g+6iRRjGxqwTy+ubx9jQaV/ OgEtws1V/j/zsGGoN97m4+aKomUKCmkOB0/pK3rVJAzu505ZAQWT0/U20E1aJr9LAq w1tz/f1UZrBMyNByUXdAhm/JaRR4EAWrY8xLy3u4BycMIqK1J5KuKBzd5SblM2iVxv ODLz/UQHwuWqQpxsGTQRM6LMNxkZIFA8yGnmc1xnr47c8jt1W3AZFs3dPgdl4DeKMl cJ0dIXhkVq5sg== Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 403CA63326 for ; Fri, 12 Aug 2022 11:09:02 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="UWaX9zKV"; dkim-atps=neutral Received: by mail-pf1-x42c.google.com with SMTP id g12so434048pfb.3 for ; Fri, 12 Aug 2022 02:09:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=c+5pmxEp18nuPmAb1DA+iCZFZ8j9iB6RgdwRFw1tOxU=; b=UWaX9zKVm/F9ghKdmZmjn80z8CNPER5ptR7VQLeNSgnMiKm2Sllieb2OgxAsB0Z5fk hqtt5qPiaNR0ibLxyJt14KANmhKUJn/vJFtJQ96g8k6/VO+HekNETu14LrNY9Gqf1Ja2 mHI72D3KtKE1Qs5/aUh3qZ1GqdBmRJwlTIc/c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=c+5pmxEp18nuPmAb1DA+iCZFZ8j9iB6RgdwRFw1tOxU=; b=gvydsM0ET5gI7zaaypLLEGmnyzM/j8zCz6S4ke5VhOAjqHdKjui9bcVoQZ77JFUfyz ZXYnoszM6cwt+YHH2fZMhSAfFBGAyMmrOZyBOGOiq64DFrFRXSZj9wKhZTcz4/pPJOTF EaQbp/9zycHKlaDEZuhWY3hhS7+yEHgvPX4QgVgiXmGhxtejRvBcndbKEOpXqLSiMId5 N1mqhvPns323SBl7QFezF8e7zMNkvU1fMBo4Qwjx+argtXIGLXsUijGudA5l+pKLByZ5 mw1Tlepbm4XFyqWGMUDa8f/VdwYlE7FkMqmBVQWfuk7es2kkDke7ASidu70mgz/pyCl6 Cyrg== X-Gm-Message-State: ACgBeo3vDKhHWGIPwrU/kBDCUi0X+KULyHMQ7By0CZATlEqKGICARaD6 tWGsr/fHL6PDOGSRAQ5LfAuODpxfzJR7nQ== X-Google-Smtp-Source: AA6agR6EC+CiGbhtOkqTui6TkgUp4/uLUk2GSE9bFI21lzRBGmC4IFNsKHKc89sisfH4KQS8t0F17g== X-Received: by 2002:a05:6a00:23d1:b0:52f:39e9:9150 with SMTP id g17-20020a056a0023d100b0052f39e99150mr3130866pfc.16.1660295339471; Fri, 12 Aug 2022 02:08:59 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:1705:d284:d114:2e24]) by smtp.gmail.com with UTF8SMTPSA id j15-20020a170902da8f00b0016f04c098ddsm1166674plx.226.2022.08.12.02.08.58 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 12 Aug 2022 02:08:59 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Fri, 12 Aug 2022 17:08:37 +0800 Message-Id: <20220812090838.1784703-7-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog In-Reply-To: <20220812090838.1784703-1-hanlinchen@chromium.org> References: <20220812090838.1784703-1-hanlinchen@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 6/7] android: Implement partial result feature based on partialResultCompleted signal X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" To impelement partial result feature. The patch includes the following: 1. Return partial results on Camera::partialResultCompleted signal, and recycle requests on Camera::requestCompleted. 2. Reorder the result according to the FIFO restriction of the same stream Signed-off-by: Han-Lin Chen --- src/android/camera_capabilities.cpp | 2 +- src/android/camera_capabilities.h | 2 + src/android/camera_device.cpp | 848 +++++++++++++++-------- src/android/camera_device.h | 36 +- src/android/camera_request.cpp | 61 +- src/android/camera_request.h | 56 +- src/android/camera_stream.cpp | 1 + src/android/jpeg/post_processor_jpeg.cpp | 2 +- 8 files changed, 693 insertions(+), 315 deletions(-) diff --git a/src/android/camera_capabilities.cpp b/src/android/camera_capabilities.cpp index 64bd8dde..6ed040d0 100644 --- a/src/android/camera_capabilities.cpp +++ b/src/android/camera_capabilities.cpp @@ -1374,7 +1374,7 @@ int CameraCapabilities::initializeStaticMetadata() staticMetadata_->addEntry(ANDROID_SCALER_CROPPING_TYPE, croppingType); /* Request static metadata. */ - int32_t partialResultCount = 1; + int32_t partialResultCount = MaxMetadataPackIndex; staticMetadata_->addEntry(ANDROID_REQUEST_PARTIAL_RESULT_COUNT, partialResultCount); diff --git a/src/android/camera_capabilities.h b/src/android/camera_capabilities.h index 6f66f221..b8fc20f1 100644 --- a/src/android/camera_capabilities.h +++ b/src/android/camera_capabilities.h @@ -23,6 +23,8 @@ class CameraCapabilities { public: + static constexpr int32_t MaxMetadataPackIndex = 64; + CameraCapabilities() = default; int initialize(std::shared_ptr camera, diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index a14d5de9..c999b198 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -24,9 +24,8 @@ #include #include -#include "system/graphics.h" - #include "camera_buffer.h" +#include "camera_capabilities.h" #include "camera_hal_config.h" #include "camera_ops.h" #include "camera_request.h" @@ -249,7 +248,11 @@ CameraDevice::CameraDevice(unsigned int id, std::shared_ptr camera) : id_(id), state_(State::Stopped), camera_(std::move(camera)), facing_(CAMERA_FACING_FRONT), orientation_(0) { + /* Set RequestCompletionMode to Immediately to send result early */ + camera_->setRequestCompletionMode(Camera::Immediately); + camera_->requestCompleted.connect(this, &CameraDevice::requestComplete); + camera_->partialResultCompleted.connect(this, &CameraDevice::partialResultComplete); maker_ = "libcamera"; model_ = "cameraModel"; @@ -438,8 +441,10 @@ void CameraDevice::stop() camera_->stop(); { - MutexLocker descriptorsLock(descriptorsMutex_); - descriptors_ = {}; + MutexLocker descriptorsLock(pendingRequestMutex_); + pendingRequests_.clear(); + pendingResults_.clear(); + pendingStreamBuffers_.clear(); } streams_.clear(); @@ -855,14 +860,37 @@ int CameraDevice::processControls(Camera3RequestDescriptor *descriptor) return 0; } -void CameraDevice::abortRequest(Camera3RequestDescriptor *descriptor) const +void CameraDevice::abortRequest(Camera3RequestDescriptor *descriptor) { - notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_REQUEST); + /* + * Since the failed buffers do not have to follow the strict ordering + * valid buffers do, and could be out-of-order with respect to valid + * buffers, it's safe to send the aborted result back to the framework + * direcly. + */ + Camera3ResultDescriptor *result = new Camera3ResultDescriptor(descriptor); + result->metadataPackIndex_ = 0; - for (auto &buffer : descriptor->buffers_) - buffer.status = StreamBuffer::Status::Error; + for (auto &buffer : descriptor->buffers_) { + setBufferStatus(buffer, StreamBuffer::Status::Error); + result->buffers_.emplace_back(&buffer); + } + + { + MutexLocker lock(descriptor->resultsMutex_); + descriptor->status_ = Camera3RequestDescriptor::Status::Aborted; + descriptor->finalResult_.reset(result); + } - descriptor->status_ = Camera3RequestDescriptor::Status::Error; + /* + * After CAMERA3_MSG_ERROR_REQUEST is notified, for a given frame, + * only process_capture_results with buffers of the status + * CAMERA3_BUFFER_STATUS_ERROR are allowed. No further notifys or + * process_capture_result with non-null metadata is allowed. + */ + notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_REQUEST); + + sendCaptureResult(result); } bool CameraDevice::isValidRequest(camera3_capture_request_t *camera3Request) const @@ -962,9 +990,10 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques * to a libcamera stream. Streams of type Mapped will be handled later. * * Collect the CameraStream associated to each requested capture stream. - * Since requestedStreams is an std:set<>, no duplications can happen. + * Since requestedDirectBuffers is an std:map<>, no duplications can + * happen. */ - std::set requestedStreams; + std::map requestedDirectBuffers; for (const auto &[i, buffer] : utils::enumerate(descriptor->buffers_)) { CameraStream *cameraStream = buffer.stream; camera3_stream_t *camera3Stream = cameraStream->camera3Stream(); @@ -983,8 +1012,6 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques FrameBuffer *frameBuffer = nullptr; UniqueFD acquireFence; - MutexLocker lock(descriptor->streamsProcessMutex_); - switch (cameraStream->type()) { case CameraStream::Type::Mapped: /* Mapped streams will be handled in the next loop. */ @@ -1003,23 +1030,23 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques cameraStream->configuration().size); frameBuffer = buffer.frameBuffer.get(); acquireFence = std::move(buffer.fence); + + requestedDirectBuffers[cameraStream] = frameBuffer; LOG(HAL, Debug) << ss.str() << " (direct)"; break; case CameraStream::Type::Internal: /* - * Get the frame buffer from the CameraStream internal - * buffer pool. - * - * The buffer has to be returned to the CameraStream - * once it has been processed. + * Get the frame buffer from the source stream's + * internal buffer pool. The buffer has to be returned + * to the source stream once it has been processed. */ frameBuffer = cameraStream->getBuffer(); - buffer.internalBuffer = frameBuffer; - LOG(HAL, Debug) << ss.str() << " (internal)"; + buffer.srcBuffer = frameBuffer; - descriptor->pendingStreamsToProcess_.insert( - { cameraStream, &buffer }); + /* Track the allocated internal buffer */ + descriptor->internalBuffers_[cameraStream] = frameBuffer; + LOG(HAL, Debug) << ss.str() << " (internal)"; break; } @@ -1031,8 +1058,6 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques auto fence = std::make_unique(std::move(acquireFence)); descriptor->request_->addBuffer(cameraStream->stream(), frameBuffer, std::move(fence)); - - requestedStreams.insert(cameraStream); } /* @@ -1054,29 +1079,54 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques << cameraStream->configuration().pixelFormat << "]" << " (mapped)"; - MutexLocker lock(descriptor->streamsProcessMutex_); - descriptor->pendingStreamsToProcess_.insert({ cameraStream, &buffer }); - /* * Make sure the CameraStream this stream is mapped on has been * added to the request. */ CameraStream *sourceStream = cameraStream->sourceStream(); + ASSERT(sourceStream); - if (requestedStreams.find(sourceStream) != requestedStreams.end()) + ASSERT(sourceStream->type() == CameraStream::Type::Direct); + + /* + * If the buffer for the source stream has been requested as + * Direct, use its framebuffer as the source buffer for + * post-processing. No need to recycle the buffer since it's + * owned by Android. + */ + auto iterDirectBuffer = requestedDirectBuffers.find(sourceStream); + if (iterDirectBuffer != requestedDirectBuffers.end()) { + buffer.srcBuffer = iterDirectBuffer->second; continue; + } /* - * If that's not the case, we need to add a buffer to the request - * for this stream. + * If that's not the case, we use an internal buffer allocated + * from the source stream. + * + * If an internal buffer has been requested for the source + * stream before, we should reuse it. */ - FrameBuffer *frameBuffer = cameraStream->getBuffer(); - buffer.internalBuffer = frameBuffer; + auto iterInternalBuffer = descriptor->internalBuffers_.find(sourceStream); + if (iterInternalBuffer != descriptor->internalBuffers_.end()) { + buffer.srcBuffer = iterInternalBuffer->second; + continue; + } + + /* + * Otherwise, we need to create an internal buffer to the + * request for the source stream. Get the frame buffer from the + * source stream's internal buffer pool. The buffer has to be + * returned to the source stream once it has been processed. + */ + FrameBuffer *frameBuffer = sourceStream->getBuffer(); + buffer.srcBuffer = frameBuffer; descriptor->request_->addBuffer(sourceStream->stream(), frameBuffer, nullptr); - requestedStreams.erase(sourceStream); + /* Track the allocated internal buffer. */ + descriptor->internalBuffers_[sourceStream] = frameBuffer; } /* @@ -1095,14 +1145,7 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques MutexLocker stateLock(stateMutex_); if (state_ == State::Flushing) { - Camera3RequestDescriptor *rawDescriptor = descriptor.get(); - { - MutexLocker descriptorsLock(descriptorsMutex_); - descriptors_.push(std::move(descriptor)); - } - abortRequest(rawDescriptor); - completeDescriptor(rawDescriptor); - + abortRequest(descriptor.get()); return 0; } @@ -1119,8 +1162,10 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques Request *request = descriptor->request_.get(); { - MutexLocker descriptorsLock(descriptorsMutex_); - descriptors_.push(std::move(descriptor)); + MutexLocker descriptorsLock(pendingRequestMutex_); + for (auto &buffer : descriptor->buffers_) + pendingStreamBuffers_[buffer.stream].push_back(&buffer); + pendingRequests_.push_back(std::move(descriptor)); } camera_->queueRequest(request); @@ -1128,223 +1173,444 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques return 0; } -void CameraDevice::requestComplete(Request *request) +void CameraDevice::partialResultComplete(Request *request, Result *result) { Camera3RequestDescriptor *descriptor = reinterpret_cast(request->cookie()); - /* - * Prepare the capture result for the Android camera stack. - * - * The buffer status is set to Success and later changed to Error if - * post-processing/compression fails. - */ + if (result->buffers().empty() && result->metadata().empty()) + LOG(HAL, Fatal) + << "Empty partial result is not allowed. Frame number: " + << descriptor->frameNumber_; + + Camera3ResultDescriptor *camera3Result = new Camera3ResultDescriptor(descriptor); + + const ControlList &metadata = result->metadata(); + if (!metadata.empty()) { + /* + * Notify shutter as soon as we have received SensorTimestamp. + */ + const auto ×tamp = metadata.get(controls::SensorTimestamp); + if (timestamp) { + notifyShutter(descriptor->frameNumber_, *timestamp); + LOG(HAL, Debug) << "Request " << request->cookie() << " notifies shutter"; + } + + camera3Result->resultMetadata_ = getDynamicResultMetadata(metadata); + } + + MutexLocker locker(camera3Result->streamsProcessMutex_); + for (auto &buffer : descriptor->buffers_) { - CameraStream *stream = buffer.stream; + CameraStream *cameraStream = buffer.stream; + for (auto *frameBuffer : result->buffers()) { + if (buffer.srcBuffer != frameBuffer && + buffer.frameBuffer.get() != frameBuffer) + continue; + + buffer.result = camera3Result; + camera3Result->buffers_.emplace_back(&buffer); + StreamBuffer::Status status = StreamBuffer::Status::Success; + if (frameBuffer->metadata().status != FrameMetadata::FrameSuccess) { + status = StreamBuffer::Status::Error; + } + setBufferStatus(buffer, status); + + switch (cameraStream->type()) { + case CameraStream::Type::Direct: { + ASSERT(buffer.frameBuffer.get() == frameBuffer); + /* + * Streams of type Direct have been queued to the + * libcamera::Camera and their acquire fences have + * already been waited on by the library. + */ + std::unique_ptr fence = buffer.frameBuffer->releaseFence(); + if (fence) + buffer.fence = fence->release(); + break; + } + case CameraStream::Type::Mapped: + case CameraStream::Type::Internal: + ASSERT(buffer.srcBuffer == frameBuffer); + if (status == StreamBuffer::Status::Error) + break; + + /* + * Acquire fences of streams of type Internal and Mapped + * will be handled during post-processing. + */ + buffer.srcBuffer = frameBuffer; + camera3Result->pendingBuffersToProcess_.emplace_back(&buffer); + + if (cameraStream->isJpegStream()) { + generateJpegExifMetadata(descriptor, &buffer); + + /* + * Allocate for post-processor to fill + * in JPEG related metadata. + */ + if (!camera3Result->resultMetadata_) + camera3Result->resultMetadata_ = getDynamicResultMetadata(metadata); + } + break; + } + } + } + + { /* - * Streams of type Direct have been queued to the - * libcamera::Camera and their acquire fences have - * already been waited on by the library. - * - * Acquire fences of streams of type Internal and Mapped - * will be handled during post-processing. + * Adding result to the request before sending it to the + * post-processing threads, so the streamProcessingComplete() + * slot can safely call compeleteResult(). */ - if (stream->type() == CameraStream::Type::Direct) { - /* If handling of the fence has failed restore buffer.fence. */ - std::unique_ptr fence = buffer.frameBuffer->releaseFence(); - if (fence) - buffer.fence = fence->release(); + MutexLocker lock(descriptor->resultsMutex_); + descriptor->partialResults_.emplace_back(camera3Result); + } + + /* + * Queue all the post-processing streams request at once. The completion + * slot streamProcessingComplete() can only execute when we are out + * this critical section (result->streamsProcessMutex_). This helps to + * handle synchronous errors here itself. + */ + for (auto *buffer : camera3Result->pendingBuffersToProcess_) { + int ret = buffer->stream->process(buffer); + if (ret) { + setBufferStatus(*buffer, StreamBuffer::Status::Error); + LOG(HAL, Error) << "Failed to run post process of request " + << descriptor->frameNumber_; } - buffer.status = StreamBuffer::Status::Success; } + if (!camera3Result->pendingBuffersToProcess_.empty()) + return; + + locker.unlock(); + tryCompleteResultDescriptor(camera3Result); +} + +void CameraDevice::requestComplete(Request *request) +{ + Camera3RequestDescriptor *camera3Request = + reinterpret_cast(request->cookie()); + /* - * If the Request has failed, abort the request by notifying the error - * and complete the request with all buffers in error state. + * On Android, each new partial result with a metadata must set a + * field (partial_result) to a distinct inclusive value between + * 1 and ANDROID_REQUEST_PARTIAL_RESULT_COUNT and the final result with + * metadata has to set the field as ANDROID_REQUEST_PARTIAL_RESULT_COUNT. + * + * An empty metadata with ANDROID_REQUEST_PARTIAL_RESULT_COUNT is not + * allowed. Add a result with a fixed metadata on requestComplete() + * in case all of previous metadata are sent early, and no more metadata + * can be sent as the final with ANDROID_REQUEST_PARTIAL_RESULT_COUNT. */ - if (request->status() != Request::RequestComplete) { - LOG(HAL, Error) << "Request " << request->cookie() - << " not successfully completed: " - << request->status(); + Camera3ResultDescriptor *result = new Camera3ResultDescriptor(camera3Request); + result->resultMetadata_ = getFixedResultMetadata(camera3Request->settings_); + result->metadataPackIndex_ = CameraCapabilities::MaxMetadataPackIndex; - abortRequest(descriptor); - completeDescriptor(descriptor); + { + MutexLocker lock(camera3Request->resultsMutex_); + camera3Request->finalResult_.reset(result); - return; + switch (request->status()) { + case Request::RequestComplete: + camera3Request->status_ = Camera3RequestDescriptor::Status::Success; + break; + case Request::RequestCancelled: + camera3Request->status_ = Camera3RequestDescriptor::Status::Cancelled; + break; + case Request::RequestPending: + LOG(HAL, Fatal) << "Try to complete an unfinished request"; + break; + } } + completeResultDescriptor(result); + return; +} + +void CameraDevice::tryCompleteResultDescriptor(Camera3ResultDescriptor *result) +{ /* - * Notify shutter as soon as we have verified we have a valid request. + * Android requires buffers for a given stream must be returned in FIFO + * order. However, different streams are independent of each other, so + * it is acceptable and expected that the buffer for request 5 for + * stream A may be returned after the buffer for request 6 for stream + * B is. And it is acceptable that the result metadata for request 6 + * for stream B is returned before the buffer for request 5 for stream + * A is. In result, if all of buffers of a result are the most front + * buffers of each stream, or the result contains no buffers, the result + * is allowed to send. Collect ready results in the order which follows + * the above rule. * - * \todo The shutter event notification should be sent to the framework - * as soon as possible, earlier than request completion time. + * \todo The reprocessing result can be returned ahead of the pending + * normal output results. But the FIFO ordering must be maintained for + * all reprocessing results. Track the reprocessing buffer's order + * independently when we have reprocessing API. */ - uint64_t sensorTimestamp = static_cast(request->metadata() - .get(controls::SensorTimestamp) - .value_or(0)); - notifyShutter(descriptor->frameNumber_, sensorTimestamp); + MutexLocker lock(pendingRequestMutex_); - LOG(HAL, Debug) << "Request " << request->cookie() << " completed with " - << descriptor->request_->buffers().size() << " streams"; + pendingResults_.push_front(result); + std::list readyResults; /* - * Generate the metadata associated with the captured buffers. - * - * Notify if the metadata generation has failed, but continue processing - * buffers and return an empty metadata pack. + * Error buffers do not have to follow the strict ordering valid buffers + * do, and ready to sent directly, removes them from the pendingBuffers + * so it won't blocking following valid buffers. */ - descriptor->resultMetadata_ = getResultMetadata(*descriptor); - if (!descriptor->resultMetadata_) { - notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_RESULT); + for (auto &buffer : result->buffers_) + if (buffer->status == StreamBuffer::Status::Error) + pendingStreamBuffers_[buffer->stream].remove(buffer); - /* - * The camera framework expects an empty metadata pack on error. - * - * \todo Check that the post-processor code handles this situation - * correctly. - */ - descriptor->resultMetadata_ = std::make_unique(0, 0); + /* + * Exhaustly collect results which is ready to sent. + */ + bool keepChecking; + do { + keepChecking = false; + auto iter = pendingResults_.begin(); + while (iter != pendingResults_.end()) { + /* + * A result is considered as ready when all of the valid + * buffers of the result are at the front of the pending + * buffers associated with its stream. + */ + bool ready = true; + for (auto &buffer : (*iter)->buffers_) { + if (buffer->status == StreamBuffer::Status::Error) + continue; + + auto &pendingBuffers = pendingStreamBuffers_[buffer->stream]; + ASSERT(!pendingBuffers.empty()); + + if (pendingBuffers.front() != buffer) { + ready = false; + break; + } + } + if (!ready) { + iter++; + continue; + } + + for (auto &buffer : (*iter)->buffers_) + if (buffer->status != StreamBuffer::Status::Error) + pendingStreamBuffers_[buffer->stream].pop_front(); + + /* Keep checking since pendingStreamBuffers has updated */ + keepChecking = true; + + readyResults.push_back(*iter); + iter = pendingResults_.erase(iter); + } + } while (keepChecking); + + lock.unlock(); + + for (auto &res : readyResults) { + completeResultDescriptor(res); } +} + +void CameraDevice::completeResultDescriptor(Camera3ResultDescriptor *result) +{ + Camera3RequestDescriptor *request = result->request_; - /* Handle post-processing. */ - MutexLocker locker(descriptor->streamsProcessMutex_); + MutexLocker lock(request->resultsMutex_); + result->complete_ = true; /* - * Queue all the post-processing streams request at once. The completion - * slot streamProcessingComplete() can only execute when we are out - * this critical section. This helps to handle synchronous errors here - * itself. + * Android requires value of metadataPackIndex follows the rules: + * + * Set to MaxMetadataPackIndex if its the final result of the request. + * Set Incrementally from 1 if its the result containing metadata. + * Set to 0 if the result contains only buffers. + * + * The metadataPackIndex should be sent incrementally. Since the + * function may be called from post-processing threads, ensure the + * strict sending order by the mutex. */ - auto iter = descriptor->pendingStreamsToProcess_.begin(); - while (iter != descriptor->pendingStreamsToProcess_.end()) { - CameraStream *stream = iter->first; - StreamBuffer *buffer = iter->second; + bool isFinalResult = (result == request->finalResult_.get()); - if (stream->isJpegStream()) { - generateJpegExifMetadata(descriptor, buffer); - } + uint32_t &metadataPackIndex = result->metadataPackIndex_ = 0; - FrameBuffer *src = request->findBuffer(stream->stream()); - if (!src) { - LOG(HAL, Error) << "Failed to find a source stream buffer"; - setBufferStatus(*buffer, StreamBuffer::Status::Error); - iter = descriptor->pendingStreamsToProcess_.erase(iter); - continue; - } + if (isFinalResult) + metadataPackIndex = CameraCapabilities::MaxMetadataPackIndex; + else if (result->resultMetadata_) + metadataPackIndex = request->nextPartialResultIndex_++; + else + metadataPackIndex = 0; - buffer->srcBuffer = src; + if (metadataPackIndex > CameraCapabilities::MaxMetadataPackIndex) + LOG(HAL, Fatal) << "Partial result count exceed limitation " + << CameraCapabilities::MaxMetadataPackIndex; - ++iter; - int ret = stream->process(buffer); - if (ret) { - setBufferStatus(*buffer, StreamBuffer::Status::Error); - descriptor->pendingStreamsToProcess_.erase(stream); + /* + * The final result will be returned in completeRequestDescriptor() + * to follow the rules that the final result should be returned in + * its submission order. + */ + if (!isFinalResult) + sendCaptureResult(result); - /* - * If the framebuffer is internal to CameraStream return - * it back now that we're done processing it. - */ - if (buffer->internalBuffer) - stream->putBuffer(buffer->internalBuffer); + if (request->status_ == Camera3RequestDescriptor::Status::Pending) + return; + + /* + * No more partial results shall be created once the status has + * changed from Pending. Check whether all of the pending + * results are sent before we complete the request. The function + * shall be re-called everytime a result is completed, and the final + * call shall complete the request. + */ + bool hasPendingResult = false; + for (auto &r : request->partialResults_) { + if (!r->complete_) { + hasPendingResult = true; } } - if (descriptor->pendingStreamsToProcess_.empty()) { - locker.unlock(); - completeDescriptor(descriptor); - } -} + if (hasPendingResult) + return; -/** - * \brief Complete the Camera3RequestDescriptor - * \param[in] descriptor The Camera3RequestDescriptor that has completed - * - * The function marks the Camera3RequestDescriptor as 'complete'. It shall be - * called when all the streams in the Camera3RequestDescriptor have completed - * capture (or have been generated via post-processing) and the request is ready - * to be sent back to the framework. - * - * \context This function is \threadsafe. - */ -void CameraDevice::completeDescriptor(Camera3RequestDescriptor *descriptor) -{ - MutexLocker lock(descriptorsMutex_); - descriptor->complete_ = true; + lock.unlock(); - sendCaptureResults(); + completeRequestDescriptor(result->request_); } + /** - * \brief Sequentially send capture results to the framework + * \brief Complete the Camera3RequestDescriptor + * \param[in] descriptor The Camera3RequestDescriptor * - * Iterate over the descriptors queue to send completed descriptors back to the - * framework, in the same order as they have been queued. For each complete - * descriptor, populate a locally-scoped camera3_capture_result_t from the - * descriptor, send the capture result back by calling the - * process_capture_result() callback, and remove the descriptor from the queue. - * Stop iterating if the descriptor at the front of the queue is not complete. + * The function shall be called when one of a partial result of the descriptor + * has completed. The function shall complete the descriptor only when all of + * the partial result has sent back to the framework, and send the final result + * according to the submission order of the requests. * - * This function should never be called directly in the codebase. Use - * completeDescriptor() instead. + * \context This function is \threadsafe. */ -void CameraDevice::sendCaptureResults() +void CameraDevice::completeRequestDescriptor(Camera3RequestDescriptor *request) { - while (!descriptors_.empty() && !descriptors_.front()->isPending()) { - auto descriptor = std::move(descriptors_.front()); - descriptors_.pop(); - - camera3_capture_result_t captureResult = {}; - - captureResult.frame_number = descriptor->frameNumber_; - - if (descriptor->resultMetadata_) - captureResult.result = - descriptor->resultMetadata_->getMetadata(); - - std::vector resultBuffers; - resultBuffers.reserve(descriptor->buffers_.size()); + MutexLocker locker(pendingRequestMutex_); + request->completed_ = true; - for (auto &buffer : descriptor->buffers_) { - camera3_buffer_status status = CAMERA3_BUFFER_STATUS_ERROR; - - if (buffer.status == StreamBuffer::Status::Success) - status = CAMERA3_BUFFER_STATUS_OK; + /* + * Android requires the final result of each request returns in + * the submission order. + */ + while (!pendingRequests_.empty()) { + auto &descriptor = pendingRequests_.front(); + if (!descriptor->completed_) + break; - /* - * Pass the buffer fence back to the camera framework as - * a release fence. This instructs the framework to wait - * on the acquire fence in case we haven't done so - * ourselves for any reason. - */ - resultBuffers.push_back({ buffer.stream->camera3Stream(), - buffer.camera3Buffer, status, - -1, buffer.fence.release() }); + Camera3RequestDescriptor::Status status; + { + MutexLocker lock(descriptor->resultsMutex_); + status = descriptor->status_; + if (descriptor->finalResult_) + sendCaptureResult(descriptor->finalResult_.get()); } - captureResult.num_output_buffers = resultBuffers.size(); - captureResult.output_buffers = resultBuffers.data(); - - if (descriptor->status_ == Camera3RequestDescriptor::Status::Success) - captureResult.partial_result = 1; + /* + * Call notify with CAMERA3_MSG_ERROR_RESULT to indicate some + * of the expected result metadata might not be available + * because the capture is cancelled by the camera. Only notify + * it when the final result is sent, since Android will ignore + * the following metadata. + */ + if (status == Camera3RequestDescriptor::Status::Cancelled) + notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_RESULT); - callbacks_->process_capture_result(callbacks_, &captureResult); + pendingRequests_.pop_front(); } } void CameraDevice::setBufferStatus(StreamBuffer &streamBuffer, - StreamBuffer::Status status) + StreamBuffer::Status status) const { streamBuffer.status = status; if (status != StreamBuffer::Status::Success) { notifyError(streamBuffer.request->frameNumber_, streamBuffer.stream->camera3Stream(), CAMERA3_MSG_ERROR_BUFFER); + } +} + +void CameraDevice::sendPartialResult(Camera3ResultDescriptor *result) +{ + Camera3RequestDescriptor *request = result->request_; + + MutexLocker lock(request->resultsMutex_); + + /* + * Android requires value of metadataPackIndex follows the rules: + * + * Set to 0: Result contains no metadata (buffers only). + * Set between 1 and (MaxMetadataPackIndex - 1): Result contains metadata. + * Set to MaxMetadataPackIndex: The result of the request. + */ + uint32_t &metadataPackIndex = result->metadataPackIndex_ = 0; + if (result->resultMetadata_) { + metadataPackIndex = request->nextPartialResultIndex_++; + } + + if (metadataPackIndex > CameraCapabilities::MaxMetadataPackIndex) + LOG(HAL, Fatal) << "Partial result count exceed limitation " + << CameraCapabilities::MaxMetadataPackIndex; + + /* + * The metadataPackIndex should be sent incrementally. Since the + * function may be called from post-processing threads, ensure the + * strict sending order by the result mutex. + */ + sendCaptureResult(result); + result->complete_ = true; +} + +void CameraDevice::sendCaptureResult(Camera3ResultDescriptor *result) const +{ + LOG(HAL, Debug) << "Send result of frameNumber: " + << result->request_->frameNumber_ + << " index: " << result->metadataPackIndex_ + << " has metadata: " << (!!result->resultMetadata_) + << " has buffers " << result->buffers_.size(); + + camera3_capture_result_t captureResult = {}; - /* Also set error status on entire request descriptor. */ - streamBuffer.request->status_ = - Camera3RequestDescriptor::Status::Error; + captureResult.frame_number = result->request_->frameNumber_; + + std::vector resultBuffers; + resultBuffers.reserve(result->buffers_.size()); + + for (auto &buffer : result->buffers_) { + camera3_buffer_status status = CAMERA3_BUFFER_STATUS_ERROR; + + if (buffer->status == StreamBuffer::Status::Success) + status = CAMERA3_BUFFER_STATUS_OK; + + /* + * Pass the buffer fence back to the camera framework as + * a release fence. This instructs the framework to wait + * on the acquire fence in case we haven't done so + * ourselves for any reason. + */ + resultBuffers.push_back({ buffer->stream->camera3Stream(), + buffer->camera3Buffer, status, + -1, buffer->fence.release() }); } + + captureResult.num_output_buffers = resultBuffers.size(); + captureResult.output_buffers = resultBuffers.data(); + + if (result->resultMetadata_) + captureResult.result = result->resultMetadata_->getMetadata(); + + captureResult.partial_result = result->metadataPackIndex_; + + callbacks_->process_capture_result(callbacks_, &captureResult); } /** @@ -1354,35 +1620,28 @@ void CameraDevice::setBufferStatus(StreamBuffer &streamBuffer, * * This function is called from the post-processor's thread whenever a camera * stream has finished post processing. The corresponding entry is dropped from - * the descriptor's pendingStreamsToProcess_ map. + * the result's pendingBufferToProcess_ list. * - * If the pendingStreamsToProcess_ map is then empty, all streams requiring to - * be generated from post-processing have been completed. Mark the descriptor as - * complete using completeDescriptor() in that case. + * If the pendingBufferToProcess_ list is then empty, all streams requiring to + * be generated from post-processing have been completed. Mark the result as + * complete using completeResultDescriptor() in that case. */ void CameraDevice::streamProcessingComplete(StreamBuffer *streamBuffer, StreamBuffer::Status status) { setBufferStatus(*streamBuffer, status); + streamBuffer->dstBuffer = nullptr; - /* - * If the framebuffer is internal to CameraStream return it back now - * that we're done processing it. - */ - if (streamBuffer->internalBuffer) - streamBuffer->stream->putBuffer(streamBuffer->internalBuffer); - - Camera3RequestDescriptor *request = streamBuffer->request; - + Camera3ResultDescriptor *result = streamBuffer->result; { - MutexLocker locker(request->streamsProcessMutex_); + MutexLocker locker(result->streamsProcessMutex_); + result->pendingBuffersToProcess_.remove(streamBuffer); - request->pendingStreamsToProcess_.erase(streamBuffer->stream); - if (!request->pendingStreamsToProcess_.empty()) + if (!result->pendingBuffersToProcess_.empty()) return; } - completeDescriptor(streamBuffer->request); + tryCompleteResultDescriptor(streamBuffer->result); } std::string CameraDevice::logPrefix() const @@ -1414,6 +1673,99 @@ void CameraDevice::notifyError(uint32_t frameNumber, camera3_stream_t *stream, callbacks_->notify(callbacks_, ¬ify); } +std::unique_ptr +CameraDevice::getDynamicResultMetadata(const ControlList &metadata) const +{ + /* + * \todo Keep this in sync with the actual number of entries. + * + * Reserve capacity for the metadata larger than 4 bytes which cannot + * store in entries. + * Currently: 6 entries, 40 bytes extra capaticy. + * + * ANDROID_SENSOR_TIMESTAMP (int64) = 8 bytes + * ANDROID_SENSOR_EXPOSURE_TIME (int64) = 8 bytes + * ANDROID_SENSOR_FRAME_DURATION (int64) = 8 bytes + * ANDROID_SCALER_CROP_REGION (int32 X 4) = 16 bytes + * Total bytes for capacity: 40 + * + * Reserve more capacity for the JPEG metadata set by the post-processor. + * Currently: 8 entries, 72 bytes extra capaticy. + * + * ANDROID_JPEG_GPS_COORDINATES (double x 3) = 24 bytes + * ANDROID_JPEG_GPS_PROCESSING_METHOD (byte x 32) = 32 bytes + * ANDROID_JPEG_GPS_TIMESTAMP (int64) = 8 bytes + * ANDROID_JPEG_SIZE (int32_t) = 4 bytes + * ANDROID_JPEG_QUALITY (byte) = 1 byte + * ANDROID_JPEG_ORIENTATION (int32_t) = 4 bytes + * ANDROID_JPEG_THUMBNAIL_QUALITY (byte) = 1 byte + * ANDROID_JPEG_THUMBNAIL_SIZE (int32 x 2) = 8 bytes + * Total bytes for JPEG metadata: 72 + * + * \todo Calculate the entries and capacity by the input ControlList. + */ + std::unique_ptr resultMetadata = + std::make_unique(14, 112); + if (!resultMetadata->isValid()) { + LOG(HAL, Error) << "Failed to allocate result metadata"; + return nullptr; + } + + /* Add metadata tags reported by libcamera. */ + const auto ×tamp = metadata.get(controls::SensorTimestamp); + if (timestamp) + resultMetadata->addEntry(ANDROID_SENSOR_TIMESTAMP, *timestamp); + + const auto &pipelineDepth = metadata.get(controls::draft::PipelineDepth); + if (pipelineDepth) + resultMetadata->addEntry(ANDROID_REQUEST_PIPELINE_DEPTH, + *pipelineDepth); + + const auto &exposureTime = metadata.get(controls::ExposureTime); + if (exposureTime) + resultMetadata->addEntry(ANDROID_SENSOR_EXPOSURE_TIME, + *exposureTime * 1000ULL); + + const auto &frameDuration = metadata.get(controls::FrameDuration); + if (frameDuration) + resultMetadata->addEntry(ANDROID_SENSOR_FRAME_DURATION, + *frameDuration * 1000); + + const auto &scalerCrop = metadata.get(controls::ScalerCrop); + if (scalerCrop) { + const Rectangle &crop = *scalerCrop; + int32_t cropRect[] = { + crop.x, + crop.y, + static_cast(crop.width), + static_cast(crop.height), + }; + resultMetadata->addEntry(ANDROID_SCALER_CROP_REGION, cropRect); + } + + const auto &testPatternMode = metadata.get(controls::draft::TestPatternMode); + if (testPatternMode) + resultMetadata->addEntry(ANDROID_SENSOR_TEST_PATTERN_MODE, + *testPatternMode); + + /* + * Return the result metadata pack even is not valid: get() will return + * nullptr. + */ + if (!resultMetadata->isValid()) { + LOG(HAL, Error) << "Failed to construct result metadata"; + } + + if (resultMetadata->resized()) { + auto [entryCount, dataCount] = resultMetadata->usage(); + LOG(HAL, Info) + << "Result metadata resized: " << entryCount + << " entries and " << dataCount << " bytes used"; + } + + return resultMetadata; +} + /* * Set jpeg metadata used to generate EXIF in the JPEG post processing. */ @@ -1424,10 +1776,8 @@ void CameraDevice::generateJpegExifMetadata(Camera3RequestDescriptor *request, auto &jpegExifMetadata = buffer->jpegExifMetadata; jpegExifMetadata.emplace(StreamBuffer::JpegExifMetadata()); - jpegExifMetadata->sensorExposureTime = 0; - if (metadata.contains(controls::ExposureTime)) { - jpegExifMetadata->sensorExposureTime = metadata.get(controls::ExposureTime) * 1000ULL; - } + const int64_t exposureTime = metadata.get(controls::ExposureTime).value_or(0); + jpegExifMetadata->sensorExposureTime = exposureTime; /* * todo: Android Sensitivity = analog gain X digital gain only on sensor. @@ -1441,31 +1791,26 @@ void CameraDevice::generateJpegExifMetadata(Camera3RequestDescriptor *request, * Produce a set of fixed result metadata. */ std::unique_ptr -CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor) const +CameraDevice::getFixedResultMetadata(const CameraMetadata &settings) const { - const ControlList &metadata = descriptor.request_->metadata(); - const CameraMetadata &settings = descriptor.settings_; camera_metadata_ro_entry_t entry; bool found; /* + * \todo Retrieve metadata from corresponding libcamera controls. * \todo Keep this in sync with the actual number of entries. - * Currently: 40 entries, 156 bytes * - * Reserve more space for the JPEG metadata set by the post-processor. - * Currently: - * ANDROID_JPEG_GPS_COORDINATES (double x 3) = 24 bytes - * ANDROID_JPEG_GPS_PROCESSING_METHOD (byte x 32) = 32 bytes - * ANDROID_JPEG_GPS_TIMESTAMP (int64) = 8 bytes - * ANDROID_JPEG_SIZE (int32_t) = 4 bytes - * ANDROID_JPEG_QUALITY (byte) = 1 byte - * ANDROID_JPEG_ORIENTATION (int32_t) = 4 bytes - * ANDROID_JPEG_THUMBNAIL_QUALITY (byte) = 1 byte - * ANDROID_JPEG_THUMBNAIL_SIZE (int32 x 2) = 8 bytes - * Total bytes for JPEG metadata: 82 + * Reserve capacity for the metadata larger than 4 bytes which cannot + * store in entries. + * Currently: 31 entries, 16 bytes + * + * ANDROID_CONTROL_AE_TARGET_FPS_RANGE (int32 X 2) = 8 bytes + * ANDROID_SENSOR_ROLLING_SHUTTER_SKEW (int64) = 8 bytes + * + * Total bytes: 16 */ std::unique_ptr resultMetadata = - std::make_unique(88, 166); + std::make_unique(31, 16); if (!resultMetadata->isValid()) { LOG(HAL, Error) << "Failed to allocate result metadata"; return nullptr; @@ -1563,9 +1908,6 @@ CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor) cons resultMetadata->addEntry(ANDROID_LENS_OPTICAL_STABILIZATION_MODE, value); - value32 = ANDROID_SENSOR_TEST_PATTERN_MODE_OFF; - resultMetadata->addEntry(ANDROID_SENSOR_TEST_PATTERN_MODE, value32); - value = ANDROID_STATISTICS_FACE_DETECT_MODE_OFF; resultMetadata->addEntry(ANDROID_STATISTICS_FACE_DETECT_MODE, value); @@ -1587,40 +1929,6 @@ CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor) cons resultMetadata->addEntry(ANDROID_SENSOR_ROLLING_SHUTTER_SKEW, rolling_shutter_skew); - /* Add metadata tags reported by libcamera. */ - const int64_t timestamp = metadata.get(controls::SensorTimestamp).value_or(0); - resultMetadata->addEntry(ANDROID_SENSOR_TIMESTAMP, timestamp); - - const auto &pipelineDepth = metadata.get(controls::draft::PipelineDepth); - if (pipelineDepth) - resultMetadata->addEntry(ANDROID_REQUEST_PIPELINE_DEPTH, - *pipelineDepth); - - const auto &exposureTime = metadata.get(controls::ExposureTime); - if (exposureTime) - resultMetadata->addEntry(ANDROID_SENSOR_EXPOSURE_TIME, - *exposureTime * 1000ULL); - - const auto &frameDuration = metadata.get(controls::FrameDuration); - if (frameDuration) - resultMetadata->addEntry(ANDROID_SENSOR_FRAME_DURATION, - *frameDuration * 1000); - - const auto &scalerCrop = metadata.get(controls::ScalerCrop); - if (scalerCrop) { - const Rectangle &crop = *scalerCrop; - int32_t cropRect[] = { - crop.x, crop.y, static_cast(crop.width), - static_cast(crop.height), - }; - resultMetadata->addEntry(ANDROID_SCALER_CROP_REGION, cropRect); - } - - const auto &testPatternMode = metadata.get(controls::draft::TestPatternMode); - if (testPatternMode) - resultMetadata->addEntry(ANDROID_SENSOR_TEST_PATTERN_MODE, - *testPatternMode); - /* * Return the result metadata pack even is not valid: get() will return * nullptr. diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 7b279895..613e419f 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -20,6 +20,7 @@ #include #include +#include #include #include #include @@ -29,7 +30,6 @@ #include "camera_capabilities.h" #include "camera_metadata.h" #include "camera_stream.h" -#include "jpeg/encoder.h" class Camera3RequestDescriptor; struct CameraConfigData; @@ -63,6 +63,8 @@ public: const camera_metadata_t *constructDefaultRequestSettings(int type); int configureStreams(camera3_stream_configuration_t *stream_list); int processCaptureRequest(camera3_capture_request_t *request); + void partialResultComplete(libcamera::Request *request, + libcamera::Result *result); void requestComplete(libcamera::Request *request); void streamProcessingComplete(StreamBuffer *bufferStream, StreamBuffer::Status status); @@ -87,20 +89,25 @@ private: createFrameBuffer(const buffer_handle_t camera3buffer, libcamera::PixelFormat pixelFormat, const libcamera::Size &size); - void abortRequest(Camera3RequestDescriptor *descriptor) const; bool isValidRequest(camera3_capture_request_t *request) const; void notifyShutter(uint32_t frameNumber, uint64_t timestamp); void notifyError(uint32_t frameNumber, camera3_stream_t *stream, camera3_error_msg_code code) const; int processControls(Camera3RequestDescriptor *descriptor); - void completeDescriptor(Camera3RequestDescriptor *descriptor) - LIBCAMERA_TSA_EXCLUDES(descriptorsMutex_); - void sendCaptureResults() LIBCAMERA_TSA_REQUIRES(descriptorsMutex_); - void setBufferStatus(StreamBuffer &buffer, StreamBuffer::Status status); + void abortRequest(Camera3RequestDescriptor *descriptor); + void tryCompleteResultDescriptor(Camera3ResultDescriptor *result); + void completeResultDescriptor(Camera3ResultDescriptor *result); + void completeRequestDescriptor(Camera3RequestDescriptor *descriptor); + void sendPartialResult(Camera3ResultDescriptor *result); + void sendCaptureResult(Camera3ResultDescriptor *partialResult) const; + void setBufferStatus(StreamBuffer &buffer, StreamBuffer::Status status) const; void generateJpegExifMetadata(Camera3RequestDescriptor *request, StreamBuffer *buffer) const; - std::unique_ptr getResultMetadata( - const Camera3RequestDescriptor &descriptor) const; + + std::unique_ptr getDynamicResultMetadata( + const libcamera::ControlList &metadata) const; + std::unique_ptr getFixedResultMetadata( + const CameraMetadata &settings) const; unsigned int id_; camera3_device_t camera3Device_; @@ -117,9 +124,16 @@ private: std::vector streams_; - libcamera::Mutex descriptorsMutex_ LIBCAMERA_TSA_ACQUIRED_AFTER(stateMutex_); - std::queue> descriptors_ - LIBCAMERA_TSA_GUARDED_BY(descriptorsMutex_); + libcamera::Mutex pendingRequestMutex_ LIBCAMERA_TSA_ACQUIRED_AFTER(stateMutex_); + + std::list> pendingRequests_ + LIBCAMERA_TSA_GUARDED_BY(pendingRequestMutex_); + + std::list pendingResults_ + LIBCAMERA_TSA_GUARDED_BY(pendingRequestMutex_); + + std::map> pendingStreamBuffers_ + LIBCAMERA_TSA_GUARDED_BY(pendingRequestMutex_); std::string maker_; std::string model_; diff --git a/src/android/camera_request.cpp b/src/android/camera_request.cpp index f5d4d314..ab829760 100644 --- a/src/android/camera_request.cpp +++ b/src/android/camera_request.cpp @@ -7,9 +7,11 @@ #include "camera_request.h" +#include #include #include "camera_buffer.h" +#include "camera_stream.h" using namespace libcamera; @@ -42,25 +44,25 @@ using namespace libcamera; * │ processCaptureRequest(camera3_capture_request_t request) │ * │ │ * │ - Create Camera3RequestDescriptor tracking this request │ - * │ - Streams requiring post-processing are stored in the │ - * │ pendingStreamsToProcess map │ + * │ - Buffer requiring post-processing are marked by the │ + * │ CameraStream::Type as Mapped or Internal │ * │ - Add this Camera3RequestDescriptor to descriptors' queue │ * │ CameraDevice::descriptors_ │ - * │ │ ┌─────────────────────────┐ - * │ - Queue the capture request to libcamera core ────────────┤►│libcamera core │ - * │ │ ├─────────────────────────┤ - * │ │ │- Capture from Camera │ - * │ │ │ │ - * │ │ │- Emit │ - * │ │ │ Camera::requestComplete│ - * │ requestCompleted(Request *request) ◄───────────────────────┼─┼──── │ - * │ │ │ │ - * │ - Check request completion status │ └─────────────────────────┘ + * │ │ ┌───────────────────────────────┐ + * │ - Queue the capture request to libcamera core ────────────┤►│libcamera core │ + * │ │ ├───────────────────────────────┤ + * │ │ │- Capture from Camera │ + * │ │ │ │ + * │ │ │- Emit │ + * │ │ │ Camera::partialResultComplete│ + * │ partialResultComplete(Request *request, Result result*) ◄──┼─┼──── │ + * │ │ │ │ + * │ - Check request completion status │ └───────────────────────────────┘ * │ │ - * │ - if (pendingStreamsToProcess > 0) │ - * │ Queue all entries from pendingStreamsToProcess │ + * │ - if (pendingBuffersToProcess > 0) │ + * │ Queue all entries from pendingBuffersToProcess │ * │ else │ │ - * │ completeDescriptor() │ └──────────────────────┐ + * │ completeResultDescriptor() │ └──────────────────────┐ * │ │ │ * │ ┌──────────────────────────┴───┬──────────────────┐ │ * │ │ │ │ │ @@ -93,10 +95,10 @@ using namespace libcamera; * │ | | | | │ * │ | - Check and set buffer status | | .... | │ * │ | - Remove post+processing entry | | | │ - * │ | from pendingStreamsToProcess | | | │ + * │ | from pendingBuffersToProcess | | | │ * │ | | | | │ - * │ | - if (pendingStreamsToProcess.empty())| | | │ - * │ | completeDescriptor | | | │ + * │ | - if (pendingBuffersToProcess.empty())| | | │ + * │ | completeResultDescriptor | | | │ * │ | | | | │ * │ +---------------------------------------+ +--------------+ │ * │ │ @@ -110,6 +112,7 @@ using namespace libcamera; Camera3RequestDescriptor::Camera3RequestDescriptor( Camera *camera, const camera3_capture_request_t *camera3Request) + : status_(Status::Pending), nextPartialResultIndex_(1), completed_(false) { frameNumber_ = camera3Request->frame_number; @@ -138,7 +141,27 @@ Camera3RequestDescriptor::Camera3RequestDescriptor( request_ = camera->createRequest(reinterpret_cast(this)); } -Camera3RequestDescriptor::~Camera3RequestDescriptor() = default; +Camera3RequestDescriptor::~Camera3RequestDescriptor() +{ + /* + * Recycle the allocated internal buffer back to its source stream. + */ + for (auto &[sourceStream, frameBuffer] : internalBuffers_) + sourceStream->putBuffer(frameBuffer); +} + +/* + * \class Camera3ResultDescriptor + * + * A utility class that groups information about a capture result to be later + * sent to framework. + */ +Camera3ResultDescriptor::Camera3ResultDescriptor(Camera3RequestDescriptor *request) + : request_(request), metadataPackIndex_(1), complete_(false) +{ +} + +Camera3ResultDescriptor::~Camera3ResultDescriptor() = default; /** * \class StreamBuffer diff --git a/src/android/camera_request.h b/src/android/camera_request.h index f91de955..fd8dad24 100644 --- a/src/android/camera_request.h +++ b/src/android/camera_request.h @@ -24,6 +24,7 @@ class CameraBuffer; class CameraStream; +class Camera3ResultDescriptor; class Camera3RequestDescriptor; class StreamBuffer @@ -52,45 +53,74 @@ public: std::unique_ptr frameBuffer; libcamera::UniqueFD fence; Status status = Status::Success; - libcamera::FrameBuffer *internalBuffer = nullptr; const libcamera::FrameBuffer *srcBuffer = nullptr; std::unique_ptr dstBuffer; std::optional jpegExifMetadata; + Camera3ResultDescriptor *result; Camera3RequestDescriptor *request; private: LIBCAMERA_DISABLE_COPY(StreamBuffer) }; +class Camera3ResultDescriptor +{ +public: + Camera3ResultDescriptor(Camera3RequestDescriptor *request); + ~Camera3ResultDescriptor(); + + Camera3RequestDescriptor *request_; + uint32_t metadataPackIndex_; + + std::unique_ptr resultMetadata_; + std::vector buffers_; + + /* Keeps track of buffers waiting for post-processing. */ + std::list pendingBuffersToProcess_ + LIBCAMERA_TSA_GUARDED_BY(streamsProcessMutex_); + libcamera::Mutex streamsProcessMutex_; + + bool complete_; + +private: + LIBCAMERA_DISABLE_COPY(Camera3ResultDescriptor) +}; + class Camera3RequestDescriptor { public: enum class Status { + Pending, Success, - Error, + Cancelled, + Aborted, }; - /* Keeps track of streams requiring post-processing. */ - std::map pendingStreamsToProcess_ - LIBCAMERA_TSA_GUARDED_BY(streamsProcessMutex_); - libcamera::Mutex streamsProcessMutex_; - Camera3RequestDescriptor(libcamera::Camera *camera, const camera3_capture_request_t *camera3Request); ~Camera3RequestDescriptor(); - bool isPending() const { return !complete_; } - uint32_t frameNumber_ = 0; std::vector buffers_; - CameraMetadata settings_; + std::unique_ptr request_; - std::unique_ptr resultMetadata_; - bool complete_ = false; - Status status_ = Status::Success; + std::map internalBuffers_; + + Status status_ + LIBCAMERA_TSA_GUARDED_BY(resultsMutex_); + uint32_t nextPartialResultIndex_ + LIBCAMERA_TSA_GUARDED_BY(resultsMutex_); + std::unique_ptr finalResult_ + LIBCAMERA_TSA_GUARDED_BY(resultsMutex_); + std::vector> partialResults_ + LIBCAMERA_TSA_GUARDED_BY(resultsMutex_); + + libcamera::Mutex resultsMutex_; + + bool completed_; private: LIBCAMERA_DISABLE_COPY(Camera3RequestDescriptor) diff --git a/src/android/camera_stream.cpp b/src/android/camera_stream.cpp index 02dc8922..3215025c 100644 --- a/src/android/camera_stream.cpp +++ b/src/android/camera_stream.cpp @@ -187,6 +187,7 @@ int CameraStream::process(StreamBuffer *streamBuffer) *streamBuffer->camera3Buffer, output.pixelFormat, output.size, PROT_READ | PROT_WRITE); if (!streamBuffer->dstBuffer->isValid()) { + streamBuffer->dstBuffer = nullptr; LOG(HAL, Error) << "Failed to create destination buffer"; return -EINVAL; } diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp index 7e3713f5..d61c566b 100644 --- a/src/android/jpeg/post_processor_jpeg.cpp +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -111,7 +111,7 @@ void PostProcessorJpeg::process(StreamBuffer *streamBuffer) ASSERT(jpegExifMetadata.has_value()); const CameraMetadata &requestMetadata = streamBuffer->request->settings_; - CameraMetadata *resultMetadata = streamBuffer->request->resultMetadata_.get(); + CameraMetadata *resultMetadata = streamBuffer->result->resultMetadata_.get(); camera_metadata_ro_entry_t entry; int ret; From patchwork Fri Aug 12 09:08:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 17093 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B295DC3272 for ; Fri, 12 Aug 2022 09:09:05 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 651E36333B; Fri, 12 Aug 2022 11:09:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660295345; bh=hj4y2etmQvTa2zfgoufeaZVpycvrNRjZWyY8FheEp7Q=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=Q1Tt6u3qiECT5VO0JWvWymxQz/MtTCYqhqyRyvHYR4Lz2p2MFqs7lzSXRJqEd+J/y Qokw/Fw3ReptJM1JgPkXEVSXr6piTkbaqKIBsl+/ujcGV1urb1Jy57QWkSQzT7VCPn 9g4b+903MqPRAl1fO/wpz8g7j6dZyFMYiRBpSFmyAPSJkrBV2rGqA0qrn65YtxIbbJ wq7FS/dw6AdgwvNFybUQ452rd7AMsJAAkC2SC6ptK+XxJz2vCbAMTdWuyZlpDYc+Ck /sWSMc2j+7JaN/LrXKoFWIQnxzLXzvulqFK6D1TlgrKExIrCktW+onS9uBhihvvyEw E2EbHBxHbOkng== Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1B7C363326 for ; Fri, 12 Aug 2022 11:09:03 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="hefMcY0d"; dkim-atps=neutral Received: by mail-pl1-x62d.google.com with SMTP id 17so241447plj.10 for ; Fri, 12 Aug 2022 02:09:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=OsOnH9HZxPNXSKbjxXufN+4s2HjN2Z6vEES5lBQtYVI=; b=hefMcY0dniXZBVss2fHzjA19utSHHbzNF+FNI2l3VMHJypH++ihczLWhMnjh6KpXOR zSUoRhmmxKAa2LnNUsMNga+Kj1mC3/ai/XC6Pnu0zsHDlqJgoajO/FiuZZ/+X6gdAxHD DhKwn3CywDisLMemxf3NrBrqpdG3xziJTDpo0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=OsOnH9HZxPNXSKbjxXufN+4s2HjN2Z6vEES5lBQtYVI=; b=2E4ysklOkqmUlWQ4qJKTQ92KEezvoOIoWZ7sGIkxME/LoXX2itNirL5SCChB+qdYvw yeCHHEhoumfub2+jVtyhO1oBhZCkkt8HMrKMH/WRBG37gd2EWDk4FtmFQuQ0okYcHCU2 FO4UYwGH+c01uGKj1wGYxW5//Zd4cL9d5CxwSMeRV/TfgYH4lD5BE5emqM4VwG1tCM5p 3QjZ+4fl3iH7lTvjNnW2eORkAEGNakFWLP36nGgl70JGQ7WH8CyEfZSpILK4nUUKwHuQ 6+9nlCvSXQQvf01wTnbJBov+sWz74mwknvIQCtTV50bVEg8BErq4LCi9LO06voXIPUQy 0c6w== X-Gm-Message-State: ACgBeo0av7evGM8N3T6LfbsIcAz7wdDn54Uognhu9RQe10mDIXxh5rjh kkdFu0XzbnLiJl9vM+znBJeia81v6sqwoQ== X-Google-Smtp-Source: AA6agR7sqZoBcoyQnTDioXD9uMk3ZTQvgfV9rc6hotlhk47PMJQS2tVUVYnOsEKSqvxZQSsjx3CayA== X-Received: by 2002:a17:902:c405:b0:170:88b3:c108 with SMTP id k5-20020a170902c40500b0017088b3c108mr3112877plk.172.1660295341554; Fri, 12 Aug 2022 02:09:01 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:1705:d284:d114:2e24]) by smtp.gmail.com with UTF8SMTPSA id 125-20020a620683000000b0052d3ea78858sm1082031pfg.127.2022.08.12.02.09.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 12 Aug 2022 02:09:01 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Fri, 12 Aug 2022 17:08:38 +0800 Message-Id: <20220812090838.1784703-8-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog In-Reply-To: <20220812090838.1784703-1-hanlinchen@chromium.org> References: <20220812090838.1784703-1-hanlinchen@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 7/7] android: function flush() waits for all requests completed before return X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Android requires that flush() only returns when there are no more outstanding buffers or requests left in the HAL. Add a contition variable to wait for all pending requests are completed before returning the call. Signed-off-by: Han-Lin Chen --- src/android/camera_device.cpp | 23 ++++++++++++++++++++--- src/android/camera_device.h | 2 ++ 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index c999b198..2672a5eb 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -428,8 +428,20 @@ void CameraDevice::flush() camera_->stop(); - MutexLocker stateLock(stateMutex_); - state_ = State::Stopped; + { + MutexLocker locker(pendingRequestMutex_); + pendingRequestsCv_.wait( + locker, + [&]() LIBCAMERA_TSA_REQUIRES(pendingRequestMutex_) { + return pendingRequests_.empty(); + }); + ASSERT(pendingRequests_.empty()); + } + + { + MutexLocker stateLock(stateMutex_); + state_ = State::Stopped; + } } void CameraDevice::stop() @@ -1480,7 +1492,6 @@ void CameraDevice::completeResultDescriptor(Camera3ResultDescriptor *result) completeRequestDescriptor(result->request_); } - /** * \brief Complete the Camera3RequestDescriptor * \param[in] descriptor The Camera3RequestDescriptor @@ -1526,6 +1537,12 @@ void CameraDevice::completeRequestDescriptor(Camera3RequestDescriptor *request) pendingRequests_.pop_front(); } + + if (pendingRequests_.empty()) { + locker.unlock(); + pendingRequestsCv_.notify_one(); + return; + } } void CameraDevice::setBufferStatus(StreamBuffer &streamBuffer, diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 613e419f..aa976db2 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -135,6 +135,8 @@ private: std::map> pendingStreamBuffers_ LIBCAMERA_TSA_GUARDED_BY(pendingRequestMutex_); + libcamera::ConditionVariable pendingRequestsCv_; + std::string maker_; std::string model_;