From patchwork Tue Jul 26 18:24:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 16802 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id DE37EBE173 for ; Tue, 26 Jul 2022 18:25:30 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A7A1563315; Tue, 26 Jul 2022 20:25:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1658859930; bh=ltpFuCAuihvbOmL8Fyz176w5YjlXklIGUIsQYWVnMoM=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=LCuH9XH5VrCbNoSAotXoqMajbAsX1wCaLVjhvHR9PNh+HOuo7agItAIk/HBZtamdn ty/lXoXG02Pwf/oc3/h6505Nc3qwAQ/G2bmI/9ffqVwyHOoUrLvBeYFX/8RJD+RlGR xOdIuxAfEiaz3pDqmt0TJthzxSE0/8K1mLIpJnEboxW0j1I8J9sFa7JfG93HR2OBGT QzD4m3P1HFUcSf15lgDfQd+j4BlHWE4w8j8Jw5d6RWJobfLlVvb/X0n/0WND/CPlsl duXP1sTMDe0KM7lIbx8zwR3HNJEk/WQvXTmXaiaxKKOIIoGwqRHxxqIWPBIeH8neP0 LtSP/tjDZVfwQ== Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C819360487 for ; Tue, 26 Jul 2022 20:25:28 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="jwUXHmJ2"; dkim-atps=neutral Received: by mail-pl1-x62b.google.com with SMTP id x1so12319610plb.3 for ; Tue, 26 Jul 2022 11:25:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uIpXw/38Bee1gJ74s8fAiVV6faeIlVK6IkLgUHBeyLs=; b=jwUXHmJ2UgLxCtpqOwO2/WU3Q1UZDl3KLugBHcXPf8L1ssySEkpZf5Lg5GcOonEgIF N9e4SJZEYhBdjLP51Dnb5pYw1XiWXBX4WaKtjmEbOt/s/RXCIOag58Gm79/kgBgcaQCj vZ+UDLe+NtkzjRYWmi2biB1wL9MlbvuaUCNho= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uIpXw/38Bee1gJ74s8fAiVV6faeIlVK6IkLgUHBeyLs=; b=nH41PQ9o3rjV2LDZszWh9fnV2Xx8ha+wi6R1oAyFZfrjq63Cqf865WRJr2f/NYnVBR GxOdliSZmyXQScP1CLrDBaskG8gGDz4HySOkDLY2WO1UE0EgXmhm2qpkxNZ5remeHeZE TvTrkZb1J6z2rYhS2qjY7KweibIF+lYs/3I8R1Dt7xp3MDaHOFD/0PqsBPcYdE/A9Pk3 ATJmLYLPrq8pqzEiB+2DLGYY7ltY/EBwrzErhlSVfawjD7B6KAoF1vpjypZEhdgoILwR 7LAaOCXA7ZDCkVCu//xUaNquZkD7SLgIkmqINWLzaI8n6bW70cywTouLDWx/+Bj5k/UO pC9g== X-Gm-Message-State: AJIora/plJKZxpOKztTHUvWf9Z/mFo05E4ONJzquAtYbpf+MZJU8d3ye 4p+QupQ1JLZlTYt16YIp5vVSAHt6y3dlMw== X-Google-Smtp-Source: AGRyM1tuukfknGyAQ7pxixW71f0OyFJdX2DWXPSOk2eWB28Gr3Z+31ESTv4EFRqfqZVMR6AuX6XO0g== X-Received: by 2002:a17:902:eb8f:b0:16d:4215:5f23 with SMTP id q15-20020a170902eb8f00b0016d42155f23mr18258091plg.70.1658859927170; Tue, 26 Jul 2022 11:25:27 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:739:6574:4032:5911]) by smtp.gmail.com with UTF8SMTPSA id y14-20020a17090a644e00b001f217ec21efsm13470920pjm.13.2022.07.26.11.25.26 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 26 Jul 2022 11:25:26 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Wed, 27 Jul 2022 02:24:55 +0800 Message-Id: <20220726182500.425182-2-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.359.gd136c6c3e2-goog In-Reply-To: <20220726182500.425182-1-hanlinchen@chromium.org> References: <20220726182500.425182-1-hanlinchen@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 1/6] libcamera: Camera: Add RequestCompletionMode to configure the completion order X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add enum RequestCompletionMode to Camera with two value: InSubmissionOrder and Immediately. The purpose is to allow the application configure the order of signaling requestCompleted. The InSubmissionOrder mode is the default mode which signals according to the request submission order. The Immediately mode allows the pipeline handler to signal as soon as a request is completed. Applications need to reconstruct the order by self. Signed-off-by: Han-Lin Chen --- include/libcamera/camera.h | 8 ++++ include/libcamera/internal/camera.h | 4 ++ src/libcamera/camera.cpp | 68 ++++++++++++++++++++++++++++- src/libcamera/pipeline_handler.cpp | 20 ++++++--- 4 files changed, 92 insertions(+), 8 deletions(-) diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h index 5bb06584..5aa4bf69 100644 --- a/include/libcamera/camera.h +++ b/include/libcamera/camera.h @@ -88,6 +88,11 @@ class Camera final : public Object, public std::enable_shared_from_this, LIBCAMERA_DECLARE_PRIVATE() public: + enum RequestCompletionMode { + InSubmissionOrder, + Immediately, + }; + static std::shared_ptr create(std::unique_ptr d, const std::string &id, const std::set &streams); @@ -101,6 +106,9 @@ public: int acquire(); int release(); + int setRequestCompletionMode(RequestCompletionMode order); + RequestCompletionMode requestCompletionMode() const; + const ControlInfoMap &controls() const; const ControlList &properties() const; diff --git a/include/libcamera/internal/camera.h b/include/libcamera/internal/camera.h index 597426a6..68e8d952 100644 --- a/include/libcamera/internal/camera.h +++ b/include/libcamera/internal/camera.h @@ -33,6 +33,9 @@ public: PipelineHandler *pipe() { return pipe_.get(); } + void setRequestCompletionMode(RequestCompletionMode mode); + RequestCompletionMode requestCompletionMode() const; + std::list queuedRequests_; ControlInfoMap controlInfo_; ControlList properties_; @@ -67,6 +70,7 @@ private: bool disconnected_; std::atomic state_; + RequestCompletionMode requestCompletionMode_; std::unique_ptr validator_; }; diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp index 2a8ef60e..4d3523e2 100644 --- a/src/libcamera/camera.cpp +++ b/src/libcamera/camera.cpp @@ -435,7 +435,8 @@ CameraConfiguration::Status CameraConfiguration::validateColorSpaces(ColorSpaceF */ Camera::Private::Private(PipelineHandler *pipe) : requestSequence_(0), pipe_(pipe->shared_from_this()), - disconnected_(false), state_(CameraAvailable) + disconnected_(false), state_(CameraAvailable), + requestCompletionMode_(Camera::InSubmissionOrder) { } @@ -573,6 +574,28 @@ void Camera::Private::setState(State state) state_.store(state, std::memory_order_release); } +/** + * \brief Set the request completion mode + * \param[in] mode The RequestCompletionMode + * + * This function set the request completion mode. + * InSubmissionOrder is the default mode. + */ +void Camera::Private::setRequestCompletionMode(RequestCompletionMode mode) +{ + requestCompletionMode_ = mode; +} + +/** + * \brief Get the request completion mode + * + * \return The current RequestCompletionMode. + */ +Camera::RequestCompletionMode Camera::Private::requestCompletionMode() const +{ + return requestCompletionMode_; +} + /** * \class Camera * \brief Camera device @@ -702,6 +725,15 @@ std::shared_ptr Camera::create(std::unique_ptr d, return std::shared_ptr(camera, Deleter()); } +/** + * \enum Camera::RequestCompletionMode + * \brief The mode of request completion behavior + * \var Camera::RequestCompletionMode::InSubmissionOrder + * \brief requestCompleted will be emited according to the request submission order + * \var Camera::RequestCompletionMode::Immediately + * \brief requestCompleted will be emited immediately when a request is completed. + */ + /** * \brief Retrieve the ID of the camera * @@ -1237,6 +1269,40 @@ int Camera::stop() return 0; } +/** + * \brief Set the request completion mode + * \param[in] mode The RequestCompletionMode + * + * This function set the request completion mode. + * InSubmissionOrder is the default mode. + * + * \return 0 on success or a negative error code otherwise + * \retval -EACCES The camera is running so can't change the behavior + */ +int Camera::setRequestCompletionMode(RequestCompletionMode mode) +{ + Private *const d = _d(); + + int ret = d->isAccessAllowed(Private::CameraAvailable, + Private::CameraConfigured); + if (ret < 0) + return -EACCES; + + d->setRequestCompletionMode(mode); + + return 0; +} + +/** + * \brief Get the request completion mode + * + * \return The current RequestCompletionMode. + */ +Camera::RequestCompletionMode Camera::requestCompletionMode() const +{ + return _d()->requestCompletionMode(); +} + /** * \brief Handle request completion and notify application * \param[in] request The request that has completed diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp index 67540533..7c180a8e 100644 --- a/src/libcamera/pipeline_handler.cpp +++ b/src/libcamera/pipeline_handler.cpp @@ -494,14 +494,20 @@ void PipelineHandler::completeRequest(Request *request) Camera::Private *data = camera->_d(); - while (!data->queuedRequests_.empty()) { - Request *req = data->queuedRequests_.front(); - if (req->status() == Request::RequestPending) - break; + auto iter = data->queuedRequests_.begin(); + while (iter != data->queuedRequests_.end()) { + if ((*iter)->status() != Request::RequestPending) { + ASSERT(!(*iter)->hasPendingBuffers()); + camera->requestComplete((*iter)); + iter = data->queuedRequests_.erase(iter); + continue; + } - ASSERT(!req->hasPendingBuffers()); - data->queuedRequests_.pop_front(); - camera->requestComplete(req); + /* Break at the pending result to reflect the submission order */ + if (camera->requestCompletionMode() == Camera::InSubmissionOrder) { + break; + } + iter++; } } From patchwork Tue Jul 26 18:24:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 16803 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A0A7FBE173 for ; Tue, 26 Jul 2022 18:25:32 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5DC4763318; Tue, 26 Jul 2022 20:25:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1658859932; bh=/J1uvqY5cxlHrHovY8ssTqJmGQWBuqdFF6AammIt0tk=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=ThvY191oImr5lMI1jjL/lsUUGBj2NXmJrDo9tiZwQsjCjINRTXS2ayfWnqPsgVBC5 aq+h0oqJ7zQpvwLR/r9mwO8qJ2UXYZ5vZ76xXkPzEYnqmz5QSC2mA0m4CjWichUwgf Q2mdxfyNNpej31TJqNGmROkoxtAEqRSmv1lhkCgMylgp5lvQB6K846EaBLNAj7IJeH qEVfsjiK3CYplNAjDTeca4k3cuXUTDz8lMuo7zgK0zpj5Yg1uKU046rfdaDoSftEa1 /ISTKYZBi0iVRwU98lDO6rQr+nYn+04x8lO3InevsXrfJdxk4Eww0FugpP1KTUuotu oT1fdpxNJjDlA== Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 3670363317 for ; Tue, 26 Jul 2022 20:25:31 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="XfKOWlei"; dkim-atps=neutral Received: by mail-pj1-x1029.google.com with SMTP id o5-20020a17090a3d4500b001ef76490983so14057906pjf.2 for ; Tue, 26 Jul 2022 11:25:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zl9BEdReWHmY1cqNo3JDLudkoxv111RZ4PNfY4ukwoY=; b=XfKOWlei3hAIpmQh/3s+N7zBJp1dwvvyb4L/1Uctq/HjTaTHAvfrXihaHvIQxQmqgM inD/0kmqnRvHnerGt69PVuCE9BKkjP+/SMagcZVzFRi+kbD3zqlj+80hS/lHsnnclMDQ tAI7uBXa8i7cird8hdu6DQwDkmmkey/O6Qoyc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zl9BEdReWHmY1cqNo3JDLudkoxv111RZ4PNfY4ukwoY=; b=6tNHFK0zZDx/S1u3bfZ+35fW/oYmOEFmD1Bhu4J4VYOtGS9/wihM3W5g3IhyUgpD3Y gWlauDhEhWS1neXBbkMywC3QWq5rWe+Xd9rzVahj2yQ/LW9ydTRkLGZrFMyFDABRtXD7 +LU4VGSMuuka2aJRGY0TbHObtNA/CgviwNB+pc6Vm59dxIpTnM2u0qcjfQeJ4t5GLJDA eXiCaeXTXnPYJVDeNCeXlmF4HNFYhh6eRbN567QXP14q/onmFmXMZnFcaBsa5WSdZAVR Ewc3dt3qsSji4FhPE9MtoU6zDEYWmnEG3nmcdGG4JWcjFqicSYm+EE2PZEYu5Nqn23QI yzrg== X-Gm-Message-State: AJIora844G+T/sGzKb+20W0k0h7Q/ASZYN2aZLMzW1BMJppIw0l5tWYl sFTc5AnjiBA6JAD2J+hWVoqhcKUaTQ6MJQ== X-Google-Smtp-Source: AGRyM1t49SkoqTu1YGVDI55gDC1vSe7cqV+kUqOHANCDKiAMx52X53v+SZwX8dJCkbRGfMm8iqBBvg== X-Received: by 2002:a17:90a:5aa3:b0:1f2:2bb7:aea9 with SMTP id n32-20020a17090a5aa300b001f22bb7aea9mr434836pji.197.1658859929224; Tue, 26 Jul 2022 11:25:29 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:739:6574:4032:5911]) by smtp.gmail.com with UTF8SMTPSA id ip13-20020a17090b314d00b001f2fee30a59sm904883pjb.57.2022.07.26.11.25.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 26 Jul 2022 11:25:29 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Wed, 27 Jul 2022 02:24:56 +0800 Message-Id: <20220726182500.425182-3-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.359.gd136c6c3e2-goog In-Reply-To: <20220726182500.425182-1-hanlinchen@chromium.org> References: <20220726182500.425182-1-hanlinchen@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 2/6] libcamera: Camera: Add signals for completion of metadata and partial result X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Allows pipelien handler to signal partial result completion by adding the following signals to Camera: Signal metadataCompleted; Signal partialResultCompleted; Together with the bufferCompleted signal, the pipeline handler is allowed to return buffers, partial metadata and combination of both at any stage of processing. Signed-off-by: Han-Lin Chen --- include/libcamera/camera.h | 2 + include/libcamera/internal/pipeline_handler.h | 3 + include/libcamera/request.h | 40 +++++ src/libcamera/camera.cpp | 10 ++ src/libcamera/pipeline_handler.cpp | 100 ++++++++++- src/libcamera/request.cpp | 159 ++++++++++++++++++ 6 files changed, 310 insertions(+), 4 deletions(-) diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h index 5aa4bf69..187e01d8 100644 --- a/include/libcamera/camera.h +++ b/include/libcamera/camera.h @@ -99,7 +99,9 @@ public: const std::string &id() const; + Signal partialResultCompleted; Signal bufferCompleted; + Signal metadataCompleted; Signal requestCompleted; Signal<> disconnected; diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h index c3e4c258..03d46367 100644 --- a/include/libcamera/internal/pipeline_handler.h +++ b/include/libcamera/internal/pipeline_handler.h @@ -33,6 +33,7 @@ class FrameBuffer; class MediaDevice; class PipelineHandler; class Request; +class Result; class PipelineHandler : public std::enable_shared_from_this, public Object @@ -62,7 +63,9 @@ public: void registerRequest(Request *request); void queueRequest(Request *request); + void completeMetadata(Request *request, const ControlList &metadata); bool completeBuffer(Request *request, FrameBuffer *buffer); + void completePartialResult(Request *request, Result &&result); void completeRequest(Request *request); const char *name() const { return name_; } diff --git a/include/libcamera/request.h b/include/libcamera/request.h index dffde153..ac520226 100644 --- a/include/libcamera/request.h +++ b/include/libcamera/request.h @@ -25,8 +25,43 @@ namespace libcamera { class Camera; class CameraControlValidator; class FrameBuffer; +class Request; class Stream; +class Result +{ +public: + Result(Request *request); + Result(Result &&result); + ~Result(); + + Request *request() const { return request_; } + const ControlList &metadata() const { return metadata_; } + + const std::vector &buffers() const { return buffers_; } + int addBuffer(FrameBuffer *buffer); + + template + void set(const Control &ctrl, const V &value) + { + return metadata_.set(ctrl, value); + } + + void set(unsigned int id, const ControlValue &value); + void merge(const ControlList &source); + + std::string toString() const; + +private: + LIBCAMERA_DISABLE_COPY(Result) + + Request *request_; + ControlList metadata_; + std::vector buffers_; +}; + +std::ostream &operator<<(std::ostream &out, const Result &r); + class Request : public Extensible { LIBCAMERA_DECLARE_PRIVATE() @@ -44,6 +79,7 @@ public: }; using BufferMap = std::map; + using ResultList = std::vector; Request(Camera *camera, uint64_t cookie = 0); ~Request(); @@ -53,9 +89,12 @@ public: ControlList &controls() { return *controls_; } ControlList &metadata() { return *metadata_; } const BufferMap &buffers() const { return bufferMap_; } + ResultList &resultList() { return results_; } + Result *addResult(Result &&result); int addBuffer(const Stream *stream, FrameBuffer *buffer, std::unique_ptr fence = nullptr); FrameBuffer *findBuffer(const Stream *stream) const; + const Stream *findStream(const FrameBuffer *buffer) const; uint32_t sequence() const; uint64_t cookie() const { return cookie_; } @@ -71,6 +110,7 @@ private: ControlList *controls_; ControlList *metadata_; BufferMap bufferMap_; + ResultList results_; const uint64_t cookie_; Status status_; diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp index 4d3523e2..a5fb9b33 100644 --- a/src/libcamera/camera.cpp +++ b/src/libcamera/camera.cpp @@ -765,6 +765,16 @@ const std::string &Camera::id() const * completed */ +/** + * \var Camera::metadataCompleted + * \brief Signal emitted when a partial metadata for a request is completed + */ + +/** + * \var Camera::partialResultCompleted + * \brief Signal emitted when a partial result for a request is completed + */ + /** * \var Camera::requestCompleted * \brief Signal emitted when a request queued to the camera has completed diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp index 7c180a8e..2102da58 100644 --- a/src/libcamera/pipeline_handler.cpp +++ b/src/libcamera/pipeline_handler.cpp @@ -453,9 +453,10 @@ void PipelineHandler::doQueueRequests() * \param[in] request The request the buffer belongs to * \param[in] buffer The buffer that has completed * - * This function shall be called by pipeline handlers to signal completion of + * This function could be called by pipeline handlers to signal completion of * the \a buffer part of the \a request. It notifies applications of buffer - * completion and updates the request's internal buffer tracking. The request + * completion and updates the request's internal buffer tracking. The function + * notify completion of a partial result including the buffer. The request * is not completed automatically when the last buffer completes to give * pipeline handlers a chance to perform any operation that may still be * needed. They shall complete requests explicitly with completeRequest(). @@ -466,10 +467,74 @@ void PipelineHandler::doQueueRequests() * otherwise */ bool PipelineHandler::completeBuffer(Request *request, FrameBuffer *buffer) +{ + Result result(request); + result.addBuffer(buffer); + completePartialResult(request, std::move(result)); + + return request->_d()->hasPendingBuffers(); +} + +/** + * \brief Complete part of metadata for a request + * \param[in] request The request the buffer belongs to + * \param[in] metadata The partial metadata that has completed + * + * This function could be called by pipeline handlers to signal completion of + * the \a metadata part of the \a request. It notifies applications of metadata + * completion. The function notify completion of a partial result including the + * metadata. The request is not completed automatically when the last metadata + * completes to give pipeline handlers a chance to perform any operation that + * may still be needed. They shall complete requests explicitly with + * completeRequest(). + * + * \context This function shall be called from the CameraManager thread. + */ +void PipelineHandler::completeMetadata(Request *request, const ControlList &metadata) +{ + Result result = Result(request); + result.merge(metadata); + completePartialResult(request, std::move(result)); +} + +/** + * \brief Complete part of metadata and buffer for a request + * \param[in] request The request the buffer belongs to + * \param[in] result The partial result that has completed + * + * This function could be called by pipeline handlers to signal completion of + * the \a result part of the \a request. It notifies applications of partial + * completion. The function notify completion of buffers and metadata included + * in the result. The request is not completed automatically when the last + * result completes to give pipeline handlers a chance to perform any operation + * that may still be needed. They shall complete requests explicitly with + * completeRequest(). The function only accepts rvalue of a Result type and + * its interval content will be moved to the internal store to avoid copying + * big metadata. + * + * \context This function shall be called from the CameraManager thread. + */ +void PipelineHandler::completePartialResult(Request *request, Result &&result) { Camera *camera = request->_d()->camera(); - camera->bufferCompleted.emit(request, buffer); - return request->_d()->completeBuffer(buffer); + Result *movedResult = request->addResult(std::move(result)); + + ASSERT(movedResult->request() == request); + ASSERT(!movedResult->buffers().empty() || !movedResult->metadata().empty()); + + for (auto buffer : movedResult->buffers()) { + request->_d()->completeBuffer(buffer); + camera->bufferCompleted.emit(request, buffer); + } + + if (!movedResult->metadata().empty()) { + request->metadata().merge(movedResult->metadata()); + camera->metadataCompleted.emit( + request, + &const_cast(movedResult->metadata())); + } + + camera->partialResultCompleted.emit(request, movedResult); } /** @@ -494,6 +559,33 @@ void PipelineHandler::completeRequest(Request *request) Camera::Private *data = camera->_d(); + /* + * Collect metadata which is not yet completed by the Camera, and + * create one partial result to cover the missing metadata before + * completing the whole request. This guarantees the aggregation of + * metadata in completed partial results equals to the global metadata + * in the request. + * + * \todo: Forbid merging metadata into request.metadata() directly and + * force calling completeMetadata() and completePartialResult() to + * report metadata. + */ + std::unordered_set completedMetadata; + for (auto &result : request->resultList()) { + for (auto &[id, _] : result.metadata()) + completedMetadata.insert(id); + } + + ControlList &requestMetadata = request->metadata(); + if (requestMetadata.size() > completedMetadata.size()) { + Result result(request); + for (auto &[id, value] : requestMetadata) + if (!completedMetadata.count(id)) + result.set(id, value); + + completePartialResult(request, std::move(result)); + } + auto iter = data->queuedRequests_.begin(); while (iter != data->queuedRequests_.end()) { if ((*iter)->status() != Request::RequestPending) { diff --git a/src/libcamera/request.cpp b/src/libcamera/request.cpp index d2af1d22..275722a4 100644 --- a/src/libcamera/request.cpp +++ b/src/libcamera/request.cpp @@ -323,6 +323,11 @@ void Request::Private::timeout() * \brief A map of Stream to FrameBuffer pointers */ +/** + * \typedef Request::ResultList + * \brief A list partial results associated with a request + */ + /** * \class Request * \brief A frame capture request @@ -394,6 +399,7 @@ void Request::reuse(ReuseFlag flags) status_ = RequestPending; + results_.clear(); controls_->clear(); metadata_->clear(); } @@ -424,6 +430,12 @@ void Request::reuse(ReuseFlag flags) * \return The map of Stream to FrameBuffer */ +/** + * \fn Request::resultList() + * \brief Retrieve the request's partial results + * \return A reference to the list of results that associates with the request + */ + /** * \brief Add a FrameBuffer with its associated Stream to the Request * \param[in] stream The stream the buffer belongs to @@ -489,6 +501,20 @@ int Request::addBuffer(const Stream *stream, FrameBuffer *buffer, return 0; } +/** + * \brief Add result into the internal result list + * \param[in] result The result to add into the request + * + * The function only accepts rvalue and moves its content into the request + * + * \return The result moved into the request + */ +Result *Request::addResult(Result &&result) +{ + results_.emplace_back(std::move(result)); + return &results_.back(); +} + /** * \var Request::bufferMap_ * \brief Mapping of streams to buffers for this request @@ -513,6 +539,21 @@ FrameBuffer *Request::findBuffer(const Stream *stream) const return it->second; } +/** + * \brief Return the stream associated with a buffer + * \param[in] buffer The buffer the stream is associated to + * \return The stream associated with the buffer, or nullptr if the buffer is + * not part of this request + */ +const Stream *Request::findStream(const FrameBuffer *buffer) const +{ + for (auto &[key, value] : bufferMap_) + if (buffer == value) + return key; + + return nullptr; +} + /** * \fn Request::metadata() * \brief Retrieve the request's metadata @@ -606,4 +647,122 @@ std::ostream &operator<<(std::ostream &out, const Request &r) return out; } +/** + * \class Result + * \brief A partial result of a frame capture request + * + * A Result allows pipeline handler to report partial results to the application + */ + +/** + * \brief Create a partial result for a capture request + * \param[in] request The request the result associated to + */ +Result::Result(Request *request) + : request_(request) +{ +} + +/** + * \brief Move constructor of a Result + * \param[in] result The other result + */ +Result::Result(Result &&result) = default; + +Result::~Result() = default; + +/** + * \fn Result::request() + * \brief Retrieve the result's associated request + * \return The Request pointer associated with the result + */ + +/** + * \fn Result::metadata() + * \brief Retrieve the result's metadata + * \return The metadata contained in the result + */ + +/** + * \fn Result::buffers() + * \brief Retrieve the result's buffers + * \return The buffers contained in the result + */ + +/** + * \fn Result::set(const Control &ctrl, const V &value) + * \brief Set the control \a ctrl value to \a value into metadata + * \param[in] ctrl The control + * \param[in] value The control value + */ + +/** + * \brief Add a FrameBuffer with its associated Stream to the Result + * \param[in] buffer The FrameBuffer to add to the result + * + * \return 0 on success or a negative error code otherwise + * \retval -EINVAL The buffer does not reference a valid Stream in the request + */ +int Result::addBuffer(FrameBuffer *buffer) +{ + if (!buffer || !request_->findStream(buffer)) { + LOG(Request, Error) << "Invalid buffer reference"; + return -EINVAL; + } + + buffers_.emplace_back(buffer); + return 0; +} + +/** + * \brief Set the control \a id to \a value + * \param[in] id The control id + * \param[in] value The control value + */ +void Result::set(unsigned int id, const ControlValue &value) +{ + metadata_.set(id, value); +} + +/** + * \brief Merge the \a source into the metadata of the result + * \param[in] source The ControlList to merge into this metadata + */ +void Result::merge(const ControlList &source) +{ + metadata_.merge(source); +} + +/** + * \brief Generate a string representation of the Result internals + * + * This function facilitates debugging of Result state while it is used + * internally within libcamera. + * + * \return A string representing the current state of the result + */ +std::string Result::toString() const +{ + std::stringstream ss; + ss << *this; + + return ss.str(); +} + +/** + * \brief Insert a text representation of a Result into an output stream + * \param[in] out The output stream + * \param[in] r The Result + * \return The output stream \a out + */ +std::ostream &operator<<(std::ostream &out, const Result &r) +{ + /* Example Output: Result(55:1/2) */ + out << "Result(" << r.request()->sequence() << ":" + << r.buffers().size() << "/" + << const_cast(r).metadata().size() << ")"; + + return out; +} + } /* namespace libcamera */ From patchwork Tue Jul 26 18:24:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 16804 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 3863EBE173 for ; Tue, 26 Jul 2022 18:25:35 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E8C8A63311; Tue, 26 Jul 2022 20:25:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1658859934; bh=R9uHT+BvZkLOJpSdkQmcQIOOfX+SstO0o9lsIOOuDww=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=YIMdY+piOA+JVVFH1eYcZLvCxYyL6n0LEfiv+WmlTt9sLiOhE+Uu5HDrNRhcQwjgB QPgCryria2syzucOE7mrLZXVYzBSZHRzS0wp2umzlfnB+a7gSOve79KF90aNweJNg/ w11z9ayjmu3FSlYbyk/bBBHXBWyvJpcdDmxnEdktu7cISsw4S8PQYSML0YpCqYi5MT 22t4Qw2WUozO2HgsAk25mBK3DfqzfVwjA6r/F8wYCGeJkAf7x46x6NoasFPWzaftl9 VGhYrv075+PaS2jtYm4PC4ScuhVHi7jSyM+jWjN3X+GM+5sQGaa4obChliABSFUuyY 6DRFZh7sHNLhg== Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com [IPv6:2607:f8b0:4864:20::535]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B7BA260487 for ; Tue, 26 Jul 2022 20:25:32 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="abjCqlL1"; dkim-atps=neutral Received: by mail-pg1-x535.google.com with SMTP id s206so13842001pgs.3 for ; Tue, 26 Jul 2022 11:25:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=T5PVKcclvqX2gU0lr0MrsXleBfpUJTOUsV+0PEKgZTM=; b=abjCqlL1l/5uWSGuz7Mbr/0BjpgthMODm6RLCTE/U12iseeLl9vzvhClh3rJ5o0GKI grEUNyUwuPzaPI9cuHSUPYxMvwq/Z+4BA6JF6RzBtH5AJ4Py2YaauD06TV/Jx8gfAg+8 xOV5Xfn08AYGLIYKAk3Xqq+hUI6WA39YphI1I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=T5PVKcclvqX2gU0lr0MrsXleBfpUJTOUsV+0PEKgZTM=; b=U1nUHmNQ/5WRmiFT1QTCjR+mNeoTEH2PQtbZ6RhGSfDgyhYqHGKQl+JaMdHO7vr+Vb 6IZYAbHJtYWkE5rDUwDCIonnyo+LD0n4zkhiu1/VYJzyD2covw/iD7D3CxpEIWMXvAb4 t0mmlfloPuEMuj+vXUZ05L9yF5Y3YAkMBwrT30TXYt7rirVIwJ/Y0G2y+MWRs5L56sym DFNXwNT4J/L6+9U0SyfkrQjC200KrYYYFXrhDDE4iTNlZ78ziA8oL6DR8C7EbN2KaTBv azj3M3vWu/XGH5GjFaqb+/OhE7iw5XXRio8TA+RKvudvwvLFSG8NNlArcqOwuR+s9cwZ rVcw== X-Gm-Message-State: AJIora9H7OdVAwPCbPuEMjV4/TEeVWc1H93DwEuw/K/ZJBidlBdxe7gZ o9ebac9rkPU8ioyziTdFrV7Qs2cYC168gA== X-Google-Smtp-Source: AGRyM1uFfZ3ZQtulOHbsxYDymmtiODqJfWvc9k9bZE+GsqBhtY0OeJ5Ud1eReUQ41jgNVpqoYmINZQ== X-Received: by 2002:a63:d209:0:b0:41a:19a5:f8c0 with SMTP id a9-20020a63d209000000b0041a19a5f8c0mr15298388pgg.565.1658859931204; Tue, 26 Jul 2022 11:25:31 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:739:6574:4032:5911]) by smtp.gmail.com with UTF8SMTPSA id f17-20020a17090ace1100b001f0f132da9fsm11230722pju.5.2022.07.26.11.25.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 26 Jul 2022 11:25:31 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Wed, 27 Jul 2022 02:24:57 +0800 Message-Id: <20220726182500.425182-4-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.359.gd136c6c3e2-goog In-Reply-To: <20220726182500.425182-1-hanlinchen@chromium.org> References: <20220726182500.425182-1-hanlinchen@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 3/6] libcamera: ipu3: returning partial results X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Return metadata at earlier stage. Signed-off-by: Han-Lin Chen --- src/libcamera/pipeline/ipu3/ipu3.cpp | 46 +++++++++++++++------------- 1 file changed, 25 insertions(+), 21 deletions(-) diff --git a/src/libcamera/pipeline/ipu3/ipu3.cpp b/src/libcamera/pipeline/ipu3/ipu3.cpp index 4fe52f74..75231156 100644 --- a/src/libcamera/pipeline/ipu3/ipu3.cpp +++ b/src/libcamera/pipeline/ipu3/ipu3.cpp @@ -1300,7 +1300,7 @@ void IPU3CameraData::metadataReady(unsigned int id, const ControlList &metadata) return; Request *request = info->request; - request->metadata().merge(metadata); + pipe()->completeMetadata(request, metadata); info->metadataProcessed = true; if (frameInfos_.tryComplete(info)) @@ -1324,16 +1324,8 @@ void IPU3CameraData::imguOutputBufferReady(FrameBuffer *buffer) return; Request *request = info->request; - pipe()->completeBuffer(request, buffer); - request->metadata().set(controls::draft::PipelineDepth, 3); - /* \todo Actually apply the scaler crop region to the ImgU. */ - const auto &scalerCrop = request->controls().get(controls::ScalerCrop); - if (scalerCrop) - cropRegion_ = *scalerCrop; - request->metadata().set(controls::ScalerCrop, cropRegion_); - if (frameInfos_.tryComplete(info)) pipe()->completeRequest(request); } @@ -1372,13 +1364,24 @@ void IPU3CameraData::cio2BufferReady(FrameBuffer *buffer) * \todo The sensor timestamp should be better estimated by connecting * to the V4L2Device::frameStart signal. */ - request->metadata().set(controls::SensorTimestamp, - buffer->metadata().timestamp); info->effectiveSensorControls = delayedCtrls_->get(buffer->metadata().sequence); - if (request->findBuffer(&rawStream_)) + ControlList metadata; + metadata.set(controls::draft::PipelineDepth, 3); + metadata.set(controls::SensorTimestamp, buffer->metadata().timestamp); + + /* \todo Actually apply the scaler crop region to the ImgU. */ + const auto &scalerCrop = request->controls().get(controls::ScalerCrop); + if (scalerCrop) + cropRegion_ = *scalerCrop; + metadata.set(controls::ScalerCrop, cropRegion_); + + pipe()->completeMetadata(request, metadata); + + if (request->findBuffer(&rawStream_)) { pipe()->completeBuffer(request, buffer); + } ipa_->fillParamsBuffer(info->id, info->paramBuffer->cookie()); } @@ -1455,20 +1458,21 @@ void IPU3CameraData::frameStart(uint32_t sequence) Request *request = processingRequests_.front(); processingRequests_.pop(); - const auto &testPatternMode = request->controls().get(controls::draft::TestPatternMode); - if (!testPatternMode) - return; + int32_t testPatternMode = controls::draft::TestPatternModeOff; + const auto &testPatternControl = request->controls().get(controls::draft::TestPatternMode); + if (testPatternControl) + testPatternMode = *testPatternControl; int ret = cio2_.sensor()->setTestPatternMode( - static_cast(*testPatternMode)); + static_cast(testPatternMode)); if (ret) { - LOG(IPU3, Error) << "Failed to set test pattern mode: " - << ret; - return; + LOG(IPU3, Error) << "Failed to set test pattern mode: " << ret; + testPatternMode = controls::draft::TestPatternModeOff; } - request->metadata().set(controls::draft::TestPatternMode, - *testPatternMode); + ControlList metadata; + metadata.set(controls::draft::TestPatternMode, testPatternMode); + pipe()->completeMetadata(request, metadata); } REGISTER_PIPELINE_HANDLER(PipelineHandlerIPU3) From patchwork Tue Jul 26 18:24:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 16805 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id A6EAEBE173 for ; Tue, 26 Jul 2022 18:25:37 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 62C9C63322; Tue, 26 Jul 2022 20:25:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1658859937; bh=rrglAyE1sjLg/L4sc56fJROPJ44W0/I+XurOJgMiSRw=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=nIOBU8P6MpRuyBJP5dF9z7vL2E/G8kqJkjqdS/noi2LCHfev2Hjpmbyj4hEDHdGNI 3EFx2GFta4EWrVWvgtjYQpDWtM1s1HHo9U/b0BcSuEmvE7qtq8Qh+JG7JjkG6lEIVO JHyRmK7LirRFRJiKAHCWrzzpGsyhal0pKG6GYYF93cP81a8Fh3Jd0HUhxV60ofPyXN xq1pdjVRl2WF1g/1oQ78jbWehvQT+9urufn0aDA9UwGCeSVgyR57c7gjYgSnteZeev kfNgtwANA8iBc1PfgjPAIo01tVWO2DLOGhhbh8XoXF64pNcavYfUPL/IecFFaC/XEo 8L/zCTjYu0jnQ== Received: from mail-pg1-x52a.google.com (mail-pg1-x52a.google.com [IPv6:2607:f8b0:4864:20::52a]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 20E816331C for ; Tue, 26 Jul 2022 20:25:35 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="DEsQ+YE/"; dkim-atps=neutral Received: by mail-pg1-x52a.google.com with SMTP id q22so8676132pgt.9 for ; Tue, 26 Jul 2022 11:25:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YZiw0+qqzwMhiekqpV6jtd2f/vQRyq8xRXlSCRdyVpg=; b=DEsQ+YE/Z4XYhBp++eJaQo3ZiHDJBJxXDYAMQfA+B52nHfXPppVbFjnvbKK1CY1Fke gFmLg3T2Bd6UTnIJ3XkpMaXz1fdGEz+ieIHX7qHZ4UKtdAPlu2c4Bt5Xsyq20rTTF8uk yfJG2zI8GkkJ+UkpKb0GJEHrc2yHiXWG6YJlc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YZiw0+qqzwMhiekqpV6jtd2f/vQRyq8xRXlSCRdyVpg=; b=HDqsgDm6BKdy1rLtXNQ9YGgJ1PhcZ3hT+5Q38ro5GWy2Q1rDp2o6P4+voiSoRs/vJE RPeOgWUsgF6NhGOleOPtmSWEG/8SjDvtfgKpFdpQ4OtlLMYYsYo3TLQty70MtDMGALKe XvDlJlwgwwr7+GMnrtHhqGAYKnTFXy23qBLnZqRu5/7nHJPbCxWeZlx3id5WguKSWtL8 8ucVU+23PqkMiRD31xpbnPdF2TfoochbsKNqv2WPgWpYToifyUYl6eAqtL2si8bzSWlA 5m+5HhBv8+zp8ca00OA1JqmhDlV5DO6nzAUiL8eO4dA228mW231vc+sAo8KlvP8bikm4 Pq3Q== X-Gm-Message-State: AJIora+ICVM9JuA2y/5JO5/r30j54AEJhd2p9JzGHJTNUTvi1Bnu5LAK WQQQUTjz1ZFbuTcwjnBM1em8FZXmLId3Xg== X-Google-Smtp-Source: AGRyM1tCpeQVrWdjjTuvz4njfgypQhhPRWN+XiqOfhs+VTagOPFQo3OVYgmgMs8OraaNjbTLAHOd5Q== X-Received: by 2002:a63:fb01:0:b0:419:699f:a0bb with SMTP id o1-20020a63fb01000000b00419699fa0bbmr15774065pgh.4.1658859933411; Tue, 26 Jul 2022 11:25:33 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:739:6574:4032:5911]) by smtp.gmail.com with UTF8SMTPSA id i1-20020aa796e1000000b0052ab54a4711sm12070780pfq.150.2022.07.26.11.25.32 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 26 Jul 2022 11:25:33 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Wed, 27 Jul 2022 02:24:58 +0800 Message-Id: <20220726182500.425182-5-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.359.gd136c6c3e2-goog In-Reply-To: <20220726182500.425182-1-hanlinchen@chromium.org> References: <20220726182500.425182-1-hanlinchen@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 4/6] android: Move StreamBuffer out of Camera3RequestDescriptor X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Move StreamBuffer out of Camera3RequestDescriptor as a refactory before applying partial result to Android adaptor. For easier adding Camera3ResultDescriptor which references both StreamBuffer and Camera3RequestDescriptor. Signed-off-by: Han-Lin Chen --- src/android/camera_device.cpp | 22 +++++----- src/android/camera_device.h | 7 ++-- src/android/camera_request.cpp | 29 +++++++------ src/android/camera_request.h | 52 ++++++++++++++---------- src/android/camera_stream.cpp | 17 ++++---- src/android/camera_stream.h | 6 +-- src/android/jpeg/post_processor_jpeg.cpp | 2 +- src/android/jpeg/post_processor_jpeg.h | 2 +- src/android/post_processor.h | 4 +- src/android/yuv/post_processor_yuv.cpp | 2 +- src/android/yuv/post_processor_yuv.h | 2 +- 11 files changed, 75 insertions(+), 70 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index b20e389b..0248f146 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -860,7 +860,7 @@ void CameraDevice::abortRequest(Camera3RequestDescriptor *descriptor) const notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_REQUEST); for (auto &buffer : descriptor->buffers_) - buffer.status = Camera3RequestDescriptor::Status::Error; + buffer.status = StreamBuffer::Status::Error; descriptor->status_ = Camera3RequestDescriptor::Status::Error; } @@ -1156,7 +1156,7 @@ void CameraDevice::requestComplete(Request *request) if (fence) buffer.fence = fence->release(); } - buffer.status = Camera3RequestDescriptor::Status::Success; + buffer.status = StreamBuffer::Status::Success; } /* @@ -1219,12 +1219,12 @@ void CameraDevice::requestComplete(Request *request) auto iter = descriptor->pendingStreamsToProcess_.begin(); while (iter != descriptor->pendingStreamsToProcess_.end()) { CameraStream *stream = iter->first; - Camera3RequestDescriptor::StreamBuffer *buffer = iter->second; + StreamBuffer *buffer = iter->second; FrameBuffer *src = request->findBuffer(stream->stream()); if (!src) { LOG(HAL, Error) << "Failed to find a source stream buffer"; - setBufferStatus(*buffer, Camera3RequestDescriptor::Status::Error); + setBufferStatus(*buffer, StreamBuffer::Status::Error); iter = descriptor->pendingStreamsToProcess_.erase(iter); continue; } @@ -1234,7 +1234,7 @@ void CameraDevice::requestComplete(Request *request) ++iter; int ret = stream->process(buffer); if (ret) { - setBufferStatus(*buffer, Camera3RequestDescriptor::Status::Error); + setBufferStatus(*buffer, StreamBuffer::Status::Error); descriptor->pendingStreamsToProcess_.erase(stream); /* @@ -1304,7 +1304,7 @@ void CameraDevice::sendCaptureResults() for (auto &buffer : descriptor->buffers_) { camera3_buffer_status status = CAMERA3_BUFFER_STATUS_ERROR; - if (buffer.status == Camera3RequestDescriptor::Status::Success) + if (buffer.status == StreamBuffer::Status::Success) status = CAMERA3_BUFFER_STATUS_OK; /* @@ -1328,11 +1328,11 @@ void CameraDevice::sendCaptureResults() } } -void CameraDevice::setBufferStatus(Camera3RequestDescriptor::StreamBuffer &streamBuffer, - Camera3RequestDescriptor::Status status) +void CameraDevice::setBufferStatus(StreamBuffer &streamBuffer, + StreamBuffer::Status status) { streamBuffer.status = status; - if (status != Camera3RequestDescriptor::Status::Success) { + if (status != StreamBuffer::Status::Success) { notifyError(streamBuffer.request->frameNumber_, streamBuffer.stream->camera3Stream(), CAMERA3_MSG_ERROR_BUFFER); @@ -1356,8 +1356,8 @@ void CameraDevice::setBufferStatus(Camera3RequestDescriptor::StreamBuffer &strea * be generated from post-processing have been completed. Mark the descriptor as * complete using completeDescriptor() in that case. */ -void CameraDevice::streamProcessingComplete(Camera3RequestDescriptor::StreamBuffer *streamBuffer, - Camera3RequestDescriptor::Status status) +void CameraDevice::streamProcessingComplete(StreamBuffer *streamBuffer, + StreamBuffer::Status status) { setBufferStatus(*streamBuffer, status); diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 64050416..9eb7221b 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -64,8 +64,8 @@ public: int configureStreams(camera3_stream_configuration_t *stream_list); int processCaptureRequest(camera3_capture_request_t *request); void requestComplete(libcamera::Request *request); - void streamProcessingComplete(Camera3RequestDescriptor::StreamBuffer *bufferStream, - Camera3RequestDescriptor::Status status); + void streamProcessingComplete(StreamBuffer *bufferStream, + StreamBuffer::Status status); protected: std::string logPrefix() const override; @@ -96,8 +96,7 @@ private: void completeDescriptor(Camera3RequestDescriptor *descriptor) LIBCAMERA_TSA_EXCLUDES(descriptorsMutex_); void sendCaptureResults() LIBCAMERA_TSA_REQUIRES(descriptorsMutex_); - void setBufferStatus(Camera3RequestDescriptor::StreamBuffer &buffer, - Camera3RequestDescriptor::Status status); + void setBufferStatus(StreamBuffer &buffer, StreamBuffer::Status status); std::unique_ptr getResultMetadata( const Camera3RequestDescriptor &descriptor) const; diff --git a/src/android/camera_request.cpp b/src/android/camera_request.cpp index 6c87adba..f5d4d314 100644 --- a/src/android/camera_request.cpp +++ b/src/android/camera_request.cpp @@ -141,7 +141,7 @@ Camera3RequestDescriptor::Camera3RequestDescriptor( Camera3RequestDescriptor::~Camera3RequestDescriptor() = default; /** - * \struct Camera3RequestDescriptor::StreamBuffer + * \class StreamBuffer * \brief Group information for per-stream buffer of Camera3RequestDescriptor * * A capture request placed to the libcamera HAL can contain multiple streams. @@ -150,35 +150,35 @@ Camera3RequestDescriptor::~Camera3RequestDescriptor() = default; * generation. The generation of the stream will depend on its type (refer to * the CameraStream::Type documentation). * - * \var Camera3RequestDescriptor::StreamBuffer::stream + * \var StreamBuffer::stream * \brief Pointer to the corresponding CameraStream * - * \var Camera3RequestDescriptor::StreamBuffer::camera3Buffer + * \var StreamBuffer::camera3Buffer * \brief Native handle to the buffer * - * \var Camera3RequestDescriptor::StreamBuffer::frameBuffer + * \var StreamBuffer::frameBuffer * \brief Encapsulate the dmabuf handle inside a libcamera::FrameBuffer for * direct streams * - * \var Camera3RequestDescriptor::StreamBuffer::fence + * \var StreamBuffer::fence * \brief Acquire fence of the buffer * - * \var Camera3RequestDescriptor::StreamBuffer::status + * \var StreamBuffer::status * \brief Track the status of the buffer * - * \var Camera3RequestDescriptor::StreamBuffer::internalBuffer + * \var StreamBuffer::internalBuffer * \brief Pointer to a buffer internally handled by CameraStream (if any) * - * \var Camera3RequestDescriptor::StreamBuffer::srcBuffer + * \var StreamBuffer::srcBuffer * \brief Pointer to the source frame buffer used for post-processing * - * \var Camera3RequestDescriptor::StreamBuffer::dstBuffer + * \var StreamBuffer::dstBuffer * \brief Pointer to the destination frame buffer used for post-processing * - * \var Camera3RequestDescriptor::StreamBuffer::request + * \var StreamBuffer::request * \brief Back pointer to the Camera3RequestDescriptor to which the StreamBuffer belongs */ -Camera3RequestDescriptor::StreamBuffer::StreamBuffer( +StreamBuffer::StreamBuffer( CameraStream *cameraStream, const camera3_stream_buffer_t &buffer, Camera3RequestDescriptor *requestDescriptor) : stream(cameraStream), camera3Buffer(buffer.buffer), @@ -186,9 +186,8 @@ Camera3RequestDescriptor::StreamBuffer::StreamBuffer( { } -Camera3RequestDescriptor::StreamBuffer::~StreamBuffer() = default; +StreamBuffer::~StreamBuffer() = default; -Camera3RequestDescriptor::StreamBuffer::StreamBuffer(StreamBuffer &&) = default; +StreamBuffer::StreamBuffer(StreamBuffer &&) = default; -Camera3RequestDescriptor::StreamBuffer & -Camera3RequestDescriptor::StreamBuffer::operator=(Camera3RequestDescriptor::StreamBuffer &&) = default; +StreamBuffer &StreamBuffer::operator=(StreamBuffer &&) = default; diff --git a/src/android/camera_request.h b/src/android/camera_request.h index 37b6ae32..5aa4eea8 100644 --- a/src/android/camera_request.h +++ b/src/android/camera_request.h @@ -24,8 +24,9 @@ class CameraBuffer; class CameraStream; +class Camera3RequestDescriptor; -class Camera3RequestDescriptor +class StreamBuffer { public: enum class Status { @@ -33,27 +34,34 @@ public: Error, }; - struct StreamBuffer { - StreamBuffer(CameraStream *stream, - const camera3_stream_buffer_t &buffer, - Camera3RequestDescriptor *request); - ~StreamBuffer(); - - StreamBuffer(StreamBuffer &&); - StreamBuffer &operator=(StreamBuffer &&); - - CameraStream *stream; - buffer_handle_t *camera3Buffer; - std::unique_ptr frameBuffer; - libcamera::UniqueFD fence; - Status status = Status::Success; - libcamera::FrameBuffer *internalBuffer = nullptr; - const libcamera::FrameBuffer *srcBuffer = nullptr; - std::unique_ptr dstBuffer; - Camera3RequestDescriptor *request; - - private: - LIBCAMERA_DISABLE_COPY(StreamBuffer) + StreamBuffer(CameraStream *stream, + const camera3_stream_buffer_t &buffer, + Camera3RequestDescriptor *request); + ~StreamBuffer(); + + StreamBuffer(StreamBuffer &&); + StreamBuffer &operator=(StreamBuffer &&); + + CameraStream *stream; + buffer_handle_t *camera3Buffer; + std::unique_ptr frameBuffer; + libcamera::UniqueFD fence; + Status status = Status::Success; + libcamera::FrameBuffer *internalBuffer = nullptr; + const libcamera::FrameBuffer *srcBuffer = nullptr; + std::unique_ptr dstBuffer; + Camera3RequestDescriptor *request; + +private: + LIBCAMERA_DISABLE_COPY(StreamBuffer) +}; + +class Camera3RequestDescriptor +{ +public: + enum class Status { + Success, + Error, }; /* Keeps track of streams requiring post-processing. */ diff --git a/src/android/camera_stream.cpp b/src/android/camera_stream.cpp index 045e6006..02dc8922 100644 --- a/src/android/camera_stream.cpp +++ b/src/android/camera_stream.cpp @@ -112,14 +112,14 @@ int CameraStream::configure() worker_ = std::make_unique(postProcessor_.get()); postProcessor_->processComplete.connect( - this, [&](Camera3RequestDescriptor::StreamBuffer *streamBuffer, + this, [&](StreamBuffer *streamBuffer, PostProcessor::Status status) { - Camera3RequestDescriptor::Status bufferStatus; + StreamBuffer::Status bufferStatus; if (status == PostProcessor::Status::Success) - bufferStatus = Camera3RequestDescriptor::Status::Success; + bufferStatus = StreamBuffer::Status::Success; else - bufferStatus = Camera3RequestDescriptor::Status::Error; + bufferStatus = StreamBuffer::Status::Error; cameraDevice_->streamProcessingComplete(streamBuffer, bufferStatus); @@ -165,7 +165,7 @@ int CameraStream::waitFence(int fence) return -errno; } -int CameraStream::process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) +int CameraStream::process(StreamBuffer *streamBuffer) { ASSERT(type_ != Type::Direct); @@ -283,7 +283,7 @@ void CameraStream::PostProcessorWorker::start() Thread::start(); } -void CameraStream::PostProcessorWorker::queueRequest(Camera3RequestDescriptor::StreamBuffer *dest) +void CameraStream::PostProcessorWorker::queueRequest(StreamBuffer *dest) { { MutexLocker lock(mutex_); @@ -306,7 +306,7 @@ void CameraStream::PostProcessorWorker::run() if (state_ != State::Running) break; - Camera3RequestDescriptor::StreamBuffer *streamBuffer = requests_.front(); + StreamBuffer *streamBuffer = requests_.front(); requests_.pop(); locker.unlock(); @@ -316,8 +316,7 @@ void CameraStream::PostProcessorWorker::run() } if (state_ == State::Flushing) { - std::queue requests = - std::move(requests_); + std::queue requests = std::move(requests_); locker.unlock(); while (!requests.empty()) { diff --git a/src/android/camera_stream.h b/src/android/camera_stream.h index 4c5078b2..c3b2e325 100644 --- a/src/android/camera_stream.h +++ b/src/android/camera_stream.h @@ -127,7 +127,7 @@ public: CameraStream *sourceStream() const { return sourceStream_; } int configure(); - int process(Camera3RequestDescriptor::StreamBuffer *streamBuffer); + int process(StreamBuffer *streamBuffer); libcamera::FrameBuffer *getBuffer(); void putBuffer(libcamera::FrameBuffer *buffer); void flush(); @@ -146,7 +146,7 @@ private: ~PostProcessorWorker(); void start(); - void queueRequest(Camera3RequestDescriptor::StreamBuffer *request); + void queueRequest(StreamBuffer *request); void flush(); protected: @@ -158,7 +158,7 @@ private: libcamera::Mutex mutex_; libcamera::ConditionVariable cv_; - std::queue requests_ + std::queue requests_ LIBCAMERA_TSA_GUARDED_BY(mutex_); State state_ LIBCAMERA_TSA_GUARDED_BY(mutex_) = State::Stopped; diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp index d72ebc3c..b9ba38ce 100644 --- a/src/android/jpeg/post_processor_jpeg.cpp +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -98,7 +98,7 @@ void PostProcessorJpeg::generateThumbnail(const FrameBuffer &source, } } -void PostProcessorJpeg::process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) +void PostProcessorJpeg::process(StreamBuffer *streamBuffer) { ASSERT(encoder_); diff --git a/src/android/jpeg/post_processor_jpeg.h b/src/android/jpeg/post_processor_jpeg.h index 98309b01..0ba3e01b 100644 --- a/src/android/jpeg/post_processor_jpeg.h +++ b/src/android/jpeg/post_processor_jpeg.h @@ -22,7 +22,7 @@ public: int configure(const libcamera::StreamConfiguration &incfg, const libcamera::StreamConfiguration &outcfg) override; - void process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) override; + void process(StreamBuffer *streamBuffer) override; private: void generateThumbnail(const libcamera::FrameBuffer &source, diff --git a/src/android/post_processor.h b/src/android/post_processor.h index 1a205b05..ac0182cd 100644 --- a/src/android/post_processor.h +++ b/src/android/post_processor.h @@ -27,7 +27,7 @@ public: virtual int configure(const libcamera::StreamConfiguration &inCfg, const libcamera::StreamConfiguration &outCfg) = 0; - virtual void process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) = 0; + virtual void process(StreamBuffer *streamBuffer) = 0; - libcamera::Signal processComplete; + libcamera::Signal processComplete; }; diff --git a/src/android/yuv/post_processor_yuv.cpp b/src/android/yuv/post_processor_yuv.cpp index ed44e6fe..de49b378 100644 --- a/src/android/yuv/post_processor_yuv.cpp +++ b/src/android/yuv/post_processor_yuv.cpp @@ -49,7 +49,7 @@ int PostProcessorYuv::configure(const StreamConfiguration &inCfg, return 0; } -void PostProcessorYuv::process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) +void PostProcessorYuv::process(StreamBuffer *streamBuffer) { const FrameBuffer &source = *streamBuffer->srcBuffer; CameraBuffer *destination = streamBuffer->dstBuffer.get(); diff --git a/src/android/yuv/post_processor_yuv.h b/src/android/yuv/post_processor_yuv.h index a7ac17c5..27cf4bbc 100644 --- a/src/android/yuv/post_processor_yuv.h +++ b/src/android/yuv/post_processor_yuv.h @@ -18,7 +18,7 @@ public: int configure(const libcamera::StreamConfiguration &incfg, const libcamera::StreamConfiguration &outcfg) override; - void process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) override; + void process(StreamBuffer *streamBuffer) override; private: bool isValidBuffers(const libcamera::FrameBuffer &source, From patchwork Tue Jul 26 18:24:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 16806 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C6A54BE173 for ; Tue, 26 Jul 2022 18:25:38 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8309D63321; Tue, 26 Jul 2022 20:25:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1658859938; bh=SGvs+XAYBHG0g8CkbMXJzzZoDY1auSkliWcYVb7uQng=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=j0i/nIrooU3OnAgnNMgujBbQpHw7k6KgsZhWtCgJJm654hxQpeHkN24s2o4IMqsLG fPKHmrYWb42sp0k3Gd9RsHC3L4Y5k36+ebGllek0GES/yCp1x1iZhSFeCKK5dobEdT cytIMEUkfe2+HpPceoc/5dXHbHZHZND4SScaG5Jc1O62w0rM2D99/F4ZaUnSDs6B81 HJRWKxatSCo4dergl1P/iEMD8C2K+GqQkUj9EqSS6Ksmqq9zGjWepVYqfc1+9eO2Ml cPpDpKggs+t94FUShVldjYXWLnJIjYpjNMgSbbzA9lG7iiqh9JfhkLQmncyPE19o8m 390ismP+uXxAQ== Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id F1B836331C for ; Tue, 26 Jul 2022 20:25:36 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="E5UHaHix"; dkim-atps=neutral Received: by mail-pj1-x1029.google.com with SMTP id 15-20020a17090a098f00b001f305b453feso293405pjo.1 for ; Tue, 26 Jul 2022 11:25:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=e1btxfjXcpjZDCkIbAtMdRbWtTDz46Fdh+JYTpzSH1s=; b=E5UHaHix21pWM+b7u6P7TDvLpYguoP9Lz5o8fxZJsuFIe9B42CxLRPrfvWhHfjnoRr pG8Fx6df/3bdX43IPWrCaLSvBUBdpgxSFsIXhxElbEpSGMWDXWQJM8Y2j4liQxlVG3yQ zC7tEJq6dPkDYO+9fFCjAfScYQQ3hEKJYYB/k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=e1btxfjXcpjZDCkIbAtMdRbWtTDz46Fdh+JYTpzSH1s=; b=A1IYF/VjNfjE9Q0HK4PXG+EVzG4J3JPDFtM/K3vibSEvHou0EnutfjZjbwPkjzr3Jx NHp8f30kUD8YPWDXITSxvtQLzi92oAKpu/d6D/0eQ4DOr1CRT/AK0iD8Lv2qkEJS3kAU 0h3PTC4/tPoMkTSTBBSLV0BdCQZ36qwFBpI9+5rPZN9wdHRS8H4J0bCvh5TmnldZhI9E 09xeibmssJHR9UVuyGXlAYrbpYfjGAeRySpdRskiILVY0DP44HufJkce70VJKyvjfu6q TABE105AoE+IIJVRMptfFhdOgoILxWrWsJWG3gjkrJ9vFhD9KrwVKn5xDzYxJc025B3z UIhQ== X-Gm-Message-State: AJIora8fNeTkLlb8LY3X/cmTaD4WO1VGUHPYK3vVV7QKRa+rBZvUh8KE tqRH14ICI69rLCCWmpDIhx1hgCMksl/Mfw== X-Google-Smtp-Source: AGRyM1s8oEy3zrWIxOO88fzd9Z5XgwhHQPsbC0euHo2sqiTAUrqvsqY418DzmRTSB3DoondqDhcp6A== X-Received: by 2002:a17:90b:3e83:b0:1f0:4233:b222 with SMTP id rj3-20020a17090b3e8300b001f04233b222mr470136pjb.218.1658859935385; Tue, 26 Jul 2022 11:25:35 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:739:6574:4032:5911]) by smtp.gmail.com with UTF8SMTPSA id w23-20020a627b17000000b0051bc5f4df1csm12039134pfc.154.2022.07.26.11.25.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 26 Jul 2022 11:25:35 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Wed, 27 Jul 2022 02:24:59 +0800 Message-Id: <20220726182500.425182-6-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.359.gd136c6c3e2-goog In-Reply-To: <20220726182500.425182-1-hanlinchen@chromium.org> References: <20220726182500.425182-1-hanlinchen@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 5/6] android: Add JpegExifMetadata to store tags setting into Exif X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" With partial result, some metadata needed to adding into Exif may be sent back to framework earlier before Jpeg post-processing. Add a type JpegExifMetadata associated with StreamBuffer to store the values, so Jpeg post-processing doesn't need to reference to current metadata for the them. Signed-off-by: Han-Lin Chen --- src/android/camera_device.cpp | 27 ++++++++++++++++++++++++ src/android/camera_device.h | 2 ++ src/android/camera_request.h | 6 ++++++ src/android/camera_stream.h | 4 ++++ src/android/jpeg/post_processor_jpeg.cpp | 13 ++++++------ 5 files changed, 46 insertions(+), 6 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index 0248f146..a14d5de9 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -1221,6 +1221,10 @@ void CameraDevice::requestComplete(Request *request) CameraStream *stream = iter->first; StreamBuffer *buffer = iter->second; + if (stream->isJpegStream()) { + generateJpegExifMetadata(descriptor, buffer); + } + FrameBuffer *src = request->findBuffer(stream->stream()); if (!src) { LOG(HAL, Error) << "Failed to find a source stream buffer"; @@ -1410,6 +1414,29 @@ void CameraDevice::notifyError(uint32_t frameNumber, camera3_stream_t *stream, callbacks_->notify(callbacks_, ¬ify); } +/* + * Set jpeg metadata used to generate EXIF in the JPEG post processing. + */ +void CameraDevice::generateJpegExifMetadata(Camera3RequestDescriptor *request, + StreamBuffer *buffer) const +{ + const ControlList &metadata = request->request_->metadata(); + auto &jpegExifMetadata = buffer->jpegExifMetadata; + jpegExifMetadata.emplace(StreamBuffer::JpegExifMetadata()); + + jpegExifMetadata->sensorExposureTime = 0; + if (metadata.contains(controls::ExposureTime)) { + jpegExifMetadata->sensorExposureTime = metadata.get(controls::ExposureTime) * 1000ULL; + } + + /* + * todo: Android Sensitivity = analog gain X digital gain only on sensor. + * Digital gain on ISP shouldn't be included. Calculate sensitivity + * accordingingly when we can differentiate the source of digital gains. + */ + jpegExifMetadata->sensorSensitivityISO = 100; +} + /* * Produce a set of fixed result metadata. */ diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 9eb7221b..7b279895 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -97,6 +97,8 @@ private: LIBCAMERA_TSA_EXCLUDES(descriptorsMutex_); void sendCaptureResults() LIBCAMERA_TSA_REQUIRES(descriptorsMutex_); void setBufferStatus(StreamBuffer &buffer, StreamBuffer::Status status); + void generateJpegExifMetadata(Camera3RequestDescriptor *request, + StreamBuffer *buffer) const; std::unique_ptr getResultMetadata( const Camera3RequestDescriptor &descriptor) const; diff --git a/src/android/camera_request.h b/src/android/camera_request.h index 5aa4eea8..f91de955 100644 --- a/src/android/camera_request.h +++ b/src/android/camera_request.h @@ -42,6 +42,11 @@ public: StreamBuffer(StreamBuffer &&); StreamBuffer &operator=(StreamBuffer &&); + struct JpegExifMetadata { + int64_t sensorExposureTime; + int32_t sensorSensitivityISO; + }; + CameraStream *stream; buffer_handle_t *camera3Buffer; std::unique_ptr frameBuffer; @@ -50,6 +55,7 @@ public: libcamera::FrameBuffer *internalBuffer = nullptr; const libcamera::FrameBuffer *srcBuffer = nullptr; std::unique_ptr dstBuffer; + std::optional jpegExifMetadata; Camera3RequestDescriptor *request; private: diff --git a/src/android/camera_stream.h b/src/android/camera_stream.h index c3b2e325..fc50d412 100644 --- a/src/android/camera_stream.h +++ b/src/android/camera_stream.h @@ -125,6 +125,10 @@ public: const libcamera::StreamConfiguration &configuration() const; libcamera::Stream *stream() const; CameraStream *sourceStream() const { return sourceStream_; } + bool isJpegStream() const + { + return camera3Stream_->format == HAL_PIXEL_FORMAT_BLOB; + } int configure(); int process(StreamBuffer *streamBuffer); diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp index b9ba38ce..91fd329c 100644 --- a/src/android/jpeg/post_processor_jpeg.cpp +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -104,8 +104,11 @@ void PostProcessorJpeg::process(StreamBuffer *streamBuffer) const FrameBuffer &source = *streamBuffer->srcBuffer; CameraBuffer *destination = streamBuffer->dstBuffer.get(); + const std::optional &jpegExifMetadata = + streamBuffer->jpegExifMetadata; ASSERT(destination->numPlanes() == 1); + ASSERT(jpegExifMetadata.has_value()); const CameraMetadata &requestMetadata = streamBuffer->request->settings_; CameraMetadata *resultMetadata = streamBuffer->request->resultMetadata_.get(); @@ -131,15 +134,13 @@ void PostProcessorJpeg::process(StreamBuffer *streamBuffer) */ exif.setTimestamp(std::time(nullptr), 0ms); - ret = resultMetadata->getEntry(ANDROID_SENSOR_EXPOSURE_TIME, &entry); - exif.setExposureTime(ret ? *entry.data.i64 : 0); + exif.setExposureTime(jpegExifMetadata->sensorExposureTime); + exif.setISO(jpegExifMetadata->sensorSensitivityISO); + ret = requestMetadata.getEntry(ANDROID_LENS_APERTURE, &entry); if (ret) exif.setAperture(*entry.data.f); - ret = resultMetadata->getEntry(ANDROID_SENSOR_SENSITIVITY, &entry); - exif.setISO(ret ? *entry.data.i32 : 100); - exif.setFlash(Exif::Flash::FlashNotPresent); exif.setWhiteBalance(Exif::WhiteBalance::Auto); @@ -152,6 +153,7 @@ void PostProcessorJpeg::process(StreamBuffer *streamBuffer) *entry.data.i64); } + std::vector thumbnail; ret = requestMetadata.getEntry(ANDROID_JPEG_THUMBNAIL_SIZE, &entry); if (ret) { const int32_t *data = entry.data.i32; @@ -163,7 +165,6 @@ void PostProcessorJpeg::process(StreamBuffer *streamBuffer) resultMetadata->addEntry(ANDROID_JPEG_THUMBNAIL_QUALITY, quality); if (thumbnailSize != Size(0, 0)) { - std::vector thumbnail; generateThumbnail(source, thumbnailSize, quality, &thumbnail); if (!thumbnail.empty()) exif.setThumbnail(thumbnail, Exif::Compression::JPEG); From patchwork Tue Jul 26 18:25:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 16807 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 4408BBE173 for ; Tue, 26 Jul 2022 18:25:42 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 00AFE63326; Tue, 26 Jul 2022 20:25:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1658859942; bh=kC/obOEbRUaRNbcrPqopvP2rrKB/+JQCu9KyLBSHlK0=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=pEKf82/5OCeaka/LCWvx/SuvtJ0I/T6ldPUabtO1YWdtddRkgonpSaKMbdumQtz00 MvP1qfl9aFqQeheJFE6yiTKts69yqGOXDIfg4MLAvnOgNto0+i6CTryy4mPg1sopXX 1icfFS5hy5Obqq783IjsY/wSbFGk6wFEND/r/6v26xeBcLGHY4Dgy4pzMe2pm/ncXY QE5ZIwW/4d61edWndIgWDp8cGbSjHEYHY9JuqHGdiON3y6yRJjrTKHsw0yUz7MHuiO LDm1Pfm6N5fuMuAlac3B+rgFmvMcfo9hGQHZ8mXXur/7v7+3oIy96p61ZvymN1gPzA 1VC7T3addQ+9Q== Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 07DEB60487 for ; Tue, 26 Jul 2022 20:25:40 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="gAuWm/Yv"; dkim-atps=neutral Received: by mail-pl1-x62f.google.com with SMTP id x7so169046pll.7 for ; Tue, 26 Jul 2022 11:25:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Q9AjxhyfkzdksonddqwU0JC18QtJBL2g9AqgOm7NYi4=; b=gAuWm/YvG2VrCEroIkL4QXy7PTqG4kFrTaE3WoV5y+076nSv3ZcvmhjVdlVBVqg01I Mn+5EUoKCNyJTO4dlk9kda6gecx5FUZZBP0T35UXf1AxJEB3jA6Gi7zjQhtol0W/CQbf +O+P2LVnz+71tuIFK8uLgfM8si1hyJFdERgFA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Q9AjxhyfkzdksonddqwU0JC18QtJBL2g9AqgOm7NYi4=; b=C9t+Ls03jeShg8x5CadzerDUDgBj9z7/RyvER05hCHhKwnfGyanKcrSas3MnGyyDG8 LBqS3i6c0DEY/+1lqr+pyMUTRCemTai9b9EEzcPKWhl64UMoHWVk7RAmvHsz/9j65OMw burBeRFpBdAJ1DHjr7u+GVKONjh0RrbJegLJDzO88K9DXUBJGDniAgdaQxWqt/rJXqpE bXGJPr6jD8LYz+R5bqGzMivDeYeA7l2AJCokBcGyrPuBnvpo5rzRo591z7D6tcX3Ib7P xjSzau1hayapu2DuLrqJti7fG8i2Kt/lzkr+9jzjeGGGPBIZYCX0aesYHWk50V08Crfd je1Q== X-Gm-Message-State: AJIora/PnU9UFnIZyjRcSplltRZCpfqRfWSHAXafeZKqLVZJNk7ANOVI BrQVMIGTOYvVa5cMtm3C1tFqszJ6XokeYw== X-Google-Smtp-Source: AGRyM1v6uXAmT/3AgvKSdiR+/UtTUev3Sjvn1spiqjDEFk/w0I/0cvlWR0sRvnJdyGqhBnmy/ffSQg== X-Received: by 2002:a17:903:124f:b0:16b:8167:e34e with SMTP id u15-20020a170903124f00b0016b8167e34emr18140779plh.52.1658859937591; Tue, 26 Jul 2022 11:25:37 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:739:6574:4032:5911]) by smtp.gmail.com with UTF8SMTPSA id e25-20020aa79819000000b0052aaff953aesm11923558pfl.115.2022.07.26.11.25.36 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 26 Jul 2022 11:25:37 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Wed, 27 Jul 2022 02:25:00 +0800 Message-Id: <20220726182500.425182-7-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.359.gd136c6c3e2-goog In-Reply-To: <20220726182500.425182-1-hanlinchen@chromium.org> References: <20220726182500.425182-1-hanlinchen@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 6/6] android: Implement partial result feature based on partialResultCompleted signal X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" To impelement partial result feature. The patch includes the following: 1. Set Camera::RequestCompletionMode to Immediately. 2. Return partial results on Camera::partialResultCompleted signal, and recycle requests on Camera::requestCompleted. 3. Refactory usage of pendingStreamsToProcess_ and starts post-processing on partial result receved. Signed-off-by: Han-Lin Chen --- src/android/camera_capabilities.cpp | 2 +- src/android/camera_capabilities.h | 2 + src/android/camera_device.cpp | 687 ++++++++++++++--------- src/android/camera_device.h | 23 +- src/android/camera_request.cpp | 50 +- src/android/camera_request.h | 46 +- src/android/jpeg/post_processor_jpeg.cpp | 2 +- 7 files changed, 522 insertions(+), 290 deletions(-) diff --git a/src/android/camera_capabilities.cpp b/src/android/camera_capabilities.cpp index 64bd8dde..6ed040d0 100644 --- a/src/android/camera_capabilities.cpp +++ b/src/android/camera_capabilities.cpp @@ -1374,7 +1374,7 @@ int CameraCapabilities::initializeStaticMetadata() staticMetadata_->addEntry(ANDROID_SCALER_CROPPING_TYPE, croppingType); /* Request static metadata. */ - int32_t partialResultCount = 1; + int32_t partialResultCount = MaxMetadataPackIndex; staticMetadata_->addEntry(ANDROID_REQUEST_PARTIAL_RESULT_COUNT, partialResultCount); diff --git a/src/android/camera_capabilities.h b/src/android/camera_capabilities.h index 6f66f221..b8fc20f1 100644 --- a/src/android/camera_capabilities.h +++ b/src/android/camera_capabilities.h @@ -23,6 +23,8 @@ class CameraCapabilities { public: + static constexpr int32_t MaxMetadataPackIndex = 64; + CameraCapabilities() = default; int initialize(std::shared_ptr camera, diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index a14d5de9..6d79fd34 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -24,9 +24,8 @@ #include #include -#include "system/graphics.h" - #include "camera_buffer.h" +#include "camera_capabilities.h" #include "camera_hal_config.h" #include "camera_ops.h" #include "camera_request.h" @@ -249,7 +248,11 @@ CameraDevice::CameraDevice(unsigned int id, std::shared_ptr camera) : id_(id), state_(State::Stopped), camera_(std::move(camera)), facing_(CAMERA_FACING_FRONT), orientation_(0) { + /* Set RequestCompletionMode to Immediately to send result early */ + camera_->setRequestCompletionMode(Camera::Immediately); + camera_->requestCompleted.connect(this, &CameraDevice::requestComplete); + camera_->partialResultCompleted.connect(this, &CameraDevice::partialResultComplete); maker_ = "libcamera"; model_ = "cameraModel"; @@ -439,7 +442,7 @@ void CameraDevice::stop() { MutexLocker descriptorsLock(descriptorsMutex_); - descriptors_ = {}; + descriptors_.clear(); } streams_.clear(); @@ -855,14 +858,30 @@ int CameraDevice::processControls(Camera3RequestDescriptor *descriptor) return 0; } -void CameraDevice::abortRequest(Camera3RequestDescriptor *descriptor) const +void CameraDevice::abortRequest(Camera3RequestDescriptor *descriptor) { - notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_REQUEST); + Camera3ResultDescriptor *result = new Camera3ResultDescriptor(descriptor); - for (auto &buffer : descriptor->buffers_) - buffer.status = StreamBuffer::Status::Error; + for (auto &buffer : descriptor->buffers_) { + setBufferStatus(buffer, StreamBuffer::Status::Error); + result->buffers_.emplace_back(&buffer); + } - descriptor->status_ = Camera3RequestDescriptor::Status::Error; + { + MutexLocker lock(descriptor->resultsMutex_); + descriptor->status_ = Camera3RequestDescriptor::Status::Flushed; + descriptor->results_.emplace_back(result); + } + + /* + * After CAMERA3_MSG_ERROR_REQUEST is notified, for a given frame, + * only process_capture_results with buffers in + * CAMERA3_BUFFER_STATUS_ERROR are allowed. No further notifys or + * process_capture_result with non-null metadata is allowed. + */ + notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_REQUEST); + + completeResultDescriptor(result); } bool CameraDevice::isValidRequest(camera3_capture_request_t *camera3Request) const @@ -962,9 +981,9 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques * to a libcamera stream. Streams of type Mapped will be handled later. * * Collect the CameraStream associated to each requested capture stream. - * Since requestedStreams is an std:set<>, no duplications can happen. + * Since requestedStreams is an std:map<>, no duplications can happen. */ - std::set requestedStreams; + std::map requestedStreams; for (const auto &[i, buffer] : utils::enumerate(descriptor->buffers_)) { CameraStream *cameraStream = buffer.stream; camera3_stream_t *camera3Stream = cameraStream->camera3Stream(); @@ -983,8 +1002,6 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques FrameBuffer *frameBuffer = nullptr; UniqueFD acquireFence; - MutexLocker lock(descriptor->streamsProcessMutex_); - switch (cameraStream->type()) { case CameraStream::Type::Mapped: /* Mapped streams will be handled in the next loop. */ @@ -1016,10 +1033,8 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques */ frameBuffer = cameraStream->getBuffer(); buffer.internalBuffer = frameBuffer; + buffer.srcBuffer = frameBuffer; LOG(HAL, Debug) << ss.str() << " (internal)"; - - descriptor->pendingStreamsToProcess_.insert( - { cameraStream, &buffer }); break; } @@ -1032,7 +1047,7 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques descriptor->request_->addBuffer(cameraStream->stream(), frameBuffer, std::move(fence)); - requestedStreams.insert(cameraStream); + requestedStreams[cameraStream] = frameBuffer; } /* @@ -1054,29 +1069,38 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques << cameraStream->configuration().pixelFormat << "]" << " (mapped)"; - MutexLocker lock(descriptor->streamsProcessMutex_); - descriptor->pendingStreamsToProcess_.insert({ cameraStream, &buffer }); - /* * Make sure the CameraStream this stream is mapped on has been * added to the request. */ CameraStream *sourceStream = cameraStream->sourceStream(); + ASSERT(sourceStream); - if (requestedStreams.find(sourceStream) != requestedStreams.end()) + ASSERT(sourceStream->type() == CameraStream::Type::Direct); + + /* + * If the buffer for the source stream is requested, use its + * framebuffer as the source buffer for post-processing. + */ + auto iter = requestedStreams.find(sourceStream); + if (iter != requestedStreams.end()) { + buffer.srcBuffer = iter->second; continue; + } /* - * If that's not the case, we need to add a buffer to the request - * for this stream. + * If that's not the case, we need to add an internal buffer + * to the request for this stream. + * + * \todo Handle the case that multiple mapped streams need to + * create one internal buffer for the same source stream. */ - FrameBuffer *frameBuffer = cameraStream->getBuffer(); + FrameBuffer *frameBuffer = sourceStream->getBuffer(); buffer.internalBuffer = frameBuffer; + buffer.srcBuffer = frameBuffer; descriptor->request_->addBuffer(sourceStream->stream(), frameBuffer, nullptr); - - requestedStreams.erase(sourceStream); } /* @@ -1098,11 +1122,9 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques Camera3RequestDescriptor *rawDescriptor = descriptor.get(); { MutexLocker descriptorsLock(descriptorsMutex_); - descriptors_.push(std::move(descriptor)); + descriptors_.push_back(std::move(descriptor)); } abortRequest(rawDescriptor); - completeDescriptor(rawDescriptor); - return 0; } @@ -1120,7 +1142,7 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques { MutexLocker descriptorsLock(descriptorsMutex_); - descriptors_.push(std::move(descriptor)); + descriptors_.push_back(std::move(descriptor)); } camera_->queueRequest(request); @@ -1128,131 +1150,238 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques return 0; } -void CameraDevice::requestComplete(Request *request) +void CameraDevice::partialResultComplete(Request *request, Result *result) { Camera3RequestDescriptor *descriptor = reinterpret_cast(request->cookie()); - /* - * Prepare the capture result for the Android camera stack. - * - * The buffer status is set to Success and later changed to Error if - * post-processing/compression fails. - */ - for (auto &buffer : descriptor->buffers_) { - CameraStream *stream = buffer.stream; + if (result->buffers().empty() && result->metadata().empty()) + LOG(HAL, Fatal) + << "Partial result should have at least buffer or metadata " + << descriptor->frameNumber_; + Camera3ResultDescriptor *camera3Result = new Camera3ResultDescriptor(descriptor); + + const ControlList &metadata = result->metadata(); + if (!metadata.empty()) { /* - * Streams of type Direct have been queued to the - * libcamera::Camera and their acquire fences have - * already been waited on by the library. - * - * Acquire fences of streams of type Internal and Mapped - * will be handled during post-processing. + * Notify shutter as soon as we have received SensorTimestamp. */ - if (stream->type() == CameraStream::Type::Direct) { - /* If handling of the fence has failed restore buffer.fence. */ - std::unique_ptr fence = buffer.frameBuffer->releaseFence(); - if (fence) - buffer.fence = fence->release(); + + const auto ×tamp = metadata.get(controls::SensorTimestamp); + if (timestamp) { + notifyShutter(descriptor->frameNumber_, *timestamp); + LOG(HAL, Debug) << "Request " << request->cookie() << " notifies shutter"; } - buffer.status = StreamBuffer::Status::Success; + + camera3Result->resultMetadata_ = getDynamicResultMetadata(metadata); } - /* - * If the Request has failed, abort the request by notifying the error - * and complete the request with all buffers in error state. - */ - if (request->status() != Request::RequestComplete) { - LOG(HAL, Error) << "Request " << request->cookie() - << " not successfully completed: " - << request->status(); + MutexLocker locker(camera3Result->streamsProcessMutex_); - abortRequest(descriptor); - completeDescriptor(descriptor); + for (auto &buffer : descriptor->buffers_) { + CameraStream *cameraStream = buffer.stream; + for (auto *frameBuffer : result->buffers()) { + if (buffer.srcBuffer != frameBuffer && + buffer.frameBuffer.get() != frameBuffer) + continue; - return; + buffer.result = camera3Result; + camera3Result->buffers_.emplace_back(&buffer); + + StreamBuffer::Status status = StreamBuffer::Status::Success; + if (frameBuffer->metadata().status != FrameMetadata::FrameSuccess) { + status = StreamBuffer::Status::Error; + } + setBufferStatus(buffer, status); + + switch (cameraStream->type()) { + case CameraStream::Type::Direct: { + ASSERT(buffer.frameBuffer.get() == frameBuffer); + /* + * Streams of type Direct have been queued to the + * libcamera::Camera and their acquire fences have + * already been waited on by the library. + */ + std::unique_ptr fence = buffer.frameBuffer->releaseFence(); + if (fence) + buffer.fence = fence->release(); + break; + } + case CameraStream::Type::Mapped: + case CameraStream::Type::Internal: + ASSERT(buffer.srcBuffer == frameBuffer); + if (status == StreamBuffer::Status::Error) { + /* + * If the framebuffer is internal to CameraStream return + * it back now that we're done processing it. + */ + if (buffer.internalBuffer) + returnInternalBuffer(buffer); + break; + } + + /* + * Acquire fences of streams of type Internal and Mapped + * will be handled during post-processing. + */ + buffer.srcBuffer = frameBuffer; + camera3Result->pendingBuffersToProcess_.emplace_back(&buffer); + + if (cameraStream->isJpegStream()) { + generateJpegExifMetadata(descriptor, &buffer); + + /* + * Allocate for post-processor to fill + * in JPEG related metadata. + */ + if (!camera3Result->resultMetadata_) + camera3Result->resultMetadata_ = getDynamicResultMetadata(metadata); + } + break; + } + } + } + + { + /* + * Adding result to the request before sending it to the + * post-processing threads, so the streamProcessingComplete() + * slot can safely call compeleteResult(). + */ + MutexLocker lock(descriptor->resultsMutex_); + descriptor->results_.emplace_back(camera3Result); } /* - * Notify shutter as soon as we have verified we have a valid request. - * - * \todo The shutter event notification should be sent to the framework - * as soon as possible, earlier than request completion time. + * Queue all the post-processing streams request at once. The completion + * slot streamProcessingComplete() can only execute when we are out + * this critical section (result->streamsProcessMutex_). This helps to + * handle synchronous errors here itself. */ - uint64_t sensorTimestamp = static_cast(request->metadata() - .get(controls::SensorTimestamp) - .value_or(0)); - notifyShutter(descriptor->frameNumber_, sensorTimestamp); + for (auto *buffer : camera3Result->pendingBuffersToProcess_) { + int ret = buffer->stream->process(buffer); + if (ret) { + setBufferStatus(*buffer, StreamBuffer::Status::Error); + + /* + * If the framebuffer is internal to CameraStream return + * it back now that we're done processing it. + */ + if (buffer->internalBuffer) + buffer->stream->putBuffer(buffer->internalBuffer); - LOG(HAL, Debug) << "Request " << request->cookie() << " completed with " - << descriptor->request_->buffers().size() << " streams"; + LOG(HAL, Error) << "Failed to run post process of request " << descriptor->frameNumber_; + } + } + + if (!camera3Result->pendingBuffersToProcess_.empty()) + return; + + locker.unlock(); + completeResultDescriptor(camera3Result); +} + +void CameraDevice::requestComplete(Request *request) +{ + Camera3RequestDescriptor *camera3Request = + reinterpret_cast(request->cookie()); /* - * Generate the metadata associated with the captured buffers. + * On Android, each new partial result with a metadata must set a + * field (partial_result) to a distinct inclusive value between + * 1 and ANDROID_REQUEST_PARTIAL_RESULT_COUNT and the final result with + * metadata has to set the field as ANDROID_REQUEST_PARTIAL_RESULT_COUNT. + * + * An empty metadata with ANDROID_REQUEST_PARTIAL_RESULT_COUNT is not + * allowed. Add a result with a fixed metadata on requestComplete() + * in case all of previous metadata are sent early, and no more metadata + * can be sent as the final with ANDROID_REQUEST_PARTIAL_RESULT_COUNT. * - * Notify if the metadata generation has failed, but continue processing - * buffers and return an empty metadata pack. + * See comments in completeResultDescriptor(). */ - descriptor->resultMetadata_ = getResultMetadata(*descriptor); - if (!descriptor->resultMetadata_) { - notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_RESULT); + Camera3ResultDescriptor *result = new Camera3ResultDescriptor(camera3Request); + result->resultMetadata_ = getFixedResultMetadata(camera3Request->settings_); - /* - * The camera framework expects an empty metadata pack on error. - * - * \todo Check that the post-processor code handles this situation - * correctly. - */ - descriptor->resultMetadata_ = std::make_unique(0, 0); + { + MutexLocker lock(camera3Request->resultsMutex_); + + switch (request->status()) { + case Request::RequestComplete: + camera3Request->status_ = Camera3RequestDescriptor::Status::Success; + break; + case Request::RequestCancelled: + camera3Request->status_ = Camera3RequestDescriptor::Status::Cancelled; + break; + case Request::RequestPending: + LOG(HAL, Fatal) << "Try to complete an unfinished request"; + break; + } + + camera3Request->results_.emplace_back(result); } - /* Handle post-processing. */ - MutexLocker locker(descriptor->streamsProcessMutex_); + completeResultDescriptor(result); + return; +} + +void CameraDevice::completeResultDescriptor(Camera3ResultDescriptor *result) +{ + Camera3RequestDescriptor *request = result->request_; + result->complete_ = true; + + MutexLocker lock(request->resultsMutex_); + + bool requestCompleted = (request->status_ != Camera3RequestDescriptor::Status::Pending); + bool requestCancelled = (request->status_ == Camera3RequestDescriptor::Status::Cancelled); + + bool hasPendingMetadata = false; + bool hasPendingResult = false; + + for (auto &r : request->results_) { + if (!r->complete_) { + hasPendingResult = true; + if (r->resultMetadata_) + hasPendingMetadata = true; + } + } /* - * Queue all the post-processing streams request at once. The completion - * slot streamProcessingComplete() can only execute when we are out - * this critical section. This helps to handle synchronous errors here - * itself. + * Android requires value of metadataPackIndex follows the rules: + * + * Set to 0: Result contains no metadata (buffers only). + * Set between 1 and (MaxMetadataPackIndex - 1): Result contains metadata. + * Set to MaxMetadataPackIndex: The final result having metadata. */ - auto iter = descriptor->pendingStreamsToProcess_.begin(); - while (iter != descriptor->pendingStreamsToProcess_.end()) { - CameraStream *stream = iter->first; - StreamBuffer *buffer = iter->second; + uint32_t &metadataPackIndex = result->metadataPackIndex_ = 0; + if (result->resultMetadata_) { + metadataPackIndex = request->nextPartialResultIndex_++; - if (stream->isJpegStream()) { - generateJpegExifMetadata(descriptor, buffer); + if (requestCompleted && !hasPendingMetadata) { + metadataPackIndex = CameraCapabilities::MaxMetadataPackIndex; } - - FrameBuffer *src = request->findBuffer(stream->stream()); - if (!src) { - LOG(HAL, Error) << "Failed to find a source stream buffer"; - setBufferStatus(*buffer, StreamBuffer::Status::Error); - iter = descriptor->pendingStreamsToProcess_.erase(iter); - continue; + if (metadataPackIndex > CameraCapabilities::MaxMetadataPackIndex) { + LOG(HAL, Fatal) << "Partial result exceed limited count " + << CameraCapabilities::MaxMetadataPackIndex; } + } - buffer->srcBuffer = src; + lock.unlock(); - ++iter; - int ret = stream->process(buffer); - if (ret) { - setBufferStatus(*buffer, StreamBuffer::Status::Error); - descriptor->pendingStreamsToProcess_.erase(stream); + sendCaptureResult(result); - /* - * If the framebuffer is internal to CameraStream return - * it back now that we're done processing it. - */ - if (buffer->internalBuffer) - stream->putBuffer(buffer->internalBuffer); - } + /* + * Call notify with CAMERA3_MSG_ERROR_RESULT to indicate some of the + * expected result metadata might not be available for the capture. Only + * calls when all pending metadata are sent, since Android ignores + * the following metadata after the notice. + */ + if (requestCancelled && !hasPendingMetadata) { + notifyError(request->frameNumber_, nullptr, CAMERA3_MSG_ERROR_RESULT); } - if (descriptor->pendingStreamsToProcess_.empty()) { - locker.unlock(); - completeDescriptor(descriptor); + if (requestCompleted && !hasPendingResult) { + completeRequestDescriptor(request); } } @@ -1260,91 +1389,99 @@ void CameraDevice::requestComplete(Request *request) * \brief Complete the Camera3RequestDescriptor * \param[in] descriptor The Camera3RequestDescriptor that has completed * - * The function marks the Camera3RequestDescriptor as 'complete'. It shall be - * called when all the streams in the Camera3RequestDescriptor have completed - * capture (or have been generated via post-processing) and the request is ready - * to be sent back to the framework. + * The function shall be called when all the buffers and metadata in the + * Camera3RequestDescriptor have completed (or have been generated via + * post-processing) and all results are sent back to the framework. * * \context This function is \threadsafe. */ -void CameraDevice::completeDescriptor(Camera3RequestDescriptor *descriptor) +void CameraDevice::completeRequestDescriptor(Camera3RequestDescriptor *descriptor) { MutexLocker lock(descriptorsMutex_); - descriptor->complete_ = true; - sendCaptureResults(); + auto iter = find_if(descriptors_.begin(), descriptors_.end(), + [&descriptor](auto &item) { + return item.get() == descriptor; + }); + + if (iter == descriptors_.end()) { + LOG(HAL, Error) << "Complete an unrecognized request"; + return; + } + + descriptors_.erase(iter); } -/** - * \brief Sequentially send capture results to the framework - * - * Iterate over the descriptors queue to send completed descriptors back to the - * framework, in the same order as they have been queued. For each complete - * descriptor, populate a locally-scoped camera3_capture_result_t from the - * descriptor, send the capture result back by calling the - * process_capture_result() callback, and remove the descriptor from the queue. - * Stop iterating if the descriptor at the front of the queue is not complete. - * - * This function should never be called directly in the codebase. Use - * completeDescriptor() instead. - */ -void CameraDevice::sendCaptureResults() +void CameraDevice::setBufferStatus(StreamBuffer &streamBuffer, + StreamBuffer::Status status) const { - while (!descriptors_.empty() && !descriptors_.front()->isPending()) { - auto descriptor = std::move(descriptors_.front()); - descriptors_.pop(); + streamBuffer.status = status; + if (status != StreamBuffer::Status::Success) { + notifyError(streamBuffer.request->frameNumber_, + streamBuffer.stream->camera3Stream(), + CAMERA3_MSG_ERROR_BUFFER); + } +} - camera3_capture_result_t captureResult = {}; +void CameraDevice::returnInternalBuffer(StreamBuffer &streamBuffer) const +{ + if (!streamBuffer.internalBuffer) { + return; + } - captureResult.frame_number = descriptor->frameNumber_; + CameraStream *stream = streamBuffer.stream; + switch (stream->type()) { + case CameraStream::Type::Internal: + stream->putBuffer(streamBuffer.internalBuffer); + break; + case CameraStream::Type::Mapped: + /* + * The internal buffer of a mapped stream is allocated from + * its source stream + */ + stream->sourceStream()->putBuffer(streamBuffer.internalBuffer); + break; + case CameraStream::Type::Direct: + LOG(HAL, Fatal) << "Direct stream buffer has internal buffer"; + break; + } +} - if (descriptor->resultMetadata_) - captureResult.result = - descriptor->resultMetadata_->getMetadata(); +void CameraDevice::sendCaptureResult(Camera3ResultDescriptor *result) const +{ + camera3_capture_result_t captureResult = {}; - std::vector resultBuffers; - resultBuffers.reserve(descriptor->buffers_.size()); + captureResult.frame_number = result->request_->frameNumber_; - for (auto &buffer : descriptor->buffers_) { - camera3_buffer_status status = CAMERA3_BUFFER_STATUS_ERROR; + std::vector resultBuffers; + resultBuffers.reserve(result->buffers_.size()); - if (buffer.status == StreamBuffer::Status::Success) - status = CAMERA3_BUFFER_STATUS_OK; + for (auto &buffer : result->buffers_) { + camera3_buffer_status status = CAMERA3_BUFFER_STATUS_ERROR; - /* - * Pass the buffer fence back to the camera framework as - * a release fence. This instructs the framework to wait - * on the acquire fence in case we haven't done so - * ourselves for any reason. - */ - resultBuffers.push_back({ buffer.stream->camera3Stream(), - buffer.camera3Buffer, status, - -1, buffer.fence.release() }); - } + if (buffer->status == StreamBuffer::Status::Success) + status = CAMERA3_BUFFER_STATUS_OK; - captureResult.num_output_buffers = resultBuffers.size(); - captureResult.output_buffers = resultBuffers.data(); + /* + * Pass the buffer fence back to the camera framework as + * a release fence. This instructs the framework to wait + * on the acquire fence in case we haven't done so + * ourselves for any reason. + */ + resultBuffers.push_back({ buffer->stream->camera3Stream(), + buffer->camera3Buffer, status, + -1, buffer->fence.release() }); + } - if (descriptor->status_ == Camera3RequestDescriptor::Status::Success) - captureResult.partial_result = 1; + captureResult.num_output_buffers = resultBuffers.size(); + captureResult.output_buffers = resultBuffers.data(); - callbacks_->process_capture_result(callbacks_, &captureResult); - } -} + if (result->resultMetadata_) + captureResult.result = result->resultMetadata_->getMetadata(); -void CameraDevice::setBufferStatus(StreamBuffer &streamBuffer, - StreamBuffer::Status status) -{ - streamBuffer.status = status; - if (status != StreamBuffer::Status::Success) { - notifyError(streamBuffer.request->frameNumber_, - streamBuffer.stream->camera3Stream(), - CAMERA3_MSG_ERROR_BUFFER); + captureResult.partial_result = result->metadataPackIndex_; - /* Also set error status on entire request descriptor. */ - streamBuffer.request->status_ = - Camera3RequestDescriptor::Status::Error; - } + callbacks_->process_capture_result(callbacks_, &captureResult); } /** @@ -1354,11 +1491,11 @@ void CameraDevice::setBufferStatus(StreamBuffer &streamBuffer, * * This function is called from the post-processor's thread whenever a camera * stream has finished post processing. The corresponding entry is dropped from - * the descriptor's pendingStreamsToProcess_ map. + * the result's pendingBufferToProcess_ list. * - * If the pendingStreamsToProcess_ map is then empty, all streams requiring to - * be generated from post-processing have been completed. Mark the descriptor as - * complete using completeDescriptor() in that case. + * If the pendingBufferToProcess_ list is then empty, all streams requiring to + * be generated from post-processing have been completed. Mark the result as + * complete using completeResultDescriptor() in that case. */ void CameraDevice::streamProcessingComplete(StreamBuffer *streamBuffer, StreamBuffer::Status status) @@ -1370,19 +1507,18 @@ void CameraDevice::streamProcessingComplete(StreamBuffer *streamBuffer, * that we're done processing it. */ if (streamBuffer->internalBuffer) - streamBuffer->stream->putBuffer(streamBuffer->internalBuffer); - - Camera3RequestDescriptor *request = streamBuffer->request; + returnInternalBuffer(*streamBuffer); + Camera3ResultDescriptor *result = streamBuffer->result; { - MutexLocker locker(request->streamsProcessMutex_); + MutexLocker locker(result->streamsProcessMutex_); + result->pendingBuffersToProcess_.remove(streamBuffer); - request->pendingStreamsToProcess_.erase(streamBuffer->stream); - if (!request->pendingStreamsToProcess_.empty()) + if (!result->pendingBuffersToProcess_.empty()) return; } - completeDescriptor(streamBuffer->request); + completeResultDescriptor(streamBuffer->result); } std::string CameraDevice::logPrefix() const @@ -1414,6 +1550,99 @@ void CameraDevice::notifyError(uint32_t frameNumber, camera3_stream_t *stream, callbacks_->notify(callbacks_, ¬ify); } +std::unique_ptr +CameraDevice::getDynamicResultMetadata(const ControlList &metadata) const +{ + /* + * \todo Keep this in sync with the actual number of entries. + * + * Reserve capacity for the metadata larger than 4 bytes which cannot + * store in entries. + * Currently: 6 entries, 40 bytes extra capaticy. + * + * ANDROID_SENSOR_TIMESTAMP (int64) = 8 bytes + * ANDROID_SENSOR_EXPOSURE_TIME (int64) = 8 bytes + * ANDROID_SENSOR_FRAME_DURATION (int64) = 8 bytes + * ANDROID_SCALER_CROP_REGION (int32 X 4) = 16 bytes + * Total bytes for capacity: 40 + * + * Reserve more capacity for the JPEG metadata set by the post-processor. + * Currently: 8 entries, 72 bytes extra capaticy. + * + * ANDROID_JPEG_GPS_COORDINATES (double x 3) = 24 bytes + * ANDROID_JPEG_GPS_PROCESSING_METHOD (byte x 32) = 32 bytes + * ANDROID_JPEG_GPS_TIMESTAMP (int64) = 8 bytes + * ANDROID_JPEG_SIZE (int32_t) = 4 bytes + * ANDROID_JPEG_QUALITY (byte) = 1 byte + * ANDROID_JPEG_ORIENTATION (int32_t) = 4 bytes + * ANDROID_JPEG_THUMBNAIL_QUALITY (byte) = 1 byte + * ANDROID_JPEG_THUMBNAIL_SIZE (int32 x 2) = 8 bytes + * Total bytes for JPEG metadata: 72 + * + * \todo Calculate the entries and capacity by the input ControlList. + */ + std::unique_ptr resultMetadata = + std::make_unique(14, 112); + if (!resultMetadata->isValid()) { + LOG(HAL, Error) << "Failed to allocate result metadata"; + return nullptr; + } + + /* Add metadata tags reported by libcamera. */ + const auto ×tamp = metadata.get(controls::SensorTimestamp); + if (timestamp) + resultMetadata->addEntry(ANDROID_SENSOR_TIMESTAMP, *timestamp); + + const auto &pipelineDepth = metadata.get(controls::draft::PipelineDepth); + if (pipelineDepth) + resultMetadata->addEntry(ANDROID_REQUEST_PIPELINE_DEPTH, + *pipelineDepth); + + const auto &exposureTime = metadata.get(controls::ExposureTime); + if (exposureTime) + resultMetadata->addEntry(ANDROID_SENSOR_EXPOSURE_TIME, + *exposureTime * 1000ULL); + + const auto &frameDuration = metadata.get(controls::FrameDuration); + if (frameDuration) + resultMetadata->addEntry(ANDROID_SENSOR_FRAME_DURATION, + *frameDuration * 1000); + + const auto &scalerCrop = metadata.get(controls::ScalerCrop); + if (scalerCrop) { + const Rectangle &crop = *scalerCrop; + int32_t cropRect[] = { + crop.x, + crop.y, + static_cast(crop.width), + static_cast(crop.height), + }; + resultMetadata->addEntry(ANDROID_SCALER_CROP_REGION, cropRect); + } + + const auto &testPatternMode = metadata.get(controls::draft::TestPatternMode); + if (testPatternMode) + resultMetadata->addEntry(ANDROID_SENSOR_TEST_PATTERN_MODE, + *testPatternMode); + + /* + * Return the result metadata pack even is not valid: get() will return + * nullptr. + */ + if (!resultMetadata->isValid()) { + LOG(HAL, Error) << "Failed to construct result metadata"; + } + + if (resultMetadata->resized()) { + auto [entryCount, dataCount] = resultMetadata->usage(); + LOG(HAL, Info) + << "Result metadata resized: " << entryCount + << " entries and " << dataCount << " bytes used"; + } + + return resultMetadata; +} + /* * Set jpeg metadata used to generate EXIF in the JPEG post processing. */ @@ -1424,10 +1653,8 @@ void CameraDevice::generateJpegExifMetadata(Camera3RequestDescriptor *request, auto &jpegExifMetadata = buffer->jpegExifMetadata; jpegExifMetadata.emplace(StreamBuffer::JpegExifMetadata()); - jpegExifMetadata->sensorExposureTime = 0; - if (metadata.contains(controls::ExposureTime)) { - jpegExifMetadata->sensorExposureTime = metadata.get(controls::ExposureTime) * 1000ULL; - } + const int64_t exposureTime = metadata.get(controls::ExposureTime).value_or(0); + jpegExifMetadata->sensorExposureTime = exposureTime; /* * todo: Android Sensitivity = analog gain X digital gain only on sensor. @@ -1441,31 +1668,26 @@ void CameraDevice::generateJpegExifMetadata(Camera3RequestDescriptor *request, * Produce a set of fixed result metadata. */ std::unique_ptr -CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor) const +CameraDevice::getFixedResultMetadata(const CameraMetadata &settings) const { - const ControlList &metadata = descriptor.request_->metadata(); - const CameraMetadata &settings = descriptor.settings_; camera_metadata_ro_entry_t entry; bool found; /* + * \todo Retrieve metadata from corresponding libcamera controls. * \todo Keep this in sync with the actual number of entries. - * Currently: 40 entries, 156 bytes * - * Reserve more space for the JPEG metadata set by the post-processor. - * Currently: - * ANDROID_JPEG_GPS_COORDINATES (double x 3) = 24 bytes - * ANDROID_JPEG_GPS_PROCESSING_METHOD (byte x 32) = 32 bytes - * ANDROID_JPEG_GPS_TIMESTAMP (int64) = 8 bytes - * ANDROID_JPEG_SIZE (int32_t) = 4 bytes - * ANDROID_JPEG_QUALITY (byte) = 1 byte - * ANDROID_JPEG_ORIENTATION (int32_t) = 4 bytes - * ANDROID_JPEG_THUMBNAIL_QUALITY (byte) = 1 byte - * ANDROID_JPEG_THUMBNAIL_SIZE (int32 x 2) = 8 bytes - * Total bytes for JPEG metadata: 82 + * Reserve capacity for the metadata larger than 4 bytes which cannot + * store in entries. + * Currently: 31 entries, 16 bytes + * + * ANDROID_CONTROL_AE_TARGET_FPS_RANGE (int32 X 2) = 8 bytes + * ANDROID_SENSOR_ROLLING_SHUTTER_SKEW (int64) = 8 bytes + * + * Total bytes: 16 */ std::unique_ptr resultMetadata = - std::make_unique(88, 166); + std::make_unique(31, 16); if (!resultMetadata->isValid()) { LOG(HAL, Error) << "Failed to allocate result metadata"; return nullptr; @@ -1563,9 +1785,6 @@ CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor) cons resultMetadata->addEntry(ANDROID_LENS_OPTICAL_STABILIZATION_MODE, value); - value32 = ANDROID_SENSOR_TEST_PATTERN_MODE_OFF; - resultMetadata->addEntry(ANDROID_SENSOR_TEST_PATTERN_MODE, value32); - value = ANDROID_STATISTICS_FACE_DETECT_MODE_OFF; resultMetadata->addEntry(ANDROID_STATISTICS_FACE_DETECT_MODE, value); @@ -1587,40 +1806,6 @@ CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor) cons resultMetadata->addEntry(ANDROID_SENSOR_ROLLING_SHUTTER_SKEW, rolling_shutter_skew); - /* Add metadata tags reported by libcamera. */ - const int64_t timestamp = metadata.get(controls::SensorTimestamp).value_or(0); - resultMetadata->addEntry(ANDROID_SENSOR_TIMESTAMP, timestamp); - - const auto &pipelineDepth = metadata.get(controls::draft::PipelineDepth); - if (pipelineDepth) - resultMetadata->addEntry(ANDROID_REQUEST_PIPELINE_DEPTH, - *pipelineDepth); - - const auto &exposureTime = metadata.get(controls::ExposureTime); - if (exposureTime) - resultMetadata->addEntry(ANDROID_SENSOR_EXPOSURE_TIME, - *exposureTime * 1000ULL); - - const auto &frameDuration = metadata.get(controls::FrameDuration); - if (frameDuration) - resultMetadata->addEntry(ANDROID_SENSOR_FRAME_DURATION, - *frameDuration * 1000); - - const auto &scalerCrop = metadata.get(controls::ScalerCrop); - if (scalerCrop) { - const Rectangle &crop = *scalerCrop; - int32_t cropRect[] = { - crop.x, crop.y, static_cast(crop.width), - static_cast(crop.height), - }; - resultMetadata->addEntry(ANDROID_SCALER_CROP_REGION, cropRect); - } - - const auto &testPatternMode = metadata.get(controls::draft::TestPatternMode); - if (testPatternMode) - resultMetadata->addEntry(ANDROID_SENSOR_TEST_PATTERN_MODE, - *testPatternMode); - /* * Return the result metadata pack even is not valid: get() will return * nullptr. diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 7b279895..0c4237a6 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -20,6 +20,7 @@ #include #include +#include #include #include #include @@ -29,7 +30,6 @@ #include "camera_capabilities.h" #include "camera_metadata.h" #include "camera_stream.h" -#include "jpeg/encoder.h" class Camera3RequestDescriptor; struct CameraConfigData; @@ -63,6 +63,8 @@ public: const camera_metadata_t *constructDefaultRequestSettings(int type); int configureStreams(camera3_stream_configuration_t *stream_list); int processCaptureRequest(camera3_capture_request_t *request); + void partialResultComplete(libcamera::Request *request, + libcamera::Result *result); void requestComplete(libcamera::Request *request); void streamProcessingComplete(StreamBuffer *bufferStream, StreamBuffer::Status status); @@ -87,20 +89,25 @@ private: createFrameBuffer(const buffer_handle_t camera3buffer, libcamera::PixelFormat pixelFormat, const libcamera::Size &size); - void abortRequest(Camera3RequestDescriptor *descriptor) const; bool isValidRequest(camera3_capture_request_t *request) const; void notifyShutter(uint32_t frameNumber, uint64_t timestamp); void notifyError(uint32_t frameNumber, camera3_stream_t *stream, camera3_error_msg_code code) const; int processControls(Camera3RequestDescriptor *descriptor); - void completeDescriptor(Camera3RequestDescriptor *descriptor) + void abortRequest(Camera3RequestDescriptor *descriptor); + void completeResultDescriptor(Camera3ResultDescriptor *result); + void completeRequestDescriptor(Camera3RequestDescriptor *descriptor) LIBCAMERA_TSA_EXCLUDES(descriptorsMutex_); - void sendCaptureResults() LIBCAMERA_TSA_REQUIRES(descriptorsMutex_); - void setBufferStatus(StreamBuffer &buffer, StreamBuffer::Status status); + void sendCaptureResult(Camera3ResultDescriptor *partialResult) const; + void setBufferStatus(StreamBuffer &buffer, StreamBuffer::Status status) const; + void returnInternalBuffer(StreamBuffer &buffer) const; void generateJpegExifMetadata(Camera3RequestDescriptor *request, StreamBuffer *buffer) const; - std::unique_ptr getResultMetadata( - const Camera3RequestDescriptor &descriptor) const; + + std::unique_ptr getDynamicResultMetadata( + const libcamera::ControlList &metadata) const; + std::unique_ptr getFixedResultMetadata( + const CameraMetadata &settings) const; unsigned int id_; camera3_device_t camera3Device_; @@ -118,7 +125,7 @@ private: std::vector streams_; libcamera::Mutex descriptorsMutex_ LIBCAMERA_TSA_ACQUIRED_AFTER(stateMutex_); - std::queue> descriptors_ + std::list> descriptors_ LIBCAMERA_TSA_GUARDED_BY(descriptorsMutex_); std::string maker_; diff --git a/src/android/camera_request.cpp b/src/android/camera_request.cpp index f5d4d314..31751b5c 100644 --- a/src/android/camera_request.cpp +++ b/src/android/camera_request.cpp @@ -42,25 +42,25 @@ using namespace libcamera; * │ processCaptureRequest(camera3_capture_request_t request) │ * │ │ * │ - Create Camera3RequestDescriptor tracking this request │ - * │ - Streams requiring post-processing are stored in the │ - * │ pendingStreamsToProcess map │ + * │ - Buffer requiring post-processing are marked by the │ + * │ CameraStream::Type as Mapped or Internal │ * │ - Add this Camera3RequestDescriptor to descriptors' queue │ * │ CameraDevice::descriptors_ │ - * │ │ ┌─────────────────────────┐ - * │ - Queue the capture request to libcamera core ────────────┤►│libcamera core │ - * │ │ ├─────────────────────────┤ - * │ │ │- Capture from Camera │ - * │ │ │ │ - * │ │ │- Emit │ - * │ │ │ Camera::requestComplete│ - * │ requestCompleted(Request *request) ◄───────────────────────┼─┼──── │ - * │ │ │ │ - * │ - Check request completion status │ └─────────────────────────┘ + * │ │ ┌───────────────────────────────┐ + * │ - Queue the capture request to libcamera core ────────────┤►│libcamera core │ + * │ │ ├───────────────────────────────┤ + * │ │ │- Capture from Camera │ + * │ │ │ │ + * │ │ │- Emit │ + * │ │ │ Camera::partialResultComplete│ + * │ partialResultComplete(Request *request, Result result*) ◄──┼─┼──── │ + * │ │ │ │ + * │ - Check request completion status │ └───────────────────────────────┘ * │ │ - * │ - if (pendingStreamsToProcess > 0) │ - * │ Queue all entries from pendingStreamsToProcess │ + * │ - if (pendingBuffersToProcess > 0) │ + * │ Queue all entries from pendingBuffersToProcess │ * │ else │ │ - * │ completeDescriptor() │ └──────────────────────┐ + * │ completeResultDescriptor() │ └──────────────────────┐ * │ │ │ * │ ┌──────────────────────────┴───┬──────────────────┐ │ * │ │ │ │ │ @@ -93,10 +93,10 @@ using namespace libcamera; * │ | | | | │ * │ | - Check and set buffer status | | .... | │ * │ | - Remove post+processing entry | | | │ - * │ | from pendingStreamsToProcess | | | │ + * │ | from pendingBuffersToProcess | | | │ * │ | | | | │ - * │ | - if (pendingStreamsToProcess.empty())| | | │ - * │ | completeDescriptor | | | │ + * │ | - if (pendingBuffersToProcess.empty())| | | │ + * │ | completeResultDescriptor | | | │ * │ | | | | │ * │ +---------------------------------------+ +--------------+ │ * │ │ @@ -110,6 +110,7 @@ using namespace libcamera; Camera3RequestDescriptor::Camera3RequestDescriptor( Camera *camera, const camera3_capture_request_t *camera3Request) + : status_(Status::Pending), nextPartialResultIndex_(1) { frameNumber_ = camera3Request->frame_number; @@ -140,6 +141,19 @@ Camera3RequestDescriptor::Camera3RequestDescriptor( Camera3RequestDescriptor::~Camera3RequestDescriptor() = default; +/* + * \class Camera3ResultDescriptor + * + * A utility class that groups information about a capture result to be later + * sent to framework. + */ +Camera3ResultDescriptor::Camera3ResultDescriptor(Camera3RequestDescriptor *request) + : complete_(false), request_(request), metadataPackIndex_(1) +{ +} + +Camera3ResultDescriptor::~Camera3ResultDescriptor() = default; + /** * \class StreamBuffer * \brief Group information for per-stream buffer of Camera3RequestDescriptor diff --git a/src/android/camera_request.h b/src/android/camera_request.h index f91de955..b5d9e3ee 100644 --- a/src/android/camera_request.h +++ b/src/android/camera_request.h @@ -24,6 +24,7 @@ class CameraBuffer; class CameraStream; +class Camera3ResultDescriptor; class Camera3RequestDescriptor; class StreamBuffer @@ -56,41 +57,64 @@ public: const libcamera::FrameBuffer *srcBuffer = nullptr; std::unique_ptr dstBuffer; std::optional jpegExifMetadata; + Camera3ResultDescriptor *result; Camera3RequestDescriptor *request; private: LIBCAMERA_DISABLE_COPY(StreamBuffer) }; +class Camera3ResultDescriptor +{ +public: + Camera3ResultDescriptor(Camera3RequestDescriptor *request); + ~Camera3ResultDescriptor(); + + bool complete_; + Camera3RequestDescriptor *request_; + uint32_t metadataPackIndex_; + + std::unique_ptr resultMetadata_; + std::vector buffers_; + + /* Keeps track of buffers waiting for post-processing. */ + std::list pendingBuffersToProcess_ + LIBCAMERA_TSA_GUARDED_BY(streamsProcessMutex_); + libcamera::Mutex streamsProcessMutex_; + +private: + LIBCAMERA_DISABLE_COPY(Camera3ResultDescriptor) +}; + class Camera3RequestDescriptor { public: enum class Status { + Pending, Success, - Error, + Cancelled, + Flushed, }; - /* Keeps track of streams requiring post-processing. */ - std::map pendingStreamsToProcess_ - LIBCAMERA_TSA_GUARDED_BY(streamsProcessMutex_); - libcamera::Mutex streamsProcessMutex_; - Camera3RequestDescriptor(libcamera::Camera *camera, const camera3_capture_request_t *camera3Request); ~Camera3RequestDescriptor(); - bool isPending() const { return !complete_; } - uint32_t frameNumber_ = 0; std::vector buffers_; CameraMetadata settings_; std::unique_ptr request_; - std::unique_ptr resultMetadata_; - bool complete_ = false; - Status status_ = Status::Success; + std::vector> results_ + LIBCAMERA_TSA_GUARDED_BY(resultsMutex_); + + Status status_ LIBCAMERA_TSA_GUARDED_BY(resultsMutex_); + + uint32_t nextPartialResultIndex_ LIBCAMERA_TSA_GUARDED_BY(resultsMutex_); + + libcamera::Mutex resultsMutex_; private: LIBCAMERA_DISABLE_COPY(Camera3RequestDescriptor) diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp index 91fd329c..0cb5fd9a 100644 --- a/src/android/jpeg/post_processor_jpeg.cpp +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -111,7 +111,7 @@ void PostProcessorJpeg::process(StreamBuffer *streamBuffer) ASSERT(jpegExifMetadata.has_value()); const CameraMetadata &requestMetadata = streamBuffer->request->settings_; - CameraMetadata *resultMetadata = streamBuffer->request->resultMetadata_.get(); + CameraMetadata *resultMetadata = streamBuffer->result->resultMetadata_.get(); camera_metadata_ro_entry_t entry; int ret;