From patchwork Wed Nov 27 09:25:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cheng-Hao Yang X-Patchwork-Id: 22112 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 712C5C3213 for ; Wed, 27 Nov 2024 09:26:46 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D6C0B660C3; Wed, 27 Nov 2024 10:26:45 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="jBFo3plp"; dkim-atps=neutral Received: from mail-ot1-x335.google.com (mail-ot1-x335.google.com [IPv6:2607:f8b0:4864:20::335]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 14DAE660AD for ; Wed, 27 Nov 2024 10:26:41 +0100 (CET) Received: by mail-ot1-x335.google.com with SMTP id 46e09a7af769-71d42a79a14so1652478a34.3 for ; Wed, 27 Nov 2024 01:26:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732699599; x=1733304399; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=W9hw/h8DkwpQ2JaAwAse5GJuP66gmF3X+z6FR3Gk7gc=; b=jBFo3plpHI1hVtllOfs4k5J8lyHGdWlfJe8vG3PYZQ7jOg1d4ooyVX47FQD4UZ0Les bf6yOP0sxEhnlAvQL9j/bppbvl+jNTHVEcY9qxg8bhqG33EasY6p0d8IRJdXY1nJXKeK EzlsnW4Ow1/meumLAn958CK4Zg66GDnG2iA4Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732699599; x=1733304399; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W9hw/h8DkwpQ2JaAwAse5GJuP66gmF3X+z6FR3Gk7gc=; b=TH2DpyrLu06StJiVOtJYJu28iQRlsR6OHsZYbfnpXfWbwOdky+yxl1kMKl3vILc1X+ Wusim/j1mQlCJRahOFDlgkH6oLx+268EFlD4Zgw5NeVYlA1jPsnkw2syVkZdmfVbfVkb GIQIMMtltGz7fB7ZY5yCzm15aogw4eXlpicJvyajLuWRRPnfCA0kAKNG6QkTwFCyjigQ TSmXDWXT/FOwBF6SzMrzkqTGPyRuKs5jW4katJEWs5P9b6yM3lF2GJtWjECFoE0Ydyon 2p47R6htjmuecZSmayN7xmqLZbyFYXkKottH84RqfzMIhZ2rOW6xV4bKJpHZX68kT+Ny n75w== X-Gm-Message-State: AOJu0YydG6Zk9YbUEnErEq5Mi78UMydWzJYNNfOhxay+zIt0JYxUUQFh cV7D/jjbbRXFZXXDvibjLc54Wytd8OG5G0dTsNEUZuKBX5ekJiZmeWI/1afQavonf+JzHtBTPR8 = X-Gm-Gg: ASbGncvFY50wSq1HfIhX+3UUyZn1mirC/BmvKVYPWsNDOUtcj1WTtjPT73FvquH6NOf B0J+rAGD03Uz6ONzGmXSx3AGLhBHmK65OTKfs1uhcdtuosRYejV6euRl5TnMZ/AoLNmX3U402H3 f6+OW6v/FqsjjsQaj8X6MKXvYsTn0I67YIxCj51XVjGirTfh1B9hkeNTW2LVm2VnxNdi9OpLCVz xiC+gl8ivMrQdXO5C5TEDSQJhXuMWpgE1UZFpJSp7GBTQCGwVdP+7VoNTq3NX6tDhyvXPAO9qlu /MNiJAUbL4RWVS72EBlBjkUBz0IIUSpXF3GYsJy0jil2REmehg/FnA== X-Google-Smtp-Source: AGHT+IGX39X0MMD9q3e5WIa5y/wqcLhDtSLtr/WxnTf/+Ultwg7Caumei5Wdr09Df7d+iUNRbB17qw== X-Received: by 2002:a05:6830:6dc6:b0:718:17b6:56f5 with SMTP id 46e09a7af769-71d65d01867mr2225754a34.29.1732699599128; Wed, 27 Nov 2024 01:26:39 -0800 (PST) Received: from chenghaoyang-low.c.googlers.com.com (27.247.221.35.bc.googleusercontent.com. [35.221.247.27]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fbcbfc41f9sm8693027a12.8.2024.11.27.01.26.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Nov 2024 01:26:38 -0800 (PST) From: Harvey Yang To: libcamera-devel@lists.libcamera.org Cc: Han-Lin Chen , Harvey Yang Subject: [PATCH v2 1/9] libcamera: Camera: Add signals for completion of metadata as a partial result Date: Wed, 27 Nov 2024 09:25:51 +0000 Message-ID: <20241127092632.3145984-2-chenghaoyang@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241127092632.3145984-1-chenghaoyang@chromium.org> References: <20241127092632.3145984-1-chenghaoyang@chromium.org> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Han-Lin Chen Allows pipeline handler to signal metadata completion by adding the following signals to Camera: Signal metadataAvailable; Together with the bufferCompleted signal, the pipeline handler is allowed to return buffers and partial metadata at any stage of processing. Signed-off-by: Han-Lin Chen Co-developed-by: Harvey Yang --- include/libcamera/camera.h | 1 + include/libcamera/internal/pipeline_handler.h | 1 + include/libcamera/internal/request.h | 4 ++ include/libcamera/request.h | 1 + src/libcamera/camera.cpp | 6 +++ src/libcamera/pipeline_handler.cpp | 41 +++++++++++++++++++ src/libcamera/request.cpp | 21 ++++++++++ 7 files changed, 75 insertions(+) diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h index 94cee7bd8..eb7cdf81b 100644 --- a/include/libcamera/camera.h +++ b/include/libcamera/camera.h @@ -122,6 +122,7 @@ public: const std::string &id() const; + Signal metadataAvailable; Signal bufferCompleted; Signal requestCompleted; Signal<> disconnected; diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h index fb28a18d0..6c6cad66f 100644 --- a/include/libcamera/internal/pipeline_handler.h +++ b/include/libcamera/internal/pipeline_handler.h @@ -58,6 +58,7 @@ public: void registerRequest(Request *request); void queueRequest(Request *request); + void completeMetadata(Request *request, const ControlList &metadata); bool completeBuffer(Request *request, FrameBuffer *buffer); void completeRequest(Request *request); void cancelRequest(Request *request); diff --git a/include/libcamera/internal/request.h b/include/libcamera/internal/request.h index 4e7d05b1e..286cd9d76 100644 --- a/include/libcamera/internal/request.h +++ b/include/libcamera/internal/request.h @@ -43,6 +43,8 @@ public: void prepare(std::chrono::milliseconds timeout = 0ms); Signal<> prepared; + ControlList addCompletedMetadata(const ControlList &metadata); + private: friend class PipelineHandler; friend std::ostream &operator<<(std::ostream &out, const Request &r); @@ -60,6 +62,8 @@ private: std::unordered_set pending_; std::map> notifiers_; std::unique_ptr timer_; + + std::unordered_set completedMetadata_; }; } /* namespace libcamera */ diff --git a/include/libcamera/request.h b/include/libcamera/request.h index e214a9d13..2c78d9bb4 100644 --- a/include/libcamera/request.h +++ b/include/libcamera/request.h @@ -12,6 +12,7 @@ #include #include #include +#include #include #include diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp index 7507e9dda..22484721a 100644 --- a/src/libcamera/camera.cpp +++ b/src/libcamera/camera.cpp @@ -892,6 +892,12 @@ const std::string &Camera::id() const * completed */ +/** + * \var Camera::metadataAvailable + * \brief Signal emitted when some metadata for a request is available as a + * partial result + */ + /** * \var Camera::requestCompleted * \brief Signal emitted when a request queued to the camera has completed diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp index 991b06f26..4ba96cfa2 100644 --- a/src/libcamera/pipeline_handler.cpp +++ b/src/libcamera/pipeline_handler.cpp @@ -531,6 +531,32 @@ bool PipelineHandler::completeBuffer(Request *request, FrameBuffer *buffer) return request->_d()->completeBuffer(buffer); } +/** + * \brief Complete part of metadata for a request + * \param[in] request The request the metadata belongs to + * \param[in] metadata The partial metadata that has completed + * + * This function could be called by pipeline handlers to signal availability of + * \a metadata before \a request completes. Early metadata completion allows to + * notify applications about the availability of a partial metadata buffer + * before the associated Request has completed. + * + * A metadata key is expected to be completed at most once. If it's completed + * more than once, the key will be dropped since the second time. + * + * \context This function shall be called from the CameraManager thread. + */ +void PipelineHandler::completeMetadata(Request *request, const ControlList &metadata) +{ + const ControlList validMetadata = request->_d()->addCompletedMetadata(metadata); + if (!validMetadata.empty()) { + request->metadata().merge(validMetadata); + + Camera *camera = request->_d()->camera(); + camera->metadataAvailable.emit(request, validMetadata); + } +} + /** * \brief Signal request completion * \param[in] request The request that has completed @@ -553,6 +579,21 @@ void PipelineHandler::completeRequest(Request *request) Camera::Private *data = camera->_d(); + /* + * Collect metadata which is not yet completed by the Camera, and + * create one partial result to cover the missing metadata before + * completing the whole request. This guarantees the aggregation of + * metadata in completed partial results equals to the global metadata + * in the request. + * + * \todo: Forbid merging metadata into request.metadata() directly and + * force calling completeMetadata() to report metadata. + */ + const ControlList validMetadata = request->_d()->addCompletedMetadata( + request->metadata()); + if (!validMetadata.empty()) + camera->metadataAvailable.emit(request, validMetadata); + while (!data->queuedRequests_.empty()) { Request *req = data->queuedRequests_.front(); if (req->status() == Request::RequestPending) diff --git a/src/libcamera/request.cpp b/src/libcamera/request.cpp index 8c56ed30d..ae5cdeb19 100644 --- a/src/libcamera/request.cpp +++ b/src/libcamera/request.cpp @@ -178,6 +178,7 @@ void Request::Private::reset() pending_.clear(); notifiers_.clear(); timer_.reset(); + completedMetadata_.clear(); } /* @@ -270,6 +271,26 @@ void Request::Private::prepare(std::chrono::milliseconds timeout) * if they have failed preparing. */ +/** + * \brief Add completed metadata, as a partial result + * \param[in] metadata The metadata completed + * + * Request will record the entries that has been sent to the application, to + * prevent duplicated controls. + * + * \return ControlList that hasn't been completed before + */ +ControlList Request::Private::addCompletedMetadata(const ControlList &metadata) +{ + ControlList resultMetadata; + for (auto &[id, value] : metadata) { + if (!completedMetadata_.count(id)) + resultMetadata.set(id, value); + } + + return resultMetadata; +} + void Request::Private::notifierActivated(FrameBuffer *buffer) { /* Close the fence if successfully signalled. */ From patchwork Wed Nov 27 09:25:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cheng-Hao Yang X-Patchwork-Id: 22113 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 85CCFC3213 for ; Wed, 27 Nov 2024 09:26:48 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 43308660BF; Wed, 27 Nov 2024 10:26:48 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="UvALSEnR"; dkim-atps=neutral Received: from mail-oa1-x2b.google.com (mail-oa1-x2b.google.com [IPv6:2001:4860:4864:20::2b]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8F97F660B0 for ; Wed, 27 Nov 2024 10:26:42 +0100 (CET) Received: by mail-oa1-x2b.google.com with SMTP id 586e51a60fabf-296bf028170so2466822fac.3 for ; Wed, 27 Nov 2024 01:26:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732699601; x=1733304401; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=un0WBdxIu/s/iOAW8gEYlK97KZDKB9FH2DicPzAafSA=; b=UvALSEnRb9FUs8/aW+zbx7jsvx4kYvTksvrm5Zl3UvpCuc0nhkESg0uRjs5jAnfgT6 umD6kpDTrfN4EL9U2IBYmTI+1bROHj8Ngc6rV79vw3shw1G/B2DMOz4S2jRRh0qhksbF 0fQAc9J3hUXBvpdNNrN5h+FHskXu71+l3L/B4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732699601; x=1733304401; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=un0WBdxIu/s/iOAW8gEYlK97KZDKB9FH2DicPzAafSA=; b=JSmCjmQvQ7frY+15BvgFBFP7nPZFYJ+u6ed1q5bkvuBKEcmU00w+BHKeROFq+lXLIq /gDw9FW99RQtu/JcPcQr7RohhcTFfeSt5xVbnravMJLByeFgl3ATaARINYedP2E/5Tvs btKqprK3rQWhnfz1njx5piWyiHJSX9kINd6BzCxTd578MMk/I8gwiAZsXiihXE+NjHXU 8AXQE5N4ijzhtwuGWhypjlmgZOytm2rJKrecPZ6BqoCRHkLuQeT8h1RoDSjgI8VgYtAv CiPHalpc2L6wDBVP7Yt0KT3IB6EyZ0/hXVtUQddXh+HVlfz3QJaep0tnnMQhmJOgNZ2O 6zqw== X-Gm-Message-State: AOJu0YzD923KseJyykzzaHFGR7sSPdqs0Z4kFFRiqudeUodFwule5Ioj Z08rFJ8FKzDMx1y1HBzyuXTDQ67C7ZFu0sqHxzDWtg/lC3eVzoueicqU+8wRcygdpRFJ7Iz+QJo = X-Gm-Gg: ASbGncvGhg+x7u4j2H/fIRE2gPjvSM/4H7kKIVTQfNWOXogJuD+H8rp7l/aNWbaZZbt +nGtfKdBtmdEmSyHC0mOp6HcXCSOrhc99trSmBJuSJPbtumyqZNS5mfgvuqXf4RifQT1fAL0w+Y viOrtjb+Egk/8HkMIqWMtRqBCHu65OlNA2zEpyZqJtOfUJh4beGRt+ArikZo/WFjv3zrjk0pC7G 9k8maZVfFjq7KsgBFDq08s9x21JWZe+saJ51D3EnU0B4g5aQP/scO5u/jKaQgtvhYVyYJ2iicXy q7jMO5tw+/8osJy8NqXwwJ5Q/vxRf6LDA7zM1ZJmVJjJyFFH9Gu6QA== X-Google-Smtp-Source: AGHT+IFxVRiPZienS3w/tteUk2qfpNFAqb+U27sbWA61UZae2NDjSIcZF869Of4jt6/2OpCuuVz6TQ== X-Received: by 2002:a05:6871:c48d:b0:29d:c832:841c with SMTP id 586e51a60fabf-29dc832e5eemr1030016fac.16.1732699600726; Wed, 27 Nov 2024 01:26:40 -0800 (PST) Received: from chenghaoyang-low.c.googlers.com.com (27.247.221.35.bc.googleusercontent.com. [35.221.247.27]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fbcbfc41f9sm8693027a12.8.2024.11.27.01.26.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Nov 2024 01:26:40 -0800 (PST) From: Harvey Yang To: libcamera-devel@lists.libcamera.org Cc: Harvey Yang , Han-Lin Chen Subject: [PATCH v2 2/9] android: Move StreamBuffer out of Camera3RequestDescriptor Date: Wed, 27 Nov 2024 09:25:52 +0000 Message-ID: <20241127092632.3145984-3-chenghaoyang@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241127092632.3145984-1-chenghaoyang@chromium.org> References: <20241127092632.3145984-1-chenghaoyang@chromium.org> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Move StreamBuffer out of Camera3RequestDescriptor as a refactory before applying partial result to Android adaptor. For easier adding Camera3ResultDescriptor which references both StreamBuffer and Camera3RequestDescriptor. Signed-off-by: Han-Lin Chen Co-developed-by: Harvey Yang Signed-off-by: Harvey Yang Reviewed-by: Jacopo Mondi --- src/android/camera_device.cpp | 22 +++++----- src/android/camera_device.h | 8 ++-- src/android/camera_request.cpp | 29 +++++++------ src/android/camera_request.h | 53 ++++++++++++++---------- src/android/camera_stream.cpp | 16 +++---- src/android/camera_stream.h | 6 +-- src/android/jpeg/encoder.h | 2 +- src/android/jpeg/encoder_jea.cpp | 2 +- src/android/jpeg/encoder_jea.h | 2 +- src/android/jpeg/encoder_libjpeg.cpp | 2 +- src/android/jpeg/encoder_libjpeg.h | 2 +- src/android/jpeg/post_processor_jpeg.cpp | 2 +- src/android/jpeg/post_processor_jpeg.h | 2 +- src/android/post_processor.h | 4 +- src/android/yuv/post_processor_yuv.cpp | 2 +- src/android/yuv/post_processor_yuv.h | 2 +- 16 files changed, 82 insertions(+), 74 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index a038131ae..4e3bdc9cc 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -865,7 +865,7 @@ void CameraDevice::abortRequest(Camera3RequestDescriptor *descriptor) const notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_REQUEST); for (auto &buffer : descriptor->buffers_) - buffer.status = Camera3RequestDescriptor::Status::Error; + buffer.status = StreamBuffer::Status::Error; descriptor->status_ = Camera3RequestDescriptor::Status::Error; } @@ -1163,7 +1163,7 @@ void CameraDevice::requestComplete(Request *request) if (fence) buffer.fence = fence->release(); } - buffer.status = Camera3RequestDescriptor::Status::Success; + buffer.status = StreamBuffer::Status::Success; } /* @@ -1226,12 +1226,12 @@ void CameraDevice::requestComplete(Request *request) auto iter = descriptor->pendingStreamsToProcess_.begin(); while (iter != descriptor->pendingStreamsToProcess_.end()) { CameraStream *stream = iter->first; - Camera3RequestDescriptor::StreamBuffer *buffer = iter->second; + StreamBuffer *buffer = iter->second; FrameBuffer *src = request->findBuffer(stream->stream()); if (!src) { LOG(HAL, Error) << "Failed to find a source stream buffer"; - setBufferStatus(*buffer, Camera3RequestDescriptor::Status::Error); + setBufferStatus(*buffer, StreamBuffer::Status::Error); iter = descriptor->pendingStreamsToProcess_.erase(iter); continue; } @@ -1241,7 +1241,7 @@ void CameraDevice::requestComplete(Request *request) ++iter; int ret = stream->process(buffer); if (ret) { - setBufferStatus(*buffer, Camera3RequestDescriptor::Status::Error); + setBufferStatus(*buffer, StreamBuffer::Status::Error); descriptor->pendingStreamsToProcess_.erase(stream); /* @@ -1311,7 +1311,7 @@ void CameraDevice::sendCaptureResults() for (auto &buffer : descriptor->buffers_) { camera3_buffer_status status = CAMERA3_BUFFER_STATUS_ERROR; - if (buffer.status == Camera3RequestDescriptor::Status::Success) + if (buffer.status == StreamBuffer::Status::Success) status = CAMERA3_BUFFER_STATUS_OK; /* @@ -1335,11 +1335,11 @@ void CameraDevice::sendCaptureResults() } } -void CameraDevice::setBufferStatus(Camera3RequestDescriptor::StreamBuffer &streamBuffer, - Camera3RequestDescriptor::Status status) +void CameraDevice::setBufferStatus(StreamBuffer &streamBuffer, + StreamBuffer::Status status) { streamBuffer.status = status; - if (status != Camera3RequestDescriptor::Status::Success) { + if (status != StreamBuffer::Status::Success) { notifyError(streamBuffer.request->frameNumber_, streamBuffer.stream->camera3Stream(), CAMERA3_MSG_ERROR_BUFFER); @@ -1363,8 +1363,8 @@ void CameraDevice::setBufferStatus(Camera3RequestDescriptor::StreamBuffer &strea * be generated from post-processing have been completed. Mark the descriptor as * complete using completeDescriptor() in that case. */ -void CameraDevice::streamProcessingComplete(Camera3RequestDescriptor::StreamBuffer *streamBuffer, - Camera3RequestDescriptor::Status status) +void CameraDevice::streamProcessingComplete(StreamBuffer *streamBuffer, + StreamBuffer::Status status) { setBufferStatus(*streamBuffer, status); diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 194ca3030..c92ee1aa4 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -65,8 +65,8 @@ public: int configureStreams(camera3_stream_configuration_t *stream_list); int processCaptureRequest(camera3_capture_request_t *request); void requestComplete(libcamera::Request *request); - void streamProcessingComplete(Camera3RequestDescriptor::StreamBuffer *bufferStream, - Camera3RequestDescriptor::Status status); + void streamProcessingComplete(StreamBuffer *bufferStream, + StreamBuffer::Status status); protected: std::string logPrefix() const override; @@ -97,8 +97,8 @@ private: void completeDescriptor(Camera3RequestDescriptor *descriptor) LIBCAMERA_TSA_EXCLUDES(descriptorsMutex_); void sendCaptureResults() LIBCAMERA_TSA_REQUIRES(descriptorsMutex_); - void setBufferStatus(Camera3RequestDescriptor::StreamBuffer &buffer, - Camera3RequestDescriptor::Status status); + void setBufferStatus(StreamBuffer &buffer, + StreamBuffer::Status status); std::unique_ptr getResultMetadata( const Camera3RequestDescriptor &descriptor) const; diff --git a/src/android/camera_request.cpp b/src/android/camera_request.cpp index 0d45960d9..52a3ac1f7 100644 --- a/src/android/camera_request.cpp +++ b/src/android/camera_request.cpp @@ -141,7 +141,7 @@ Camera3RequestDescriptor::Camera3RequestDescriptor( Camera3RequestDescriptor::~Camera3RequestDescriptor() = default; /** - * \struct Camera3RequestDescriptor::StreamBuffer + * \class StreamBuffer * \brief Group information for per-stream buffer of Camera3RequestDescriptor * * A capture request placed to the libcamera HAL can contain multiple streams. @@ -150,35 +150,35 @@ Camera3RequestDescriptor::~Camera3RequestDescriptor() = default; * generation. The generation of the stream will depend on its type (refer to * the CameraStream::Type documentation). * - * \var Camera3RequestDescriptor::StreamBuffer::stream + * \var StreamBuffer::stream * \brief Pointer to the corresponding CameraStream * - * \var Camera3RequestDescriptor::StreamBuffer::camera3Buffer + * \var StreamBuffer::camera3Buffer * \brief Native handle to the buffer * - * \var Camera3RequestDescriptor::StreamBuffer::frameBuffer + * \var StreamBuffer::frameBuffer * \brief Encapsulate the dmabuf handle inside a libcamera::FrameBuffer for * direct streams * - * \var Camera3RequestDescriptor::StreamBuffer::fence + * \var StreamBuffer::fence * \brief Acquire fence of the buffer * - * \var Camera3RequestDescriptor::StreamBuffer::status + * \var StreamBuffer::status * \brief Track the status of the buffer * - * \var Camera3RequestDescriptor::StreamBuffer::internalBuffer + * \var StreamBuffer::internalBuffer * \brief Pointer to a buffer internally handled by CameraStream (if any) * - * \var Camera3RequestDescriptor::StreamBuffer::srcBuffer + * \var StreamBuffer::srcBuffer * \brief Pointer to the source frame buffer used for post-processing * - * \var Camera3RequestDescriptor::StreamBuffer::dstBuffer + * \var StreamBuffer::dstBuffer * \brief Pointer to the destination frame buffer used for post-processing * - * \var Camera3RequestDescriptor::StreamBuffer::request + * \var StreamBuffer::request * \brief Back pointer to the Camera3RequestDescriptor to which the StreamBuffer belongs */ -Camera3RequestDescriptor::StreamBuffer::StreamBuffer( +StreamBuffer::StreamBuffer( CameraStream *cameraStream, const camera3_stream_buffer_t &buffer, Camera3RequestDescriptor *requestDescriptor) : stream(cameraStream), camera3Buffer(buffer.buffer), @@ -186,9 +186,8 @@ Camera3RequestDescriptor::StreamBuffer::StreamBuffer( { } -Camera3RequestDescriptor::StreamBuffer::~StreamBuffer() = default; +StreamBuffer::~StreamBuffer() = default; -Camera3RequestDescriptor::StreamBuffer::StreamBuffer(StreamBuffer &&) = default; +StreamBuffer::StreamBuffer(StreamBuffer &&) = default; -Camera3RequestDescriptor::StreamBuffer & -Camera3RequestDescriptor::StreamBuffer::operator=(Camera3RequestDescriptor::StreamBuffer &&) = default; +StreamBuffer &StreamBuffer::operator=(StreamBuffer &&) = default; diff --git a/src/android/camera_request.h b/src/android/camera_request.h index 5b479180f..335f1985d 100644 --- a/src/android/camera_request.h +++ b/src/android/camera_request.h @@ -26,7 +26,9 @@ class CameraBuffer; class CameraStream; -class Camera3RequestDescriptor +class Camera3RequestDescriptor; + +class StreamBuffer { public: enum class Status { @@ -34,27 +36,34 @@ public: Error, }; - struct StreamBuffer { - StreamBuffer(CameraStream *stream, - const camera3_stream_buffer_t &buffer, - Camera3RequestDescriptor *request); - ~StreamBuffer(); - - StreamBuffer(StreamBuffer &&); - StreamBuffer &operator=(StreamBuffer &&); - - CameraStream *stream; - buffer_handle_t *camera3Buffer; - std::unique_ptr frameBuffer; - libcamera::UniqueFD fence; - Status status = Status::Success; - libcamera::FrameBuffer *internalBuffer = nullptr; - const libcamera::FrameBuffer *srcBuffer = nullptr; - std::unique_ptr dstBuffer; - Camera3RequestDescriptor *request; - - private: - LIBCAMERA_DISABLE_COPY(StreamBuffer) + StreamBuffer(CameraStream *stream, + const camera3_stream_buffer_t &buffer, + Camera3RequestDescriptor *request); + ~StreamBuffer(); + + StreamBuffer(StreamBuffer &&); + StreamBuffer &operator=(StreamBuffer &&); + + CameraStream *stream; + buffer_handle_t *camera3Buffer; + std::unique_ptr frameBuffer; + libcamera::UniqueFD fence; + Status status = Status::Success; + libcamera::FrameBuffer *internalBuffer = nullptr; + const libcamera::FrameBuffer *srcBuffer = nullptr; + std::unique_ptr dstBuffer; + Camera3RequestDescriptor *request; + +private: + LIBCAMERA_DISABLE_COPY(StreamBuffer) +}; + +class Camera3RequestDescriptor +{ +public: + enum class Status { + Success, + Error, }; /* Keeps track of streams requiring post-processing. */ diff --git a/src/android/camera_stream.cpp b/src/android/camera_stream.cpp index 1d68540d7..53f292d4b 100644 --- a/src/android/camera_stream.cpp +++ b/src/android/camera_stream.cpp @@ -112,14 +112,14 @@ int CameraStream::configure() worker_ = std::make_unique(postProcessor_.get()); postProcessor_->processComplete.connect( - this, [&](Camera3RequestDescriptor::StreamBuffer *streamBuffer, + this, [&](StreamBuffer *streamBuffer, PostProcessor::Status status) { - Camera3RequestDescriptor::Status bufferStatus; + StreamBuffer::Status bufferStatus; if (status == PostProcessor::Status::Success) - bufferStatus = Camera3RequestDescriptor::Status::Success; + bufferStatus = StreamBuffer::Status::Success; else - bufferStatus = Camera3RequestDescriptor::Status::Error; + bufferStatus = StreamBuffer::Status::Error; cameraDevice_->streamProcessingComplete(streamBuffer, bufferStatus); @@ -165,7 +165,7 @@ int CameraStream::waitFence(int fence) return -errno; } -int CameraStream::process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) +int CameraStream::process(StreamBuffer *streamBuffer) { ASSERT(type_ != Type::Direct); @@ -283,7 +283,7 @@ void CameraStream::PostProcessorWorker::start() Thread::start(); } -void CameraStream::PostProcessorWorker::queueRequest(Camera3RequestDescriptor::StreamBuffer *dest) +void CameraStream::PostProcessorWorker::queueRequest(StreamBuffer *dest) { { MutexLocker lock(mutex_); @@ -306,7 +306,7 @@ void CameraStream::PostProcessorWorker::run() if (state_ != State::Running) break; - Camera3RequestDescriptor::StreamBuffer *streamBuffer = requests_.front(); + StreamBuffer *streamBuffer = requests_.front(); requests_.pop(); locker.unlock(); @@ -316,7 +316,7 @@ void CameraStream::PostProcessorWorker::run() } if (state_ == State::Flushing) { - std::queue requests = + std::queue requests = std::move(requests_); locker.unlock(); diff --git a/src/android/camera_stream.h b/src/android/camera_stream.h index 395552dac..30f64f690 100644 --- a/src/android/camera_stream.h +++ b/src/android/camera_stream.h @@ -127,7 +127,7 @@ public: CameraStream *sourceStream() const { return sourceStream_; } int configure(); - int process(Camera3RequestDescriptor::StreamBuffer *streamBuffer); + int process(StreamBuffer *streamBuffer); libcamera::FrameBuffer *getBuffer(); void putBuffer(libcamera::FrameBuffer *buffer); void flush(); @@ -146,7 +146,7 @@ private: ~PostProcessorWorker(); void start(); - void queueRequest(Camera3RequestDescriptor::StreamBuffer *request); + void queueRequest(StreamBuffer *request); void flush(); protected: @@ -158,7 +158,7 @@ private: libcamera::Mutex mutex_; libcamera::ConditionVariable cv_; - std::queue requests_ + std::queue requests_ LIBCAMERA_TSA_GUARDED_BY(mutex_); State state_ LIBCAMERA_TSA_GUARDED_BY(mutex_) = State::Stopped; diff --git a/src/android/jpeg/encoder.h b/src/android/jpeg/encoder.h index ed033c191..25425d92c 100644 --- a/src/android/jpeg/encoder.h +++ b/src/android/jpeg/encoder.h @@ -20,7 +20,7 @@ public: virtual ~Encoder() = default; virtual int configure(const libcamera::StreamConfiguration &cfg) = 0; - virtual int encode(Camera3RequestDescriptor::StreamBuffer *buffer, + virtual int encode(StreamBuffer *buffer, libcamera::Span exifData, unsigned int quality) = 0; }; diff --git a/src/android/jpeg/encoder_jea.cpp b/src/android/jpeg/encoder_jea.cpp index 25dc43173..b3ad04638 100644 --- a/src/android/jpeg/encoder_jea.cpp +++ b/src/android/jpeg/encoder_jea.cpp @@ -33,7 +33,7 @@ int EncoderJea::configure(const libcamera::StreamConfiguration &cfg) return 0; } -int EncoderJea::encode(Camera3RequestDescriptor::StreamBuffer *buffer, +int EncoderJea::encode(StreamBuffer *buffer, libcamera::Span exifData, unsigned int quality) { diff --git a/src/android/jpeg/encoder_jea.h b/src/android/jpeg/encoder_jea.h index 91115d2e9..79265ebc6 100644 --- a/src/android/jpeg/encoder_jea.h +++ b/src/android/jpeg/encoder_jea.h @@ -20,7 +20,7 @@ public: ~EncoderJea(); int configure(const libcamera::StreamConfiguration &cfg) override; - int encode(Camera3RequestDescriptor::StreamBuffer *buffer, + int encode(StreamBuffer *buffer, libcamera::Span exifData, unsigned int quality) override; diff --git a/src/android/jpeg/encoder_libjpeg.cpp b/src/android/jpeg/encoder_libjpeg.cpp index cb242b5ec..cb123576a 100644 --- a/src/android/jpeg/encoder_libjpeg.cpp +++ b/src/android/jpeg/encoder_libjpeg.cpp @@ -180,7 +180,7 @@ void EncoderLibJpeg::compressNV(const std::vector> &planes) } } -int EncoderLibJpeg::encode(Camera3RequestDescriptor::StreamBuffer *buffer, +int EncoderLibJpeg::encode(StreamBuffer *buffer, libcamera::Span exifData, unsigned int quality) { diff --git a/src/android/jpeg/encoder_libjpeg.h b/src/android/jpeg/encoder_libjpeg.h index 4ac85c22e..de04b17d1 100644 --- a/src/android/jpeg/encoder_libjpeg.h +++ b/src/android/jpeg/encoder_libjpeg.h @@ -22,7 +22,7 @@ public: ~EncoderLibJpeg(); int configure(const libcamera::StreamConfiguration &cfg) override; - int encode(Camera3RequestDescriptor::StreamBuffer *buffer, + int encode(StreamBuffer *buffer, libcamera::Span exifData, unsigned int quality) override; int encode(const std::vector> &planes, diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp index 89b8a401e..f5a90785d 100644 --- a/src/android/jpeg/post_processor_jpeg.cpp +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -106,7 +106,7 @@ void PostProcessorJpeg::generateThumbnail(const FrameBuffer &source, } } -void PostProcessorJpeg::process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) +void PostProcessorJpeg::process(StreamBuffer *streamBuffer) { ASSERT(encoder_); diff --git a/src/android/jpeg/post_processor_jpeg.h b/src/android/jpeg/post_processor_jpeg.h index 6fe214577..5421f233e 100644 --- a/src/android/jpeg/post_processor_jpeg.h +++ b/src/android/jpeg/post_processor_jpeg.h @@ -22,7 +22,7 @@ public: int configure(const libcamera::StreamConfiguration &incfg, const libcamera::StreamConfiguration &outcfg) override; - void process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) override; + void process(StreamBuffer *streamBuffer) override; private: void generateThumbnail(const libcamera::FrameBuffer &source, diff --git a/src/android/post_processor.h b/src/android/post_processor.h index b504a3796..f2e00d2b1 100644 --- a/src/android/post_processor.h +++ b/src/android/post_processor.h @@ -27,7 +27,7 @@ public: virtual int configure(const libcamera::StreamConfiguration &inCfg, const libcamera::StreamConfiguration &outCfg) = 0; - virtual void process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) = 0; + virtual void process(StreamBuffer *streamBuffer) = 0; - libcamera::Signal processComplete; + libcamera::Signal processComplete; }; diff --git a/src/android/yuv/post_processor_yuv.cpp b/src/android/yuv/post_processor_yuv.cpp index c998807b0..b05623b23 100644 --- a/src/android/yuv/post_processor_yuv.cpp +++ b/src/android/yuv/post_processor_yuv.cpp @@ -49,7 +49,7 @@ int PostProcessorYuv::configure(const StreamConfiguration &inCfg, return 0; } -void PostProcessorYuv::process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) +void PostProcessorYuv::process(StreamBuffer *streamBuffer) { const FrameBuffer &source = *streamBuffer->srcBuffer; CameraBuffer *destination = streamBuffer->dstBuffer.get(); diff --git a/src/android/yuv/post_processor_yuv.h b/src/android/yuv/post_processor_yuv.h index ed7bb1fbe..459c6685e 100644 --- a/src/android/yuv/post_processor_yuv.h +++ b/src/android/yuv/post_processor_yuv.h @@ -18,7 +18,7 @@ public: int configure(const libcamera::StreamConfiguration &incfg, const libcamera::StreamConfiguration &outcfg) override; - void process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) override; + void process(StreamBuffer *streamBuffer) override; private: bool isValidBuffers(const libcamera::FrameBuffer &source, From patchwork Wed Nov 27 09:25:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cheng-Hao Yang X-Patchwork-Id: 22114 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2F0D2C3213 for ; Wed, 27 Nov 2024 09:26:50 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 14DD1660C2; Wed, 27 Nov 2024 10:26:49 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="INTFfrxP"; dkim-atps=neutral Received: from mail-oa1-x33.google.com (mail-oa1-x33.google.com [IPv6:2001:4860:4864:20::33]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 00AB2660AD for ; Wed, 27 Nov 2024 10:26:43 +0100 (CET) Received: by mail-oa1-x33.google.com with SMTP id 586e51a60fabf-2958ddf99a7so5985587fac.2 for ; Wed, 27 Nov 2024 01:26:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732699602; x=1733304402; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gkEafkQKLiXPZ/Jkv3BAtdyW3Syk3pi66hwIh5mX3Kg=; b=INTFfrxP0DlQMMccxCIangQVSC0HkMMXCkdjvUMgLdrEF75fPyMKfn5JWGYYvyWE7a aXzTqNv5sGR4MP+i7bAVmqvlllVexAtJF0hA8QyLJ47Ug2exoS6aBrrgHTSTDj587xvZ r1rvnbWhDpEPQry14KXhUkzPAhD9xepmTHi5I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732699602; x=1733304402; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gkEafkQKLiXPZ/Jkv3BAtdyW3Syk3pi66hwIh5mX3Kg=; b=ayEPQJthG79l/dDEDV/OWIEy3CB/rt3H5QZnebmIcqSgTG3qzXqnWu4jgwGQfQj4A3 GWi50ejcUrg0iI6JNR0UFgMbVHTA+30Alv/rrJ6x9PTJUVui6P2cdV/dJI0OT8LicVng TBLQuvIQMwhqOuaLT+p+yDefyFsg33Eu+Mi+atj/npKei9hQlwSmLaByYofmTCcmHXz+ P8uKLt4Sook6RwNVWsRKl9Nl2yBXdw4HgWjqKw7zV2H4QKnzspfok7iZEpaiR1cZuKnm 9MkY6OX79J8McJa1swYOyAeIH6r1bZTe/1EQsxfTekMuaxpNKXm5GxAmYfd/qb1NL6Xa 26MA== X-Gm-Message-State: AOJu0YxacKNJZnepM3+MKefWuWDhBUpwannRChsYrHJzZW6uw9oRBVoC B7WDbUPeo0hfhByanY23Y9SdjfCkwb9iSuudxozk40KDQVyngSLw8drndhBQe9yqDx1YeR9LYbU = X-Gm-Gg: ASbGnctmESWD/NG3Jreqo7YFcbNFgtEKhmFf8Fusy6UJl9R30P+CxSYTAT0lOGvOfdH fc/f3sppUOmMg5x0FzmpXP78gn0pFy43nSpshfP8OMTvLI/35jkcHL5qX9jzDid4DQ1arLjjGM3 CbJnvO3uCJuN+ayKTLYXw2yMvQRT6FtCpvQgG8Jy6uLy6fcgJyl5eovK19JsHDakh+7PKzqeMQQ 7kTzvm6AMqua9jOTlAMP3qLFIEYTkp2GnZgUZ17eWD/BS2vISzMMJVpbsz6LwiCrGIKSEgU5HON wKOle4Ut3ehhgZokSURWbCanPj3y7OXt6L5ya4RCmkyMnqFY3W4agw== X-Google-Smtp-Source: AGHT+IGX9O2+8wobsriZv6lS9Aiw/HSGc3NBv9b4jgBUH4erXX0pxZ0f2Yxzqz5DWIlSeaj896fO1Q== X-Received: by 2002:a05:6870:ab07:b0:297:2bdc:f533 with SMTP id 586e51a60fabf-29dc43056fdmr2179644fac.31.1732699602218; Wed, 27 Nov 2024 01:26:42 -0800 (PST) Received: from chenghaoyang-low.c.googlers.com.com (27.247.221.35.bc.googleusercontent.com. [35.221.247.27]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fbcbfc41f9sm8693027a12.8.2024.11.27.01.26.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Nov 2024 01:26:41 -0800 (PST) From: Harvey Yang To: libcamera-devel@lists.libcamera.org Cc: Harvey Yang , Han-Lin Chen Subject: [PATCH v2 3/9] android: Set StreamBuffer::srcBuffer in CameraDevice::processCaptureRequest Date: Wed, 27 Nov 2024 09:25:53 +0000 Message-ID: <20241127092632.3145984-4-chenghaoyang@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241127092632.3145984-1-chenghaoyang@chromium.org> References: <20241127092632.3145984-1-chenghaoyang@chromium.org> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" StreamBuffer::srcBuffer was set right before being processed, while it was already determined when being constructed. This patch sets the value in CameraDevice::processCaptureRequest. Signed-off-by: Han-Lin Chen Co-developed-by: Harvey Yang Signed-off-by: Harvey Yang --- src/android/camera_device.cpp | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index 4e3bdc9cc..11613fac9 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -967,9 +967,9 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques * to a libcamera stream. Streams of type Mapped will be handled later. * * Collect the CameraStream associated to each requested capture stream. - * Since requestedStreams is an std:set<>, no duplications can happen. + * Since requestedStreams is an std:map<>, no duplications can happen. */ - std::set requestedStreams; + std::map requestedBuffers; for (const auto &[i, buffer] : utils::enumerate(descriptor->buffers_)) { CameraStream *cameraStream = buffer.stream; camera3_stream_t *camera3Stream = cameraStream->camera3Stream(); @@ -1008,6 +1008,8 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques cameraStream->configuration().size); frameBuffer = buffer.frameBuffer.get(); acquireFence = std::move(buffer.fence); + + requestedBuffers[cameraStream] = frameBuffer; LOG(HAL, Debug) << ss.str() << " (direct)"; break; @@ -1021,6 +1023,9 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques */ frameBuffer = cameraStream->getBuffer(); buffer.internalBuffer = frameBuffer; + buffer.srcBuffer = frameBuffer; + + requestedBuffers[cameraStream] = frameBuffer; LOG(HAL, Debug) << ss.str() << " (internal)"; descriptor->pendingStreamsToProcess_.insert( @@ -1036,8 +1041,6 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques auto fence = std::make_unique(std::move(acquireFence)); descriptor->request_->addBuffer(cameraStream->stream(), frameBuffer, std::move(fence)); - - requestedStreams.insert(cameraStream); } /* @@ -1068,8 +1071,19 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques */ CameraStream *sourceStream = cameraStream->sourceStream(); ASSERT(sourceStream); - if (requestedStreams.find(sourceStream) != requestedStreams.end()) + ASSERT(sourceStream->type() == CameraStream::Type::Direct); + + /* + * If the buffer for the source stream has been requested as + * Direct, use its framebuffer as the source buffer for + * post-processing. No need to recycle the buffer since it's + * owned by Android. + */ + auto iterDirectBuffer = requestedBuffers.find(sourceStream); + if (iterDirectBuffer != requestedBuffers.end()) { + buffer.srcBuffer = iterDirectBuffer->second; continue; + } /* * If that's not the case, we need to add a buffer to the request @@ -1077,11 +1091,12 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques */ FrameBuffer *frameBuffer = cameraStream->getBuffer(); buffer.internalBuffer = frameBuffer; + buffer.srcBuffer = frameBuffer; descriptor->request_->addBuffer(sourceStream->stream(), frameBuffer, nullptr); - requestedStreams.insert(sourceStream); + requestedBuffers[sourceStream] = frameBuffer; } /* @@ -1236,8 +1251,6 @@ void CameraDevice::requestComplete(Request *request) continue; } - buffer->srcBuffer = src; - ++iter; int ret = stream->process(buffer); if (ret) { From patchwork Wed Nov 27 09:25:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cheng-Hao Yang X-Patchwork-Id: 22115 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1563DC3213 for ; Wed, 27 Nov 2024 09:26:54 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C8D07660C2; Wed, 27 Nov 2024 10:26:53 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="UmKjDph6"; dkim-atps=neutral Received: from mail-ot1-x32c.google.com (mail-ot1-x32c.google.com [IPv6:2607:f8b0:4864:20::32c]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 366F9660BF for ; Wed, 27 Nov 2024 10:26:45 +0100 (CET) Received: by mail-ot1-x32c.google.com with SMTP id 46e09a7af769-71d554dedc6so1265692a34.1 for ; Wed, 27 Nov 2024 01:26:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732699604; x=1733304404; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kAnvO/uH9Pu56foWQBN126UILV1EQphGGewh74RvqyA=; b=UmKjDph6ERLdz627fzUkmT78sMMZ9meLpNAUzER4wH8WNhwYlzsaVQgcpmZkMpw5JS wuG1QY+erYOdt5mskt4jRV/icqm/ciFzBdvJQ4ydosHjdbATUoKy6Np7bZg6m5msHul8 XE601k+KPKHqLpMKHL6yJ0xPN21J9YmzbAZhI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732699604; x=1733304404; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kAnvO/uH9Pu56foWQBN126UILV1EQphGGewh74RvqyA=; b=eEoSicBijKLrG8bB1D2goZ/RvdiUqDZpj4SsjsYvSH1FsqNUcoZsKv6wOlQJO8WTN1 NrwARhV4H1pJpEVtsweoQtx7D5aQVORWQcTvfHDWbBbRXWJLoyZwjZfaVuEqdaPrU2TQ ivrFVu0UJbzkyRlp2pQSyOcGMEV5CIJjhEKixFc102MfecIR9H10St8AEqsDvj1x4Y2f mEG2dQpyraUDog0/vMyNrlurI0r8C0y4kfWJKW2HXvpXzdEnrNrjWpKZg07WsROEdr9t s480NT5mwvZ3c7Pnh4q4Fz73z5LzwyaJKAz3W3PmKaKF2BKKo4CR7Ngzmt0q4Vv5zV05 xCCw== X-Gm-Message-State: AOJu0Ywt1aeE+72Xf1A0Je2oMLYqJPl/o4y991CRWPOs37Jfvvu3EvgR NDAkJK0jRv9rWtPVz9HYHozMoJgBVsQhkZGrTMTQBUR5dombWRsKsORJtv82+I7swPajEKacPCY = X-Gm-Gg: ASbGncsMqFwSPW8QBoGwQ8vu5ALpvNcfaIfcWGMO/c5i5yjMNLD9W7y3wEjmwR1MGhw 2YH7YbLpt2TMQefyNAf1QDmv1W90Q2umTgjlhnq/kCnB9YroKw5sVkN5SeZRmQX2oU+qFX+7POy OxvbL7dEQODLWGgZ8u8nDvHLfx/Z4VwNo8fDPndE/hMEV6T7jlmShR2ROgI9g6h72ZOa77lwHQf 1Ats8rNWnoF3OPAPa8WisPb78TnxVCARiMO0dhLcW0pgqjjg6XKGWVxPIeN/DcRcYsVWhB8weRE DHPfKpGVI2N138Roix1XK5ukmiw5JqprwT0zi36PvdIZK1nEbqvfOw== X-Google-Smtp-Source: AGHT+IG1POox6Chy0wKAmoDBhSATg6WuLQ0htCvG79teD0c8dDEKe60LndYub7JwDianEcpKTyKzzQ== X-Received: by 2002:a05:6830:258a:b0:71d:5a17:75c6 with SMTP id 46e09a7af769-71d65d2a90fmr2240309a34.29.1732699603649; Wed, 27 Nov 2024 01:26:43 -0800 (PST) Received: from chenghaoyang-low.c.googlers.com.com (27.247.221.35.bc.googleusercontent.com. [35.221.247.27]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fbcbfc41f9sm8693027a12.8.2024.11.27.01.26.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Nov 2024 01:26:43 -0800 (PST) From: Harvey Yang To: libcamera-devel@lists.libcamera.org Cc: Harvey Yang , Han-Lin Chen Subject: [PATCH v2 4/9] android: Migrate StreamBuffer::internalBuffer to Camera3RequestDescriptor Date: Wed, 27 Nov 2024 09:25:54 +0000 Message-ID: <20241127092632.3145984-5-chenghaoyang@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241127092632.3145984-1-chenghaoyang@chromium.org> References: <20241127092632.3145984-1-chenghaoyang@chromium.org> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Previously there's a potential issue in StreamBuffer::internalBuffer's lifetime, when more than one streams depend on the same internal buffer. This patch fixes the issue by returning the buffer when the whole Camera3RequestDescriptor is destructed. Signed-off-by: Han-Lin Chen Co-developed-by: Harvey Yang Signed-off-by: Harvey Yang --- src/android/camera_device.cpp | 63 +++++++++++++++++++--------------- src/android/camera_request.cpp | 13 ++++--- src/android/camera_request.h | 3 +- 3 files changed, 46 insertions(+), 33 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index 11613fac9..62f724041 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -967,9 +967,10 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques * to a libcamera stream. Streams of type Mapped will be handled later. * * Collect the CameraStream associated to each requested capture stream. - * Since requestedStreams is an std:map<>, no duplications can happen. + * Since requestedDirectBuffers is an std:map<>, no duplications can + * happen. */ - std::map requestedBuffers; + std::map requestedDirectBuffers; for (const auto &[i, buffer] : utils::enumerate(descriptor->buffers_)) { CameraStream *cameraStream = buffer.stream; camera3_stream_t *camera3Stream = cameraStream->camera3Stream(); @@ -1009,7 +1010,7 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques frameBuffer = buffer.frameBuffer.get(); acquireFence = std::move(buffer.fence); - requestedBuffers[cameraStream] = frameBuffer; + requestedDirectBuffers[cameraStream] = frameBuffer; LOG(HAL, Debug) << ss.str() << " (direct)"; break; @@ -1018,14 +1019,17 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques * Get the frame buffer from the CameraStream internal * buffer pool. * - * The buffer has to be returned to the CameraStream - * once it has been processed. + * The buffer will be returned to the CameraStream when + * the request is destructed. */ frameBuffer = cameraStream->getBuffer(); - buffer.internalBuffer = frameBuffer; buffer.srcBuffer = frameBuffer; - requestedBuffers[cameraStream] = frameBuffer; + /* + * Track the allocated internal buffers, which will be + * recycled when the descriptor destroyed. + * */ + descriptor->internalBuffers_[cameraStream] = frameBuffer; LOG(HAL, Debug) << ss.str() << " (internal)"; descriptor->pendingStreamsToProcess_.insert( @@ -1079,24 +1083,41 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques * post-processing. No need to recycle the buffer since it's * owned by Android. */ - auto iterDirectBuffer = requestedBuffers.find(sourceStream); - if (iterDirectBuffer != requestedBuffers.end()) { + auto iterDirectBuffer = requestedDirectBuffers.find(sourceStream); + if (iterDirectBuffer != requestedDirectBuffers.end()) { buffer.srcBuffer = iterDirectBuffer->second; continue; } /* - * If that's not the case, we need to add a buffer to the request - * for this stream. + * If that's not the case, we use an internal buffer allocated + * from the source stream. + * + * If an internal buffer has been requested for the source + * stream before, we should reuse it. + */ + auto iterInternalBuffer = descriptor->internalBuffers_.find(sourceStream); + if (iterInternalBuffer != descriptor->internalBuffers_.end()) { + buffer.srcBuffer = iterInternalBuffer->second; + continue; + } + + /* + * Otherwise, we need to create an internal buffer to the + * request for the source stream. Get the frame buffer from the + * source stream's internal buffer pool. + * + * The buffer will be returned to the CameraStream when the + * request is destructed. */ - FrameBuffer *frameBuffer = cameraStream->getBuffer(); - buffer.internalBuffer = frameBuffer; + FrameBuffer *frameBuffer = sourceStream->getBuffer(); buffer.srcBuffer = frameBuffer; descriptor->request_->addBuffer(sourceStream->stream(), frameBuffer, nullptr); - requestedBuffers[sourceStream] = frameBuffer; + /* Track the allocated internal buffer. */ + descriptor->internalBuffers_[sourceStream] = frameBuffer; } /* @@ -1256,13 +1277,6 @@ void CameraDevice::requestComplete(Request *request) if (ret) { setBufferStatus(*buffer, StreamBuffer::Status::Error); descriptor->pendingStreamsToProcess_.erase(stream); - - /* - * If the framebuffer is internal to CameraStream return - * it back now that we're done processing it. - */ - if (buffer->internalBuffer) - stream->putBuffer(buffer->internalBuffer); } } @@ -1381,13 +1395,6 @@ void CameraDevice::streamProcessingComplete(StreamBuffer *streamBuffer, { setBufferStatus(*streamBuffer, status); - /* - * If the framebuffer is internal to CameraStream return it back now - * that we're done processing it. - */ - if (streamBuffer->internalBuffer) - streamBuffer->stream->putBuffer(streamBuffer->internalBuffer); - Camera3RequestDescriptor *request = streamBuffer->request; { diff --git a/src/android/camera_request.cpp b/src/android/camera_request.cpp index 52a3ac1f7..a9240a83c 100644 --- a/src/android/camera_request.cpp +++ b/src/android/camera_request.cpp @@ -10,6 +10,7 @@ #include #include "camera_buffer.h" +#include "camera_stream.h" using namespace libcamera; @@ -138,7 +139,14 @@ Camera3RequestDescriptor::Camera3RequestDescriptor( request_ = camera->createRequest(reinterpret_cast(this)); } -Camera3RequestDescriptor::~Camera3RequestDescriptor() = default; +Camera3RequestDescriptor::~Camera3RequestDescriptor() +{ + /* + * Recycle the allocated internal buffer back to its source stream. + */ + for (auto &[sourceStream, frameBuffer] : internalBuffers_) + sourceStream->putBuffer(frameBuffer); +} /** * \class StreamBuffer @@ -166,9 +174,6 @@ Camera3RequestDescriptor::~Camera3RequestDescriptor() = default; * \var StreamBuffer::status * \brief Track the status of the buffer * - * \var StreamBuffer::internalBuffer - * \brief Pointer to a buffer internally handled by CameraStream (if any) - * * \var StreamBuffer::srcBuffer * \brief Pointer to the source frame buffer used for post-processing * diff --git a/src/android/camera_request.h b/src/android/camera_request.h index 335f1985d..6b2a00795 100644 --- a/src/android/camera_request.h +++ b/src/android/camera_request.h @@ -49,7 +49,6 @@ public: std::unique_ptr frameBuffer; libcamera::UniqueFD fence; Status status = Status::Success; - libcamera::FrameBuffer *internalBuffer = nullptr; const libcamera::FrameBuffer *srcBuffer = nullptr; std::unique_ptr dstBuffer; Camera3RequestDescriptor *request; @@ -85,6 +84,8 @@ public: std::unique_ptr request_; std::unique_ptr resultMetadata_; + std::map internalBuffers_; + bool complete_ = false; Status status_ = Status::Success; From patchwork Wed Nov 27 09:25:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cheng-Hao Yang X-Patchwork-Id: 22116 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id CD28EC3213 for ; Wed, 27 Nov 2024 09:26:54 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4CA05660B9; Wed, 27 Nov 2024 10:26:54 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="UJkZ593m"; dkim-atps=neutral Received: from mail-ot1-x32f.google.com (mail-ot1-x32f.google.com [IPv6:2607:f8b0:4864:20::32f]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id ADD13660C2 for ; Wed, 27 Nov 2024 10:26:46 +0100 (CET) Received: by mail-ot1-x32f.google.com with SMTP id 46e09a7af769-71d46ace74aso1722954a34.3 for ; Wed, 27 Nov 2024 01:26:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732699605; x=1733304405; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pdGP+F5FAemY1y8l4UaN9IsfqrmqR26H1y8tizh1Imo=; b=UJkZ593mOFmNVw1u0Grq05ZEb+3btfLWoKVpplO/JGacuN5qYKJ96471vGXcFmZDqz RDDHFZBVgWDZjk81OS2F0BooP1nJSu+Zm96bDptJ4lmfl+hngE05DWwuzRGb5bFQbqba soQuEWxfF/x8js39b5RFETgE4+MV1vhFMPpMM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732699605; x=1733304405; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pdGP+F5FAemY1y8l4UaN9IsfqrmqR26H1y8tizh1Imo=; b=cTyr/r0RPJh1tnoFfaBBL3wa9bqYJvjGQHVrhTvBRT2MdF+hRVVisckTmFsMD7qhmy lfwo7y4G9Lo8VVu6qJ4chAd7/bNkCcElbvWN0pkECjoIY6bkCJJkoJ1sk9mKTZISktTQ E4QP3wkCC8IMkUFLbfw9PKetTx/sc6uCsImy+C3Q9JUL3X2wN1P46lxEPVsjj0uBWPO4 eCamA6ol5qmTYaq+Tpki+ou3dzwbkK0YYBpJn3YaLkwRpk+HPOc7Sv1edNM19Q5Y12pb HVZZOjfYUI2GLaYT9WmkrlRyyJrqf1aOY7SGIIg+q88LFBQcnlRmB4UB6fTLz90iRwN1 aP7A== X-Gm-Message-State: AOJu0Yx1+nLvbJDrMSOT+LQp1WgGhCpwQCTkS0pa+U4IoZwMwk27fUdy iHVWNZsVZORwMTpbhbECS3IUQkG2600TBMqBpL+hE4KSd9A2bPfUeynvGbA+M3Et4hy0LEjiLlI = X-Gm-Gg: ASbGncudYqQkZzJ2iDODLmwyalFyqG+y/yNsiJH4LLzUDuYokxRDdRbVsr1sK316DB/ cV9R477MRVIBKDY833e2o3XGi/G9sbSxEZhpaOTqFSSAW3ELjbtGgn5mpsJSMkEYgmN+cqt+Kpk SGTMoR+2mm0v4dHhBgENbzl2boRMZjboseH9xCIA6YaoeAHO2+h1t495LLdCidT0oF6MRd1jEin 6YfXouoo4OcmaSvPNc3gyM9HgiRo0rHbw7EPawiNwUVD+WUd+X1rtBsq93nFWh3BGYIbJD9MlS2 YJ1Yd5QNgce1aj+Ife1N5czR05gVYWOMo/zBVGdaNVW2QRUuDD+tqA== X-Google-Smtp-Source: AGHT+IHOqV76tqcIXHAHI6bqlt7KBWEEhU5HDabDmlEgvsrYEcYZ7A+GrEggmiRV6SHUxewvNJqp/Q== X-Received: by 2002:a05:6830:f94:b0:71d:5f65:ce48 with SMTP id 46e09a7af769-71d65d2809cmr2297014a34.28.1732699605234; Wed, 27 Nov 2024 01:26:45 -0800 (PST) Received: from chenghaoyang-low.c.googlers.com.com (27.247.221.35.bc.googleusercontent.com. [35.221.247.27]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fbcbfc41f9sm8693027a12.8.2024.11.27.01.26.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Nov 2024 01:26:44 -0800 (PST) From: Harvey Yang To: libcamera-devel@lists.libcamera.org Cc: Harvey Yang , Han-Lin Chen Subject: [PATCH v2 5/9] android: Add CameraDevice::sendCaptureResult() Date: Wed, 27 Nov 2024 09:25:55 +0000 Message-ID: <20241127092632.3145984-6-chenghaoyang@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241127092632.3145984-1-chenghaoyang@chromium.org> References: <20241127092632.3145984-1-chenghaoyang@chromium.org> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The new function is separated from sendCaptureResults(), which allows other functions to send a result out of order, like aborting. It'll be updated in the upcoming patch, when the usage of error result is separated. This function also allows the upcoming partial results to be sent back to the application without waiting for the requests. Signed-off-by: Han-Lin Chen Co-developed-by: Harvey Yang Signed-off-by: Harvey Yang Reviewed-by: Jacopo Mondi --- src/android/camera_device.cpp | 57 ++++++++++++++++++----------------- src/android/camera_device.h | 1 + 2 files changed, 31 insertions(+), 27 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index 62f724041..0377cf215 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -1324,42 +1324,45 @@ void CameraDevice::sendCaptureResults() auto descriptor = std::move(descriptors_.front()); descriptors_.pop(); - camera3_capture_result_t captureResult = {}; + sendCaptureResult(descriptor.get()); + } +} - captureResult.frame_number = descriptor->frameNumber_; +void CameraDevice::sendCaptureResult(Camera3RequestDescriptor *request) const +{ + std::vector resultBuffers; + resultBuffers.reserve(request->buffers_.size()); - if (descriptor->resultMetadata_) - captureResult.result = - descriptor->resultMetadata_->getMetadata(); + for (auto &buffer : request->buffers_) { + camera3_buffer_status status = CAMERA3_BUFFER_STATUS_ERROR; - std::vector resultBuffers; - resultBuffers.reserve(descriptor->buffers_.size()); + if (buffer.status == StreamBuffer::Status::Success) + status = CAMERA3_BUFFER_STATUS_OK; - for (auto &buffer : descriptor->buffers_) { - camera3_buffer_status status = CAMERA3_BUFFER_STATUS_ERROR; + /* + * Pass the buffer fence back to the camera framework as + * a release fence. This instructs the framework to wait + * on the acquire fence in case we haven't done so + * ourselves for any reason. + */ + resultBuffers.push_back({ buffer.stream->camera3Stream(), + buffer.camera3Buffer, status, + -1, buffer.fence.release() }); + } - if (buffer.status == StreamBuffer::Status::Success) - status = CAMERA3_BUFFER_STATUS_OK; + camera3_capture_result_t captureResult = {}; - /* - * Pass the buffer fence back to the camera framework as - * a release fence. This instructs the framework to wait - * on the acquire fence in case we haven't done so - * ourselves for any reason. - */ - resultBuffers.push_back({ buffer.stream->camera3Stream(), - buffer.camera3Buffer, status, - -1, buffer.fence.release() }); - } + captureResult.frame_number = request->frameNumber_; + captureResult.num_output_buffers = resultBuffers.size(); + captureResult.output_buffers = resultBuffers.data(); - captureResult.num_output_buffers = resultBuffers.size(); - captureResult.output_buffers = resultBuffers.data(); + if (request->status_ == Camera3RequestDescriptor::Status::Success) + captureResult.partial_result = 1; - if (descriptor->status_ == Camera3RequestDescriptor::Status::Success) - captureResult.partial_result = 1; + if (request->resultMetadata_) + captureResult.result = request->resultMetadata_->getMetadata(); - callbacks_->process_capture_result(callbacks_, &captureResult); - } + callbacks_->process_capture_result(callbacks_, &captureResult); } void CameraDevice::setBufferStatus(StreamBuffer &streamBuffer, diff --git a/src/android/camera_device.h b/src/android/camera_device.h index c92ee1aa4..699aa8f17 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -97,6 +97,7 @@ private: void completeDescriptor(Camera3RequestDescriptor *descriptor) LIBCAMERA_TSA_EXCLUDES(descriptorsMutex_); void sendCaptureResults() LIBCAMERA_TSA_REQUIRES(descriptorsMutex_); + void sendCaptureResult(Camera3RequestDescriptor *request) const; void setBufferStatus(StreamBuffer &buffer, StreamBuffer::Status status); std::unique_ptr getResultMetadata( From patchwork Wed Nov 27 09:25:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cheng-Hao Yang X-Patchwork-Id: 22117 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 43C61C3213 for ; Wed, 27 Nov 2024 09:26:57 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 924A6660B9; Wed, 27 Nov 2024 10:26:56 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="YYtWI9nA"; dkim-atps=neutral Received: from mail-ot1-x334.google.com (mail-ot1-x334.google.com [IPv6:2607:f8b0:4864:20::334]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 314D3660C7 for ; Wed, 27 Nov 2024 10:26:48 +0100 (CET) Received: by mail-ot1-x334.google.com with SMTP id 46e09a7af769-71d4d0516e6so1696532a34.2 for ; Wed, 27 Nov 2024 01:26:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732699607; x=1733304407; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7qf7TjoxNHo5OLDoWBsF8q4aXijLrJm4Gl0VuJh1ic0=; b=YYtWI9nAgdyIEDo6UZU9eei0bgK9iPDVCTowryk1ztNoC8ZBuG8RENRpNTwS2h6i8i 4KIGJdBShV0AYpLdSmIcZevV+IeNdqnjr4A51HJBzb4l2Z4ArT+es+T/QnC9SCzWwGs4 GbmJgffafwjkbTwFzizGhnbmjXGm/+lh/ipWY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732699607; x=1733304407; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7qf7TjoxNHo5OLDoWBsF8q4aXijLrJm4Gl0VuJh1ic0=; b=QgehKqgbBZZf7Vx/R8i8vphYph/jTb7Zj0mKZi/OIMXsQvyfuoaDq9BLlCWysSO5lx ZAesw18X4HeRHLp9jb0B2veAEXboSSb24Xi2Z79rujcXEDyQuzzgzePG56pXZYERSnw1 qR0zn5vib00HveE1FP+wL6nrZNRF5CdOkqyjHRKZSInf3sE6/ZYJJiNH/HXZ9GMVqz5G YhRR8e0AD++itOpWp+9WYEQuvBh7vC1CZVqpLa3ztfj07rM5YlMHlXj41OD2gPBPMzl9 IgDu4oHOJ96mi86JYi1Vuw9w4/CQhRrHIx09lDSdTj+xWG8sAFZSqwy8hxTMRNpno4TL +3Pg== X-Gm-Message-State: AOJu0YyhlSLvZrLd0OF/+viwlY/DufV8QQ+m8LTk8JsxZogl4VRa1M6l 9dhhtfTVIqBk6dTi77QW7FFRlWxPqaAAO2j40tkr3yQDNP2gkqlEPE7wfRUh0kBjU6huUSaQLDc = X-Gm-Gg: ASbGnctXXC3VzdItL97FzqO5YdEEmDWL5+HPqZwtBwacjmgZZZZH3CvSByz7rv4g4g8 D38P7yP/yKKrq0Ems/B/aaRuUBPIYwMKFSfL1+Eb2QLiMD/lktBLNHste7wJFFZVAtqgyakKADX nBA/mKgksJm8FnSd+UTuXf7DSSARXai//Ze7OhaI2jLo1qb6dIy8j9IX0kjhaJ2TnU/qz6cvtGW G46GJeCpovFf2TQveg1FIAyHhr09isz2hvoIj7CGktSe3Gi3ekg8QzqaPuLIoFjXBDbMAl3wC2M oz581frw7j8vF3/rKqTPIOscnUE2dXxxLM7EZ++k3DOdt/PhkW3Msg== X-Google-Smtp-Source: AGHT+IErrD10G3jBi+mWN3Zjm+Ew7Cwjj1wFpafYaRzKTRTL+0s/QFh7z1QoDsMjCii7PQxoU/V0RQ== X-Received: by 2002:a05:6830:3805:b0:71d:4d88:2293 with SMTP id 46e09a7af769-71d65cf62c7mr2174243a34.23.1732699606734; Wed, 27 Nov 2024 01:26:46 -0800 (PST) Received: from chenghaoyang-low.c.googlers.com.com (27.247.221.35.bc.googleusercontent.com. [35.221.247.27]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fbcbfc41f9sm8693027a12.8.2024.11.27.01.26.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Nov 2024 01:26:46 -0800 (PST) From: Harvey Yang To: libcamera-devel@lists.libcamera.org Cc: Harvey Yang , Han-Lin Chen Subject: [PATCH v2 6/9] android: Cleanup CAMERA3_MSG_ERROR_REQUEST Date: Wed, 27 Nov 2024 09:25:56 +0000 Message-ID: <20241127092632.3145984-7-chenghaoyang@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241127092632.3145984-1-chenghaoyang@chromium.org> References: <20241127092632.3145984-1-chenghaoyang@chromium.org> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When a request is completed with failure, CAMERA3_MSG_ERROR_RESULT should be used instead. This patch also cleans up aborting when flushing with sendCaptureResult(). Signed-off-by: Han-Lin Chen Co-developed-by: Harvey Yang Signed-off-by: Harvey Yang --- src/android/camera_device.cpp | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index 0377cf215..3fb92268e 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -868,6 +868,8 @@ void CameraDevice::abortRequest(Camera3RequestDescriptor *descriptor) const buffer.status = StreamBuffer::Status::Error; descriptor->status_ = Camera3RequestDescriptor::Status::Error; + + sendCaptureResult(descriptor); } bool CameraDevice::isValidRequest(camera3_capture_request_t *camera3Request) const @@ -1136,14 +1138,7 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques MutexLocker stateLock(stateMutex_); if (state_ == State::Flushing) { - Camera3RequestDescriptor *rawDescriptor = descriptor.get(); - { - MutexLocker descriptorsLock(descriptorsMutex_); - descriptors_.push(std::move(descriptor)); - } - abortRequest(rawDescriptor); - completeDescriptor(rawDescriptor); - + abortRequest(descriptor.get()); return 0; } @@ -1211,10 +1206,7 @@ void CameraDevice::requestComplete(Request *request) << " not successfully completed: " << request->status(); - abortRequest(descriptor); - completeDescriptor(descriptor); - - return; + descriptor->status_ = Camera3RequestDescriptor::Status::Error; } /* @@ -1239,7 +1231,7 @@ void CameraDevice::requestComplete(Request *request) */ descriptor->resultMetadata_ = getResultMetadata(*descriptor); if (!descriptor->resultMetadata_) { - notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_RESULT); + descriptor->status_ = Camera3RequestDescriptor::Status::Error; /* * The camera framework expects an empty metadata pack on error. @@ -1325,6 +1317,16 @@ void CameraDevice::sendCaptureResults() descriptors_.pop(); sendCaptureResult(descriptor.get()); + + /* + * Call notify with CAMERA3_MSG_ERROR_RESULT to indicate some + * of the expected result metadata might not be available + * because the capture is cancelled by the camera. Only notify + * it when the final result is sent, since Android will ignore + * the following metadata. + */ + if (descriptor->status_ == Camera3RequestDescriptor::Status::Error) + notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_RESULT); } } From patchwork Wed Nov 27 09:25:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cheng-Hao Yang X-Patchwork-Id: 22118 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 29ED3C3213 for ; Wed, 27 Nov 2024 09:26:59 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A59F5660D7; Wed, 27 Nov 2024 10:26:58 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="csaFehX9"; dkim-atps=neutral Received: from mail-ot1-x330.google.com (mail-ot1-x330.google.com [IPv6:2607:f8b0:4864:20::330]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C2947660C4 for ; Wed, 27 Nov 2024 10:26:49 +0100 (CET) Received: by mail-ot1-x330.google.com with SMTP id 46e09a7af769-71d4c462ab1so1390747a34.0 for ; Wed, 27 Nov 2024 01:26:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732699608; x=1733304408; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vtXzRx/bGtcOeUBwLnIhf0TSMieqneRSDdL6wR8sPsI=; b=csaFehX9ijCuzV5eHRoRACyV6/AlhPdJAO69mrUz20/1RU6VlbOyGyTpzpgZ5TtxeD DqqNt2ZBqyLG4eIdCp1M2Fdm8COtHbuTkzTS56siRekopRzoOuHAvaes81rk0jogcad6 jU3kQkbOihAMi2yLP+196Mt5dnRA3bArFJQKY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732699608; x=1733304408; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vtXzRx/bGtcOeUBwLnIhf0TSMieqneRSDdL6wR8sPsI=; b=qKYVJJNfuLZ+amFoVvwu8qlf+d1sGYuWsQjVos1bxYms8SpMWUMVHTmS5okJN80pId VL9fGhRoXnYOp+vC23EHLFG4n+hW5+sIzFc+UWr2lF8zn1v8LUQClAl8zkT5KJY+88wh oVg1EPW4gjbgcKdgNBjrM6OjiG+1qj7xDIax2fhA6eu21ih738PO6Xii31mrkH1K9tpU 2OBVTre9yk2hZcIkJyrRnHKgYQaFJVyqkpgQSiv5UlyyKaPy/VDOLj96bvNZn+E9E+BY SBqQ/YfRSWoFVxZJpswEhqwQ6gtleqZs6oESbH8WAx1sTRf/ommLFcBZUXdEAWwaZIBT jYxw== X-Gm-Message-State: AOJu0YztDp4Eigr/9dIlQwdmIkYkJGFhbo4NxhSZHBfdhm11iDeGGaSu ojLU/34/7y0pnOmc2OnTGo+ctvxxDFHO4afOhqkm/1diDmKaQHPBd9meWCCs1tJ3Yx/gGHcajuU = X-Gm-Gg: ASbGnctNMSHoxpK/6Fi0Gaw5vU49U+46/CW/cKP89i5NXUzxVBdedr3jvl9FYSBiJdF owHxsxRtM/A2+jWgqV1YIEsJn30fSLxTEP/wJOGbttbvFnUSIVwDSekw/ZDFgK1tKrsoIQ+4uZK NLp+shnX/es78SqPoDF+f9Afl51I1XjsTxuhDlLhZnmLBQw+KDF0Cl3uwrv4BH7yDw93qjoRwCb iAmfvjr1Y+v+WAyTxKncbu3BF/45gw4NNNGCdunlWm+ntWNcXJEBB4ZedR1k6hB3tzAodVkizlc F7gnZnUjhkBUPR2rrW/0ZHqxBH1LcBdimAMCZJTJtJYVNekLx3WXUg== X-Google-Smtp-Source: AGHT+IEcBk0l7j0pXW0yfCYpqfM5tIad4tZ+ngME+MjUvHI6hhelFajbwNIyRv4Lmbfhcmgw0XKGng== X-Received: by 2002:a05:6830:6c09:b0:71d:5a2d:a1d9 with SMTP id 46e09a7af769-71d65c94f26mr1887976a34.12.1732699608267; Wed, 27 Nov 2024 01:26:48 -0800 (PST) Received: from chenghaoyang-low.c.googlers.com.com (27.247.221.35.bc.googleusercontent.com. [35.221.247.27]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fbcbfc41f9sm8693027a12.8.2024.11.27.01.26.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Nov 2024 01:26:47 -0800 (PST) From: Harvey Yang To: libcamera-devel@lists.libcamera.org Cc: Harvey Yang , Han-Lin Chen Subject: [PATCH v2 7/9] android: Remove Camera3RequestDescriptor::streamsProcessMutex_ Date: Wed, 27 Nov 2024 09:25:57 +0000 Message-ID: <20241127092632.3145984-8-chenghaoyang@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241127092632.3145984-1-chenghaoyang@chromium.org> References: <20241127092632.3145984-1-chenghaoyang@chromium.org> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This mutex was needed when CameraStream's worker thread posts a result back to CameraDevice. We can simplify it by calling CameraDevice's function on libcamera::Camera's owner thread. With this delegation, `Camera3RequestDescriptor::pendingStreamsToProcess_` will be firstly setup in the application's thread in processCaptureRequest(), and the rest accesses will be done in libcamera::CameraManager's thread. Signed-off-by: Han-Lin Chen Co-developed-by: Harvey Yang Signed-off-by: Harvey Yang --- src/android/camera_device.cpp | 33 +++++++++++++++++---------------- src/android/camera_device.h | 2 ++ src/android/camera_request.h | 4 +--- 3 files changed, 20 insertions(+), 19 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index 3fb92268e..9fd851bc8 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -991,8 +991,6 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques FrameBuffer *frameBuffer = nullptr; UniqueFD acquireFence; - MutexLocker lock(descriptor->streamsProcessMutex_); - switch (cameraStream->type()) { case CameraStream::Type::Mapped: /* Mapped streams will be handled in the next loop. */ @@ -1068,7 +1066,6 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques << cameraStream->configuration().pixelFormat << "]" << " (mapped)"; - MutexLocker lock(descriptor->streamsProcessMutex_); descriptor->pendingStreamsToProcess_.insert({ cameraStream, &buffer }); /* @@ -1242,9 +1239,6 @@ void CameraDevice::requestComplete(Request *request) descriptor->resultMetadata_ = std::make_unique(0, 0); } - /* Handle post-processing. */ - MutexLocker locker(descriptor->streamsProcessMutex_); - /* * Queue all the post-processing streams request at once. The completion * slot streamProcessingComplete() can only execute when we are out @@ -1272,10 +1266,8 @@ void CameraDevice::requestComplete(Request *request) } } - if (descriptor->pendingStreamsToProcess_.empty()) { - locker.unlock(); + if (descriptor->pendingStreamsToProcess_.empty()) completeDescriptor(descriptor); - } } /** @@ -1382,6 +1374,19 @@ void CameraDevice::setBufferStatus(StreamBuffer &streamBuffer, } } +void CameraDevice::streamProcessingCompleteDelegate(StreamBuffer *streamBuffer, + StreamBuffer::Status status) +{ + /* + * Delegate the callback to the camera manager thread to simplify race condition. + */ + auto *method = new BoundMethodMember{ + this, camera_.get(), &CameraDevice::streamProcessingComplete, ConnectionTypeQueued + }; + + method->activate(streamBuffer, status); +} + /** * \brief Handle post-processing completion of a stream in a capture request * \param[in] streamBuffer The StreamBuffer for which processing is complete @@ -1402,13 +1407,9 @@ void CameraDevice::streamProcessingComplete(StreamBuffer *streamBuffer, Camera3RequestDescriptor *request = streamBuffer->request; - { - MutexLocker locker(request->streamsProcessMutex_); - - request->pendingStreamsToProcess_.erase(streamBuffer->stream); - if (!request->pendingStreamsToProcess_.empty()) - return; - } + request->pendingStreamsToProcess_.erase(streamBuffer->stream); + if (!request->pendingStreamsToProcess_.empty()) + return; completeDescriptor(streamBuffer->request); } diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 699aa8f17..815a695d1 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -65,6 +65,8 @@ public: int configureStreams(camera3_stream_configuration_t *stream_list); int processCaptureRequest(camera3_capture_request_t *request); void requestComplete(libcamera::Request *request); + void streamProcessingCompleteDelegate(StreamBuffer *bufferStream, + StreamBuffer::Status status); void streamProcessingComplete(StreamBuffer *bufferStream, StreamBuffer::Status status); diff --git a/src/android/camera_request.h b/src/android/camera_request.h index 6b2a00795..bd75d4595 100644 --- a/src/android/camera_request.h +++ b/src/android/camera_request.h @@ -66,9 +66,7 @@ public: }; /* Keeps track of streams requiring post-processing. */ - std::map pendingStreamsToProcess_ - LIBCAMERA_TSA_GUARDED_BY(streamsProcessMutex_); - libcamera::Mutex streamsProcessMutex_; + std::map pendingStreamsToProcess_; Camera3RequestDescriptor(libcamera::Camera *camera, const camera3_capture_request_t *camera3Request); From patchwork Wed Nov 27 09:25:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cheng-Hao Yang X-Patchwork-Id: 22119 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 3853EC3213 for ; Wed, 27 Nov 2024 09:27:01 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A11EC660B9; Wed, 27 Nov 2024 10:27:00 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="cnuCQ81b"; dkim-atps=neutral Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DD997660C2 for ; Wed, 27 Nov 2024 10:26:51 +0100 (CET) Received: by mail-pg1-x52b.google.com with SMTP id 41be03b00d2f7-7e6cbf6cd1dso4453571a12.3 for ; Wed, 27 Nov 2024 01:26:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732699610; x=1733304410; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AoZO/hfgntIi/tXlmTlmoFRfHC/LykvzY6f4ocf5hXU=; b=cnuCQ81bvlVoz5eoK2Q42jr9WzIELJUokfbYRkXOqm04Ezz7ucLqfIaZG3OrpJTRXB 5pcNBLBf1oqbGRXe6dE98In4EkeebumMq260BqyFZmogajr9y2qs1sEMx3T5j73rQwcb lraqpFsMAwP0vUBQJEiRHfB48jI2YXlf2Pu1M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732699610; x=1733304410; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AoZO/hfgntIi/tXlmTlmoFRfHC/LykvzY6f4ocf5hXU=; b=gs2nJbvk/HSyp41uXnQa6bblZm1Q0WwyZ1ozBR9qOGgGczewEUHFj5NCYo4L+lQI5x WEUyR2kIqqzekBvPgdl8pCjXs365AlI/wLQ5c5itCnfureOS081/LfTzxLemZk6mOQkQ 3BVbeD750t3p+xOznzM9OhDrhLRSKpipWfXbaq18kLADCMxdZGj1qoKbQpkM5D5iiHVG Mbx/OaK31PHcm1xS0Tn5Vm8jdeTELd3Q/3hp5mHWb9q4XrezqJl4AiHx5jYFnEff7WlI OqPgruvyC5wvV0oisFKuPrvR6g11PCwysjP2RQr4CcZs10nNOvger8Gwgp5m6JxgRlbK zz6A== X-Gm-Message-State: AOJu0YzixCRTB5uxvpgW0DIPvIHDHsPsAxwelPFeQhAz+9M1gw6Y7+TA /PmN99UTKRcX9FoTlkU5hU3Nv0SHfNUdsciFTvkvioGQ4EAuyqR/KFgXi3OfpejSK5gGvt2dudY = X-Gm-Gg: ASbGnctbefb6Y4XCeV8T1C6Xq+FQ/fkUickn5rhFFRiZl7AE3Kmst+cv7QqsMwOr5DS rzJmRynEnRgFoDizs2jMmmLwVsc+fn5M1ArniBRDeHeEbr8TWGya2w1Pc8Wzvm1PZuU7643sUtr 9znYmDhhqMAUzjpPNI9vySlz/5tNJt/0KZ7i8zMmY/wQqKf3N/RtCjv3E6SGIomXS9QFd4ID9Mw 4lLLlcsW971YabYAKZpn+QaMDYuqj5Sm1FaiOHlBXdyKqTJ3Dyty2Q2iBnfXuH07BgxTsAwkzzj Ge/WXBGs/Rmxg1VGIjyKzgsA+YFijYoGOugniKb+jHGOnvEjZtkFQA== X-Google-Smtp-Source: AGHT+IHv6ezMs3I6FehAsRDLUMqTqh8pWgFh4NDxShAem9+bDFVkJY8laV14HZD4vuUotwKdgYYWJA== X-Received: by 2002:a05:6a20:2583:b0:1db:dae1:22e5 with SMTP id adf61e73a8af0-1e0e0b2c74emr5364897637.20.1732699609722; Wed, 27 Nov 2024 01:26:49 -0800 (PST) Received: from chenghaoyang-low.c.googlers.com.com (27.247.221.35.bc.googleusercontent.com. [35.221.247.27]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fbcbfc41f9sm8693027a12.8.2024.11.27.01.26.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Nov 2024 01:26:49 -0800 (PST) From: Harvey Yang To: libcamera-devel@lists.libcamera.org Cc: Han-Lin Chen , Harvey Yang Subject: [PATCH v2 8/9] android: Add JpegExifMetadata to store tags setting into Exif Date: Wed, 27 Nov 2024 09:25:58 +0000 Message-ID: <20241127092632.3145984-9-chenghaoyang@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241127092632.3145984-1-chenghaoyang@chromium.org> References: <20241127092632.3145984-1-chenghaoyang@chromium.org> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Han-Lin Chen With partial result, some metadata, which needs to be added into Exif, may be sent back to framework earlier before Jpeg post-processing. Add a type JpegExifMetadata associated with StreamBuffer to store the values, so Jpeg post-processing doesn't need to reference to current metadata. Signed-off-by: Han-Lin Chen Co-developed-by: Harvey Yang Signed-off-by: Harvey Yang --- src/android/camera_device.cpp | 26 ++++++++++++++++++++++++ src/android/camera_device.h | 2 ++ src/android/camera_request.h | 6 ++++++ src/android/camera_stream.h | 4 ++++ src/android/jpeg/post_processor_jpeg.cpp | 12 ++++++----- 5 files changed, 45 insertions(+), 5 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index 9fd851bc8..e085e18b2 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -1250,6 +1250,10 @@ void CameraDevice::requestComplete(Request *request) CameraStream *stream = iter->first; StreamBuffer *buffer = iter->second; + if (stream->isJpegStream()) { + generateJpegExifMetadata(descriptor, buffer); + } + FrameBuffer *src = request->findBuffer(stream->stream()); if (!src) { LOG(HAL, Error) << "Failed to find a source stream buffer"; @@ -1443,6 +1447,28 @@ void CameraDevice::notifyError(uint32_t frameNumber, camera3_stream_t *stream, callbacks_->notify(callbacks_, ¬ify); } +/* + * Set jpeg metadata used to generate EXIF in the JPEG post processing. + */ +void CameraDevice::generateJpegExifMetadata(Camera3RequestDescriptor *request, + StreamBuffer *buffer) const +{ + const ControlList &metadata = request->request_->metadata(); + auto &jpegExifMetadata = buffer->jpegExifMetadata; + jpegExifMetadata.emplace(StreamBuffer::JpegExifMetadata()); + + const int64_t exposureTime = metadata.get(controls::ExposureTime).value_or(0); + jpegExifMetadata->sensorExposureTime = exposureTime; + + /* + * todo: Android Sensitivity should only include analog gain X digital + * gain from sensor. Digital gain on ISP shouldn't be included. + * Calculate sensitivity accordingly when we can differentiate + * the source of digital gains. + */ + jpegExifMetadata->sensorSensitivityISO = 100; +} + /* * Produce a set of fixed result metadata. */ diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 815a695d1..3c46ff918 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -102,6 +102,8 @@ private: void sendCaptureResult(Camera3RequestDescriptor *request) const; void setBufferStatus(StreamBuffer &buffer, StreamBuffer::Status status); + void generateJpegExifMetadata(Camera3RequestDescriptor *request, + StreamBuffer *buffer) const; std::unique_ptr getResultMetadata( const Camera3RequestDescriptor &descriptor) const; diff --git a/src/android/camera_request.h b/src/android/camera_request.h index bd75d4595..bd87b36fd 100644 --- a/src/android/camera_request.h +++ b/src/android/camera_request.h @@ -44,6 +44,11 @@ public: StreamBuffer(StreamBuffer &&); StreamBuffer &operator=(StreamBuffer &&); + struct JpegExifMetadata { + int64_t sensorExposureTime; + int32_t sensorSensitivityISO; + }; + CameraStream *stream; buffer_handle_t *camera3Buffer; std::unique_ptr frameBuffer; @@ -51,6 +56,7 @@ public: Status status = Status::Success; const libcamera::FrameBuffer *srcBuffer = nullptr; std::unique_ptr dstBuffer; + std::optional jpegExifMetadata; Camera3RequestDescriptor *request; private: diff --git a/src/android/camera_stream.h b/src/android/camera_stream.h index 30f64f690..47cd7ab85 100644 --- a/src/android/camera_stream.h +++ b/src/android/camera_stream.h @@ -125,6 +125,10 @@ public: const libcamera::StreamConfiguration &configuration() const; libcamera::Stream *stream() const; CameraStream *sourceStream() const { return sourceStream_; } + bool isJpegStream() const + { + return camera3Stream_->format == HAL_PIXEL_FORMAT_BLOB; + } int configure(); int process(StreamBuffer *streamBuffer); diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp index f5a90785d..48782b574 100644 --- a/src/android/jpeg/post_processor_jpeg.cpp +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -112,8 +112,11 @@ void PostProcessorJpeg::process(StreamBuffer *streamBuffer) const FrameBuffer &source = *streamBuffer->srcBuffer; CameraBuffer *destination = streamBuffer->dstBuffer.get(); + const std::optional &jpegExifMetadata = + streamBuffer->jpegExifMetadata; ASSERT(destination->numPlanes() == 1); + ASSERT(jpegExifMetadata.has_value()); const CameraMetadata &requestMetadata = streamBuffer->request->settings_; CameraMetadata *resultMetadata = streamBuffer->request->resultMetadata_.get(); @@ -139,15 +142,14 @@ void PostProcessorJpeg::process(StreamBuffer *streamBuffer) */ exif.setTimestamp(std::time(nullptr), 0ms); - ret = resultMetadata->getEntry(ANDROID_SENSOR_EXPOSURE_TIME, &entry); - exif.setExposureTime(ret ? *entry.data.i64 : 0); + /* Exif requires nsec for exposure time */ + exif.setExposureTime(jpegExifMetadata->sensorExposureTime * 1000); + exif.setISO(jpegExifMetadata->sensorSensitivityISO); + ret = requestMetadata.getEntry(ANDROID_LENS_APERTURE, &entry); if (ret) exif.setAperture(*entry.data.f); - ret = resultMetadata->getEntry(ANDROID_SENSOR_SENSITIVITY, &entry); - exif.setISO(ret ? *entry.data.i32 : 100); - exif.setFlash(Exif::Flash::FlashNotPresent); exif.setWhiteBalance(Exif::WhiteBalance::Auto); From patchwork Wed Nov 27 09:25:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Cheng-Hao Yang X-Patchwork-Id: 22120 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 65A83C3213 for ; Wed, 27 Nov 2024 09:27:02 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D9EC6660DA; Wed, 27 Nov 2024 10:27:01 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="n6+11Lh4"; dkim-atps=neutral Received: from mail-pg1-x52e.google.com (mail-pg1-x52e.google.com [IPv6:2607:f8b0:4864:20::52e]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id EA850660D1 for ; Wed, 27 Nov 2024 10:26:53 +0100 (CET) Received: by mail-pg1-x52e.google.com with SMTP id 41be03b00d2f7-7fbc29b3145so459468a12.0 for ; Wed, 27 Nov 2024 01:26:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732699612; x=1733304412; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=h4NeHvcJcw/iNl+9WLU3pCWOhShUrqZjicvR0HM/C9M=; b=n6+11Lh4fUEy4HPVW2HGQhI1jhLPSUVx2WGC5Uy3iI8bW3Gzxplu/RIyPIIT9kb5ms 4qI7hUqiTockGKa1F2EGWn34OorXgsF2q/YGS5Ui4H+c/hXk6GrXA638Y+eix9fgrhDM ambH9H/36QxDSP4DazuQspFQLHpAy37llRobY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732699612; x=1733304412; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h4NeHvcJcw/iNl+9WLU3pCWOhShUrqZjicvR0HM/C9M=; b=rt2mNhD3t6JcpvxAWxNjf2z/koh7AM8MW+P2xh3lVL3F27uNxy32OrJctazuPkjqSP DdYxWpsO2g/Um/p9AAM2HVsr+s4RppoNY5HvKZfDyml79+BpcpP6Q0YqxlhyhJgAdDJE rDRzY7dWJNTUWEEiKbOHgCeQCh9+qbV8CoWfbSBnv0j+86zkNqP0PKO5tPkJas2dOlua 7EwgxVza9sS/j+Q7vIjsv4ATS6hMNyl9+OZBbevNHLUc2YI9SnmEtUHng+ESvouf0nAM OFUZzw3EFrtkAzo3m5FVbrT7H5PPFjXMkd9hLDaNbC9/FGxf4A2Pnyg41L0KpTKVy20P jc/w== X-Gm-Message-State: AOJu0YxyogkgoU0b0UyeZmDbm4JGZKoxJcXXp9quG+4lCXOCh03meENu vBECNpQ9vZoPPfeLypqwLXsjvAniL/K0KOJEA9xBJFYFJwBY0nZtCjnhVq13K2uUJzJoj5/pvlI = X-Gm-Gg: ASbGncuxUsnRuNphpj4nFZJZFKRDI0xjGjR46ARnNF9AsVkk9+uEaGDP8kgoCcinw5H zGLzdNIpbQBIgO/7JWfKkDgPKiqvOHhp58XpfI/eNPm0Sp5YFq+QG9XAJeBMthpTblDjtHlm60e Gnk84k2JL7vuA1/TdGmXl6FBRwqTlUCS/p3mCEWYiSTCVI96vl7C1O/FQVCLE3k4NufJ1eGop5B yXhPcUbxEAAMO2xGRTFli1NSPN5+OYkz4o+iTFJMVYOQbLCtd2U1ofWUpOsz4utt/QXSEoBE88A gIYEmg+/ZWFWxAmsV92nnJ1XvaTwvKd+ppeQv6bj7Sb+he5/RWN51g== X-Google-Smtp-Source: AGHT+IHyW5+aq2GchDfLw2B+xVi6QvHTr/Wa3l6PDNmnoaDNY3QtE4hij+hz1jbccypPmStM+omGdQ== X-Received: by 2002:a05:6a20:4393:b0:1e0:d5be:bf75 with SMTP id adf61e73a8af0-1e0e11badd8mr3569104637.17.1732699611400; Wed, 27 Nov 2024 01:26:51 -0800 (PST) Received: from chenghaoyang-low.c.googlers.com.com (27.247.221.35.bc.googleusercontent.com. [35.221.247.27]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fbcbfc41f9sm8693027a12.8.2024.11.27.01.26.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Nov 2024 01:26:50 -0800 (PST) From: Harvey Yang To: libcamera-devel@lists.libcamera.org Cc: Harvey Yang , Han-Lin Chen Subject: [PATCH v2 9/9] android: Support partial results Date: Wed, 27 Nov 2024 09:25:59 +0000 Message-ID: <20241127092632.3145984-10-chenghaoyang@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241127092632.3145984-1-chenghaoyang@chromium.org> References: <20241127092632.3145984-1-chenghaoyang@chromium.org> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" With bufferCompleted and metadataAvailable signals, CameraDevice can support partial results. This allows applications to get results earlier, especially for buffers that would be blocked by other streams. Signed-off-by: Han-Lin Chen Co-developed-by: Harvey Yang Signed-off-by: Harvey Yang --- src/android/camera_capabilities.cpp | 11 +- src/android/camera_capabilities.h | 2 + src/android/camera_device.cpp | 750 ++++++++++++++++------- src/android/camera_device.h | 39 +- src/android/camera_request.cpp | 54 +- src/android/camera_request.h | 36 +- src/android/camera_stream.cpp | 4 +- src/android/jpeg/post_processor_jpeg.cpp | 2 +- 8 files changed, 621 insertions(+), 277 deletions(-) diff --git a/src/android/camera_capabilities.cpp b/src/android/camera_capabilities.cpp index b161bc6b3..bb0a3b755 100644 --- a/src/android/camera_capabilities.cpp +++ b/src/android/camera_capabilities.cpp @@ -223,6 +223,14 @@ std::vector setMetadata(CameraMetadata *metadata, uint32_t tag, } /* namespace */ +/** + * \var CameraCapabilities::kMaxMetadataPackIndex + * + * It defines how many sub-components a result will be composed of. This enables + * partial results. It's currently identical to + * ANDROID_REQUEST_PARTIAL_RESULT_COUNT. + */ + bool CameraCapabilities::validateManualSensorCapability() { const char *noMode = "Manual sensor capability unavailable: "; @@ -1416,9 +1424,8 @@ int CameraCapabilities::initializeStaticMetadata() staticMetadata_->addEntry(ANDROID_SCALER_CROPPING_TYPE, croppingType); /* Request static metadata. */ - int32_t partialResultCount = 1; staticMetadata_->addEntry(ANDROID_REQUEST_PARTIAL_RESULT_COUNT, - partialResultCount); + kMaxMetadataPackIndex); { /* Default the value to 2 if not reported by the camera. */ diff --git a/src/android/camera_capabilities.h b/src/android/camera_capabilities.h index 56ac1efeb..b11f93241 100644 --- a/src/android/camera_capabilities.h +++ b/src/android/camera_capabilities.h @@ -23,6 +23,8 @@ class CameraCapabilities { public: + static constexpr int32_t kMaxMetadataPackIndex = 64; + CameraCapabilities() = default; int initialize(std::shared_ptr camera, diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index e085e18b2..f03440b79 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -252,6 +252,8 @@ CameraDevice::CameraDevice(unsigned int id, std::shared_ptr camera) facing_(CAMERA_FACING_FRONT), orientation_(0) { camera_->requestCompleted.connect(this, &CameraDevice::requestComplete); + camera_->bufferCompleted.connect(this, &CameraDevice::bufferComplete); + camera_->metadataAvailable.connect(this, &CameraDevice::metadataAvailable); maker_ = "libcamera"; model_ = "cameraModel"; @@ -438,8 +440,9 @@ void CameraDevice::stop() camera_->stop(); { - MutexLocker descriptorsLock(descriptorsMutex_); - descriptors_ = {}; + MutexLocker descriptorsLock(pendingRequestMutex_); + pendingRequests_ = {}; + pendingStreamBuffers_ = {}; } streams_.clear(); @@ -860,16 +863,39 @@ int CameraDevice::processControls(Camera3RequestDescriptor *descriptor) return 0; } +/* + * abortRequest() is only called before the request is queued into the device, + * i.e., there is no need to remove it from pendingRequests_ and + * pendingStreamBuffers_. + */ void CameraDevice::abortRequest(Camera3RequestDescriptor *descriptor) const { - notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_REQUEST); + /* + * Since the failed buffers do not have to follow the strict ordering + * valid buffers do, and could be out-of-order with respect to valid + * buffers, it's safe to send the aborted result back to the framework + * immediately. + */ + descriptor->status_ = Camera3RequestDescriptor::Status::Error; + descriptor->finalResult_ = std::make_unique(descriptor); - for (auto &buffer : descriptor->buffers_) + Camera3ResultDescriptor *result = descriptor->finalResult_.get(); + + result->metadataPackIndex_ = 0; + for (auto &buffer : descriptor->buffers_) { buffer.status = StreamBuffer::Status::Error; + result->buffers_.emplace_back(&buffer); + } - descriptor->status_ = Camera3RequestDescriptor::Status::Error; + /* + * After CAMERA3_MSG_ERROR_REQUEST is notified, for a given frame, + * only process_capture_results with buffers of the status + * CAMERA3_BUFFER_STATUS_ERROR are allowed. No further notifies or + * process_capture_result with non-null metadata is allowed. + */ + notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_REQUEST); - sendCaptureResult(descriptor); + sendCaptureResult(result); } bool CameraDevice::isValidRequest(camera3_capture_request_t *camera3Request) const @@ -1031,9 +1057,6 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques * */ descriptor->internalBuffers_[cameraStream] = frameBuffer; LOG(HAL, Debug) << ss.str() << " (internal)"; - - descriptor->pendingStreamsToProcess_.insert( - { cameraStream, &buffer }); break; } @@ -1066,8 +1089,6 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques << cameraStream->configuration().pixelFormat << "]" << " (mapped)"; - descriptor->pendingStreamsToProcess_.insert({ cameraStream, &buffer }); - /* * Make sure the CameraStream this stream is mapped on has been * added to the request. @@ -1154,8 +1175,10 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques Request *request = descriptor->request_.get(); { - MutexLocker descriptorsLock(descriptorsMutex_); - descriptors_.push(std::move(descriptor)); + MutexLocker descriptorsLock(pendingRequestMutex_); + for (auto &buffer : descriptor->buffers_) + pendingStreamBuffers_[buffer.stream].push_back(&buffer); + pendingRequests_.emplace(std::move(descriptor)); } camera_->queueRequest(request); @@ -1163,132 +1186,279 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques return 0; } -void CameraDevice::requestComplete(Request *request) +void CameraDevice::bufferComplete(libcamera::Request *request, + libcamera::FrameBuffer *frameBuffer) { Camera3RequestDescriptor *descriptor = reinterpret_cast(request->cookie()); - /* - * Prepare the capture result for the Android camera stack. - * - * The buffer status is set to Success and later changed to Error if - * post-processing/compression fails. - */ + descriptor->partialResults_.emplace_back(new Camera3ResultDescriptor(descriptor)); + Camera3ResultDescriptor *camera3Result = descriptor->partialResults_.back().get(); + for (auto &buffer : descriptor->buffers_) { - CameraStream *stream = buffer.stream; + CameraStream *cameraStream = buffer.stream; + if (buffer.srcBuffer != frameBuffer && + buffer.frameBuffer.get() != frameBuffer) + continue; - /* - * Streams of type Direct have been queued to the - * libcamera::Camera and their acquire fences have - * already been waited on by the library. - * - * Acquire fences of streams of type Internal and Mapped - * will be handled during post-processing. - */ - if (stream->type() == CameraStream::Type::Direct) { - /* If handling of the fence has failed restore buffer.fence. */ + buffer.result = camera3Result; + camera3Result->buffers_.emplace_back(&buffer); + + StreamBuffer::Status status = StreamBuffer::Status::Success; + if (frameBuffer->metadata().status != FrameMetadata::FrameSuccess) { + status = StreamBuffer::Status::Error; + } + setBufferStatus(buffer, status); + + switch (cameraStream->type()) { + case CameraStream::Type::Direct: { + ASSERT(buffer.frameBuffer.get() == frameBuffer); + /* + * Streams of type Direct have been queued to the + * libcamera::Camera and their acquire fences have + * already been waited on by the library. + */ std::unique_ptr fence = buffer.frameBuffer->releaseFence(); if (fence) buffer.fence = fence->release(); + break; + } + case CameraStream::Type::Mapped: + case CameraStream::Type::Internal: + ASSERT(buffer.srcBuffer == frameBuffer); + if (status == StreamBuffer::Status::Error) + break; + + camera3Result->pendingBuffersToProcess_.emplace_back(&buffer); + + if (cameraStream->isJpegStream()) { + generateJpegExifMetadata(descriptor, &buffer); + + /* + * Allocate for post-processor to fill + * in JPEG related metadata. + */ + camera3Result->resultMetadata_ = getJpegPartialResultMetadata(); + } + + break; + } + } + + for (auto iter = camera3Result->pendingBuffersToProcess_.begin(); + iter != camera3Result->pendingBuffersToProcess_.end();) { + StreamBuffer *buffer = *iter; + int ret = buffer->stream->process(buffer); + if (ret) { + iter = camera3Result->pendingBuffersToProcess_.erase(iter); + setBufferStatus(*buffer, StreamBuffer::Status::Error); + LOG(HAL, Error) << "Failed to run post process of request " + << descriptor->frameNumber_; + } else { + iter++; } - buffer.status = StreamBuffer::Status::Success; } + if (camera3Result->pendingBuffersToProcess_.empty()) + checkAndCompleteReadyPartialResults(camera3Result); +} + +void CameraDevice::metadataAvailable(libcamera::Request *request, + const libcamera::ControlList &metadata) +{ + ASSERT(!metadata.empty()); + + Camera3RequestDescriptor *descriptor = + reinterpret_cast(request->cookie()); + + descriptor->partialResults_.emplace_back(new Camera3ResultDescriptor(descriptor)); + Camera3ResultDescriptor *camera3Result = descriptor->partialResults_.back().get(); + /* - * If the Request has failed, abort the request by notifying the error - * and complete the request with all buffers in error state. + * Notify shutter as soon as we have received SensorTimestamp. */ - if (request->status() != Request::RequestComplete) { - LOG(HAL, Error) << "Request " << request->cookie() - << " not successfully completed: " - << request->status(); + const auto ×tamp = metadata.get(controls::SensorTimestamp); + if (timestamp) { + notifyShutter(descriptor->frameNumber_, *timestamp); + LOG(HAL, Debug) << "Request " << request->cookie() << " notifies shutter"; + } + + camera3Result->resultMetadata_ = getPartialResultMetadata(metadata); + + checkAndCompleteReadyPartialResults(camera3Result); +} + +void CameraDevice::requestComplete(Request *request) +{ + Camera3RequestDescriptor *camera3Request = + reinterpret_cast(request->cookie()); - descriptor->status_ = Camera3RequestDescriptor::Status::Error; + switch (request->status()) { + case Request::RequestComplete: + camera3Request->status_ = Camera3RequestDescriptor::Status::Success; + break; + case Request::RequestCancelled: + camera3Request->status_ = Camera3RequestDescriptor::Status::Error; + break; + case Request::RequestPending: + LOG(HAL, Fatal) << "Try to complete an unfinished request"; + break; } + camera3Request->finalResult_ = std::make_unique(camera3Request); + Camera3ResultDescriptor *result = camera3Request->finalResult_.get(); + /* - * Notify shutter as soon as we have verified we have a valid request. - * - * \todo The shutter event notification should be sent to the framework - * as soon as possible, earlier than request completion time. + * On Android, The final result with metadata has to set the field as + * CameraCapabilities::MaxMetadataPackIndex, and should be returned by + * the submission order of the requests. Create a result as the final + * result which is guranteed be sent in order by CompleteRequestDescriptor(). + */ + result->resultMetadata_ = getFinalResultMetadata(camera3Request->settings_); + result->metadataPackIndex_ = CameraCapabilities::kMaxMetadataPackIndex; + + /* + * We need to check whether there are partial results pending for + * post-processing, before we complete the request descriptor. Otherwise, + * the callback of post-processing will complete the request instead. */ - uint64_t sensorTimestamp = static_cast(request->metadata() - .get(controls::SensorTimestamp) - .value_or(0)); - notifyShutter(descriptor->frameNumber_, sensorTimestamp); + for (auto &r : camera3Request->partialResults_) + if (!r->completed_) + return; - LOG(HAL, Debug) << "Request " << request->cookie() << " completed with " - << descriptor->request_->buffers().size() << " streams"; + completeRequestDescriptor(camera3Request); +} +void CameraDevice::checkAndCompleteReadyPartialResults(Camera3ResultDescriptor *result) +{ /* - * Generate the metadata associated with the captured buffers. + * Android requires buffers for a given stream must be returned in FIFO + * order. However, different streams are independent of each other, so + * it is acceptable and expected that the buffer for request 5 for + * stream A may be returned after the buffer for request 6 for stream + * B is. And it is acceptable that the result metadata for request 6 + * for stream B is returned before the buffer for request 5 for stream + * A is. As a result, if all buffers of a result are the most front + * buffers of each stream, or the result contains no buffers, the result + * is allowed to send. Collect ready results to send in the order which + * follows the above rule. * - * Notify if the metadata generation has failed, but continue processing - * buffers and return an empty metadata pack. + * \todo The reprocessing result can be returned ahead of the pending + * normal output results. But the FIFO ordering must be maintained for + * all reprocessing results. Track the reprocessing buffer's order + * independently when we have reprocessing API. */ - descriptor->resultMetadata_ = getResultMetadata(*descriptor); - if (!descriptor->resultMetadata_) { - descriptor->status_ = Camera3RequestDescriptor::Status::Error; + MutexLocker lock(pendingRequestMutex_); - /* - * The camera framework expects an empty metadata pack on error. - * - * \todo Check that the post-processor code handles this situation - * correctly. - */ - descriptor->resultMetadata_ = std::make_unique(0, 0); - } + pendingPartialResults_.emplace_front(result); + std::list readyResults; /* - * Queue all the post-processing streams request at once. The completion - * slot streamProcessingComplete() can only execute when we are out - * this critical section. This helps to handle synchronous errors here - * itself. + * Error buffers do not have to follow the strict ordering as valid + * buffers do. They're ready to be sent directly. Therefore, remove them + * from the pendingBuffers so it won't block following valid buffers. */ - auto iter = descriptor->pendingStreamsToProcess_.begin(); - while (iter != descriptor->pendingStreamsToProcess_.end()) { - CameraStream *stream = iter->first; - StreamBuffer *buffer = iter->second; + for (auto &buffer : result->buffers_) + if (buffer->status == StreamBuffer::Status::Error) + pendingStreamBuffers_[buffer->stream].remove(buffer); - if (stream->isJpegStream()) { - generateJpegExifMetadata(descriptor, buffer); - } + /* + * Exhaustly collect results which is ready to sent. + */ + bool keepChecking; + do { + keepChecking = false; + auto iter = pendingPartialResults_.begin(); + while (iter != pendingPartialResults_.end()) { + /* + * A result is considered as ready when all of the valid + * buffers of the result are at the front of the pending + * buffers associated with its stream. + */ + bool ready = true; + for (auto &buffer : (*iter)->buffers_) { + if (buffer->status == StreamBuffer::Status::Error) + continue; - FrameBuffer *src = request->findBuffer(stream->stream()); - if (!src) { - LOG(HAL, Error) << "Failed to find a source stream buffer"; - setBufferStatus(*buffer, StreamBuffer::Status::Error); - iter = descriptor->pendingStreamsToProcess_.erase(iter); - continue; - } + auto &pendingBuffers = pendingStreamBuffers_[buffer->stream]; - ++iter; - int ret = stream->process(buffer); - if (ret) { - setBufferStatus(*buffer, StreamBuffer::Status::Error); - descriptor->pendingStreamsToProcess_.erase(stream); + ASSERT(!pendingBuffers.empty()); + + if (pendingBuffers.front() != buffer) { + ready = false; + break; + } + } + + if (!ready) { + iter++; + continue; + } + + for (auto &buffer : (*iter)->buffers_) + if (buffer->status != StreamBuffer::Status::Error) + pendingStreamBuffers_[buffer->stream].pop_front(); + + /* Keep checking since pendingStreamBuffers has updated */ + keepChecking = true; + + readyResults.emplace_back(*iter); + iter = pendingPartialResults_.erase(iter); } + } while (keepChecking); + + lock.unlock(); + + for (auto &res : readyResults) { + completePartialResultDescriptor(res); } +} + +void CameraDevice::completePartialResultDescriptor(Camera3ResultDescriptor *result) +{ + Camera3RequestDescriptor *request = result->request_; + result->completed_ = true; + + /* + * Android requires value of metadataPackIndex of partial results + * set it to 0 if the result contains only buffers, Otherwise set it + * Incrementally from 1 to MaxMetadataPackIndex - 1. + */ + if (result->resultMetadata_) + result->metadataPackIndex_ = request->nextPartialResultIndex_++; + else + result->metadataPackIndex_ = 0; + + sendCaptureResult(result); - if (descriptor->pendingStreamsToProcess_.empty()) - completeDescriptor(descriptor); + /* + * The Status would be changed from Pending to Success or Error only + * when the requestComplete() has been called. It's garanteed that no + * more partial results will be added to the request and the final result + * is ready. In the case, if all partial results are completed, we can + * complete the request. + */ + if (request->status_ == Camera3RequestDescriptor::Status::Pending) + return; + + for (auto &r : request->partialResults_) + if (!r->completed_) + return; + + completeRequestDescriptor(request); } /** * \brief Complete the Camera3RequestDescriptor - * \param[in] descriptor The Camera3RequestDescriptor that has completed + * \param[in] descriptor The Camera3RequestDescriptor * - * The function marks the Camera3RequestDescriptor as 'complete'. It shall be - * called when all the streams in the Camera3RequestDescriptor have completed - * capture (or have been generated via post-processing) and the request is ready - * to be sent back to the framework. - * - * \context This function is \threadsafe. + * The function shall complete the descriptor only when all of the partial + * result has sent back to the framework, and send the final result according + * to the submission order of the requests. */ -void CameraDevice::completeDescriptor(Camera3RequestDescriptor *descriptor) +void CameraDevice::completeRequestDescriptor(Camera3RequestDescriptor *request) { - MutexLocker lock(descriptorsMutex_); - descriptor->complete_ = true; + request->complete_ = true; sendCaptureResults(); } @@ -1304,15 +1474,23 @@ void CameraDevice::completeDescriptor(Camera3RequestDescriptor *descriptor) * Stop iterating if the descriptor at the front of the queue is not complete. * * This function should never be called directly in the codebase. Use - * completeDescriptor() instead. + * completeRequestDescriptor() instead. */ void CameraDevice::sendCaptureResults() { - while (!descriptors_.empty() && !descriptors_.front()->isPending()) { - auto descriptor = std::move(descriptors_.front()); - descriptors_.pop(); + MutexLocker descriptorsLock(pendingRequestMutex_); + + while (!pendingRequests_.empty()) { + auto &descriptor = pendingRequests_.front(); + if (!descriptor->complete_) + break; - sendCaptureResult(descriptor.get()); + /* + * Android requires the final result of each request returns in + * their submission order. + */ + ASSERT(descriptor->finalResult_); + sendCaptureResult(descriptor->finalResult_.get()); /* * Call notify with CAMERA3_MSG_ERROR_RESULT to indicate some @@ -1323,18 +1501,20 @@ void CameraDevice::sendCaptureResults() */ if (descriptor->status_ == Camera3RequestDescriptor::Status::Error) notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_RESULT); + + pendingRequests_.pop(); } } -void CameraDevice::sendCaptureResult(Camera3RequestDescriptor *request) const +void CameraDevice::sendCaptureResult(Camera3ResultDescriptor *result) const { std::vector resultBuffers; - resultBuffers.reserve(request->buffers_.size()); + resultBuffers.reserve(result->buffers_.size()); - for (auto &buffer : request->buffers_) { + for (auto &buffer : result->buffers_) { camera3_buffer_status status = CAMERA3_BUFFER_STATUS_ERROR; - if (buffer.status == StreamBuffer::Status::Success) + if (buffer->status == StreamBuffer::Status::Success) status = CAMERA3_BUFFER_STATUS_OK; /* @@ -1343,22 +1523,20 @@ void CameraDevice::sendCaptureResult(Camera3RequestDescriptor *request) const * on the acquire fence in case we haven't done so * ourselves for any reason. */ - resultBuffers.push_back({ buffer.stream->camera3Stream(), - buffer.camera3Buffer, status, - -1, buffer.fence.release() }); + resultBuffers.push_back({ buffer->stream->camera3Stream(), + buffer->camera3Buffer, status, + -1, buffer->fence.release() }); } camera3_capture_result_t captureResult = {}; - captureResult.frame_number = request->frameNumber_; + captureResult.frame_number = result->request_->frameNumber_; captureResult.num_output_buffers = resultBuffers.size(); captureResult.output_buffers = resultBuffers.data(); + captureResult.partial_result = result->metadataPackIndex_; - if (request->status_ == Camera3RequestDescriptor::Status::Success) - captureResult.partial_result = 1; - - if (request->resultMetadata_) - captureResult.result = request->resultMetadata_->getMetadata(); + if (result->resultMetadata_) + captureResult.result = result->resultMetadata_->getMetadata(); callbacks_->process_capture_result(callbacks_, &captureResult); } @@ -1371,10 +1549,6 @@ void CameraDevice::setBufferStatus(StreamBuffer &streamBuffer, notifyError(streamBuffer.request->frameNumber_, streamBuffer.stream->camera3Stream(), CAMERA3_MSG_ERROR_BUFFER); - - /* Also set error status on entire request descriptor. */ - streamBuffer.request->status_ = - Camera3RequestDescriptor::Status::Error; } } @@ -1396,26 +1570,25 @@ void CameraDevice::streamProcessingCompleteDelegate(StreamBuffer *streamBuffer, * \param[in] streamBuffer The StreamBuffer for which processing is complete * \param[in] status Stream post-processing status * - * This function is called from the post-processor's thread whenever a camera + * This function is called from the camera's thread whenever a camera * stream has finished post processing. The corresponding entry is dropped from - * the descriptor's pendingStreamsToProcess_ map. + * the result's pendingBufferToProcess_ list. * - * If the pendingStreamsToProcess_ map is then empty, all streams requiring to - * be generated from post-processing have been completed. Mark the descriptor as - * complete using completeDescriptor() in that case. + * If the pendingBufferToProcess_ list is then empty, all streams requiring to + * be generated from post-processing have been completed. */ void CameraDevice::streamProcessingComplete(StreamBuffer *streamBuffer, StreamBuffer::Status status) { setBufferStatus(*streamBuffer, status); - Camera3RequestDescriptor *request = streamBuffer->request; + Camera3ResultDescriptor *result = streamBuffer->result; + result->pendingBuffersToProcess_.remove(streamBuffer); - request->pendingStreamsToProcess_.erase(streamBuffer->stream); - if (!request->pendingStreamsToProcess_.empty()) + if (!result->pendingBuffersToProcess_.empty()) return; - completeDescriptor(streamBuffer->request); + checkAndCompleteReadyPartialResults(result); } std::string CameraDevice::logPrefix() const @@ -1469,23 +1642,12 @@ void CameraDevice::generateJpegExifMetadata(Camera3RequestDescriptor *request, jpegExifMetadata->sensorSensitivityISO = 100; } -/* - * Produce a set of fixed result metadata. - */ -std::unique_ptr -CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor) const +std::unique_ptr CameraDevice::getJpegPartialResultMetadata() const { - const ControlList &metadata = descriptor.request_->metadata(); - const CameraMetadata &settings = descriptor.settings_; - camera_metadata_ro_entry_t entry; - bool found; - /* - * \todo Keep this in sync with the actual number of entries. - * Currently: 40 entries, 156 bytes + * Reserve more capacity for the JPEG metadata set by the post-processor. + * Currently: 8 entries, 82 bytes extra capaticy. * - * Reserve more space for the JPEG metadata set by the post-processor. - * Currently: * ANDROID_JPEG_GPS_COORDINATES (double x 3) = 24 bytes * ANDROID_JPEG_GPS_PROCESSING_METHOD (byte x 32) = 32 bytes * ANDROID_JPEG_GPS_TIMESTAMP (int64) = 8 bytes @@ -1497,7 +1659,215 @@ CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor) cons * Total bytes for JPEG metadata: 82 */ std::unique_ptr resultMetadata = - std::make_unique(88, 166); + std::make_unique(8, 82); + if (!resultMetadata->isValid()) { + LOG(HAL, Error) << "Failed to allocate result metadata"; + return nullptr; + } + + return resultMetadata; +} + +std::unique_ptr +CameraDevice::getPartialResultMetadata(const ControlList &metadata) const +{ + /* + * \todo Keep this in sync with the actual number of entries. + * + * Reserve capacity for the metadata larger than 4 bytes which cannot + * store in entries. + * Currently: 6 entries, 40 bytes extra capaticy. + * + * ANDROID_SENSOR_TIMESTAMP (int64) = 8 bytes + * ANDROID_REQUEST_PIPELINE_DEPTH (byte) = 1 byte + * ANDROID_SENSOR_EXPOSURE_TIME (int64) = 8 bytes + * ANDROID_CONTROL_AE_STATE (enum) = 4 bytes + * ANDROID_CONTROL_AF_STATE (enum) = 4 bytes + * ANDROID_SENSOR_SENSITIVITY (int32) = 4 bytes + * ANDROID_CONTROL_AWB_STATE (enum) = 4 bytes + * ANDROID_SENSOR_FRAME_DURATION (int64) = 8 bytes + * ANDROID_SCALER_CROP_REGION (int32 X 4) = 16 bytes + * ANDROID_SENSOR_TEST_PATTERN_MODE (enum) = 4 bytes + * Total bytes for capacity: 61 + * + * ANDROID_STATISTICS_FACE_RECTANGLES (int32[]) = 4*4*n bytes + * ANDROID_STATISTICS_FACE_SCORES (byte[]) = n bytes + * ANDROID_STATISTICS_FACE_LANDMARKS (int32[]) = 4*2*n bytes + * ANDROID_STATISTICS_FACE_IDS (int32[]) = 4*n bytes + * + * \todo Calculate the entries and capacity by the input ControlList. + */ + + const auto &faceDetectRectangles = + metadata.get(controls::draft::FaceDetectFaceRectangles); + + size_t entryCapacity = 10; + size_t dataCapacity = 61; + if (faceDetectRectangles) { + entryCapacity += 4; + dataCapacity += faceDetectRectangles->size() * 29; + } + + std::unique_ptr resultMetadata = + std::make_unique(entryCapacity, dataCapacity); + if (!resultMetadata->isValid()) { + LOG(HAL, Error) << "Failed to allocate result metadata"; + return nullptr; + } + + if (faceDetectRectangles) { + std::vector flatRectangles; + for (const Rectangle &rect : *faceDetectRectangles) { + flatRectangles.push_back(rect.x); + flatRectangles.push_back(rect.y); + flatRectangles.push_back(rect.x + rect.width); + flatRectangles.push_back(rect.y + rect.height); + } + resultMetadata->addEntry( + ANDROID_STATISTICS_FACE_RECTANGLES, flatRectangles); + } + + const auto &faceDetectFaceScores = + metadata.get(controls::draft::FaceDetectFaceScores); + if (faceDetectRectangles && faceDetectFaceScores) { + if (faceDetectFaceScores->size() != faceDetectRectangles->size()) { + LOG(HAL, Error) << "Pipeline returned wrong number of face scores; " + << "Expected: " << faceDetectRectangles->size() + << ", got: " << faceDetectFaceScores->size(); + } + resultMetadata->addEntry(ANDROID_STATISTICS_FACE_SCORES, + *faceDetectFaceScores); + } + + const auto &faceDetectFaceLandmarks = + metadata.get(controls::draft::FaceDetectFaceLandmarks); + if (faceDetectRectangles && faceDetectFaceLandmarks) { + size_t expectedLandmarks = faceDetectRectangles->size() * 3; + if (faceDetectFaceLandmarks->size() != expectedLandmarks) { + LOG(HAL, Error) << "Pipeline returned wrong number of face landmarks; " + << "Expected: " << expectedLandmarks + << ", got: " << faceDetectFaceLandmarks->size(); + } + + std::vector androidLandmarks; + for (const Point &landmark : *faceDetectFaceLandmarks) { + androidLandmarks.push_back(landmark.x); + androidLandmarks.push_back(landmark.y); + } + resultMetadata->addEntry( + ANDROID_STATISTICS_FACE_LANDMARKS, androidLandmarks); + } + + const auto &faceDetectFaceIds = metadata.get(controls::draft::FaceDetectFaceIds); + if (faceDetectRectangles && faceDetectFaceIds) { + if (faceDetectFaceIds->size() != faceDetectRectangles->size()) { + LOG(HAL, Error) << "Pipeline returned wrong number of face ids; " + << "Expected: " << faceDetectRectangles->size() + << ", got: " << faceDetectFaceIds->size(); + } + resultMetadata->addEntry(ANDROID_STATISTICS_FACE_IDS, *faceDetectFaceIds); + } + + /* Add metadata tags reported by libcamera. */ + const auto ×tamp = metadata.get(controls::SensorTimestamp); + if (timestamp) + resultMetadata->addEntry(ANDROID_SENSOR_TIMESTAMP, *timestamp); + + const auto &pipelineDepth = metadata.get(controls::draft::PipelineDepth); + if (pipelineDepth) + resultMetadata->addEntry(ANDROID_REQUEST_PIPELINE_DEPTH, + *pipelineDepth); + + if (metadata.contains(controls::EXPOSURE_TIME)) { + const auto &exposureTime = metadata.get(controls::ExposureTime); + int64_t exposure_time = static_cast(exposureTime.value_or(33'333)); + resultMetadata->addEntry(ANDROID_SENSOR_EXPOSURE_TIME, exposure_time * 1000ULL); + } + + if (metadata.contains(controls::draft::AE_STATE)) { + const auto &aeState = metadata.get(controls::draft::AeState); + resultMetadata->addEntry(ANDROID_CONTROL_AE_STATE, aeState.value_or(0)); + } + + if (metadata.contains(controls::AF_STATE)) { + const auto &afState = metadata.get(controls::AfState); + resultMetadata->addEntry(ANDROID_CONTROL_AF_STATE, afState.value_or(0)); + } + + if (metadata.contains(controls::ANALOGUE_GAIN)) { + const auto &sensorSensitivity = metadata.get(controls::AnalogueGain).value_or(100); + resultMetadata->addEntry(ANDROID_SENSOR_SENSITIVITY, static_cast(sensorSensitivity)); + } + + const auto &awbState = metadata.get(controls::draft::AwbState); + if (metadata.contains(controls::draft::AWB_STATE)) { + resultMetadata->addEntry(ANDROID_CONTROL_AWB_STATE, awbState.value_or(0)); + } + + const auto &frameDuration = metadata.get(controls::FrameDuration); + if (metadata.contains(controls::FRAME_DURATION)) { + resultMetadata->addEntry(ANDROID_SENSOR_FRAME_DURATION, frameDuration.value_or(33'333'333)); + } + + const auto &scalerCrop = metadata.get(controls::ScalerCrop); + if (scalerCrop) { + const Rectangle &crop = *scalerCrop; + int32_t cropRect[] = { + crop.x, + crop.y, + static_cast(crop.width), + static_cast(crop.height), + }; + resultMetadata->addEntry(ANDROID_SCALER_CROP_REGION, cropRect); + } + + const auto &testPatternMode = metadata.get(controls::draft::TestPatternMode); + if (testPatternMode) + resultMetadata->addEntry(ANDROID_SENSOR_TEST_PATTERN_MODE, + *testPatternMode); + + /* + * Return the result metadata pack even is not valid: get() will return + * nullptr. + */ + if (!resultMetadata->isValid()) { + LOG(HAL, Error) << "Failed to construct result metadata"; + } + + if (resultMetadata->resized()) { + auto [entryCount, dataCount] = resultMetadata->usage(); + LOG(HAL, Info) + << "Result metadata resized: " << entryCount + << " entries and " << dataCount << " bytes used"; + } + + return resultMetadata; +} + +/* + * Produce a set of fixed result metadata. + */ +std::unique_ptr +CameraDevice::getFinalResultMetadata(const CameraMetadata &settings) const +{ + camera_metadata_ro_entry_t entry; + bool found; + + /* + * \todo Retrieve metadata from corresponding libcamera controls. + * \todo Keep this in sync with the actual number of entries. + * + * Reserve capacity for the metadata larger than 4 bytes which cannot + * store in entries. + * Currently: 31 entries, 16 bytes + * + * ANDROID_CONTROL_AE_TARGET_FPS_RANGE (int32 X 2) = 8 bytes + * ANDROID_SENSOR_ROLLING_SHUTTER_SKEW (int64) = 8 bytes + * + * Total bytes: 16 + */ + std::unique_ptr resultMetadata = + std::make_unique(31, 16); if (!resultMetadata->isValid()) { LOG(HAL, Error) << "Failed to allocate result metadata"; return nullptr; @@ -1536,8 +1906,7 @@ CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor) cons entry.data.i32, 2); found = settings.getEntry(ANDROID_CONTROL_AE_PRECAPTURE_TRIGGER, &entry); - value = found ? *entry.data.u8 : - (uint8_t)ANDROID_CONTROL_AE_PRECAPTURE_TRIGGER_IDLE; + value = found ? *entry.data.u8 : (uint8_t)ANDROID_CONTROL_AE_PRECAPTURE_TRIGGER_IDLE; resultMetadata->addEntry(ANDROID_CONTROL_AE_PRECAPTURE_TRIGGER, value); value = ANDROID_CONTROL_AE_STATE_CONVERGED; @@ -1620,95 +1989,6 @@ CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor) cons resultMetadata->addEntry(ANDROID_SENSOR_ROLLING_SHUTTER_SKEW, rolling_shutter_skew); - /* Add metadata tags reported by libcamera. */ - const int64_t timestamp = metadata.get(controls::SensorTimestamp).value_or(0); - resultMetadata->addEntry(ANDROID_SENSOR_TIMESTAMP, timestamp); - - const auto &pipelineDepth = metadata.get(controls::draft::PipelineDepth); - if (pipelineDepth) - resultMetadata->addEntry(ANDROID_REQUEST_PIPELINE_DEPTH, - *pipelineDepth); - - const auto &exposureTime = metadata.get(controls::ExposureTime); - if (exposureTime) - resultMetadata->addEntry(ANDROID_SENSOR_EXPOSURE_TIME, - *exposureTime * 1000ULL); - - const auto &frameDuration = metadata.get(controls::FrameDuration); - if (frameDuration) - resultMetadata->addEntry(ANDROID_SENSOR_FRAME_DURATION, - *frameDuration * 1000); - - const auto &faceDetectRectangles = - metadata.get(controls::draft::FaceDetectFaceRectangles); - if (faceDetectRectangles) { - std::vector flatRectangles; - for (const Rectangle &rect : *faceDetectRectangles) { - flatRectangles.push_back(rect.x); - flatRectangles.push_back(rect.y); - flatRectangles.push_back(rect.x + rect.width); - flatRectangles.push_back(rect.y + rect.height); - } - resultMetadata->addEntry( - ANDROID_STATISTICS_FACE_RECTANGLES, flatRectangles); - } - - const auto &faceDetectFaceScores = - metadata.get(controls::draft::FaceDetectFaceScores); - if (faceDetectRectangles && faceDetectFaceScores) { - if (faceDetectFaceScores->size() != faceDetectRectangles->size()) { - LOG(HAL, Error) << "Pipeline returned wrong number of face scores; " - << "Expected: " << faceDetectRectangles->size() - << ", got: " << faceDetectFaceScores->size(); - } - resultMetadata->addEntry(ANDROID_STATISTICS_FACE_SCORES, - *faceDetectFaceScores); - } - - const auto &faceDetectFaceLandmarks = - metadata.get(controls::draft::FaceDetectFaceLandmarks); - if (faceDetectRectangles && faceDetectFaceLandmarks) { - size_t expectedLandmarks = faceDetectRectangles->size() * 3; - if (faceDetectFaceLandmarks->size() != expectedLandmarks) { - LOG(HAL, Error) << "Pipeline returned wrong number of face landmarks; " - << "Expected: " << expectedLandmarks - << ", got: " << faceDetectFaceLandmarks->size(); - } - - std::vector androidLandmarks; - for (const Point &landmark : *faceDetectFaceLandmarks) { - androidLandmarks.push_back(landmark.x); - androidLandmarks.push_back(landmark.y); - } - resultMetadata->addEntry( - ANDROID_STATISTICS_FACE_LANDMARKS, androidLandmarks); - } - - const auto &faceDetectFaceIds = metadata.get(controls::draft::FaceDetectFaceIds); - if (faceDetectRectangles && faceDetectFaceIds) { - if (faceDetectFaceIds->size() != faceDetectRectangles->size()) { - LOG(HAL, Error) << "Pipeline returned wrong number of face ids; " - << "Expected: " << faceDetectRectangles->size() - << ", got: " << faceDetectFaceIds->size(); - } - resultMetadata->addEntry(ANDROID_STATISTICS_FACE_IDS, *faceDetectFaceIds); - } - - const auto &scalerCrop = metadata.get(controls::ScalerCrop); - if (scalerCrop) { - const Rectangle &crop = *scalerCrop; - int32_t cropRect[] = { - crop.x, crop.y, static_cast(crop.width), - static_cast(crop.height), - }; - resultMetadata->addEntry(ANDROID_SCALER_CROP_REGION, cropRect); - } - - const auto &testPatternMode = metadata.get(controls::draft::TestPatternMode); - if (testPatternMode) - resultMetadata->addEntry(ANDROID_SENSOR_TEST_PATTERN_MODE, - *testPatternMode); - /* * Return the result metadata pack even is not valid: get() will return * nullptr. diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 3c46ff918..2aa6b2c09 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -64,11 +64,13 @@ public: const camera_metadata_t *constructDefaultRequestSettings(int type); int configureStreams(camera3_stream_configuration_t *stream_list); int processCaptureRequest(camera3_capture_request_t *request); + void bufferComplete(libcamera::Request *request, + libcamera::FrameBuffer *buffer); + void metadataAvailable(libcamera::Request *request, + const libcamera::ControlList &metadata); void requestComplete(libcamera::Request *request); void streamProcessingCompleteDelegate(StreamBuffer *bufferStream, StreamBuffer::Status status); - void streamProcessingComplete(StreamBuffer *bufferStream, - StreamBuffer::Status status); protected: std::string logPrefix() const override; @@ -96,16 +98,26 @@ private: void notifyError(uint32_t frameNumber, camera3_stream_t *stream, camera3_error_msg_code code) const; int processControls(Camera3RequestDescriptor *descriptor); - void completeDescriptor(Camera3RequestDescriptor *descriptor) - LIBCAMERA_TSA_EXCLUDES(descriptorsMutex_); - void sendCaptureResults() LIBCAMERA_TSA_REQUIRES(descriptorsMutex_); - void sendCaptureResult(Camera3RequestDescriptor *request) const; + + void checkAndCompleteReadyPartialResults(Camera3ResultDescriptor *result); + void completePartialResultDescriptor(Camera3ResultDescriptor *result); + void completeRequestDescriptor(Camera3RequestDescriptor *descriptor); + + void streamProcessingComplete(StreamBuffer *bufferStream, + StreamBuffer::Status status); + + void sendCaptureResults(); + void sendCaptureResult(Camera3ResultDescriptor *result) const; void setBufferStatus(StreamBuffer &buffer, StreamBuffer::Status status); void generateJpegExifMetadata(Camera3RequestDescriptor *request, StreamBuffer *buffer) const; - std::unique_ptr getResultMetadata( - const Camera3RequestDescriptor &descriptor) const; + + std::unique_ptr getJpegPartialResultMetadata() const; + std::unique_ptr getPartialResultMetadata( + const libcamera::ControlList &metadata) const; + std::unique_ptr getFinalResultMetadata( + const CameraMetadata &settings) const; unsigned int id_; camera3_device_t camera3Device_; @@ -122,9 +134,14 @@ private: std::vector streams_; - libcamera::Mutex descriptorsMutex_ LIBCAMERA_TSA_ACQUIRED_AFTER(stateMutex_); - std::queue> descriptors_ - LIBCAMERA_TSA_GUARDED_BY(descriptorsMutex_); + /* Protects access to the pending requests and stream buffers. */ + libcamera::Mutex pendingRequestMutex_; + std::queue> pendingRequests_ + LIBCAMERA_TSA_GUARDED_BY(pendingRequestMutex_); + std::map> pendingStreamBuffers_ + LIBCAMERA_TSA_GUARDED_BY(pendingRequestMutex_); + + std::list pendingPartialResults_; std::string maker_; std::string model_; diff --git a/src/android/camera_request.cpp b/src/android/camera_request.cpp index a9240a83c..20e1b3e54 100644 --- a/src/android/camera_request.cpp +++ b/src/android/camera_request.cpp @@ -43,25 +43,25 @@ using namespace libcamera; * │ processCaptureRequest(camera3_capture_request_t request) │ * │ │ * │ - Create Camera3RequestDescriptor tracking this request │ - * │ - Streams requiring post-processing are stored in the │ - * │ pendingStreamsToProcess map │ + * │ - Buffers requiring post-processing are marked by the │ + * │ CameraStream::Type as Mapped or Internal │ * │ - Add this Camera3RequestDescriptor to descriptors' queue │ - * │ CameraDevice::descriptors_ │ - * │ │ ┌─────────────────────────┐ - * │ - Queue the capture request to libcamera core ────────────┤►│libcamera core │ - * │ │ ├─────────────────────────┤ - * │ │ │- Capture from Camera │ - * │ │ │ │ - * │ │ │- Emit │ - * │ │ │ Camera::requestComplete│ - * │ requestCompleted(Request *request) ◄───────────────────────┼─┼──── │ - * │ │ │ │ - * │ - Check request completion status │ └─────────────────────────┘ + * │ CameraDevice::pendingRequests_ │ + * │ │ ┌────────────────────────────────┐ + * │ - Queue the capture request to libcamera core ────────────┤►│libcamera core │ + * │ │ ├────────────────────────────────┤ + * │ │ │- Capture from Camera │ + * │ │ │ │ + * │ │ │- Emit │ + * │ │ │ Camera::partialResultCompleted│ + * │ partialResultComplete(Request *request, Result result*) ◄──┼─┼──── │ + * │ │ │ │ + * │ - Check request completion status │ └────────────────────────────────┘ * │ │ - * │ - if (pendingStreamsToProcess > 0) │ - * │ Queue all entries from pendingStreamsToProcess │ + * │ - if (pendingBuffersToProcess > 0) │ + * │ Queue all entries from pendingBuffersToProcess │ * │ else │ │ - * │ completeDescriptor() │ └──────────────────────┐ + * │ completeResultDescriptor() │ └──────────────────────┐ * │ │ │ * │ ┌──────────────────────────┴───┬──────────────────┐ │ * │ │ │ │ │ @@ -94,10 +94,10 @@ using namespace libcamera; * │ | | | | │ * │ | - Check and set buffer status | | .... | │ * │ | - Remove post+processing entry | | | │ - * │ | from pendingStreamsToProcess | | | │ + * │ | from pendingBuffersToProcess | | | │ * │ | | | | │ - * │ | - if (pendingStreamsToProcess.empty())| | | │ - * │ | completeDescriptor | | | │ + * │ | - if (pendingBuffersToProcess.empty())| | | │ + * │ | completeResultDescriptor | | | │ * │ | | | | │ * │ +---------------------------------------+ +--------------+ │ * │ │ @@ -148,6 +148,19 @@ Camera3RequestDescriptor::~Camera3RequestDescriptor() sourceStream->putBuffer(frameBuffer); } +/* + * \class Camera3ResultDescriptor + * + * A utility class that groups information about a capture result to be sent to + * framework. + */ +Camera3ResultDescriptor::Camera3ResultDescriptor(Camera3RequestDescriptor *request) + : request_(request), metadataPackIndex_(1), completed_(false) +{ +} + +Camera3ResultDescriptor::~Camera3ResultDescriptor() = default; + /** * \class StreamBuffer * \brief Group information for per-stream buffer of Camera3RequestDescriptor @@ -182,6 +195,9 @@ Camera3RequestDescriptor::~Camera3RequestDescriptor() * * \var StreamBuffer::request * \brief Back pointer to the Camera3RequestDescriptor to which the StreamBuffer belongs + * + * \var StreamBuffer::result + * \brief Back pointer to the Camera3ResultDescriptor to which the StreamBuffer belongs */ StreamBuffer::StreamBuffer( CameraStream *cameraStream, const camera3_stream_buffer_t &buffer, diff --git a/src/android/camera_request.h b/src/android/camera_request.h index bd87b36fd..18386a905 100644 --- a/src/android/camera_request.h +++ b/src/android/camera_request.h @@ -26,6 +26,7 @@ class CameraBuffer; class CameraStream; +class Camera3ResultDescriptor; class Camera3RequestDescriptor; class StreamBuffer @@ -57,41 +58,62 @@ public: const libcamera::FrameBuffer *srcBuffer = nullptr; std::unique_ptr dstBuffer; std::optional jpegExifMetadata; + Camera3ResultDescriptor *result; Camera3RequestDescriptor *request; private: LIBCAMERA_DISABLE_COPY(StreamBuffer) }; +class Camera3ResultDescriptor +{ +public: + Camera3ResultDescriptor(Camera3RequestDescriptor *request); + ~Camera3ResultDescriptor(); + + Camera3RequestDescriptor *request_; + uint32_t metadataPackIndex_; + + std::unique_ptr resultMetadata_; + std::vector buffers_; + + /* Keeps track of buffers waiting for post-processing. */ + std::list pendingBuffersToProcess_; + + bool completed_; + +private: + LIBCAMERA_DISABLE_COPY(Camera3ResultDescriptor) +}; + class Camera3RequestDescriptor { public: enum class Status { + Pending, Success, Error, }; - /* Keeps track of streams requiring post-processing. */ - std::map pendingStreamsToProcess_; - Camera3RequestDescriptor(libcamera::Camera *camera, const camera3_capture_request_t *camera3Request); ~Camera3RequestDescriptor(); - bool isPending() const { return !complete_; } - uint32_t frameNumber_ = 0; std::vector buffers_; CameraMetadata settings_; std::unique_ptr request_; - std::unique_ptr resultMetadata_; std::map internalBuffers_; bool complete_ = false; - Status status_ = Status::Success; + Status status_ = Status::Pending; + + uint32_t nextPartialResultIndex_ = 1; + std::unique_ptr finalResult_; + std::vector> partialResults_; private: LIBCAMERA_DISABLE_COPY(Camera3RequestDescriptor) diff --git a/src/android/camera_stream.cpp b/src/android/camera_stream.cpp index 53f292d4b..7837fd7aa 100644 --- a/src/android/camera_stream.cpp +++ b/src/android/camera_stream.cpp @@ -121,8 +121,8 @@ int CameraStream::configure() else bufferStatus = StreamBuffer::Status::Error; - cameraDevice_->streamProcessingComplete(streamBuffer, - bufferStatus); + cameraDevice_->streamProcessingCompleteDelegate(streamBuffer, + bufferStatus); }); worker_->start(); diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp index 48782b574..671e560ec 100644 --- a/src/android/jpeg/post_processor_jpeg.cpp +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -119,7 +119,7 @@ void PostProcessorJpeg::process(StreamBuffer *streamBuffer) ASSERT(jpegExifMetadata.has_value()); const CameraMetadata &requestMetadata = streamBuffer->request->settings_; - CameraMetadata *resultMetadata = streamBuffer->request->resultMetadata_.get(); + CameraMetadata *resultMetadata = streamBuffer->result->resultMetadata_.get(); camera_metadata_ro_entry_t entry; int ret;