From patchwork Tue Sep 7 19:57:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 13753 X-Patchwork-Delegate: umang.jain@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 6D796BDC71 for ; Tue, 7 Sep 2021 19:57:28 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3AB7A6916D; Tue, 7 Sep 2021 21:57:28 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="nqpNwoQl"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id CBFC369170 for ; Tue, 7 Sep 2021 21:57:26 +0200 (CEST) Received: from perceval.ideasonboard.com (unknown [103.251.226.2]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id BE507499; Tue, 7 Sep 2021 21:57:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631044646; bh=4MV9WVM3H+SvBCE2rQKm8NsrlZsuki7KrEWI8O02328=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nqpNwoQlVc9l9TIIB8xyQMZeecRHDwJyTErT8w1sFZPVwZzxfcw36PiBSPOA61G/p sRJ4I6DrPhKRk0QNVkwTZ73kguS7YegmA3HahrI/mzdepNRYtxdl91k0aJGZfk7zXY Cd5zM4u247QNI0xSJB3GTjxhmdUhZwkOh0x0bM2U= From: Umang Jain To: libcamera-devel@lists.libcamera.org Date: Wed, 8 Sep 2021 01:27:02 +0530 Message-Id: <20210907195704.338079-5-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210907195704.338079-1-umang.jain@ideasonboard.com> References: <20210907195704.338079-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 4/6] android: camera_device: Add a queue for sending capture results X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When a camera capture request completes, the next step is to send the capture results to the framework via process_capture_results(). All the capture associated result metadata is prepared and populated. If any post-processing is required, it will also take place in the same thread (which can be blocking for a subtaintial amount of time) before the results can be sent back to the framework. A follow up commit will move the post-processing to run in a separate thread. In order to do so, there is few bits of groundwork which this patch entails. Mainly, we need to preserve the order of sending the capture results back in which they were queued by the framework to the HAL (i.e. the order is sequential). Hence, we need to add a queue in which capture results can be queued with context. As per this patch, the post-processor still runs synchronously as before, but it will queue up the current capture results with context. Later down the line, the capture results are dequeud in order and sent back to the framework. The context is preserved using Camera3RequestDescriptor utility structure. It has been expanded accordingly to address the needs of the functionality. Signed-off-by: Umang Jain --- src/android/camera_device.cpp | 113 +++++++++++++++++++++++++++++++--- src/android/camera_device.h | 23 +++++++ 2 files changed, 126 insertions(+), 10 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index f2f36f32..9d4ec02e 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -240,6 +240,8 @@ CameraDevice::Camera3RequestDescriptor::Camera3RequestDescriptor( /* Clone the controls associated with the camera3 request. */ settings_ = CameraMetadata(camera3Request->settings); + internalBuffer_ = nullptr; + /* * Create the CaptureRequest, stored as a unique_ptr<> to tie its * lifetime to the descriptor. @@ -989,6 +991,7 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques * once it has been processed. */ buffer = cameraStream->getBuffer(); + descriptor.internalBuffer_ = buffer; LOG(HAL, Debug) << ss.str() << " (internal)"; break; } @@ -1148,25 +1151,115 @@ void CameraDevice::requestComplete(Request *request) continue; } + /* + * Save the current context of capture result and queue the + * descriptor before processing the camera stream. + * + * When the processing completes, the descriptor will be + * dequeued and capture results will be sent to the framework. + */ + descriptor.status_ = Camera3RequestDescriptor::NOT_FINISHED; + descriptor.resultMetadata_ = std::move(resultMetadata); + descriptor.captureResult_ = captureResult; + + std::unique_ptr reqDescriptor = + std::make_unique(); + *reqDescriptor = std::move(descriptor); + queuedDescriptor_.push_back(std::move(reqDescriptor)); + Camera3RequestDescriptor *currentDescriptor = queuedDescriptor_.back().get(); + + CameraMetadata *metadata = currentDescriptor->resultMetadata_.get(); + cameraStream->processComplete.connect( + this, [=](CameraStream::ProcessStatus status) { + streamProcessingComplete(cameraStream, status, + metadata); + }); + int ret = cameraStream->process(src, *buffer.buffer, - descriptor.settings_, - resultMetadata.get()); + currentDescriptor->settings_, + metadata); + return; + } + + if (queuedDescriptor_.empty()) { + captureResult.result = resultMetadata->get(); + callbacks_->process_capture_result(callbacks_, &captureResult); + } else { + /* + * Previous capture results waiting to be sent to framework + * hence, queue the current capture results as well. After that, + * check if any results are ready to be sent back to the + * framework. + */ + descriptor.status_ = Camera3RequestDescriptor::FINISHED_SUCCESS; + descriptor.resultMetadata_ = std::move(resultMetadata); + descriptor.captureResult_ = captureResult; + std::unique_ptr reqDescriptor = + std::make_unique(); + *reqDescriptor = std::move(descriptor); + queuedDescriptor_.push_back(std::move(reqDescriptor)); + + sendQueuedCaptureResults(); + } +} + +void CameraDevice::streamProcessingComplete(CameraStream *cameraStream, + CameraStream::ProcessStatus status, + CameraMetadata *resultMetadata) +{ + for (auto &d : queuedDescriptor_) { + if (d->resultMetadata_.get() != resultMetadata) + continue; + /* * Return the FrameBuffer to the CameraStream now that we're * done processing it. */ if (cameraStream->type() == CameraStream::Type::Internal) - cameraStream->putBuffer(src); - - if (ret) { - buffer.status = CAMERA3_BUFFER_STATUS_ERROR; - notifyError(descriptor.frameNumber_, buffer.stream, - CAMERA3_MSG_ERROR_BUFFER); + cameraStream->putBuffer(d->internalBuffer_); + + if (status == CameraStream::ProcessStatus::Success) { + d->status_ = Camera3RequestDescriptor::FINISHED_SUCCESS; + } else { + d->status_ = Camera3RequestDescriptor::FINISHED_FAILED; + d->captureResult_.partial_result = 0; + for (camera3_stream_buffer_t &buffer : d->buffers_) { + CameraStream *cs = static_cast(buffer.stream->priv); + + if (cs->camera3Stream().format != HAL_PIXEL_FORMAT_BLOB) + continue; + + buffer.status = CAMERA3_BUFFER_STATUS_ERROR; + notifyError(d->frameNumber_, buffer.stream, + CAMERA3_MSG_ERROR_BUFFER); + } } + + break; } - captureResult.result = resultMetadata->get(); - callbacks_->process_capture_result(callbacks_, &captureResult); + /* + * Send back capture results to the framework by inspecting the queue. + * The framework can defer queueing further requests to the HAL (via + * process_capture_request) unless until it receives the capture results + * for already queued requests. + */ + sendQueuedCaptureResults(); +} + +void CameraDevice::sendQueuedCaptureResults() +{ + while (!queuedDescriptor_.empty()) { + std::unique_ptr &d = queuedDescriptor_.front(); + if (d->status_ != Camera3RequestDescriptor::NOT_FINISHED) { + d->captureResult_.result = d->resultMetadata_->get(); + callbacks_->process_capture_result(callbacks_, + &(d->captureResult_)); + queuedDescriptor_.pop_front(); + } else { + break; + } + } } std::string CameraDevice::logPrefix() const diff --git a/src/android/camera_device.h b/src/android/camera_device.h index a5576927..36425773 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -7,6 +7,7 @@ #ifndef __ANDROID_CAMERA_DEVICE_H__ #define __ANDROID_CAMERA_DEVICE_H__ +#include #include #include #include @@ -83,6 +84,21 @@ private: std::vector> frameBuffers_; CameraMetadata settings_; std::unique_ptr request_; + + /* + * Below are utility placeholders used when a capture result + * needs to be queued before completion via + * process_capture_result() callback. + */ + enum completionStatus { + NOT_FINISHED, + FINISHED_SUCCESS, + FINISHED_FAILED, + }; + std::unique_ptr resultMetadata_; + camera3_capture_result_t captureResult_; + libcamera::FrameBuffer *internalBuffer_; + completionStatus status_; }; enum class State { @@ -105,6 +121,11 @@ private: std::unique_ptr getResultMetadata( const Camera3RequestDescriptor &descriptor) const; + void sendQueuedCaptureResults(); + void streamProcessingComplete(CameraStream *stream, + CameraStream::ProcessStatus status, + CameraMetadata *resultMetadata); + unsigned int id_; camera3_device_t camera3Device_; @@ -125,6 +146,8 @@ private: libcamera::Mutex descriptorsMutex_; /* Protects descriptors_. */ std::map descriptors_; + std::deque> queuedDescriptor_; + std::string maker_; std::string model_;