From patchwork Fri Nov 19 13:15:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 14659 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 80995BDB13 for ; Fri, 19 Nov 2021 13:15:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 37F096038B; Fri, 19 Nov 2021 14:15:24 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="C5etcH7W"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6B69460233 for ; Fri, 19 Nov 2021 14:15:23 +0100 (CET) Received: from perceval.ideasonboard.com (unknown [103.238.109.11]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id B14741959; Fri, 19 Nov 2021 14:15:21 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637327723; bh=58Z3SQwHRGnHnO2jwIudlDet8HwYzd65YgRqh3OmxYQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=C5etcH7W3bRlYOtynhwDxEzlldRKOPvJp2CalQHrY/JWmaSxKOEZJDY5vnr42IBNG 5R9E2VvbSxBnmZ4zquFI6E9BGreDfnf8iXoXs0hAGYzFJrCQUfIcDojAJYpdIqwY4c W8xTKTp4nZs8TV/YZIKymDWbFSiGwLIP5KbVynWY= From: Umang Jain To: libcamera-devel@lists.libcamera.org Date: Fri, 19 Nov 2021 18:45:05 +0530 Message-Id: <20211119131506.382462-2-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211119131506.382462-1-umang.jain@ideasonboard.com> References: <20211119131506.382462-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH 1/2] android: Document the structures and functions for post-processing X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Specifically document: - CameraDevice::sendCaptureResults() - CameraDevice::completeDescriptor() - CameraDevice::streamProcessingComplete() - CameraStream::PostProcessorWorker class - Camera3RequestDescriptor::StreamBuffer structure Signed-off-by: Umang Jain Reviewed-by: Laurent Pinchart --- src/android/camera_device.cpp | 35 +++++++++++++++++++++++++++++++ src/android/camera_request.cpp | 38 ++++++++++++++++++++++++++++++++++ src/android/camera_stream.cpp | 12 +++++++++++ 3 files changed, 85 insertions(+) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index f2e0bdbd..fb46be4d 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -1147,6 +1147,16 @@ void CameraDevice::requestComplete(Request *request) } } +/** + * \brief Complete the Camera3RequestDescriptor + * \param[in] descriptor The Camera3RequestDescriptor deemed to be complete + * + * Marks the Camera3RequestDescriptor as 'complete'. This means all the streams + * in the Camera3RequestDescriptor have completed capture (or generated via + * post-processing) and is ready be sent back to the framework. + * + * \context This function is \threadsafe. + */ void CameraDevice::completeDescriptor(Camera3RequestDescriptor *descriptor) { MutexLocker lock(descriptorsMutex_); @@ -1155,6 +1165,17 @@ void CameraDevice::completeDescriptor(Camera3RequestDescriptor *descriptor) sendCaptureResults(); } +/** + * \brief Sequentially send capture results to the framework + * + * Inspect the head of the descriptors' queue and see if it is ready to be sent + * back to the framework. Populate a locally-scoped camera3_capture_result_t + * using the fields of the descriptor and send the capture result back by + * calling the process_capture_result() callback. + * + * This function should never be called directly in the codebase. Use + * completeDescriptor() instead. + */ void CameraDevice::sendCaptureResults() { while (!descriptors_.empty() && !descriptors_.front()->isPending()) { @@ -1214,6 +1235,20 @@ void CameraDevice::setBufferStatus(Camera3RequestDescriptor::StreamBuffer &strea } } +/** + * \brief Handle post-processing completion of a stream in a capture request + * \param[in] streamBuffer The StreamBuffer for which processing is complete + * \param[in] status Stream post-processing status + * + * This is invoked in the post-processor's thread whenever a camera stream has + * finished post processing. The corresponding entry is dropped from the + * descriptor's pendingStreamsToProcess_ map. + * + * If the pendingStreamsToProcess_ map is found to be empty, it is perceived + * that all the streams requiring to be generated from post-processing have been + * completed. Mark the descriptor as complete using completeDescriptor() in that + * case. + */ void CameraDevice::streamProcessingComplete(Camera3RequestDescriptor::StreamBuffer *streamBuffer, Camera3RequestDescriptor::Status status) { diff --git a/src/android/camera_request.cpp b/src/android/camera_request.cpp index 8162aa78..4e017792 100644 --- a/src/android/camera_request.cpp +++ b/src/android/camera_request.cpp @@ -53,6 +53,44 @@ Camera3RequestDescriptor::Camera3RequestDescriptor( Camera3RequestDescriptor::~Camera3RequestDescriptor() = default; +/** + * \struct Camera3RequestDescriptor::StreamBuffer + * \brief Groups information for per-stream buffer of Camera3RequestDescriptor + * + * A capture request placed to the libcamera HAL can contain multiple streams. + * Each stream will have an associated buffer to be filled. StreamBuffer struct + * tracks this buffer with contextual information which aids in the stream's + * generation. The generation of stream will depend on its type (refer + * CameraStream::Type documentation). + * + * \var Camera3RequestDescriptor::StreamBuffer::stream + * \brief Corresponding pointer to CameraStream + * + * \var Camera3RequestDescriptor::StreamBuffer::camera3Buffer + * \brief Native handle to the buffer + * + * \var Camera3RequestDescriptor::StreamBuffer::frameBuffer + * \brief Encapsulate the dmabuf handle inside a libcamera::FrameBuffer for + * direct streams + * + * \var Camera3RequestDescriptor::StreamBuffer::fence + * \brief Acquire fence of the buffer + * + * \var Camera3RequestDescriptor::StreamBuffer::status + * \brief Track the status of the buffer + * + * \var Camera3RequestDescriptor::StreamBuffer::internalBuffer + * \brief Pointer to a buffer internally handled by CameraStream (if any) + * + * \var Camera3RequestDescriptor::StreamBuffer::srcBuffer + * \brief Pointer to source frame buffer required for post-processing + * + * \var Camera3RequestDescriptor::StreamBuffer::dstBuffer + * \brief Pointer to destination frame buffer required for post-processing + * + * \var Camera3RequestDescriptor::StreamBuffer::request + * \brief Back pointer to Camera3RequestDescriptor to which StreamBuffer belongs + */ Camera3RequestDescriptor::StreamBuffer::StreamBuffer( CameraStream *cameraStream, const camera3_stream_buffer_t &buffer, Camera3RequestDescriptor *requestDescriptor) diff --git a/src/android/camera_stream.cpp b/src/android/camera_stream.cpp index 9023c13c..3cda58c9 100644 --- a/src/android/camera_stream.cpp +++ b/src/android/camera_stream.cpp @@ -231,6 +231,18 @@ void CameraStream::putBuffer(FrameBuffer *buffer) buffers_.push_back(buffer); } +/** + * \class CameraStream::PostProcessorWorker + * \brief Post process a CameraStream on an internal thread + * + * If the association between CameraStream and camera3_stream_t dictated by + * CameraStream::Type is internal or mapped, the stream is generated by post + * processing of a libcamera stream. Such a request is queued to + * PostProcessorWorker in CameraStream::process(). A queue of post-processing + * requests is maintained by the PostProcessorWorker and it will run the + * post-processing on an internal thread as soon as any request is available on + * its queue. + */ CameraStream::PostProcessorWorker::PostProcessorWorker(PostProcessor *postProcessor) : postProcessor_(postProcessor) { From patchwork Fri Nov 19 13:15:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 14660 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2FD89BDB13 for ; Fri, 19 Nov 2021 13:15:28 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DBA9B60376; Fri, 19 Nov 2021 14:15:27 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="VgabFReV"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id AAF3560376 for ; Fri, 19 Nov 2021 14:15:25 +0100 (CET) Received: from perceval.ideasonboard.com (unknown [103.238.109.11]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 5CED91959; Fri, 19 Nov 2021 14:15:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1637327725; bh=XFGO2dcQ0y919xqo3DAU4E4FO8ntuB5pEhSEXyG5pmE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VgabFReVyPlEPYTQnSxmvCiGP6/CEq0Pu1p7KMLouNNedkKXjhbq3DcJ75qS/m3Hs aIj9iauGKERA6rs1nEjoj/W0T/p+16J5IR3pi7YSgoIKeQUKZFH5M5NY9w1P9ectSx fIBZwKovsYDByF/lSN4xG4+E3nwWT3kIv/Gw5R5U= From: Umang Jain To: libcamera-devel@lists.libcamera.org Date: Fri, 19 Nov 2021 18:45:06 +0530 Message-Id: <20211119131506.382462-3-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211119131506.382462-1-umang.jain@ideasonboard.com> References: <20211119131506.382462-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH 2/2] android: camera_request: Lifetime of a Camera3RequestDescriptor X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This commit provides a sketch regarding Camera3RequestDescriptor which is aids tracking each capture reuquest placed by the android framework to libcamera HAL. Signed-off-by: Umang Jain Reviewed-by: Laurent Pinchart --- src/android/camera_request.cpp | 97 ++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) diff --git a/src/android/camera_request.cpp b/src/android/camera_request.cpp index 4e017792..824b667d 100644 --- a/src/android/camera_request.cpp +++ b/src/android/camera_request.cpp @@ -18,6 +18,9 @@ using namespace libcamera; * * A utility class that groups information about a capture request to be later * reused at request complete time to notify the framework. + * + * Also, refer to the Camera3RequestDescriptor's lifetime diagram at the end of + * this file. */ Camera3RequestDescriptor::Camera3RequestDescriptor( @@ -105,3 +108,97 @@ Camera3RequestDescriptor::StreamBuffer::StreamBuffer(StreamBuffer &&) = default; Camera3RequestDescriptor::StreamBuffer & Camera3RequestDescriptor::StreamBuffer::operator=(Camera3RequestDescriptor::StreamBuffer &&) = default; + +/******************************************************************************* + * Lifetime of a Camera3RequestDescriptor tracking a capture request placed by + * Android Framework + ******************************************************************************* + * + * + * Android Framework + * │ + * │ ┌──────────────────────────────────┐ + * │ │camera3_capture_request_t │ + * │ │ │ + * │ │Requested output streams │ + * │ │ stream1 stream2 stream3 ... │ + * │ └──────────────────────────────────┘ + * ▼ + * ┌─────────────────────────────────────────────────────────────┐ + * │ libcamera HAL │ + * ├─────────────────────────────────────────────────────────────┤ + * │ CameraDevice │ + * │ │ + * │ processCaptureRequest(camera3_capture_request_t request) │ + * │ │ + * │ - Create Camera3RequestDescriptor tracking this request │ + * │ - Streams requiring post-processing is stored as a │ + * │ map Camera3Requestdescriptor::pendingStreamsToProcess │ + * │ - Add this Camera3RequestDescriptor to descriptors' queue │ + * │ CameraDevice::descriptors_ │ + * │ │ ┌─────────────────────────┐ + * │ - Queue the capture request to libcamera core ────────────┤►│libcamera core │ + * │ │ ├─────────────────────────┤ + * │ │ │- Capture from Camera │ + * │ │ │ │ + * │ │ │- Emit │ + * │ │ │ Camera::requestComplete│ + * │ requestCompleted(Request *request) ◄───────────────────────┼─┼──── │ + * │ │ │ │ + * │ - Check request completion status │ └─────────────────────────┘ + * │ │ + * │ - If(pendingStreamsToProcess > 0) │ + * │ Queue all entries from pendingStreamsToProcess │ + * │ else │ │ + * │ completeDescriptor() │ └──────────────────────┐ + * │ │ │ + * │ ┌──────────────────────────┴───┬──────────────────┐ │ + * │ │ │ │ │ + * │ ┌──────────▼────────────┐ ┌───────────▼─────────┐ ▼ │ + * │ │CameraStream1 │ │CameraStream2 │ .... │ + * │ ├┬───┬───┬──────────────┤ ├┬───┬───┬────────────┤ │ + * │ ││ │ │ │ ││ │ │ │ │ + * │ │▼───▼───▼──────────────┤ │▼───▼───▼────────────┤ │ + * │ │PostProcessorWorker ├─ │PostProcessorWorker │ │ + * │ │ │ │ │ │ + * │ │ xxxxxxxxxxxxxxxxxxx │ │ xxxxxxxxxxxxxxxxxxx │ │ + * │ │ x PostProcessor x │ │ x PostProcessor x │ │ + * │ │ x process() x │ │ x process() x │ │ + * │ │ x x │ │ x x │ │ + * │ │ x Emit x │ │ x Emit x │ │ + * │ │ x processComplete x │ │ x processComplete x │ │ + * │ │ x x │ │ x x │ │ + * │ │ xxxxxxxxxxxxxxx│xxx │ │ xxxxxxxxxxxxxxx│xxx │ │ + * │ │ │ │ │ │ │ │ + * │ │ │ │ │ │ │ │ + * │ └────────────────┼──────┘ └────────────────┼────┘ │ + * │ │ │ │ + * │ │ │ │ + * │ │ │ │ + * │ ▼ ▼ │ + * │ xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx │ + * │ x CameraDevice x x CameraDevice x │ + * │ x x x x │ + * │ x streamProcessingComplete() x x streamProcessingComplete() x │ + * │ x x x x │ + * │ x - Check and set buffer status x x - Check and set buffer status x │ + * │ x - Remove post-processing entry x x - Remove post-processing entry x │ + * │ x from pendingStreamsToProcess x x from pendingStreamsToProcess x │ + * │ x x x x │ + * │ x - If(pendingStreamsToProcess.empty()x x - If(pendingStreamsToProcess.empty()x │ + * │ x completeDescriptor x x completeDescriptor x │ + * │ x x x x │ + * │ xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx │ + * │ │ + * └────────────────────────────────────────────────────────────────────────────────────┘ + * + * + * + * + * + * + * xxxxxxxxxxxxxx + * x x - PostProcessorWorker's thread + * x x + * xxxxxxxxxxxxxx + */