{"id":12241,"url":"https://patchwork.libcamera.org/api/1.1/patches/12241/?format=json","web_url":"https://patchwork.libcamera.org/patch/12241/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20210510105235.28319-2-jacopo@jmondi.org>","date":"2021-05-10T10:52:28","name":"[libcamera-devel,1/8] android: Rework request completion notification","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"0ce56d9b13d9ac665c777567fe604b4784cb9cf5","submitter":{"id":3,"url":"https://patchwork.libcamera.org/api/1.1/people/3/?format=json","name":"Jacopo Mondi","email":"jacopo@jmondi.org"},"delegate":{"id":15,"url":"https://patchwork.libcamera.org/api/1.1/users/15/?format=json","username":"jmondi","first_name":"Jacopo","last_name":"Mondi","email":"jacopo@jmondi.org"},"mbox":"https://patchwork.libcamera.org/patch/12241/mbox/","series":[{"id":2016,"url":"https://patchwork.libcamera.org/api/1.1/series/2016/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=2016","date":"2021-05-10T10:52:27","name":"android: Implement flush() camera operation","version":1,"mbox":"https://patchwork.libcamera.org/series/2016/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/12241/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/12241/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id E759EBF831\n\tfor ;\n\tMon, 10 May 2021 10:51:58 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id B27796890E;\n\tMon, 10 May 2021 12:51:58 +0200 (CEST)","from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net\n\t[217.70.183.196])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 6AB4B688E5\n\tfor ;\n\tMon, 10 May 2021 12:51:56 +0200 (CEST)","from uno.homenet.telecomitalia.it\n\t(host-82-59-136-116.retail.telecomitalia.it [82.59.136.116])\n\t(Authenticated sender: jacopo@jmondi.org)\n\tby relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 6ADD6E01DB;\n\tMon, 10 May 2021 10:51:55 +0000 (UTC)"],"X-Originating-IP":"82.59.136.116","From":"Jacopo Mondi ","To":"libcamera-devel@lists.libcamera.org","Date":"Mon, 10 May 2021 12:52:28 +0200","Message-Id":"<20210510105235.28319-2-jacopo@jmondi.org>","X-Mailer":"git-send-email 2.31.1","In-Reply-To":"<20210510105235.28319-1-jacopo@jmondi.org>","References":"<20210510105235.28319-1-jacopo@jmondi.org>","MIME-Version":"1.0","Subject":"[libcamera-devel] [PATCH 1/8] android: Rework request completion\n\tnotification","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"The current implementation of CameraDevice::requestComplete() which\nhandles event notification and calls the framework capture result\ncallback does not handle error notification precisely enough.\n\nIn detail:\n- Error notification is an asynchronous callback that has to be notified\n to the framework as soon as an error condition is detected, and it\n independent from the process_capture_result() callback\n\n- Error notification requires the HAL to report the precise error cause,\n by specifying the correct CAMERA3_MSG_ERROR_* error code.\n\nThe current implementation only notifies errors of type\nCAMERA3_MSG_ERROR_REQUEST at the end of the procedure, before the\ncallback invocation.\n\nRework the procedure to:\n\n- Notify CAMERA3_MSG_ERROR_DEVICE and perform library tear-down in case\n a Fatal error is detected\n\n- Notify CAMERA3_MSG_ERROR_REQUEST if the libcamera::Request::status is\n different than RequestCompleted and immediately call\n process_capture_result() with all buffers in error state.\n\n- Notify the shutter event as soon as possible\n\n- Notify CAMERA3_MSG_ERROR_RESULT in case the metadata cannot be\n generated correctly and call process_capture_result() with the right\n buffer state regardless of metadata availability.\n\n- Notify CAMERA3_MSG_ERROR_BUFFER for buffers whose post-processing\n failed\n\nWhile at it, return the CameraStream buffer by calling\ncameraStream->putBuffer() regardless of the post-processing result.\n\nNo regression detected when running CTS in LIMITED mode.\n\nSigned-off-by: Jacopo Mondi \n---\n src/android/camera_device.cpp | 138 +++++++++++++++++++++-------------\n src/android/camera_device.h | 3 +-\n 2 files changed, 86 insertions(+), 55 deletions(-)","diff":"diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp\nindex f1f26c775611..d8b7e62ef896 100644\n--- a/src/android/camera_device.cpp\n+++ b/src/android/camera_device.cpp\n@@ -2025,17 +2025,20 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques\n \n void CameraDevice::requestComplete(Request *request)\n {\n-\tcamera3_buffer_status status = CAMERA3_BUFFER_STATUS_OK;\n-\tstd::unique_ptr resultMetadata;\n-\n \tdecltype(descriptors_)::node_type node;\n \t{\n \t\tstd::scoped_lock lock(mutex_);\n \t\tauto it = descriptors_.find(request->cookie());\n \t\tif (it == descriptors_.end()) {\n+\t\t\t/*\n+\t\t\t * \\todo Clarify if the Camera has to be closed on\n+\t\t\t * ERROR_DEVICE and possibly demote the Fatal to simple\n+\t\t\t * Error.\n+\t\t\t */\n+\t\t\tnotifyError(0, nullptr, CAMERA3_MSG_ERROR_DEVICE);\n \t\t\tLOG(HAL, Fatal)\n \t\t\t\t<< \"Unknown request: \" << request->cookie();\n-\t\t\tstatus = CAMERA3_BUFFER_STATUS_ERROR;\n+\n \t\t\treturn;\n \t\t}\n \n@@ -2043,16 +2046,72 @@ void CameraDevice::requestComplete(Request *request)\n \t}\n \tCamera3RequestDescriptor &descriptor = node.mapped();\n \n+\t/*\n+\t * Prepare the capture result for the Android camera stack.\n+\t *\n+\t * The buffer status is set to OK and later changed to ERROR if\n+\t * post-processing/compression fails.\n+\t */\n+\tcamera3_capture_result_t captureResult = {};\n+\tcaptureResult.frame_number = descriptor.frameNumber_;\n+\tcaptureResult.num_output_buffers = descriptor.buffers_.size();\n+\tfor (camera3_stream_buffer_t &buffer : descriptor.buffers_) {\n+\t\tbuffer.acquire_fence = -1;\n+\t\tbuffer.release_fence = -1;\n+\t\tbuffer.status = CAMERA3_BUFFER_STATUS_OK;\n+\t}\n+\tcaptureResult.output_buffers = descriptor.buffers_.data();\n+\tcaptureResult.partial_result = 1;\n+\n+\t/*\n+\t * If the Request has failed, abort the request by notifying the error\n+\t * and complete the request with all buffers in error state.\n+\t */\n \tif (request->status() != Request::RequestComplete) {\n-\t\tLOG(HAL, Error) << \"Request not successfully completed: \"\n+\t\tLOG(HAL, Error) << \"Request \" << request->cookie()\n+\t\t\t\t<< \" not successfully completed: \"\n \t\t\t\t<< request->status();\n-\t\tstatus = CAMERA3_BUFFER_STATUS_ERROR;\n+\n+\t\tnotifyError(descriptor.frameNumber_,\n+\t\t\t descriptor.buffers_[0].stream,\n+\t\t\t CAMERA3_MSG_ERROR_REQUEST);\n+\n+\t\tcaptureResult.partial_result = 0;\n+\t\tfor (camera3_stream_buffer_t &buffer : descriptor.buffers_)\n+\t\t\tbuffer.status = CAMERA3_BUFFER_STATUS_ERROR;\n+\t\tcallbacks_->process_capture_result(callbacks_, &captureResult);\n+\n+\t\treturn;\n \t}\n \n+\t/*\n+\t * Notify shutter as soon as we have verified we have a valid request.\n+\t *\n+\t * \\todo The shutter event notification should be sent to the framework\n+\t * as soon as possible, earlier than request completion time.\n+\t */\n+\tuint64_t sensorTimestamp = static_cast(request->metadata()\n+\t\t\t\t\t\t\t .get(controls::SensorTimestamp));\n+\tnotifyShutter(descriptor.frameNumber_, sensorTimestamp);\n+\n \tLOG(HAL, Debug) << \"Request \" << request->cookie() << \" completed with \"\n \t\t\t<< descriptor.buffers_.size() << \" streams\";\n \n-\tresultMetadata = getResultMetadata(descriptor);\n+\t/*\n+\t * Generate the metadata associated with the captured buffers.\n+\t *\n+\t * Notify if the metadata generation has failed, but continue processing\n+\t * buffers and return an empty metadata pack.\n+\t */\n+\tstd::unique_ptr resultMetadata = getResultMetadata(descriptor);\n+\tif (!resultMetadata) {\n+\t\tnotifyError(descriptor.frameNumber_, descriptor.buffers_[0].stream,\n+\t\t\t CAMERA3_MSG_ERROR_RESULT);\n+\n+\t\t/* The camera framework expects an empy metadata pack on error. */\n+\t\tresultMetadata = std::make_unique(0, 0);\n+\t}\n+\tcaptureResult.result = resultMetadata->get();\n \n \t/* Handle any JPEG compression. */\n \tfor (camera3_stream_buffer_t &buffer : descriptor.buffers_) {\n@@ -2065,56 +2124,27 @@ void CameraDevice::requestComplete(Request *request)\n \t\tFrameBuffer *src = request->findBuffer(cameraStream->stream());\n \t\tif (!src) {\n \t\t\tLOG(HAL, Error) << \"Failed to find a source stream buffer\";\n+\t\t\tbuffer.status = CAMERA3_BUFFER_STATUS_ERROR;\n+\t\t\tnotifyError(descriptor.frameNumber_, buffer.stream,\n+\t\t\t\t CAMERA3_MSG_ERROR_BUFFER);\n \t\t\tcontinue;\n \t\t}\n \n-\t\tint ret = cameraStream->process(*src,\n-\t\t\t\t\t\t*buffer.buffer,\n+\t\tint ret = cameraStream->process(*src, *buffer.buffer,\n \t\t\t\t\t\tdescriptor.settings_,\n \t\t\t\t\t\tresultMetadata.get());\n-\t\tif (ret) {\n-\t\t\tstatus = CAMERA3_BUFFER_STATUS_ERROR;\n-\t\t\tcontinue;\n-\t\t}\n-\n \t\t/*\n \t\t * Return the FrameBuffer to the CameraStream now that we're\n \t\t * done processing it.\n \t\t */\n \t\tif (cameraStream->type() == CameraStream::Type::Internal)\n \t\t\tcameraStream->putBuffer(src);\n-\t}\n \n-\t/* Prepare to call back the Android camera stack. */\n-\tcamera3_capture_result_t captureResult = {};\n-\tcaptureResult.frame_number = descriptor.frameNumber_;\n-\tcaptureResult.num_output_buffers = descriptor.buffers_.size();\n-\tfor (camera3_stream_buffer_t &buffer : descriptor.buffers_) {\n-\t\tbuffer.acquire_fence = -1;\n-\t\tbuffer.release_fence = -1;\n-\t\tbuffer.status = status;\n-\t}\n-\tcaptureResult.output_buffers = descriptor.buffers_.data();\n-\n-\tif (status == CAMERA3_BUFFER_STATUS_OK) {\n-\t\tuint64_t timestamp =\n-\t\t\tstatic_cast(request->metadata()\n-\t\t\t\t\t .get(controls::SensorTimestamp));\n-\t\tnotifyShutter(descriptor.frameNumber_, timestamp);\n-\n-\t\tcaptureResult.partial_result = 1;\n-\t\tcaptureResult.result = resultMetadata->get();\n-\t}\n-\n-\tif (status == CAMERA3_BUFFER_STATUS_ERROR || !captureResult.result) {\n-\t\t/* \\todo Improve error handling. In case we notify an error\n-\t\t * because the metadata generation fails, a shutter event has\n-\t\t * already been notified for this frame number before the error\n-\t\t * is here signalled. Make sure the error path plays well with\n-\t\t * the camera stack state machine.\n-\t\t */\n-\t\tnotifyError(descriptor.frameNumber_,\n-\t\t\t descriptor.buffers_[0].stream);\n+\t\tif (ret) {\n+\t\t\tbuffer.status = CAMERA3_BUFFER_STATUS_ERROR;\n+\t\t\tnotifyError(descriptor.frameNumber_, buffer.stream,\n+\t\t\t\t CAMERA3_MSG_ERROR_BUFFER);\n+\t\t}\n \t}\n \n \tcallbacks_->process_capture_result(callbacks_, &captureResult);\n@@ -2136,23 +2166,23 @@ void CameraDevice::notifyShutter(uint32_t frameNumber, uint64_t timestamp)\n \tcallbacks_->notify(callbacks_, ¬ify);\n }\n \n-void CameraDevice::notifyError(uint32_t frameNumber, camera3_stream_t *stream)\n+void CameraDevice::notifyError(uint32_t frameNumber, camera3_stream_t *stream,\n+\t\t\t camera3_error_msg_code code)\n {\n \tcamera3_notify_msg_t notify = {};\n \n-\t/*\n-\t * \\todo Report and identify the stream number or configuration to\n-\t * clarify the stream that failed.\n-\t */\n-\tLOG(HAL, Error) << \"Error occurred on frame \" << frameNumber << \" (\"\n-\t\t\t<< toPixelFormat(stream->format).toString() << \")\";\n-\n \tnotify.type = CAMERA3_MSG_ERROR;\n \tnotify.message.error.error_stream = stream;\n \tnotify.message.error.frame_number = frameNumber;\n-\tnotify.message.error.error_code = CAMERA3_MSG_ERROR_REQUEST;\n+\tnotify.message.error.error_code = code;\n \n \tcallbacks_->notify(callbacks_, ¬ify);\n+\n+\tif (stream)\n+\t\tLOG(HAL, Error) << \"Error occurred on frame \" << frameNumber << \" (\"\n+\t\t\t\t<< toPixelFormat(stream->format).toString() << \")\";\n+\telse\n+\t\tLOG(HAL, Error) << \"Fatal error occurred on device\";\n }\n \n /*\ndiff --git a/src/android/camera_device.h b/src/android/camera_device.h\nindex 23457e47767a..8d5da8bc59e1 100644\n--- a/src/android/camera_device.h\n+++ b/src/android/camera_device.h\n@@ -101,7 +101,8 @@ private:\n \tstd::tuple calculateStaticMetadataSize();\n \tlibcamera::FrameBuffer *createFrameBuffer(const buffer_handle_t camera3buffer);\n \tvoid notifyShutter(uint32_t frameNumber, uint64_t timestamp);\n-\tvoid notifyError(uint32_t frameNumber, camera3_stream_t *stream);\n+\tvoid notifyError(uint32_t frameNumber, camera3_stream_t *stream,\n+\t\t\t camera3_error_msg_code code);\n \tstd::unique_ptr requestTemplatePreview();\n \tstd::unique_ptr requestTemplateVideo();\n \tlibcamera::PixelFormat toPixelFormat(int format) const;\n","prefixes":["libcamera-devel","1/8"]}