| Message ID | 20210504155808.67191-1-jacopo@jmondi.org |
|---|---|
| State | Superseded |
| Delegated to: | Jacopo Mondi |
| Headers | show |
| Series |
|
| Related | show |
Hi Jacopo, Thanks for your work. On 2021-05-04 17:58:08 +0200, Jacopo Mondi wrote: > The current implementation of CameraDevice::requestComplete() which > handles event notification and calls the framework capture result > callback does not handle error notification precisely enough. > > In detail: > - Error notification is an asynchronous callback that has to be notified > to the framework as soon as an error condition is detected, and it > independent from the process_capture_result() callback > > - Error notification requires the HAL to report the precise error cause, > by specifying the correct CAMERA3_MSG_ERROR_* error code. > > The current implementation only notifies errors of type > CAMERA3_MSG_ERROR_REQUEST at the end of the procedure, before the > callback invocation. > > Rework the procedure to: > > - Notify CAMERA3_MSG_ERROR_DEVICE and close the camera in case a Fatal > error is detected (Fatal does not perform library teardown in > production builds) > > - Notify CAMERA3_MSG_ERROR_REQUEST if the libcamera::Request::status is > different than RequestCompleted and immediately call > process_capture_result() with all buffers in error state. > > - Notify the shutter event as soon as possible > > - Notify CAMERA3_MSG_ERROR_RESULT in case the metadata cannot be > generated correctly and call process_capture_result() with the right > buffer state regardless of metadata availability. > > - Notify CAMERA3_MSG_ERROR_BUFFER for buffers whose post-processing > failed > > While at it, return the CameraStream buffer by calling > cameraStream->putBuffer() regardless of the post-processing result. > > No regression detected when running CTS in LIMITED mode. > > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org> > --- > src/android/camera_device.cpp | 128 ++++++++++++++++++++-------------- > src/android/camera_device.h | 3 +- > 2 files changed, 76 insertions(+), 55 deletions(-) > > diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp > index f1f26c775611..5588e99b6eba 100644 > --- a/src/android/camera_device.cpp > +++ b/src/android/camera_device.cpp > @@ -2025,9 +2025,6 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques > > void CameraDevice::requestComplete(Request *request) > { > - camera3_buffer_status status = CAMERA3_BUFFER_STATUS_OK; > - std::unique_ptr<CameraMetadata> resultMetadata; > - > decltype(descriptors_)::node_type node; > { > std::scoped_lock<std::mutex> lock(mutex_); > @@ -2035,7 +2032,8 @@ void CameraDevice::requestComplete(Request *request) > if (it == descriptors_.end()) { > LOG(HAL, Fatal) > << "Unknown request: " << request->cookie(); > - status = CAMERA3_BUFFER_STATUS_ERROR; > + close(); > + notifyError(0, nullptr, CAMERA3_MSG_ERROR_DEVICE); > return; > } > > @@ -2043,16 +2041,67 @@ void CameraDevice::requestComplete(Request *request) > } > Camera3RequestDescriptor &descriptor = node.mapped(); > > + /* > + * Prepare the capture result to call back the Android camera stack with. nit: Prepare the capture result for the Android camera stack ? > + * > + * The buffer status is set to OK and later changed to ERROR if > + * post-processing/compression fails. > + */ > + camera3_capture_result_t captureResult = {}; > + captureResult.frame_number = descriptor.frameNumber_; > + captureResult.num_output_buffers = descriptor.buffers_.size(); > + for (camera3_stream_buffer_t &buffer : descriptor.buffers_) { > + buffer.acquire_fence = -1; > + buffer.release_fence = -1; > + buffer.status = CAMERA3_BUFFER_STATUS_OK; > + } > + captureResult.output_buffers = descriptor.buffers_.data(); > + captureResult.partial_result = 1; > + > + /* > + * If the Request has failed, abort the request by notifying the error > + * and complete the request with all buffers in error state. > + */ > if (request->status() != Request::RequestComplete) { > - LOG(HAL, Error) << "Request not successfully completed: " > + LOG(HAL, Error) << "Request " << request->cookie() > + << " not successfully completed: " > << request->status(); > - status = CAMERA3_BUFFER_STATUS_ERROR; > + > + notifyError(descriptor.frameNumber_, > + descriptor.buffers_[0].stream, > + CAMERA3_MSG_ERROR_REQUEST); > + > + for (camera3_stream_buffer_t &buffer : descriptor.buffers_) > + buffer.status = CAMERA3_BUFFER_STATUS_ERROR; > + callbacks_->process_capture_result(callbacks_, &captureResult); > + > + return; > } > > + /* > + * Notify shutter as soon as we have verified we have a valid request. > + * > + * \todo The shutter event notification should be sent to the framework > + * as soon as possible, earlier than request completion time. > + */ > + uint64_t sensorTimestamp = static_cast<uint64_t>(request->metadata() > + .get(controls::SensorTimestamp)); > + notifyShutter(descriptor.frameNumber_, sensorTimestamp); > + > LOG(HAL, Debug) << "Request " << request->cookie() << " completed with " > << descriptor.buffers_.size() << " streams"; > > - resultMetadata = getResultMetadata(descriptor); > + /* > + * Generate the metadata associated with the captured buffers. > + * > + * Notify if the metadata generation has failed, but continue processing > + * buffers and return an empty metadata pack. > + */ > + std::unique_ptr<CameraMetadata> resultMetadata = getResultMetadata(descriptor); > + if (!resultMetadata) > + notifyError(descriptor.frameNumber_, descriptor.buffers_[0].stream, > + CAMERA3_MSG_ERROR_RESULT); > + captureResult.result = resultMetadata->get(); Hum if !resultMetadata will not resultMetadata->get() give us a nullptr exception here? > > /* Handle any JPEG compression. */ > for (camera3_stream_buffer_t &buffer : descriptor.buffers_) { > @@ -2065,56 +2114,27 @@ void CameraDevice::requestComplete(Request *request) > FrameBuffer *src = request->findBuffer(cameraStream->stream()); > if (!src) { > LOG(HAL, Error) << "Failed to find a source stream buffer"; > + buffer.status = CAMERA3_BUFFER_STATUS_ERROR; > + notifyError(descriptor.frameNumber_, buffer.stream, > + CAMERA3_MSG_ERROR_BUFFER); > continue; > } > > - int ret = cameraStream->process(*src, > - *buffer.buffer, > + int ret = cameraStream->process(*src, *buffer.buffer, > descriptor.settings_, > resultMetadata.get()); > - if (ret) { > - status = CAMERA3_BUFFER_STATUS_ERROR; > - continue; > - } > - > /* > * Return the FrameBuffer to the CameraStream now that we're > * done processing it. > */ > if (cameraStream->type() == CameraStream::Type::Internal) > cameraStream->putBuffer(src); > - } > - > - /* Prepare to call back the Android camera stack. */ > - camera3_capture_result_t captureResult = {}; > - captureResult.frame_number = descriptor.frameNumber_; > - captureResult.num_output_buffers = descriptor.buffers_.size(); > - for (camera3_stream_buffer_t &buffer : descriptor.buffers_) { > - buffer.acquire_fence = -1; > - buffer.release_fence = -1; > - buffer.status = status; > - } > - captureResult.output_buffers = descriptor.buffers_.data(); > - > - if (status == CAMERA3_BUFFER_STATUS_OK) { > - uint64_t timestamp = > - static_cast<uint64_t>(request->metadata() > - .get(controls::SensorTimestamp)); > - notifyShutter(descriptor.frameNumber_, timestamp); > - > - captureResult.partial_result = 1; > - captureResult.result = resultMetadata->get(); > - } > > - if (status == CAMERA3_BUFFER_STATUS_ERROR || !captureResult.result) { > - /* \todo Improve error handling. In case we notify an error > - * because the metadata generation fails, a shutter event has > - * already been notified for this frame number before the error > - * is here signalled. Make sure the error path plays well with > - * the camera stack state machine. > - */ > - notifyError(descriptor.frameNumber_, > - descriptor.buffers_[0].stream); > + if (ret) { > + buffer.status = CAMERA3_BUFFER_STATUS_ERROR; > + notifyError(descriptor.frameNumber_, buffer.stream, > + CAMERA3_MSG_ERROR_BUFFER); > + } > } > > callbacks_->process_capture_result(callbacks_, &captureResult); > @@ -2136,23 +2156,23 @@ void CameraDevice::notifyShutter(uint32_t frameNumber, uint64_t timestamp) > callbacks_->notify(callbacks_, ¬ify); > } > > -void CameraDevice::notifyError(uint32_t frameNumber, camera3_stream_t *stream) > +void CameraDevice::notifyError(uint32_t frameNumber, camera3_stream_t *stream, > + camera3_error_msg_code code) > { > camera3_notify_msg_t notify = {}; > > - /* > - * \todo Report and identify the stream number or configuration to > - * clarify the stream that failed. > - */ > - LOG(HAL, Error) << "Error occurred on frame " << frameNumber << " (" > - << toPixelFormat(stream->format).toString() << ")"; > - > notify.type = CAMERA3_MSG_ERROR; > notify.message.error.error_stream = stream; > notify.message.error.frame_number = frameNumber; > - notify.message.error.error_code = CAMERA3_MSG_ERROR_REQUEST; > + notify.message.error.error_code = code; > > callbacks_->notify(callbacks_, ¬ify); > + > + if (stream) > + LOG(HAL, Error) << "Error occurred on frame " << frameNumber << " (" > + << toPixelFormat(stream->format).toString() << ")"; > + else > + LOG(HAL, Error) << "Fatal error occurred on device"; > } > > /* > diff --git a/src/android/camera_device.h b/src/android/camera_device.h > index 23457e47767a..8d5da8bc59e1 100644 > --- a/src/android/camera_device.h > +++ b/src/android/camera_device.h > @@ -101,7 +101,8 @@ private: > std::tuple<uint32_t, uint32_t> calculateStaticMetadataSize(); > libcamera::FrameBuffer *createFrameBuffer(const buffer_handle_t camera3buffer); > void notifyShutter(uint32_t frameNumber, uint64_t timestamp); > - void notifyError(uint32_t frameNumber, camera3_stream_t *stream); > + void notifyError(uint32_t frameNumber, camera3_stream_t *stream, > + camera3_error_msg_code code); > std::unique_ptr<CameraMetadata> requestTemplatePreview(); > std::unique_ptr<CameraMetadata> requestTemplateVideo(); > libcamera::PixelFormat toPixelFormat(int format) const; > -- > 2.31.1 > > _______________________________________________ > libcamera-devel mailing list > libcamera-devel@lists.libcamera.org > https://lists.libcamera.org/listinfo/libcamera-devel
Hi Niklas, On Thu, May 06, 2021 at 10:31:10AM +0200, Niklas Söderlund wrote: > Hi Jacopo, > > Thanks for your work. > > On 2021-05-04 17:58:08 +0200, Jacopo Mondi wrote: > > The current implementation of CameraDevice::requestComplete() which > > handles event notification and calls the framework capture result > > callback does not handle error notification precisely enough. > > > > In detail: > > - Error notification is an asynchronous callback that has to be notified > > to the framework as soon as an error condition is detected, and it > > independent from the process_capture_result() callback > > > > - Error notification requires the HAL to report the precise error cause, > > by specifying the correct CAMERA3_MSG_ERROR_* error code. > > > > The current implementation only notifies errors of type > > CAMERA3_MSG_ERROR_REQUEST at the end of the procedure, before the > > callback invocation. > > > > Rework the procedure to: > > > > - Notify CAMERA3_MSG_ERROR_DEVICE and close the camera in case a Fatal > > error is detected (Fatal does not perform library teardown in > > production builds) > > > > - Notify CAMERA3_MSG_ERROR_REQUEST if the libcamera::Request::status is > > different than RequestCompleted and immediately call > > process_capture_result() with all buffers in error state. > > > > - Notify the shutter event as soon as possible > > > > - Notify CAMERA3_MSG_ERROR_RESULT in case the metadata cannot be > > generated correctly and call process_capture_result() with the right > > buffer state regardless of metadata availability. > > > > - Notify CAMERA3_MSG_ERROR_BUFFER for buffers whose post-processing > > failed > > > > While at it, return the CameraStream buffer by calling > > cameraStream->putBuffer() regardless of the post-processing result. > > > > No regression detected when running CTS in LIMITED mode. > > > > Signed-off-by: Jacopo Mondi <jacopo@jmondi.org> > > --- > > src/android/camera_device.cpp | 128 ++++++++++++++++++++-------------- > > src/android/camera_device.h | 3 +- > > 2 files changed, 76 insertions(+), 55 deletions(-) > > > > diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp > > index f1f26c775611..5588e99b6eba 100644 > > --- a/src/android/camera_device.cpp > > +++ b/src/android/camera_device.cpp > > @@ -2025,9 +2025,6 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques > > > > void CameraDevice::requestComplete(Request *request) > > { > > - camera3_buffer_status status = CAMERA3_BUFFER_STATUS_OK; > > - std::unique_ptr<CameraMetadata> resultMetadata; > > - > > decltype(descriptors_)::node_type node; > > { > > std::scoped_lock<std::mutex> lock(mutex_); > > @@ -2035,7 +2032,8 @@ void CameraDevice::requestComplete(Request *request) > > if (it == descriptors_.end()) { > > LOG(HAL, Fatal) > > << "Unknown request: " << request->cookie(); > > - status = CAMERA3_BUFFER_STATUS_ERROR; > > + close(); > > + notifyError(0, nullptr, CAMERA3_MSG_ERROR_DEVICE); > > return; > > } > > > > @@ -2043,16 +2041,67 @@ void CameraDevice::requestComplete(Request *request) > > } > > Camera3RequestDescriptor &descriptor = node.mapped(); > > > > + /* > > + * Prepare the capture result to call back the Android camera stack with. > > nit: Prepare the capture result for the Android camera stack ? > Ack, it's simpler > > + * > > + * The buffer status is set to OK and later changed to ERROR if > > + * post-processing/compression fails. > > + */ > > + camera3_capture_result_t captureResult = {}; > > + captureResult.frame_number = descriptor.frameNumber_; > > + captureResult.num_output_buffers = descriptor.buffers_.size(); > > + for (camera3_stream_buffer_t &buffer : descriptor.buffers_) { > > + buffer.acquire_fence = -1; > > + buffer.release_fence = -1; > > + buffer.status = CAMERA3_BUFFER_STATUS_OK; > > + } > > + captureResult.output_buffers = descriptor.buffers_.data(); > > + captureResult.partial_result = 1; > > + > > + /* > > + * If the Request has failed, abort the request by notifying the error > > + * and complete the request with all buffers in error state. > > + */ > > if (request->status() != Request::RequestComplete) { > > - LOG(HAL, Error) << "Request not successfully completed: " > > + LOG(HAL, Error) << "Request " << request->cookie() > > + << " not successfully completed: " > > << request->status(); > > - status = CAMERA3_BUFFER_STATUS_ERROR; > > + > > + notifyError(descriptor.frameNumber_, > > + descriptor.buffers_[0].stream, > > + CAMERA3_MSG_ERROR_REQUEST); > > + > > + for (camera3_stream_buffer_t &buffer : descriptor.buffers_) > > + buffer.status = CAMERA3_BUFFER_STATUS_ERROR; > > + callbacks_->process_capture_result(callbacks_, &captureResult); > > + > > + return; > > } > > > > + /* > > + * Notify shutter as soon as we have verified we have a valid request. > > + * > > + * \todo The shutter event notification should be sent to the framework > > + * as soon as possible, earlier than request completion time. > > + */ > > + uint64_t sensorTimestamp = static_cast<uint64_t>(request->metadata() > > + .get(controls::SensorTimestamp)); > > + notifyShutter(descriptor.frameNumber_, sensorTimestamp); > > + > > LOG(HAL, Debug) << "Request " << request->cookie() << " completed with " > > << descriptor.buffers_.size() << " streams"; > > > > - resultMetadata = getResultMetadata(descriptor); > > + /* > > + * Generate the metadata associated with the captured buffers. > > + * > > + * Notify if the metadata generation has failed, but continue processing > > + * buffers and return an empty metadata pack. > > + */ > > + std::unique_ptr<CameraMetadata> resultMetadata = getResultMetadata(descriptor); > > + if (!resultMetadata) > > + notifyError(descriptor.frameNumber_, descriptor.buffers_[0].stream, > > + CAMERA3_MSG_ERROR_RESULT); > > + captureResult.result = resultMetadata->get(); > > Hum if !resultMetadata will not resultMetadata->get() give us a nullptr > exception here? > According to the std::unique_ptr<>::get() documentation: "Returns a pointer to the managed object or nullptr if no object is owned." as getResultMetadata() returns nullptr on error, and the function signature is std::unique_ptr<CameraMetadata> CameraDevice::getResultMetadata() my understanding is that the returned unique_ptr<> will own no object, hence get() should return nullptr. Or have I missed something about the "no object is owned" vs "a unique_ptr<> that owns nullptr" ? > > > > /* Handle any JPEG compression. */ > > for (camera3_stream_buffer_t &buffer : descriptor.buffers_) { > > @@ -2065,56 +2114,27 @@ void CameraDevice::requestComplete(Request *request) > > FrameBuffer *src = request->findBuffer(cameraStream->stream()); > > if (!src) { > > LOG(HAL, Error) << "Failed to find a source stream buffer"; > > + buffer.status = CAMERA3_BUFFER_STATUS_ERROR; > > + notifyError(descriptor.frameNumber_, buffer.stream, > > + CAMERA3_MSG_ERROR_BUFFER); > > continue; > > } > > > > - int ret = cameraStream->process(*src, > > - *buffer.buffer, > > + int ret = cameraStream->process(*src, *buffer.buffer, > > descriptor.settings_, > > resultMetadata.get()); > > - if (ret) { > > - status = CAMERA3_BUFFER_STATUS_ERROR; > > - continue; > > - } > > - > > /* > > * Return the FrameBuffer to the CameraStream now that we're > > * done processing it. > > */ > > if (cameraStream->type() == CameraStream::Type::Internal) > > cameraStream->putBuffer(src); > > - } > > - > > - /* Prepare to call back the Android camera stack. */ > > - camera3_capture_result_t captureResult = {}; > > - captureResult.frame_number = descriptor.frameNumber_; > > - captureResult.num_output_buffers = descriptor.buffers_.size(); > > - for (camera3_stream_buffer_t &buffer : descriptor.buffers_) { > > - buffer.acquire_fence = -1; > > - buffer.release_fence = -1; > > - buffer.status = status; > > - } > > - captureResult.output_buffers = descriptor.buffers_.data(); > > - > > - if (status == CAMERA3_BUFFER_STATUS_OK) { > > - uint64_t timestamp = > > - static_cast<uint64_t>(request->metadata() > > - .get(controls::SensorTimestamp)); > > - notifyShutter(descriptor.frameNumber_, timestamp); > > - > > - captureResult.partial_result = 1; > > - captureResult.result = resultMetadata->get(); > > - } > > > > - if (status == CAMERA3_BUFFER_STATUS_ERROR || !captureResult.result) { > > - /* \todo Improve error handling. In case we notify an error > > - * because the metadata generation fails, a shutter event has > > - * already been notified for this frame number before the error > > - * is here signalled. Make sure the error path plays well with > > - * the camera stack state machine. > > - */ > > - notifyError(descriptor.frameNumber_, > > - descriptor.buffers_[0].stream); > > + if (ret) { > > + buffer.status = CAMERA3_BUFFER_STATUS_ERROR; > > + notifyError(descriptor.frameNumber_, buffer.stream, > > + CAMERA3_MSG_ERROR_BUFFER); > > + } > > } > > > > callbacks_->process_capture_result(callbacks_, &captureResult); > > @@ -2136,23 +2156,23 @@ void CameraDevice::notifyShutter(uint32_t frameNumber, uint64_t timestamp) > > callbacks_->notify(callbacks_, ¬ify); > > } > > > > -void CameraDevice::notifyError(uint32_t frameNumber, camera3_stream_t *stream) > > +void CameraDevice::notifyError(uint32_t frameNumber, camera3_stream_t *stream, > > + camera3_error_msg_code code) > > { > > camera3_notify_msg_t notify = {}; > > > > - /* > > - * \todo Report and identify the stream number or configuration to > > - * clarify the stream that failed. > > - */ > > - LOG(HAL, Error) << "Error occurred on frame " << frameNumber << " (" > > - << toPixelFormat(stream->format).toString() << ")"; > > - > > notify.type = CAMERA3_MSG_ERROR; > > notify.message.error.error_stream = stream; > > notify.message.error.frame_number = frameNumber; > > - notify.message.error.error_code = CAMERA3_MSG_ERROR_REQUEST; > > + notify.message.error.error_code = code; > > > > callbacks_->notify(callbacks_, ¬ify); > > + > > + if (stream) > > + LOG(HAL, Error) << "Error occurred on frame " << frameNumber << " (" > > + << toPixelFormat(stream->format).toString() << ")"; > > + else > > + LOG(HAL, Error) << "Fatal error occurred on device"; > > } > > > > /* > > diff --git a/src/android/camera_device.h b/src/android/camera_device.h > > index 23457e47767a..8d5da8bc59e1 100644 > > --- a/src/android/camera_device.h > > +++ b/src/android/camera_device.h > > @@ -101,7 +101,8 @@ private: > > std::tuple<uint32_t, uint32_t> calculateStaticMetadataSize(); > > libcamera::FrameBuffer *createFrameBuffer(const buffer_handle_t camera3buffer); > > void notifyShutter(uint32_t frameNumber, uint64_t timestamp); > > - void notifyError(uint32_t frameNumber, camera3_stream_t *stream); > > + void notifyError(uint32_t frameNumber, camera3_stream_t *stream, > > + camera3_error_msg_code code); > > std::unique_ptr<CameraMetadata> requestTemplatePreview(); > > std::unique_ptr<CameraMetadata> requestTemplateVideo(); > > libcamera::PixelFormat toPixelFormat(int format) const; > > -- > > 2.31.1 > > > > _______________________________________________ > > libcamera-devel mailing list > > libcamera-devel@lists.libcamera.org > > https://lists.libcamera.org/listinfo/libcamera-devel > > -- > Regards, > Niklas Söderlund
Hi Jacopo, On 2021-05-06 10:53:02 +0200, Jacopo Mondi wrote: <snip> > > > > > > - resultMetadata = getResultMetadata(descriptor); > > > + /* > > > + * Generate the metadata associated with the captured buffers. > > > + * > > > + * Notify if the metadata generation has failed, but continue processing > > > + * buffers and return an empty metadata pack. > > > + */ > > > + std::unique_ptr<CameraMetadata> resultMetadata = getResultMetadata(descriptor); > > > + if (!resultMetadata) > > > + notifyError(descriptor.frameNumber_, descriptor.buffers_[0].stream, > > > + CAMERA3_MSG_ERROR_RESULT); > > > + captureResult.result = resultMetadata->get(); > > > > Hum if !resultMetadata will not resultMetadata->get() give us a nullptr > > exception here? > > > > According to the std::unique_ptr<>::get() documentation: > > "Returns a pointer to the managed object or nullptr if no object is owned." > > as getResultMetadata() returns nullptr on error, and the function > signature is > std::unique_ptr<CameraMetadata> CameraDevice::getResultMetadata() > > my understanding is that the returned unique_ptr<> will own no object, > hence get() should return nullptr. Or have I missed something about > the "no object is owned" vs "a unique_ptr<> that owns nullptr" ? I share your understanding of std::unique_ptr<>::get() but that would be resultMetadata.get() and not resultMetadata->get() The later calls CameraMetadata::get() which returns camera_metadata_t * which is the correct datatype for the captureResult.result right? So I think the code calls the correct get(), only the manual protection for the nullptr is missing.
Hi Niklas, On Thu, May 06, 2021 at 11:03:09AM +0200, Niklas Söderlund wrote: > Hi Jacopo, > > On 2021-05-06 10:53:02 +0200, Jacopo Mondi wrote: > > <snip> > > > > > > > > > - resultMetadata = getResultMetadata(descriptor); > > > > + /* > > > > + * Generate the metadata associated with the captured buffers. > > > > + * > > > > + * Notify if the metadata generation has failed, but continue processing > > > > + * buffers and return an empty metadata pack. > > > > + */ > > > > + std::unique_ptr<CameraMetadata> resultMetadata = getResultMetadata(descriptor); > > > > + if (!resultMetadata) > > > > + notifyError(descriptor.frameNumber_, descriptor.buffers_[0].stream, > > > > + CAMERA3_MSG_ERROR_RESULT); > > > > + captureResult.result = resultMetadata->get(); > > > > > > Hum if !resultMetadata will not resultMetadata->get() give us a nullptr > > > exception here? > > > > > > > According to the std::unique_ptr<>::get() documentation: > > > > "Returns a pointer to the managed object or nullptr if no object is owned." > > > > as getResultMetadata() returns nullptr on error, and the function > > signature is > > std::unique_ptr<CameraMetadata> CameraDevice::getResultMetadata() > > > > my understanding is that the returned unique_ptr<> will own no object, > > hence get() should return nullptr. Or have I missed something about > > the "no object is owned" vs "a unique_ptr<> that owns nullptr" ? > > I share your understanding of std::unique_ptr<>::get() but that would be > > resultMetadata.get() > > and not > > resultMetadata->get() > > The later calls CameraMetadata::get() which returns camera_metadata_t * > which is the correct datatype for the captureResult.result right? So I > think the code calls the correct get(), only the manual protection for > the nullptr is missing. AAAA! very good catch! thanks, I'm a bit slow Yes, unique_ptr<>::operator-> of course says The behavior is undefined if get() == nullptr. so yes, I should put a check there. Sorry, I overlooked this! thanks for spotting, I'll fix Thanks j > > -- > Regards, > Niklas Söderlund
diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index f1f26c775611..5588e99b6eba 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -2025,9 +2025,6 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques void CameraDevice::requestComplete(Request *request) { - camera3_buffer_status status = CAMERA3_BUFFER_STATUS_OK; - std::unique_ptr<CameraMetadata> resultMetadata; - decltype(descriptors_)::node_type node; { std::scoped_lock<std::mutex> lock(mutex_); @@ -2035,7 +2032,8 @@ void CameraDevice::requestComplete(Request *request) if (it == descriptors_.end()) { LOG(HAL, Fatal) << "Unknown request: " << request->cookie(); - status = CAMERA3_BUFFER_STATUS_ERROR; + close(); + notifyError(0, nullptr, CAMERA3_MSG_ERROR_DEVICE); return; } @@ -2043,16 +2041,67 @@ void CameraDevice::requestComplete(Request *request) } Camera3RequestDescriptor &descriptor = node.mapped(); + /* + * Prepare the capture result to call back the Android camera stack with. + * + * The buffer status is set to OK and later changed to ERROR if + * post-processing/compression fails. + */ + camera3_capture_result_t captureResult = {}; + captureResult.frame_number = descriptor.frameNumber_; + captureResult.num_output_buffers = descriptor.buffers_.size(); + for (camera3_stream_buffer_t &buffer : descriptor.buffers_) { + buffer.acquire_fence = -1; + buffer.release_fence = -1; + buffer.status = CAMERA3_BUFFER_STATUS_OK; + } + captureResult.output_buffers = descriptor.buffers_.data(); + captureResult.partial_result = 1; + + /* + * If the Request has failed, abort the request by notifying the error + * and complete the request with all buffers in error state. + */ if (request->status() != Request::RequestComplete) { - LOG(HAL, Error) << "Request not successfully completed: " + LOG(HAL, Error) << "Request " << request->cookie() + << " not successfully completed: " << request->status(); - status = CAMERA3_BUFFER_STATUS_ERROR; + + notifyError(descriptor.frameNumber_, + descriptor.buffers_[0].stream, + CAMERA3_MSG_ERROR_REQUEST); + + for (camera3_stream_buffer_t &buffer : descriptor.buffers_) + buffer.status = CAMERA3_BUFFER_STATUS_ERROR; + callbacks_->process_capture_result(callbacks_, &captureResult); + + return; } + /* + * Notify shutter as soon as we have verified we have a valid request. + * + * \todo The shutter event notification should be sent to the framework + * as soon as possible, earlier than request completion time. + */ + uint64_t sensorTimestamp = static_cast<uint64_t>(request->metadata() + .get(controls::SensorTimestamp)); + notifyShutter(descriptor.frameNumber_, sensorTimestamp); + LOG(HAL, Debug) << "Request " << request->cookie() << " completed with " << descriptor.buffers_.size() << " streams"; - resultMetadata = getResultMetadata(descriptor); + /* + * Generate the metadata associated with the captured buffers. + * + * Notify if the metadata generation has failed, but continue processing + * buffers and return an empty metadata pack. + */ + std::unique_ptr<CameraMetadata> resultMetadata = getResultMetadata(descriptor); + if (!resultMetadata) + notifyError(descriptor.frameNumber_, descriptor.buffers_[0].stream, + CAMERA3_MSG_ERROR_RESULT); + captureResult.result = resultMetadata->get(); /* Handle any JPEG compression. */ for (camera3_stream_buffer_t &buffer : descriptor.buffers_) { @@ -2065,56 +2114,27 @@ void CameraDevice::requestComplete(Request *request) FrameBuffer *src = request->findBuffer(cameraStream->stream()); if (!src) { LOG(HAL, Error) << "Failed to find a source stream buffer"; + buffer.status = CAMERA3_BUFFER_STATUS_ERROR; + notifyError(descriptor.frameNumber_, buffer.stream, + CAMERA3_MSG_ERROR_BUFFER); continue; } - int ret = cameraStream->process(*src, - *buffer.buffer, + int ret = cameraStream->process(*src, *buffer.buffer, descriptor.settings_, resultMetadata.get()); - if (ret) { - status = CAMERA3_BUFFER_STATUS_ERROR; - continue; - } - /* * Return the FrameBuffer to the CameraStream now that we're * done processing it. */ if (cameraStream->type() == CameraStream::Type::Internal) cameraStream->putBuffer(src); - } - - /* Prepare to call back the Android camera stack. */ - camera3_capture_result_t captureResult = {}; - captureResult.frame_number = descriptor.frameNumber_; - captureResult.num_output_buffers = descriptor.buffers_.size(); - for (camera3_stream_buffer_t &buffer : descriptor.buffers_) { - buffer.acquire_fence = -1; - buffer.release_fence = -1; - buffer.status = status; - } - captureResult.output_buffers = descriptor.buffers_.data(); - - if (status == CAMERA3_BUFFER_STATUS_OK) { - uint64_t timestamp = - static_cast<uint64_t>(request->metadata() - .get(controls::SensorTimestamp)); - notifyShutter(descriptor.frameNumber_, timestamp); - - captureResult.partial_result = 1; - captureResult.result = resultMetadata->get(); - } - if (status == CAMERA3_BUFFER_STATUS_ERROR || !captureResult.result) { - /* \todo Improve error handling. In case we notify an error - * because the metadata generation fails, a shutter event has - * already been notified for this frame number before the error - * is here signalled. Make sure the error path plays well with - * the camera stack state machine. - */ - notifyError(descriptor.frameNumber_, - descriptor.buffers_[0].stream); + if (ret) { + buffer.status = CAMERA3_BUFFER_STATUS_ERROR; + notifyError(descriptor.frameNumber_, buffer.stream, + CAMERA3_MSG_ERROR_BUFFER); + } } callbacks_->process_capture_result(callbacks_, &captureResult); @@ -2136,23 +2156,23 @@ void CameraDevice::notifyShutter(uint32_t frameNumber, uint64_t timestamp) callbacks_->notify(callbacks_, ¬ify); } -void CameraDevice::notifyError(uint32_t frameNumber, camera3_stream_t *stream) +void CameraDevice::notifyError(uint32_t frameNumber, camera3_stream_t *stream, + camera3_error_msg_code code) { camera3_notify_msg_t notify = {}; - /* - * \todo Report and identify the stream number or configuration to - * clarify the stream that failed. - */ - LOG(HAL, Error) << "Error occurred on frame " << frameNumber << " (" - << toPixelFormat(stream->format).toString() << ")"; - notify.type = CAMERA3_MSG_ERROR; notify.message.error.error_stream = stream; notify.message.error.frame_number = frameNumber; - notify.message.error.error_code = CAMERA3_MSG_ERROR_REQUEST; + notify.message.error.error_code = code; callbacks_->notify(callbacks_, ¬ify); + + if (stream) + LOG(HAL, Error) << "Error occurred on frame " << frameNumber << " (" + << toPixelFormat(stream->format).toString() << ")"; + else + LOG(HAL, Error) << "Fatal error occurred on device"; } /* diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 23457e47767a..8d5da8bc59e1 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -101,7 +101,8 @@ private: std::tuple<uint32_t, uint32_t> calculateStaticMetadataSize(); libcamera::FrameBuffer *createFrameBuffer(const buffer_handle_t camera3buffer); void notifyShutter(uint32_t frameNumber, uint64_t timestamp); - void notifyError(uint32_t frameNumber, camera3_stream_t *stream); + void notifyError(uint32_t frameNumber, camera3_stream_t *stream, + camera3_error_msg_code code); std::unique_ptr<CameraMetadata> requestTemplatePreview(); std::unique_ptr<CameraMetadata> requestTemplateVideo(); libcamera::PixelFormat toPixelFormat(int format) const;
The current implementation of CameraDevice::requestComplete() which handles event notification and calls the framework capture result callback does not handle error notification precisely enough. In detail: - Error notification is an asynchronous callback that has to be notified to the framework as soon as an error condition is detected, and it independent from the process_capture_result() callback - Error notification requires the HAL to report the precise error cause, by specifying the correct CAMERA3_MSG_ERROR_* error code. The current implementation only notifies errors of type CAMERA3_MSG_ERROR_REQUEST at the end of the procedure, before the callback invocation. Rework the procedure to: - Notify CAMERA3_MSG_ERROR_DEVICE and close the camera in case a Fatal error is detected (Fatal does not perform library teardown in production builds) - Notify CAMERA3_MSG_ERROR_REQUEST if the libcamera::Request::status is different than RequestCompleted and immediately call process_capture_result() with all buffers in error state. - Notify the shutter event as soon as possible - Notify CAMERA3_MSG_ERROR_RESULT in case the metadata cannot be generated correctly and call process_capture_result() with the right buffer state regardless of metadata availability. - Notify CAMERA3_MSG_ERROR_BUFFER for buffers whose post-processing failed While at it, return the CameraStream buffer by calling cameraStream->putBuffer() regardless of the post-processing result. No regression detected when running CTS in LIMITED mode. Signed-off-by: Jacopo Mondi <jacopo@jmondi.org> --- src/android/camera_device.cpp | 128 ++++++++++++++++++++-------------- src/android/camera_device.h | 3 +- 2 files changed, 76 insertions(+), 55 deletions(-)