From patchwork Fri Aug 12 09:08:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hanlin Chen X-Patchwork-Id: 17092 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D9362C3272 for ; Fri, 12 Aug 2022 09:09:04 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A001463332; Fri, 12 Aug 2022 11:09:04 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660295344; bh=XUVXIH095CW+orozHNb+edNGl6eV+3R8nVMFXij+Gb8=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=mGZh/sPPLbEQ7pxUxL8JvF9siIFMmND9E4/7+2yH2MkALp/10TxkGjA6rXIJGOsDt Q6Tq6g2TOLwPMxdSlW/DYGkgiyZZrJEZtp+2m+Gwz92g+6iRRjGxqwTy+ubx9jQaV/ OgEtws1V/j/zsGGoN97m4+aKomUKCmkOB0/pK3rVJAzu505ZAQWT0/U20E1aJr9LAq w1tz/f1UZrBMyNByUXdAhm/JaRR4EAWrY8xLy3u4BycMIqK1J5KuKBzd5SblM2iVxv ODLz/UQHwuWqQpxsGTQRM6LMNxkZIFA8yGnmc1xnr47c8jt1W3AZFs3dPgdl4DeKMl cJ0dIXhkVq5sg== Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 403CA63326 for ; Fri, 12 Aug 2022 11:09:02 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="UWaX9zKV"; dkim-atps=neutral Received: by mail-pf1-x42c.google.com with SMTP id g12so434048pfb.3 for ; Fri, 12 Aug 2022 02:09:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=c+5pmxEp18nuPmAb1DA+iCZFZ8j9iB6RgdwRFw1tOxU=; b=UWaX9zKVm/F9ghKdmZmjn80z8CNPER5ptR7VQLeNSgnMiKm2Sllieb2OgxAsB0Z5fk hqtt5qPiaNR0ibLxyJt14KANmhKUJn/vJFtJQ96g8k6/VO+HekNETu14LrNY9Gqf1Ja2 mHI72D3KtKE1Qs5/aUh3qZ1GqdBmRJwlTIc/c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=c+5pmxEp18nuPmAb1DA+iCZFZ8j9iB6RgdwRFw1tOxU=; b=gvydsM0ET5gI7zaaypLLEGmnyzM/j8zCz6S4ke5VhOAjqHdKjui9bcVoQZ77JFUfyz ZXYnoszM6cwt+YHH2fZMhSAfFBGAyMmrOZyBOGOiq64DFrFRXSZj9wKhZTcz4/pPJOTF EaQbp/9zycHKlaDEZuhWY3hhS7+yEHgvPX4QgVgiXmGhxtejRvBcndbKEOpXqLSiMId5 N1mqhvPns323SBl7QFezF8e7zMNkvU1fMBo4Qwjx+argtXIGLXsUijGudA5l+pKLByZ5 mw1Tlepbm4XFyqWGMUDa8f/VdwYlE7FkMqmBVQWfuk7es2kkDke7ASidu70mgz/pyCl6 Cyrg== X-Gm-Message-State: ACgBeo3vDKhHWGIPwrU/kBDCUi0X+KULyHMQ7By0CZATlEqKGICARaD6 tWGsr/fHL6PDOGSRAQ5LfAuODpxfzJR7nQ== X-Google-Smtp-Source: AA6agR6EC+CiGbhtOkqTui6TkgUp4/uLUk2GSE9bFI21lzRBGmC4IFNsKHKc89sisfH4KQS8t0F17g== X-Received: by 2002:a05:6a00:23d1:b0:52f:39e9:9150 with SMTP id g17-20020a056a0023d100b0052f39e99150mr3130866pfc.16.1660295339471; Fri, 12 Aug 2022 02:08:59 -0700 (PDT) Received: from localhost ([2401:fa00:1:17:1705:d284:d114:2e24]) by smtp.gmail.com with UTF8SMTPSA id j15-20020a170902da8f00b0016f04c098ddsm1166674plx.226.2022.08.12.02.08.58 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 12 Aug 2022 02:08:59 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Fri, 12 Aug 2022 17:08:37 +0800 Message-Id: <20220812090838.1784703-7-hanlinchen@chromium.org> X-Mailer: git-send-email 2.37.1.595.g718a3a8f04-goog In-Reply-To: <20220812090838.1784703-1-hanlinchen@chromium.org> References: <20220812090838.1784703-1-hanlinchen@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 6/7] android: Implement partial result feature based on partialResultCompleted signal X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Han-Lin Chen via libcamera-devel From: Hanlin Chen Reply-To: Han-Lin Chen Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" To impelement partial result feature. The patch includes the following: 1. Return partial results on Camera::partialResultCompleted signal, and recycle requests on Camera::requestCompleted. 2. Reorder the result according to the FIFO restriction of the same stream Signed-off-by: Han-Lin Chen --- src/android/camera_capabilities.cpp | 2 +- src/android/camera_capabilities.h | 2 + src/android/camera_device.cpp | 848 +++++++++++++++-------- src/android/camera_device.h | 36 +- src/android/camera_request.cpp | 61 +- src/android/camera_request.h | 56 +- src/android/camera_stream.cpp | 1 + src/android/jpeg/post_processor_jpeg.cpp | 2 +- 8 files changed, 693 insertions(+), 315 deletions(-) diff --git a/src/android/camera_capabilities.cpp b/src/android/camera_capabilities.cpp index 64bd8dde..6ed040d0 100644 --- a/src/android/camera_capabilities.cpp +++ b/src/android/camera_capabilities.cpp @@ -1374,7 +1374,7 @@ int CameraCapabilities::initializeStaticMetadata() staticMetadata_->addEntry(ANDROID_SCALER_CROPPING_TYPE, croppingType); /* Request static metadata. */ - int32_t partialResultCount = 1; + int32_t partialResultCount = MaxMetadataPackIndex; staticMetadata_->addEntry(ANDROID_REQUEST_PARTIAL_RESULT_COUNT, partialResultCount); diff --git a/src/android/camera_capabilities.h b/src/android/camera_capabilities.h index 6f66f221..b8fc20f1 100644 --- a/src/android/camera_capabilities.h +++ b/src/android/camera_capabilities.h @@ -23,6 +23,8 @@ class CameraCapabilities { public: + static constexpr int32_t MaxMetadataPackIndex = 64; + CameraCapabilities() = default; int initialize(std::shared_ptr camera, diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index a14d5de9..c999b198 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -24,9 +24,8 @@ #include #include -#include "system/graphics.h" - #include "camera_buffer.h" +#include "camera_capabilities.h" #include "camera_hal_config.h" #include "camera_ops.h" #include "camera_request.h" @@ -249,7 +248,11 @@ CameraDevice::CameraDevice(unsigned int id, std::shared_ptr camera) : id_(id), state_(State::Stopped), camera_(std::move(camera)), facing_(CAMERA_FACING_FRONT), orientation_(0) { + /* Set RequestCompletionMode to Immediately to send result early */ + camera_->setRequestCompletionMode(Camera::Immediately); + camera_->requestCompleted.connect(this, &CameraDevice::requestComplete); + camera_->partialResultCompleted.connect(this, &CameraDevice::partialResultComplete); maker_ = "libcamera"; model_ = "cameraModel"; @@ -438,8 +441,10 @@ void CameraDevice::stop() camera_->stop(); { - MutexLocker descriptorsLock(descriptorsMutex_); - descriptors_ = {}; + MutexLocker descriptorsLock(pendingRequestMutex_); + pendingRequests_.clear(); + pendingResults_.clear(); + pendingStreamBuffers_.clear(); } streams_.clear(); @@ -855,14 +860,37 @@ int CameraDevice::processControls(Camera3RequestDescriptor *descriptor) return 0; } -void CameraDevice::abortRequest(Camera3RequestDescriptor *descriptor) const +void CameraDevice::abortRequest(Camera3RequestDescriptor *descriptor) { - notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_REQUEST); + /* + * Since the failed buffers do not have to follow the strict ordering + * valid buffers do, and could be out-of-order with respect to valid + * buffers, it's safe to send the aborted result back to the framework + * direcly. + */ + Camera3ResultDescriptor *result = new Camera3ResultDescriptor(descriptor); + result->metadataPackIndex_ = 0; - for (auto &buffer : descriptor->buffers_) - buffer.status = StreamBuffer::Status::Error; + for (auto &buffer : descriptor->buffers_) { + setBufferStatus(buffer, StreamBuffer::Status::Error); + result->buffers_.emplace_back(&buffer); + } + + { + MutexLocker lock(descriptor->resultsMutex_); + descriptor->status_ = Camera3RequestDescriptor::Status::Aborted; + descriptor->finalResult_.reset(result); + } - descriptor->status_ = Camera3RequestDescriptor::Status::Error; + /* + * After CAMERA3_MSG_ERROR_REQUEST is notified, for a given frame, + * only process_capture_results with buffers of the status + * CAMERA3_BUFFER_STATUS_ERROR are allowed. No further notifys or + * process_capture_result with non-null metadata is allowed. + */ + notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_REQUEST); + + sendCaptureResult(result); } bool CameraDevice::isValidRequest(camera3_capture_request_t *camera3Request) const @@ -962,9 +990,10 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques * to a libcamera stream. Streams of type Mapped will be handled later. * * Collect the CameraStream associated to each requested capture stream. - * Since requestedStreams is an std:set<>, no duplications can happen. + * Since requestedDirectBuffers is an std:map<>, no duplications can + * happen. */ - std::set requestedStreams; + std::map requestedDirectBuffers; for (const auto &[i, buffer] : utils::enumerate(descriptor->buffers_)) { CameraStream *cameraStream = buffer.stream; camera3_stream_t *camera3Stream = cameraStream->camera3Stream(); @@ -983,8 +1012,6 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques FrameBuffer *frameBuffer = nullptr; UniqueFD acquireFence; - MutexLocker lock(descriptor->streamsProcessMutex_); - switch (cameraStream->type()) { case CameraStream::Type::Mapped: /* Mapped streams will be handled in the next loop. */ @@ -1003,23 +1030,23 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques cameraStream->configuration().size); frameBuffer = buffer.frameBuffer.get(); acquireFence = std::move(buffer.fence); + + requestedDirectBuffers[cameraStream] = frameBuffer; LOG(HAL, Debug) << ss.str() << " (direct)"; break; case CameraStream::Type::Internal: /* - * Get the frame buffer from the CameraStream internal - * buffer pool. - * - * The buffer has to be returned to the CameraStream - * once it has been processed. + * Get the frame buffer from the source stream's + * internal buffer pool. The buffer has to be returned + * to the source stream once it has been processed. */ frameBuffer = cameraStream->getBuffer(); - buffer.internalBuffer = frameBuffer; - LOG(HAL, Debug) << ss.str() << " (internal)"; + buffer.srcBuffer = frameBuffer; - descriptor->pendingStreamsToProcess_.insert( - { cameraStream, &buffer }); + /* Track the allocated internal buffer */ + descriptor->internalBuffers_[cameraStream] = frameBuffer; + LOG(HAL, Debug) << ss.str() << " (internal)"; break; } @@ -1031,8 +1058,6 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques auto fence = std::make_unique(std::move(acquireFence)); descriptor->request_->addBuffer(cameraStream->stream(), frameBuffer, std::move(fence)); - - requestedStreams.insert(cameraStream); } /* @@ -1054,29 +1079,54 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques << cameraStream->configuration().pixelFormat << "]" << " (mapped)"; - MutexLocker lock(descriptor->streamsProcessMutex_); - descriptor->pendingStreamsToProcess_.insert({ cameraStream, &buffer }); - /* * Make sure the CameraStream this stream is mapped on has been * added to the request. */ CameraStream *sourceStream = cameraStream->sourceStream(); + ASSERT(sourceStream); - if (requestedStreams.find(sourceStream) != requestedStreams.end()) + ASSERT(sourceStream->type() == CameraStream::Type::Direct); + + /* + * If the buffer for the source stream has been requested as + * Direct, use its framebuffer as the source buffer for + * post-processing. No need to recycle the buffer since it's + * owned by Android. + */ + auto iterDirectBuffer = requestedDirectBuffers.find(sourceStream); + if (iterDirectBuffer != requestedDirectBuffers.end()) { + buffer.srcBuffer = iterDirectBuffer->second; continue; + } /* - * If that's not the case, we need to add a buffer to the request - * for this stream. + * If that's not the case, we use an internal buffer allocated + * from the source stream. + * + * If an internal buffer has been requested for the source + * stream before, we should reuse it. */ - FrameBuffer *frameBuffer = cameraStream->getBuffer(); - buffer.internalBuffer = frameBuffer; + auto iterInternalBuffer = descriptor->internalBuffers_.find(sourceStream); + if (iterInternalBuffer != descriptor->internalBuffers_.end()) { + buffer.srcBuffer = iterInternalBuffer->second; + continue; + } + + /* + * Otherwise, we need to create an internal buffer to the + * request for the source stream. Get the frame buffer from the + * source stream's internal buffer pool. The buffer has to be + * returned to the source stream once it has been processed. + */ + FrameBuffer *frameBuffer = sourceStream->getBuffer(); + buffer.srcBuffer = frameBuffer; descriptor->request_->addBuffer(sourceStream->stream(), frameBuffer, nullptr); - requestedStreams.erase(sourceStream); + /* Track the allocated internal buffer. */ + descriptor->internalBuffers_[sourceStream] = frameBuffer; } /* @@ -1095,14 +1145,7 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques MutexLocker stateLock(stateMutex_); if (state_ == State::Flushing) { - Camera3RequestDescriptor *rawDescriptor = descriptor.get(); - { - MutexLocker descriptorsLock(descriptorsMutex_); - descriptors_.push(std::move(descriptor)); - } - abortRequest(rawDescriptor); - completeDescriptor(rawDescriptor); - + abortRequest(descriptor.get()); return 0; } @@ -1119,8 +1162,10 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques Request *request = descriptor->request_.get(); { - MutexLocker descriptorsLock(descriptorsMutex_); - descriptors_.push(std::move(descriptor)); + MutexLocker descriptorsLock(pendingRequestMutex_); + for (auto &buffer : descriptor->buffers_) + pendingStreamBuffers_[buffer.stream].push_back(&buffer); + pendingRequests_.push_back(std::move(descriptor)); } camera_->queueRequest(request); @@ -1128,223 +1173,444 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques return 0; } -void CameraDevice::requestComplete(Request *request) +void CameraDevice::partialResultComplete(Request *request, Result *result) { Camera3RequestDescriptor *descriptor = reinterpret_cast(request->cookie()); - /* - * Prepare the capture result for the Android camera stack. - * - * The buffer status is set to Success and later changed to Error if - * post-processing/compression fails. - */ + if (result->buffers().empty() && result->metadata().empty()) + LOG(HAL, Fatal) + << "Empty partial result is not allowed. Frame number: " + << descriptor->frameNumber_; + + Camera3ResultDescriptor *camera3Result = new Camera3ResultDescriptor(descriptor); + + const ControlList &metadata = result->metadata(); + if (!metadata.empty()) { + /* + * Notify shutter as soon as we have received SensorTimestamp. + */ + const auto ×tamp = metadata.get(controls::SensorTimestamp); + if (timestamp) { + notifyShutter(descriptor->frameNumber_, *timestamp); + LOG(HAL, Debug) << "Request " << request->cookie() << " notifies shutter"; + } + + camera3Result->resultMetadata_ = getDynamicResultMetadata(metadata); + } + + MutexLocker locker(camera3Result->streamsProcessMutex_); + for (auto &buffer : descriptor->buffers_) { - CameraStream *stream = buffer.stream; + CameraStream *cameraStream = buffer.stream; + for (auto *frameBuffer : result->buffers()) { + if (buffer.srcBuffer != frameBuffer && + buffer.frameBuffer.get() != frameBuffer) + continue; + + buffer.result = camera3Result; + camera3Result->buffers_.emplace_back(&buffer); + StreamBuffer::Status status = StreamBuffer::Status::Success; + if (frameBuffer->metadata().status != FrameMetadata::FrameSuccess) { + status = StreamBuffer::Status::Error; + } + setBufferStatus(buffer, status); + + switch (cameraStream->type()) { + case CameraStream::Type::Direct: { + ASSERT(buffer.frameBuffer.get() == frameBuffer); + /* + * Streams of type Direct have been queued to the + * libcamera::Camera and their acquire fences have + * already been waited on by the library. + */ + std::unique_ptr fence = buffer.frameBuffer->releaseFence(); + if (fence) + buffer.fence = fence->release(); + break; + } + case CameraStream::Type::Mapped: + case CameraStream::Type::Internal: + ASSERT(buffer.srcBuffer == frameBuffer); + if (status == StreamBuffer::Status::Error) + break; + + /* + * Acquire fences of streams of type Internal and Mapped + * will be handled during post-processing. + */ + buffer.srcBuffer = frameBuffer; + camera3Result->pendingBuffersToProcess_.emplace_back(&buffer); + + if (cameraStream->isJpegStream()) { + generateJpegExifMetadata(descriptor, &buffer); + + /* + * Allocate for post-processor to fill + * in JPEG related metadata. + */ + if (!camera3Result->resultMetadata_) + camera3Result->resultMetadata_ = getDynamicResultMetadata(metadata); + } + break; + } + } + } + + { /* - * Streams of type Direct have been queued to the - * libcamera::Camera and their acquire fences have - * already been waited on by the library. - * - * Acquire fences of streams of type Internal and Mapped - * will be handled during post-processing. + * Adding result to the request before sending it to the + * post-processing threads, so the streamProcessingComplete() + * slot can safely call compeleteResult(). */ - if (stream->type() == CameraStream::Type::Direct) { - /* If handling of the fence has failed restore buffer.fence. */ - std::unique_ptr fence = buffer.frameBuffer->releaseFence(); - if (fence) - buffer.fence = fence->release(); + MutexLocker lock(descriptor->resultsMutex_); + descriptor->partialResults_.emplace_back(camera3Result); + } + + /* + * Queue all the post-processing streams request at once. The completion + * slot streamProcessingComplete() can only execute when we are out + * this critical section (result->streamsProcessMutex_). This helps to + * handle synchronous errors here itself. + */ + for (auto *buffer : camera3Result->pendingBuffersToProcess_) { + int ret = buffer->stream->process(buffer); + if (ret) { + setBufferStatus(*buffer, StreamBuffer::Status::Error); + LOG(HAL, Error) << "Failed to run post process of request " + << descriptor->frameNumber_; } - buffer.status = StreamBuffer::Status::Success; } + if (!camera3Result->pendingBuffersToProcess_.empty()) + return; + + locker.unlock(); + tryCompleteResultDescriptor(camera3Result); +} + +void CameraDevice::requestComplete(Request *request) +{ + Camera3RequestDescriptor *camera3Request = + reinterpret_cast(request->cookie()); + /* - * If the Request has failed, abort the request by notifying the error - * and complete the request with all buffers in error state. + * On Android, each new partial result with a metadata must set a + * field (partial_result) to a distinct inclusive value between + * 1 and ANDROID_REQUEST_PARTIAL_RESULT_COUNT and the final result with + * metadata has to set the field as ANDROID_REQUEST_PARTIAL_RESULT_COUNT. + * + * An empty metadata with ANDROID_REQUEST_PARTIAL_RESULT_COUNT is not + * allowed. Add a result with a fixed metadata on requestComplete() + * in case all of previous metadata are sent early, and no more metadata + * can be sent as the final with ANDROID_REQUEST_PARTIAL_RESULT_COUNT. */ - if (request->status() != Request::RequestComplete) { - LOG(HAL, Error) << "Request " << request->cookie() - << " not successfully completed: " - << request->status(); + Camera3ResultDescriptor *result = new Camera3ResultDescriptor(camera3Request); + result->resultMetadata_ = getFixedResultMetadata(camera3Request->settings_); + result->metadataPackIndex_ = CameraCapabilities::MaxMetadataPackIndex; - abortRequest(descriptor); - completeDescriptor(descriptor); + { + MutexLocker lock(camera3Request->resultsMutex_); + camera3Request->finalResult_.reset(result); - return; + switch (request->status()) { + case Request::RequestComplete: + camera3Request->status_ = Camera3RequestDescriptor::Status::Success; + break; + case Request::RequestCancelled: + camera3Request->status_ = Camera3RequestDescriptor::Status::Cancelled; + break; + case Request::RequestPending: + LOG(HAL, Fatal) << "Try to complete an unfinished request"; + break; + } } + completeResultDescriptor(result); + return; +} + +void CameraDevice::tryCompleteResultDescriptor(Camera3ResultDescriptor *result) +{ /* - * Notify shutter as soon as we have verified we have a valid request. + * Android requires buffers for a given stream must be returned in FIFO + * order. However, different streams are independent of each other, so + * it is acceptable and expected that the buffer for request 5 for + * stream A may be returned after the buffer for request 6 for stream + * B is. And it is acceptable that the result metadata for request 6 + * for stream B is returned before the buffer for request 5 for stream + * A is. In result, if all of buffers of a result are the most front + * buffers of each stream, or the result contains no buffers, the result + * is allowed to send. Collect ready results in the order which follows + * the above rule. * - * \todo The shutter event notification should be sent to the framework - * as soon as possible, earlier than request completion time. + * \todo The reprocessing result can be returned ahead of the pending + * normal output results. But the FIFO ordering must be maintained for + * all reprocessing results. Track the reprocessing buffer's order + * independently when we have reprocessing API. */ - uint64_t sensorTimestamp = static_cast(request->metadata() - .get(controls::SensorTimestamp) - .value_or(0)); - notifyShutter(descriptor->frameNumber_, sensorTimestamp); + MutexLocker lock(pendingRequestMutex_); - LOG(HAL, Debug) << "Request " << request->cookie() << " completed with " - << descriptor->request_->buffers().size() << " streams"; + pendingResults_.push_front(result); + std::list readyResults; /* - * Generate the metadata associated with the captured buffers. - * - * Notify if the metadata generation has failed, but continue processing - * buffers and return an empty metadata pack. + * Error buffers do not have to follow the strict ordering valid buffers + * do, and ready to sent directly, removes them from the pendingBuffers + * so it won't blocking following valid buffers. */ - descriptor->resultMetadata_ = getResultMetadata(*descriptor); - if (!descriptor->resultMetadata_) { - notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_RESULT); + for (auto &buffer : result->buffers_) + if (buffer->status == StreamBuffer::Status::Error) + pendingStreamBuffers_[buffer->stream].remove(buffer); - /* - * The camera framework expects an empty metadata pack on error. - * - * \todo Check that the post-processor code handles this situation - * correctly. - */ - descriptor->resultMetadata_ = std::make_unique(0, 0); + /* + * Exhaustly collect results which is ready to sent. + */ + bool keepChecking; + do { + keepChecking = false; + auto iter = pendingResults_.begin(); + while (iter != pendingResults_.end()) { + /* + * A result is considered as ready when all of the valid + * buffers of the result are at the front of the pending + * buffers associated with its stream. + */ + bool ready = true; + for (auto &buffer : (*iter)->buffers_) { + if (buffer->status == StreamBuffer::Status::Error) + continue; + + auto &pendingBuffers = pendingStreamBuffers_[buffer->stream]; + ASSERT(!pendingBuffers.empty()); + + if (pendingBuffers.front() != buffer) { + ready = false; + break; + } + } + if (!ready) { + iter++; + continue; + } + + for (auto &buffer : (*iter)->buffers_) + if (buffer->status != StreamBuffer::Status::Error) + pendingStreamBuffers_[buffer->stream].pop_front(); + + /* Keep checking since pendingStreamBuffers has updated */ + keepChecking = true; + + readyResults.push_back(*iter); + iter = pendingResults_.erase(iter); + } + } while (keepChecking); + + lock.unlock(); + + for (auto &res : readyResults) { + completeResultDescriptor(res); } +} + +void CameraDevice::completeResultDescriptor(Camera3ResultDescriptor *result) +{ + Camera3RequestDescriptor *request = result->request_; - /* Handle post-processing. */ - MutexLocker locker(descriptor->streamsProcessMutex_); + MutexLocker lock(request->resultsMutex_); + result->complete_ = true; /* - * Queue all the post-processing streams request at once. The completion - * slot streamProcessingComplete() can only execute when we are out - * this critical section. This helps to handle synchronous errors here - * itself. + * Android requires value of metadataPackIndex follows the rules: + * + * Set to MaxMetadataPackIndex if its the final result of the request. + * Set Incrementally from 1 if its the result containing metadata. + * Set to 0 if the result contains only buffers. + * + * The metadataPackIndex should be sent incrementally. Since the + * function may be called from post-processing threads, ensure the + * strict sending order by the mutex. */ - auto iter = descriptor->pendingStreamsToProcess_.begin(); - while (iter != descriptor->pendingStreamsToProcess_.end()) { - CameraStream *stream = iter->first; - StreamBuffer *buffer = iter->second; + bool isFinalResult = (result == request->finalResult_.get()); - if (stream->isJpegStream()) { - generateJpegExifMetadata(descriptor, buffer); - } + uint32_t &metadataPackIndex = result->metadataPackIndex_ = 0; - FrameBuffer *src = request->findBuffer(stream->stream()); - if (!src) { - LOG(HAL, Error) << "Failed to find a source stream buffer"; - setBufferStatus(*buffer, StreamBuffer::Status::Error); - iter = descriptor->pendingStreamsToProcess_.erase(iter); - continue; - } + if (isFinalResult) + metadataPackIndex = CameraCapabilities::MaxMetadataPackIndex; + else if (result->resultMetadata_) + metadataPackIndex = request->nextPartialResultIndex_++; + else + metadataPackIndex = 0; - buffer->srcBuffer = src; + if (metadataPackIndex > CameraCapabilities::MaxMetadataPackIndex) + LOG(HAL, Fatal) << "Partial result count exceed limitation " + << CameraCapabilities::MaxMetadataPackIndex; - ++iter; - int ret = stream->process(buffer); - if (ret) { - setBufferStatus(*buffer, StreamBuffer::Status::Error); - descriptor->pendingStreamsToProcess_.erase(stream); + /* + * The final result will be returned in completeRequestDescriptor() + * to follow the rules that the final result should be returned in + * its submission order. + */ + if (!isFinalResult) + sendCaptureResult(result); - /* - * If the framebuffer is internal to CameraStream return - * it back now that we're done processing it. - */ - if (buffer->internalBuffer) - stream->putBuffer(buffer->internalBuffer); + if (request->status_ == Camera3RequestDescriptor::Status::Pending) + return; + + /* + * No more partial results shall be created once the status has + * changed from Pending. Check whether all of the pending + * results are sent before we complete the request. The function + * shall be re-called everytime a result is completed, and the final + * call shall complete the request. + */ + bool hasPendingResult = false; + for (auto &r : request->partialResults_) { + if (!r->complete_) { + hasPendingResult = true; } } - if (descriptor->pendingStreamsToProcess_.empty()) { - locker.unlock(); - completeDescriptor(descriptor); - } -} + if (hasPendingResult) + return; -/** - * \brief Complete the Camera3RequestDescriptor - * \param[in] descriptor The Camera3RequestDescriptor that has completed - * - * The function marks the Camera3RequestDescriptor as 'complete'. It shall be - * called when all the streams in the Camera3RequestDescriptor have completed - * capture (or have been generated via post-processing) and the request is ready - * to be sent back to the framework. - * - * \context This function is \threadsafe. - */ -void CameraDevice::completeDescriptor(Camera3RequestDescriptor *descriptor) -{ - MutexLocker lock(descriptorsMutex_); - descriptor->complete_ = true; + lock.unlock(); - sendCaptureResults(); + completeRequestDescriptor(result->request_); } + /** - * \brief Sequentially send capture results to the framework + * \brief Complete the Camera3RequestDescriptor + * \param[in] descriptor The Camera3RequestDescriptor * - * Iterate over the descriptors queue to send completed descriptors back to the - * framework, in the same order as they have been queued. For each complete - * descriptor, populate a locally-scoped camera3_capture_result_t from the - * descriptor, send the capture result back by calling the - * process_capture_result() callback, and remove the descriptor from the queue. - * Stop iterating if the descriptor at the front of the queue is not complete. + * The function shall be called when one of a partial result of the descriptor + * has completed. The function shall complete the descriptor only when all of + * the partial result has sent back to the framework, and send the final result + * according to the submission order of the requests. * - * This function should never be called directly in the codebase. Use - * completeDescriptor() instead. + * \context This function is \threadsafe. */ -void CameraDevice::sendCaptureResults() +void CameraDevice::completeRequestDescriptor(Camera3RequestDescriptor *request) { - while (!descriptors_.empty() && !descriptors_.front()->isPending()) { - auto descriptor = std::move(descriptors_.front()); - descriptors_.pop(); - - camera3_capture_result_t captureResult = {}; - - captureResult.frame_number = descriptor->frameNumber_; - - if (descriptor->resultMetadata_) - captureResult.result = - descriptor->resultMetadata_->getMetadata(); - - std::vector resultBuffers; - resultBuffers.reserve(descriptor->buffers_.size()); + MutexLocker locker(pendingRequestMutex_); + request->completed_ = true; - for (auto &buffer : descriptor->buffers_) { - camera3_buffer_status status = CAMERA3_BUFFER_STATUS_ERROR; - - if (buffer.status == StreamBuffer::Status::Success) - status = CAMERA3_BUFFER_STATUS_OK; + /* + * Android requires the final result of each request returns in + * the submission order. + */ + while (!pendingRequests_.empty()) { + auto &descriptor = pendingRequests_.front(); + if (!descriptor->completed_) + break; - /* - * Pass the buffer fence back to the camera framework as - * a release fence. This instructs the framework to wait - * on the acquire fence in case we haven't done so - * ourselves for any reason. - */ - resultBuffers.push_back({ buffer.stream->camera3Stream(), - buffer.camera3Buffer, status, - -1, buffer.fence.release() }); + Camera3RequestDescriptor::Status status; + { + MutexLocker lock(descriptor->resultsMutex_); + status = descriptor->status_; + if (descriptor->finalResult_) + sendCaptureResult(descriptor->finalResult_.get()); } - captureResult.num_output_buffers = resultBuffers.size(); - captureResult.output_buffers = resultBuffers.data(); - - if (descriptor->status_ == Camera3RequestDescriptor::Status::Success) - captureResult.partial_result = 1; + /* + * Call notify with CAMERA3_MSG_ERROR_RESULT to indicate some + * of the expected result metadata might not be available + * because the capture is cancelled by the camera. Only notify + * it when the final result is sent, since Android will ignore + * the following metadata. + */ + if (status == Camera3RequestDescriptor::Status::Cancelled) + notifyError(descriptor->frameNumber_, nullptr, CAMERA3_MSG_ERROR_RESULT); - callbacks_->process_capture_result(callbacks_, &captureResult); + pendingRequests_.pop_front(); } } void CameraDevice::setBufferStatus(StreamBuffer &streamBuffer, - StreamBuffer::Status status) + StreamBuffer::Status status) const { streamBuffer.status = status; if (status != StreamBuffer::Status::Success) { notifyError(streamBuffer.request->frameNumber_, streamBuffer.stream->camera3Stream(), CAMERA3_MSG_ERROR_BUFFER); + } +} + +void CameraDevice::sendPartialResult(Camera3ResultDescriptor *result) +{ + Camera3RequestDescriptor *request = result->request_; + + MutexLocker lock(request->resultsMutex_); + + /* + * Android requires value of metadataPackIndex follows the rules: + * + * Set to 0: Result contains no metadata (buffers only). + * Set between 1 and (MaxMetadataPackIndex - 1): Result contains metadata. + * Set to MaxMetadataPackIndex: The result of the request. + */ + uint32_t &metadataPackIndex = result->metadataPackIndex_ = 0; + if (result->resultMetadata_) { + metadataPackIndex = request->nextPartialResultIndex_++; + } + + if (metadataPackIndex > CameraCapabilities::MaxMetadataPackIndex) + LOG(HAL, Fatal) << "Partial result count exceed limitation " + << CameraCapabilities::MaxMetadataPackIndex; + + /* + * The metadataPackIndex should be sent incrementally. Since the + * function may be called from post-processing threads, ensure the + * strict sending order by the result mutex. + */ + sendCaptureResult(result); + result->complete_ = true; +} + +void CameraDevice::sendCaptureResult(Camera3ResultDescriptor *result) const +{ + LOG(HAL, Debug) << "Send result of frameNumber: " + << result->request_->frameNumber_ + << " index: " << result->metadataPackIndex_ + << " has metadata: " << (!!result->resultMetadata_) + << " has buffers " << result->buffers_.size(); + + camera3_capture_result_t captureResult = {}; - /* Also set error status on entire request descriptor. */ - streamBuffer.request->status_ = - Camera3RequestDescriptor::Status::Error; + captureResult.frame_number = result->request_->frameNumber_; + + std::vector resultBuffers; + resultBuffers.reserve(result->buffers_.size()); + + for (auto &buffer : result->buffers_) { + camera3_buffer_status status = CAMERA3_BUFFER_STATUS_ERROR; + + if (buffer->status == StreamBuffer::Status::Success) + status = CAMERA3_BUFFER_STATUS_OK; + + /* + * Pass the buffer fence back to the camera framework as + * a release fence. This instructs the framework to wait + * on the acquire fence in case we haven't done so + * ourselves for any reason. + */ + resultBuffers.push_back({ buffer->stream->camera3Stream(), + buffer->camera3Buffer, status, + -1, buffer->fence.release() }); } + + captureResult.num_output_buffers = resultBuffers.size(); + captureResult.output_buffers = resultBuffers.data(); + + if (result->resultMetadata_) + captureResult.result = result->resultMetadata_->getMetadata(); + + captureResult.partial_result = result->metadataPackIndex_; + + callbacks_->process_capture_result(callbacks_, &captureResult); } /** @@ -1354,35 +1620,28 @@ void CameraDevice::setBufferStatus(StreamBuffer &streamBuffer, * * This function is called from the post-processor's thread whenever a camera * stream has finished post processing. The corresponding entry is dropped from - * the descriptor's pendingStreamsToProcess_ map. + * the result's pendingBufferToProcess_ list. * - * If the pendingStreamsToProcess_ map is then empty, all streams requiring to - * be generated from post-processing have been completed. Mark the descriptor as - * complete using completeDescriptor() in that case. + * If the pendingBufferToProcess_ list is then empty, all streams requiring to + * be generated from post-processing have been completed. Mark the result as + * complete using completeResultDescriptor() in that case. */ void CameraDevice::streamProcessingComplete(StreamBuffer *streamBuffer, StreamBuffer::Status status) { setBufferStatus(*streamBuffer, status); + streamBuffer->dstBuffer = nullptr; - /* - * If the framebuffer is internal to CameraStream return it back now - * that we're done processing it. - */ - if (streamBuffer->internalBuffer) - streamBuffer->stream->putBuffer(streamBuffer->internalBuffer); - - Camera3RequestDescriptor *request = streamBuffer->request; - + Camera3ResultDescriptor *result = streamBuffer->result; { - MutexLocker locker(request->streamsProcessMutex_); + MutexLocker locker(result->streamsProcessMutex_); + result->pendingBuffersToProcess_.remove(streamBuffer); - request->pendingStreamsToProcess_.erase(streamBuffer->stream); - if (!request->pendingStreamsToProcess_.empty()) + if (!result->pendingBuffersToProcess_.empty()) return; } - completeDescriptor(streamBuffer->request); + tryCompleteResultDescriptor(streamBuffer->result); } std::string CameraDevice::logPrefix() const @@ -1414,6 +1673,99 @@ void CameraDevice::notifyError(uint32_t frameNumber, camera3_stream_t *stream, callbacks_->notify(callbacks_, ¬ify); } +std::unique_ptr +CameraDevice::getDynamicResultMetadata(const ControlList &metadata) const +{ + /* + * \todo Keep this in sync with the actual number of entries. + * + * Reserve capacity for the metadata larger than 4 bytes which cannot + * store in entries. + * Currently: 6 entries, 40 bytes extra capaticy. + * + * ANDROID_SENSOR_TIMESTAMP (int64) = 8 bytes + * ANDROID_SENSOR_EXPOSURE_TIME (int64) = 8 bytes + * ANDROID_SENSOR_FRAME_DURATION (int64) = 8 bytes + * ANDROID_SCALER_CROP_REGION (int32 X 4) = 16 bytes + * Total bytes for capacity: 40 + * + * Reserve more capacity for the JPEG metadata set by the post-processor. + * Currently: 8 entries, 72 bytes extra capaticy. + * + * ANDROID_JPEG_GPS_COORDINATES (double x 3) = 24 bytes + * ANDROID_JPEG_GPS_PROCESSING_METHOD (byte x 32) = 32 bytes + * ANDROID_JPEG_GPS_TIMESTAMP (int64) = 8 bytes + * ANDROID_JPEG_SIZE (int32_t) = 4 bytes + * ANDROID_JPEG_QUALITY (byte) = 1 byte + * ANDROID_JPEG_ORIENTATION (int32_t) = 4 bytes + * ANDROID_JPEG_THUMBNAIL_QUALITY (byte) = 1 byte + * ANDROID_JPEG_THUMBNAIL_SIZE (int32 x 2) = 8 bytes + * Total bytes for JPEG metadata: 72 + * + * \todo Calculate the entries and capacity by the input ControlList. + */ + std::unique_ptr resultMetadata = + std::make_unique(14, 112); + if (!resultMetadata->isValid()) { + LOG(HAL, Error) << "Failed to allocate result metadata"; + return nullptr; + } + + /* Add metadata tags reported by libcamera. */ + const auto ×tamp = metadata.get(controls::SensorTimestamp); + if (timestamp) + resultMetadata->addEntry(ANDROID_SENSOR_TIMESTAMP, *timestamp); + + const auto &pipelineDepth = metadata.get(controls::draft::PipelineDepth); + if (pipelineDepth) + resultMetadata->addEntry(ANDROID_REQUEST_PIPELINE_DEPTH, + *pipelineDepth); + + const auto &exposureTime = metadata.get(controls::ExposureTime); + if (exposureTime) + resultMetadata->addEntry(ANDROID_SENSOR_EXPOSURE_TIME, + *exposureTime * 1000ULL); + + const auto &frameDuration = metadata.get(controls::FrameDuration); + if (frameDuration) + resultMetadata->addEntry(ANDROID_SENSOR_FRAME_DURATION, + *frameDuration * 1000); + + const auto &scalerCrop = metadata.get(controls::ScalerCrop); + if (scalerCrop) { + const Rectangle &crop = *scalerCrop; + int32_t cropRect[] = { + crop.x, + crop.y, + static_cast(crop.width), + static_cast(crop.height), + }; + resultMetadata->addEntry(ANDROID_SCALER_CROP_REGION, cropRect); + } + + const auto &testPatternMode = metadata.get(controls::draft::TestPatternMode); + if (testPatternMode) + resultMetadata->addEntry(ANDROID_SENSOR_TEST_PATTERN_MODE, + *testPatternMode); + + /* + * Return the result metadata pack even is not valid: get() will return + * nullptr. + */ + if (!resultMetadata->isValid()) { + LOG(HAL, Error) << "Failed to construct result metadata"; + } + + if (resultMetadata->resized()) { + auto [entryCount, dataCount] = resultMetadata->usage(); + LOG(HAL, Info) + << "Result metadata resized: " << entryCount + << " entries and " << dataCount << " bytes used"; + } + + return resultMetadata; +} + /* * Set jpeg metadata used to generate EXIF in the JPEG post processing. */ @@ -1424,10 +1776,8 @@ void CameraDevice::generateJpegExifMetadata(Camera3RequestDescriptor *request, auto &jpegExifMetadata = buffer->jpegExifMetadata; jpegExifMetadata.emplace(StreamBuffer::JpegExifMetadata()); - jpegExifMetadata->sensorExposureTime = 0; - if (metadata.contains(controls::ExposureTime)) { - jpegExifMetadata->sensorExposureTime = metadata.get(controls::ExposureTime) * 1000ULL; - } + const int64_t exposureTime = metadata.get(controls::ExposureTime).value_or(0); + jpegExifMetadata->sensorExposureTime = exposureTime; /* * todo: Android Sensitivity = analog gain X digital gain only on sensor. @@ -1441,31 +1791,26 @@ void CameraDevice::generateJpegExifMetadata(Camera3RequestDescriptor *request, * Produce a set of fixed result metadata. */ std::unique_ptr -CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor) const +CameraDevice::getFixedResultMetadata(const CameraMetadata &settings) const { - const ControlList &metadata = descriptor.request_->metadata(); - const CameraMetadata &settings = descriptor.settings_; camera_metadata_ro_entry_t entry; bool found; /* + * \todo Retrieve metadata from corresponding libcamera controls. * \todo Keep this in sync with the actual number of entries. - * Currently: 40 entries, 156 bytes * - * Reserve more space for the JPEG metadata set by the post-processor. - * Currently: - * ANDROID_JPEG_GPS_COORDINATES (double x 3) = 24 bytes - * ANDROID_JPEG_GPS_PROCESSING_METHOD (byte x 32) = 32 bytes - * ANDROID_JPEG_GPS_TIMESTAMP (int64) = 8 bytes - * ANDROID_JPEG_SIZE (int32_t) = 4 bytes - * ANDROID_JPEG_QUALITY (byte) = 1 byte - * ANDROID_JPEG_ORIENTATION (int32_t) = 4 bytes - * ANDROID_JPEG_THUMBNAIL_QUALITY (byte) = 1 byte - * ANDROID_JPEG_THUMBNAIL_SIZE (int32 x 2) = 8 bytes - * Total bytes for JPEG metadata: 82 + * Reserve capacity for the metadata larger than 4 bytes which cannot + * store in entries. + * Currently: 31 entries, 16 bytes + * + * ANDROID_CONTROL_AE_TARGET_FPS_RANGE (int32 X 2) = 8 bytes + * ANDROID_SENSOR_ROLLING_SHUTTER_SKEW (int64) = 8 bytes + * + * Total bytes: 16 */ std::unique_ptr resultMetadata = - std::make_unique(88, 166); + std::make_unique(31, 16); if (!resultMetadata->isValid()) { LOG(HAL, Error) << "Failed to allocate result metadata"; return nullptr; @@ -1563,9 +1908,6 @@ CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor) cons resultMetadata->addEntry(ANDROID_LENS_OPTICAL_STABILIZATION_MODE, value); - value32 = ANDROID_SENSOR_TEST_PATTERN_MODE_OFF; - resultMetadata->addEntry(ANDROID_SENSOR_TEST_PATTERN_MODE, value32); - value = ANDROID_STATISTICS_FACE_DETECT_MODE_OFF; resultMetadata->addEntry(ANDROID_STATISTICS_FACE_DETECT_MODE, value); @@ -1587,40 +1929,6 @@ CameraDevice::getResultMetadata(const Camera3RequestDescriptor &descriptor) cons resultMetadata->addEntry(ANDROID_SENSOR_ROLLING_SHUTTER_SKEW, rolling_shutter_skew); - /* Add metadata tags reported by libcamera. */ - const int64_t timestamp = metadata.get(controls::SensorTimestamp).value_or(0); - resultMetadata->addEntry(ANDROID_SENSOR_TIMESTAMP, timestamp); - - const auto &pipelineDepth = metadata.get(controls::draft::PipelineDepth); - if (pipelineDepth) - resultMetadata->addEntry(ANDROID_REQUEST_PIPELINE_DEPTH, - *pipelineDepth); - - const auto &exposureTime = metadata.get(controls::ExposureTime); - if (exposureTime) - resultMetadata->addEntry(ANDROID_SENSOR_EXPOSURE_TIME, - *exposureTime * 1000ULL); - - const auto &frameDuration = metadata.get(controls::FrameDuration); - if (frameDuration) - resultMetadata->addEntry(ANDROID_SENSOR_FRAME_DURATION, - *frameDuration * 1000); - - const auto &scalerCrop = metadata.get(controls::ScalerCrop); - if (scalerCrop) { - const Rectangle &crop = *scalerCrop; - int32_t cropRect[] = { - crop.x, crop.y, static_cast(crop.width), - static_cast(crop.height), - }; - resultMetadata->addEntry(ANDROID_SCALER_CROP_REGION, cropRect); - } - - const auto &testPatternMode = metadata.get(controls::draft::TestPatternMode); - if (testPatternMode) - resultMetadata->addEntry(ANDROID_SENSOR_TEST_PATTERN_MODE, - *testPatternMode); - /* * Return the result metadata pack even is not valid: get() will return * nullptr. diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 7b279895..613e419f 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -20,6 +20,7 @@ #include #include +#include #include #include #include @@ -29,7 +30,6 @@ #include "camera_capabilities.h" #include "camera_metadata.h" #include "camera_stream.h" -#include "jpeg/encoder.h" class Camera3RequestDescriptor; struct CameraConfigData; @@ -63,6 +63,8 @@ public: const camera_metadata_t *constructDefaultRequestSettings(int type); int configureStreams(camera3_stream_configuration_t *stream_list); int processCaptureRequest(camera3_capture_request_t *request); + void partialResultComplete(libcamera::Request *request, + libcamera::Result *result); void requestComplete(libcamera::Request *request); void streamProcessingComplete(StreamBuffer *bufferStream, StreamBuffer::Status status); @@ -87,20 +89,25 @@ private: createFrameBuffer(const buffer_handle_t camera3buffer, libcamera::PixelFormat pixelFormat, const libcamera::Size &size); - void abortRequest(Camera3RequestDescriptor *descriptor) const; bool isValidRequest(camera3_capture_request_t *request) const; void notifyShutter(uint32_t frameNumber, uint64_t timestamp); void notifyError(uint32_t frameNumber, camera3_stream_t *stream, camera3_error_msg_code code) const; int processControls(Camera3RequestDescriptor *descriptor); - void completeDescriptor(Camera3RequestDescriptor *descriptor) - LIBCAMERA_TSA_EXCLUDES(descriptorsMutex_); - void sendCaptureResults() LIBCAMERA_TSA_REQUIRES(descriptorsMutex_); - void setBufferStatus(StreamBuffer &buffer, StreamBuffer::Status status); + void abortRequest(Camera3RequestDescriptor *descriptor); + void tryCompleteResultDescriptor(Camera3ResultDescriptor *result); + void completeResultDescriptor(Camera3ResultDescriptor *result); + void completeRequestDescriptor(Camera3RequestDescriptor *descriptor); + void sendPartialResult(Camera3ResultDescriptor *result); + void sendCaptureResult(Camera3ResultDescriptor *partialResult) const; + void setBufferStatus(StreamBuffer &buffer, StreamBuffer::Status status) const; void generateJpegExifMetadata(Camera3RequestDescriptor *request, StreamBuffer *buffer) const; - std::unique_ptr getResultMetadata( - const Camera3RequestDescriptor &descriptor) const; + + std::unique_ptr getDynamicResultMetadata( + const libcamera::ControlList &metadata) const; + std::unique_ptr getFixedResultMetadata( + const CameraMetadata &settings) const; unsigned int id_; camera3_device_t camera3Device_; @@ -117,9 +124,16 @@ private: std::vector streams_; - libcamera::Mutex descriptorsMutex_ LIBCAMERA_TSA_ACQUIRED_AFTER(stateMutex_); - std::queue> descriptors_ - LIBCAMERA_TSA_GUARDED_BY(descriptorsMutex_); + libcamera::Mutex pendingRequestMutex_ LIBCAMERA_TSA_ACQUIRED_AFTER(stateMutex_); + + std::list> pendingRequests_ + LIBCAMERA_TSA_GUARDED_BY(pendingRequestMutex_); + + std::list pendingResults_ + LIBCAMERA_TSA_GUARDED_BY(pendingRequestMutex_); + + std::map> pendingStreamBuffers_ + LIBCAMERA_TSA_GUARDED_BY(pendingRequestMutex_); std::string maker_; std::string model_; diff --git a/src/android/camera_request.cpp b/src/android/camera_request.cpp index f5d4d314..ab829760 100644 --- a/src/android/camera_request.cpp +++ b/src/android/camera_request.cpp @@ -7,9 +7,11 @@ #include "camera_request.h" +#include #include #include "camera_buffer.h" +#include "camera_stream.h" using namespace libcamera; @@ -42,25 +44,25 @@ using namespace libcamera; * │ processCaptureRequest(camera3_capture_request_t request) │ * │ │ * │ - Create Camera3RequestDescriptor tracking this request │ - * │ - Streams requiring post-processing are stored in the │ - * │ pendingStreamsToProcess map │ + * │ - Buffer requiring post-processing are marked by the │ + * │ CameraStream::Type as Mapped or Internal │ * │ - Add this Camera3RequestDescriptor to descriptors' queue │ * │ CameraDevice::descriptors_ │ - * │ │ ┌─────────────────────────┐ - * │ - Queue the capture request to libcamera core ────────────┤►│libcamera core │ - * │ │ ├─────────────────────────┤ - * │ │ │- Capture from Camera │ - * │ │ │ │ - * │ │ │- Emit │ - * │ │ │ Camera::requestComplete│ - * │ requestCompleted(Request *request) ◄───────────────────────┼─┼──── │ - * │ │ │ │ - * │ - Check request completion status │ └─────────────────────────┘ + * │ │ ┌───────────────────────────────┐ + * │ - Queue the capture request to libcamera core ────────────┤►│libcamera core │ + * │ │ ├───────────────────────────────┤ + * │ │ │- Capture from Camera │ + * │ │ │ │ + * │ │ │- Emit │ + * │ │ │ Camera::partialResultComplete│ + * │ partialResultComplete(Request *request, Result result*) ◄──┼─┼──── │ + * │ │ │ │ + * │ - Check request completion status │ └───────────────────────────────┘ * │ │ - * │ - if (pendingStreamsToProcess > 0) │ - * │ Queue all entries from pendingStreamsToProcess │ + * │ - if (pendingBuffersToProcess > 0) │ + * │ Queue all entries from pendingBuffersToProcess │ * │ else │ │ - * │ completeDescriptor() │ └──────────────────────┐ + * │ completeResultDescriptor() │ └──────────────────────┐ * │ │ │ * │ ┌──────────────────────────┴───┬──────────────────┐ │ * │ │ │ │ │ @@ -93,10 +95,10 @@ using namespace libcamera; * │ | | | | │ * │ | - Check and set buffer status | | .... | │ * │ | - Remove post+processing entry | | | │ - * │ | from pendingStreamsToProcess | | | │ + * │ | from pendingBuffersToProcess | | | │ * │ | | | | │ - * │ | - if (pendingStreamsToProcess.empty())| | | │ - * │ | completeDescriptor | | | │ + * │ | - if (pendingBuffersToProcess.empty())| | | │ + * │ | completeResultDescriptor | | | │ * │ | | | | │ * │ +---------------------------------------+ +--------------+ │ * │ │ @@ -110,6 +112,7 @@ using namespace libcamera; Camera3RequestDescriptor::Camera3RequestDescriptor( Camera *camera, const camera3_capture_request_t *camera3Request) + : status_(Status::Pending), nextPartialResultIndex_(1), completed_(false) { frameNumber_ = camera3Request->frame_number; @@ -138,7 +141,27 @@ Camera3RequestDescriptor::Camera3RequestDescriptor( request_ = camera->createRequest(reinterpret_cast(this)); } -Camera3RequestDescriptor::~Camera3RequestDescriptor() = default; +Camera3RequestDescriptor::~Camera3RequestDescriptor() +{ + /* + * Recycle the allocated internal buffer back to its source stream. + */ + for (auto &[sourceStream, frameBuffer] : internalBuffers_) + sourceStream->putBuffer(frameBuffer); +} + +/* + * \class Camera3ResultDescriptor + * + * A utility class that groups information about a capture result to be later + * sent to framework. + */ +Camera3ResultDescriptor::Camera3ResultDescriptor(Camera3RequestDescriptor *request) + : request_(request), metadataPackIndex_(1), complete_(false) +{ +} + +Camera3ResultDescriptor::~Camera3ResultDescriptor() = default; /** * \class StreamBuffer diff --git a/src/android/camera_request.h b/src/android/camera_request.h index f91de955..fd8dad24 100644 --- a/src/android/camera_request.h +++ b/src/android/camera_request.h @@ -24,6 +24,7 @@ class CameraBuffer; class CameraStream; +class Camera3ResultDescriptor; class Camera3RequestDescriptor; class StreamBuffer @@ -52,45 +53,74 @@ public: std::unique_ptr frameBuffer; libcamera::UniqueFD fence; Status status = Status::Success; - libcamera::FrameBuffer *internalBuffer = nullptr; const libcamera::FrameBuffer *srcBuffer = nullptr; std::unique_ptr dstBuffer; std::optional jpegExifMetadata; + Camera3ResultDescriptor *result; Camera3RequestDescriptor *request; private: LIBCAMERA_DISABLE_COPY(StreamBuffer) }; +class Camera3ResultDescriptor +{ +public: + Camera3ResultDescriptor(Camera3RequestDescriptor *request); + ~Camera3ResultDescriptor(); + + Camera3RequestDescriptor *request_; + uint32_t metadataPackIndex_; + + std::unique_ptr resultMetadata_; + std::vector buffers_; + + /* Keeps track of buffers waiting for post-processing. */ + std::list pendingBuffersToProcess_ + LIBCAMERA_TSA_GUARDED_BY(streamsProcessMutex_); + libcamera::Mutex streamsProcessMutex_; + + bool complete_; + +private: + LIBCAMERA_DISABLE_COPY(Camera3ResultDescriptor) +}; + class Camera3RequestDescriptor { public: enum class Status { + Pending, Success, - Error, + Cancelled, + Aborted, }; - /* Keeps track of streams requiring post-processing. */ - std::map pendingStreamsToProcess_ - LIBCAMERA_TSA_GUARDED_BY(streamsProcessMutex_); - libcamera::Mutex streamsProcessMutex_; - Camera3RequestDescriptor(libcamera::Camera *camera, const camera3_capture_request_t *camera3Request); ~Camera3RequestDescriptor(); - bool isPending() const { return !complete_; } - uint32_t frameNumber_ = 0; std::vector buffers_; - CameraMetadata settings_; + std::unique_ptr request_; - std::unique_ptr resultMetadata_; - bool complete_ = false; - Status status_ = Status::Success; + std::map internalBuffers_; + + Status status_ + LIBCAMERA_TSA_GUARDED_BY(resultsMutex_); + uint32_t nextPartialResultIndex_ + LIBCAMERA_TSA_GUARDED_BY(resultsMutex_); + std::unique_ptr finalResult_ + LIBCAMERA_TSA_GUARDED_BY(resultsMutex_); + std::vector> partialResults_ + LIBCAMERA_TSA_GUARDED_BY(resultsMutex_); + + libcamera::Mutex resultsMutex_; + + bool completed_; private: LIBCAMERA_DISABLE_COPY(Camera3RequestDescriptor) diff --git a/src/android/camera_stream.cpp b/src/android/camera_stream.cpp index 02dc8922..3215025c 100644 --- a/src/android/camera_stream.cpp +++ b/src/android/camera_stream.cpp @@ -187,6 +187,7 @@ int CameraStream::process(StreamBuffer *streamBuffer) *streamBuffer->camera3Buffer, output.pixelFormat, output.size, PROT_READ | PROT_WRITE); if (!streamBuffer->dstBuffer->isValid()) { + streamBuffer->dstBuffer = nullptr; LOG(HAL, Error) << "Failed to create destination buffer"; return -EINVAL; } diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp index 7e3713f5..d61c566b 100644 --- a/src/android/jpeg/post_processor_jpeg.cpp +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -111,7 +111,7 @@ void PostProcessorJpeg::process(StreamBuffer *streamBuffer) ASSERT(jpegExifMetadata.has_value()); const CameraMetadata &requestMetadata = streamBuffer->request->settings_; - CameraMetadata *resultMetadata = streamBuffer->request->resultMetadata_.get(); + CameraMetadata *resultMetadata = streamBuffer->result->resultMetadata_.get(); camera_metadata_ro_entry_t entry; int ret;