From patchwork Thu Jan 6 09:43:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hirokazu Honda X-Patchwork-Id: 15256 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id F224ABF415 for ; Thu, 6 Jan 2022 09:43:35 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A1C6660928; Thu, 6 Jan 2022 10:43:35 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="C/kOC22X"; dkim-atps=neutral Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id BCAF5604F5 for ; Thu, 6 Jan 2022 10:43:33 +0100 (CET) Received: by mail-pj1-x1036.google.com with SMTP id rj2-20020a17090b3e8200b001b1944bad25so2573669pjb.5 for ; Thu, 06 Jan 2022 01:43:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KyuDPm/zxDMeMdQNU7I6ndUbNpxweiggVhp2DAYM1/4=; b=C/kOC22XlKn9ctJizGH8yZh+Ra7Qb6BVDu+q1/SnMB6sZXyUTdGCsjY8RMh3Lo+BLr zNhCB976mKnkqogqkILy+y4ELyFdMi1I3vOoVt56E5mvLDa+Vu/j5SVQ/UJ0Xm37YbsJ oslmVDjbAu0OCCUy8m8QS1zC9i9yOIG/g/SMw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KyuDPm/zxDMeMdQNU7I6ndUbNpxweiggVhp2DAYM1/4=; b=8A0XisMcPJQrJpB5A59lgv7Ga0uBxN/uWMXVYoRybRZps2P5imaSnmoMxAw2WkVf5N 2d2LwDlJ8KVatBUoq3rdcT7PRDLxzlVqOKV9c4ckIqtA6uqUlfLHV0JNCf2GGUCn/+n0 6z0HbhDHlbx7FjZsYe8dzM3LQBitA+Z1Qt22NJeIMEU8eamGSAfyGHB7FP+B2rKBIEEL RPpnDzasCYwE6ENXj5YJrURNGaySfT0L129Z3bjd0GBQ17L5Ykd0mGr+KLKa0YV/mRmZ PfJ0y7junBVjm4gulK8EhQForB1mGa1j68qZX27pHM9VlkFUIkIr7TezwRK1SmInOx+7 xP8A== X-Gm-Message-State: AOAM531qwf+vyf5AAsLYpGJBWI3+mRe8OfoXEOcZt1SQad22A+gG+Bls /yYBD6MqdBoWiYkfn/4cPPjqE6fXPIFAzw== X-Google-Smtp-Source: ABdhPJwgeOX6H3BbI/LPNBfTljMqNkP3l7TJg8HzkYPlRak1iViDQwOQnbyRAAILYe20AV60U0HFYw== X-Received: by 2002:a17:90b:3b49:: with SMTP id ot9mr9155941pjb.110.1641462211678; Thu, 06 Jan 2022 01:43:31 -0800 (PST) Received: from hiroh2.tok.corp.google.com ([2401:fa00:8f:203:cb79:fed4:e097:2139]) by smtp.gmail.com with ESMTPSA id 4sm1844860pfy.191.2022.01.06.01.43.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Jan 2022 01:43:31 -0800 (PST) From: Hirokazu Honda To: libcamera-devel@lists.libcamera.org Date: Thu, 6 Jan 2022 18:43:23 +0900 Message-Id: <20220106094323.1819801-2-hiroh@chromium.org> X-Mailer: git-send-email 2.34.1.448.ga2b2bfdf31-goog In-Reply-To: <20220106094323.1819801-1-hiroh@chromium.org> References: <20220106094323.1819801-1-hiroh@chromium.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 1/1] android: camera_device: Configure one stream for identical stream requests X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" An Android HAL client may request multiple identical streams. It is redundant that a native camera device produces a separate stream for each of the identical requests. Configure the camera with a single stream in that case. The other identical HAL streams will be produced by the YUV post-processor. The Android HAL client can provide capture requests that are resolved as they are produced by YUV post-processor. Then a buffer to be filled a camera is not given. So the HAL adaptation layer looks up buffer information to be passed to a camera and allocate the buffer by using PlatformBufferAllocator. Signed-off-by: Hirokazu Honda --- src/android/camera_device.cpp | 84 ++++++++++++++++++++++++++++++++++- src/android/camera_request.h | 2 + src/android/camera_stream.cpp | 12 ++++- src/android/camera_stream.h | 6 ++- 4 files changed, 100 insertions(+), 4 deletions(-) -- 2.34.1.448.ga2b2bfdf31-goog diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index 83825736..53533064 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -604,6 +605,35 @@ int CameraDevice::configureStreams(camera3_stream_configuration_t *stream_list) continue; } + /* + * While gralloc usage flags are supposed to report usage + * patterns to select a suitable buffer allocation strategy, in + * practice they're also used to make other decisions, such as + * selecting the actual format for the IMPLEMENTATION_DEFINED + * HAL pixel format. To avoid issues, we thus have to set the + * GRALLOC_USAGE_HW_CAMERA_WRITE flag unconditionally, even for + * streams that will be produced in software. + */ + stream->usage |= GRALLOC_USAGE_HW_CAMERA_WRITE; + + /* + * If a CameraStream with the same size and format of the + * current stream has already been requested, associate the two. + */ + auto iter = std::find_if( + streamConfigs.begin(), streamConfigs.end(), + [size, format](const Camera3StreamConfig &streamConfig) { + return streamConfig.config.size == size && + streamConfig.config.pixelFormat == format; + }); + if (iter != streamConfigs.end()) { + /* Add usage to copy the buffer in streams[0] to stream. */ + iter->streams[0].stream->usage |= GRALLOC_USAGE_SW_READ_OFTEN; + stream->usage |= GRALLOC_USAGE_SW_WRITE_OFTEN; + iter->streams.push_back({ stream, CameraStream::Type::Mapped }); + continue; + } + Camera3StreamConfig streamConfig; streamConfig.streams = { { stream, CameraStream::Type::Direct } }; streamConfig.config.size = size; @@ -681,10 +711,32 @@ int CameraDevice::configureStreams(camera3_stream_configuration_t *stream_list) for (const auto &streamConfig : streamConfigs) { config->addConfiguration(streamConfig.config); + CameraStream *directStream = nullptr; for (auto &stream : streamConfig.streams) { + CameraStream *sourceStream = nullptr; + + /* + * Sets the source stream for Mapped type streams to the + * same Direct type stream. The Direct type stream is + * put earlier than Mapped type streams in the current + * implementation. This might be broken in the future. + * + * \todo Remove this order assumption. + */ + if (stream.type == CameraStream::Type::Mapped) { + ASSERT(directStream); + sourceStream = directStream; + } + streams_.emplace_back(this, config.get(), stream.type, - stream.stream, config->size() - 1); + stream.stream, + sourceStream, + config->size() - 1); stream.stream->priv = static_cast(&streams_.back()); + if (!directStream && + stream.type == CameraStream::Type::Direct) { + directStream = &streams_.back(); + } } } @@ -917,6 +969,8 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques LOG(HAL, Debug) << "Queueing request " << descriptor->request_->cookie() << " with " << descriptor->buffers_.size() << " streams"; + std::set requiredStreams; + std::set providedStreams; for (const auto &[i, buffer] : utils::enumerate(descriptor->buffers_)) { CameraStream *cameraStream = buffer.stream; camera3_stream_t *camera3Stream = cameraStream->camera3Stream(); @@ -947,6 +1001,8 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques descriptor->pendingStreamsToProcess_.insert( { cameraStream, &buffer }); + ASSERT(cameraStream->sourceStream()); + requiredStreams.insert(cameraStream->sourceStream()); continue; case CameraStream::Type::Direct: @@ -990,8 +1046,32 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques auto fence = std::make_unique(std::move(acquireFence)); descriptor->request_->addBuffer(cameraStream->stream(), frameBuffer, std::move(fence)); + providedStreams.insert(cameraStream); } + for (CameraStream *requiredStream : requiredStreams) { + if (providedStreams.find(requiredStream) != providedStreams.end()) + continue; + + /* \todo Can we Map stream to Internal type stream? */ + ASSERT(requiredStream->type() == CameraStream::Type::Direct); + + FrameBuffer *frameBuffer = requiredStream->getBuffer(); + if (!frameBuffer) { + LOG(HAL, Error) << "Failed to create frame buffer"; + return -ENOMEM; + } + + /* + * addBuffer() lets a buffer for requiredStream be output by + * camera. Records frameBuffer with requiredStream to call + * putBuffer() after post-processing is complete. + */ + descriptor->request_->addBuffer(requiredStream->stream(), + frameBuffer, nullptr); + MutexLocker lock(descriptor->streamsProcessMutex_); + descriptor->putBuffers_.push_back({ requiredStream, frameBuffer }); + } /* * Translate controls from Android to libcamera and queue the request * to the camera. @@ -1251,6 +1331,8 @@ void CameraDevice::streamProcessingComplete(Camera3RequestDescriptor::StreamBuff request->pendingStreamsToProcess_.erase(streamBuffer->stream); if (!request->pendingStreamsToProcess_.empty()) return; + for (auto [cameraStream, buffer] : request->putBuffers_) + cameraStream->putBuffer(buffer); } completeDescriptor(streamBuffer->request); diff --git a/src/android/camera_request.h b/src/android/camera_request.h index 37b6ae32..7a41c6d9 100644 --- a/src/android/camera_request.h +++ b/src/android/camera_request.h @@ -59,6 +59,8 @@ public: /* Keeps track of streams requiring post-processing. */ std::map pendingStreamsToProcess_ LIBCAMERA_TSA_GUARDED_BY(streamsProcessMutex_); + std::vector> putBuffers_ + LIBCAMERA_TSA_GUARDED_BY(streamsProcessMutex_); libcamera::Mutex streamsProcessMutex_; Camera3RequestDescriptor(libcamera::Camera *camera, diff --git a/src/android/camera_stream.cpp b/src/android/camera_stream.cpp index c2157450..634cf319 100644 --- a/src/android/camera_stream.cpp +++ b/src/android/camera_stream.cpp @@ -52,9 +52,11 @@ LOG_DECLARE_CATEGORY(HAL) CameraStream::CameraStream(CameraDevice *const cameraDevice, CameraConfiguration *config, Type type, - camera3_stream_t *camera3Stream, unsigned int index) + camera3_stream_t *camera3Stream, + CameraStream *const sourceStream, unsigned int index) : cameraDevice_(cameraDevice), config_(config), type_(type), - camera3Stream_(camera3Stream), index_(index) + camera3Stream_(camera3Stream), sourceStream_(sourceStream), + index_(index) { } @@ -206,6 +208,12 @@ void CameraStream::flush() FrameBuffer *CameraStream::getBuffer() { + if (type_ == Type::Direct && !allocator_) { + allocator_ = std::make_unique(cameraDevice_); + ASSERT(!mutex_); + mutex_ = std::make_unique(); + } + if (!allocator_) return nullptr; diff --git a/src/android/camera_stream.h b/src/android/camera_stream.h index f9504462..4c5078b2 100644 --- a/src/android/camera_stream.h +++ b/src/android/camera_stream.h @@ -114,7 +114,9 @@ public: }; CameraStream(CameraDevice *const cameraDevice, libcamera::CameraConfiguration *config, Type type, - camera3_stream_t *camera3Stream, unsigned int index); + camera3_stream_t *camera3Stream, + CameraStream *const sourceStream, + unsigned int index); CameraStream(CameraStream &&other); ~CameraStream(); @@ -122,6 +124,7 @@ public: camera3_stream_t *camera3Stream() const { return camera3Stream_; } const libcamera::StreamConfiguration &configuration() const; libcamera::Stream *stream() const; + CameraStream *sourceStream() const { return sourceStream_; } int configure(); int process(Camera3RequestDescriptor::StreamBuffer *streamBuffer); @@ -167,6 +170,7 @@ private: const libcamera::CameraConfiguration *config_; const Type type_; camera3_stream_t *camera3Stream_; + CameraStream *const sourceStream_; const unsigned int index_; std::unique_ptr allocator_;