From patchwork Tue Sep 7 19:56:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 13750 X-Patchwork-Delegate: umang.jain@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id BF777BDC71 for ; Tue, 7 Sep 2021 19:57:21 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 7A47C6916F; Tue, 7 Sep 2021 21:57:21 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="TPRJ/GRr"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D985B60252 for ; Tue, 7 Sep 2021 21:57:19 +0200 (CEST) Received: from perceval.ideasonboard.com (unknown [103.251.226.2]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id D76AAD88; Tue, 7 Sep 2021 21:57:18 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631044639; bh=LYXRMTX2uukFCcQuz+x5e9UwRZi5hsu4Q14uBP0jejM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TPRJ/GRrn1iqHVGIK78gpwHOwp2MXONPXzBpbFEZigDhP5EaLPh+iD0dccimS9j7i EIAvdlnGBxTEWP44ke785E4xu6WRhLiMWScZxlvX1pVZ7j3UjCRKh73rGXt4QqN1nV fjn8PEpRaAyM3dg5uLa/qRQf8Co3IWr2jKlR3dqw= From: Umang Jain To: libcamera-devel@lists.libcamera.org Date: Wed, 8 Sep 2021 01:26:59 +0530 Message-Id: <20210907195704.338079-2-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210907195704.338079-1-umang.jain@ideasonboard.com> References: <20210907195704.338079-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 1/6] android: camera_stream: Pass FrameBuffer pointer instead of reference X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Pass the libcamera::FrameBuffer pointer to the post-processor instead of passing it by reference. Passing by reference is fine as long as the post processing is done synchronously. However in subsequent commits, the post processing is planned to be moved to a separate thread. This will conflict as the reference argument (in current case 'source') is copied when we will try to invoke a method on separate thread(which will run the post-processor) using Object::invokeMethod(). The cause of conflict is the LIBCAMERA_DISABLE_COPY_AND_MOVE rule applied on the FrameBuffer class. To resolve the conflict, pass in the FrameBuffer pointer instead of reference. This requires changes to the existing PostProcessor interface and all its implemented classes. Signed-off-by: Umang Jain --- src/android/camera_device.cpp | 2 +- src/android/camera_stream.cpp | 2 +- src/android/camera_stream.h | 2 +- src/android/jpeg/encoder.h | 2 +- src/android/jpeg/encoder_libjpeg.cpp | 4 ++-- src/android/jpeg/encoder_libjpeg.h | 2 +- src/android/jpeg/post_processor_jpeg.cpp | 4 ++-- src/android/jpeg/post_processor_jpeg.h | 4 ++-- src/android/jpeg/thumbnailer.cpp | 4 ++-- src/android/jpeg/thumbnailer.h | 2 +- src/android/post_processor.h | 2 +- src/android/yuv/post_processor_yuv.cpp | 18 +++++++++--------- src/android/yuv/post_processor_yuv.h | 4 ++-- 13 files changed, 26 insertions(+), 26 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index ab31bdda..f2f36f32 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -1148,7 +1148,7 @@ void CameraDevice::requestComplete(Request *request) continue; } - int ret = cameraStream->process(*src, *buffer.buffer, + int ret = cameraStream->process(src, *buffer.buffer, descriptor.settings_, resultMetadata.get()); /* diff --git a/src/android/camera_stream.cpp b/src/android/camera_stream.cpp index 0f5ae947..0fed5382 100644 --- a/src/android/camera_stream.cpp +++ b/src/android/camera_stream.cpp @@ -98,7 +98,7 @@ int CameraStream::configure() return 0; } -int CameraStream::process(const FrameBuffer &source, +int CameraStream::process(const FrameBuffer *source, buffer_handle_t camera3Dest, const CameraMetadata &requestMetadata, CameraMetadata *resultMetadata) diff --git a/src/android/camera_stream.h b/src/android/camera_stream.h index 2dab6c3a..5c232cb6 100644 --- a/src/android/camera_stream.h +++ b/src/android/camera_stream.h @@ -118,7 +118,7 @@ public: libcamera::Stream *stream() const; int configure(); - int process(const libcamera::FrameBuffer &source, + int process(const libcamera::FrameBuffer *source, buffer_handle_t camera3Dest, const CameraMetadata &requestMetadata, CameraMetadata *resultMetadata); diff --git a/src/android/jpeg/encoder.h b/src/android/jpeg/encoder.h index a28522f4..7b6140e7 100644 --- a/src/android/jpeg/encoder.h +++ b/src/android/jpeg/encoder.h @@ -18,7 +18,7 @@ public: virtual ~Encoder() = default; virtual int configure(const libcamera::StreamConfiguration &cfg) = 0; - virtual int encode(const libcamera::FrameBuffer &source, + virtual int encode(const libcamera::FrameBuffer *source, libcamera::Span destination, libcamera::Span exifData, unsigned int quality) = 0; diff --git a/src/android/jpeg/encoder_libjpeg.cpp b/src/android/jpeg/encoder_libjpeg.cpp index 21a3b33d..3114ed4b 100644 --- a/src/android/jpeg/encoder_libjpeg.cpp +++ b/src/android/jpeg/encoder_libjpeg.cpp @@ -178,10 +178,10 @@ void EncoderLibJpeg::compressNV(const std::vector> &planes) } } -int EncoderLibJpeg::encode(const FrameBuffer &source, Span dest, +int EncoderLibJpeg::encode(const FrameBuffer *source, Span dest, Span exifData, unsigned int quality) { - MappedFrameBuffer frame(&source, MappedFrameBuffer::MapFlag::Read); + MappedFrameBuffer frame(source, MappedFrameBuffer::MapFlag::Read); if (!frame.isValid()) { LOG(JPEG, Error) << "Failed to map FrameBuffer : " << strerror(frame.error()); diff --git a/src/android/jpeg/encoder_libjpeg.h b/src/android/jpeg/encoder_libjpeg.h index 45ffbd7f..ae4e1e32 100644 --- a/src/android/jpeg/encoder_libjpeg.h +++ b/src/android/jpeg/encoder_libjpeg.h @@ -22,7 +22,7 @@ public: ~EncoderLibJpeg(); int configure(const libcamera::StreamConfiguration &cfg) override; - int encode(const libcamera::FrameBuffer &source, + int encode(const libcamera::FrameBuffer *source, libcamera::Span destination, libcamera::Span exifData, unsigned int quality) override; diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp index 68d74842..fe41207a 100644 --- a/src/android/jpeg/post_processor_jpeg.cpp +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -50,7 +50,7 @@ int PostProcessorJpeg::configure(const StreamConfiguration &inCfg, return encoder_->configure(inCfg); } -void PostProcessorJpeg::generateThumbnail(const FrameBuffer &source, +void PostProcessorJpeg::generateThumbnail(const FrameBuffer *source, const Size &targetSize, unsigned int quality, std::vector *thumbnail) @@ -82,7 +82,7 @@ void PostProcessorJpeg::generateThumbnail(const FrameBuffer &source, } } -int PostProcessorJpeg::process(const FrameBuffer &source, +int PostProcessorJpeg::process(const FrameBuffer *source, CameraBuffer *destination, const CameraMetadata &requestMetadata, CameraMetadata *resultMetadata) diff --git a/src/android/jpeg/post_processor_jpeg.h b/src/android/jpeg/post_processor_jpeg.h index 6fd31022..c4b2e9ef 100644 --- a/src/android/jpeg/post_processor_jpeg.h +++ b/src/android/jpeg/post_processor_jpeg.h @@ -22,13 +22,13 @@ public: int configure(const libcamera::StreamConfiguration &incfg, const libcamera::StreamConfiguration &outcfg) override; - int process(const libcamera::FrameBuffer &source, + int process(const libcamera::FrameBuffer *source, CameraBuffer *destination, const CameraMetadata &requestMetadata, CameraMetadata *resultMetadata) override; private: - void generateThumbnail(const libcamera::FrameBuffer &source, + void generateThumbnail(const libcamera::FrameBuffer *source, const libcamera::Size &targetSize, unsigned int quality, std::vector *thumbnail); diff --git a/src/android/jpeg/thumbnailer.cpp b/src/android/jpeg/thumbnailer.cpp index 1fab8072..ffac6a15 100644 --- a/src/android/jpeg/thumbnailer.cpp +++ b/src/android/jpeg/thumbnailer.cpp @@ -37,11 +37,11 @@ void Thumbnailer::configure(const Size &sourceSize, PixelFormat pixelFormat) valid_ = true; } -void Thumbnailer::createThumbnail(const FrameBuffer &source, +void Thumbnailer::createThumbnail(const FrameBuffer *source, const Size &targetSize, std::vector *destination) { - MappedFrameBuffer frame(&source, MappedFrameBuffer::MapFlag::Read); + MappedFrameBuffer frame(source, MappedFrameBuffer::MapFlag::Read); if (!frame.isValid()) { LOG(Thumbnailer, Error) << "Failed to map FrameBuffer : " diff --git a/src/android/jpeg/thumbnailer.h b/src/android/jpeg/thumbnailer.h index 4d086c49..0f3caf40 100644 --- a/src/android/jpeg/thumbnailer.h +++ b/src/android/jpeg/thumbnailer.h @@ -19,7 +19,7 @@ public: void configure(const libcamera::Size &sourceSize, libcamera::PixelFormat pixelFormat); - void createThumbnail(const libcamera::FrameBuffer &source, + void createThumbnail(const libcamera::FrameBuffer *source, const libcamera::Size &targetSize, std::vector *dest); const libcamera::PixelFormat &pixelFormat() const { return pixelFormat_; } diff --git a/src/android/post_processor.h b/src/android/post_processor.h index ab2b2c60..61dfb6d4 100644 --- a/src/android/post_processor.h +++ b/src/android/post_processor.h @@ -21,7 +21,7 @@ public: virtual int configure(const libcamera::StreamConfiguration &inCfg, const libcamera::StreamConfiguration &outCfg) = 0; - virtual int process(const libcamera::FrameBuffer &source, + virtual int process(const libcamera::FrameBuffer *source, CameraBuffer *destination, const CameraMetadata &requestMetadata, CameraMetadata *resultMetadata) = 0; diff --git a/src/android/yuv/post_processor_yuv.cpp b/src/android/yuv/post_processor_yuv.cpp index 7b3b4960..0a874886 100644 --- a/src/android/yuv/post_processor_yuv.cpp +++ b/src/android/yuv/post_processor_yuv.cpp @@ -49,7 +49,7 @@ int PostProcessorYuv::configure(const StreamConfiguration &inCfg, return 0; } -int PostProcessorYuv::process(const FrameBuffer &source, +int PostProcessorYuv::process(const FrameBuffer *source, CameraBuffer *destination, [[maybe_unused]] const CameraMetadata &requestMetadata, [[maybe_unused]] CameraMetadata *metadata) @@ -57,7 +57,7 @@ int PostProcessorYuv::process(const FrameBuffer &source, if (!isValidBuffers(source, *destination)) return -EINVAL; - const MappedFrameBuffer sourceMapped(&source, MappedFrameBuffer::MapFlag::Read); + const MappedFrameBuffer sourceMapped(source, MappedFrameBuffer::MapFlag::Read); if (!sourceMapped.isValid()) { LOG(YUV, Error) << "Failed to mmap camera frame buffer"; return -EINVAL; @@ -83,12 +83,12 @@ int PostProcessorYuv::process(const FrameBuffer &source, return 0; } -bool PostProcessorYuv::isValidBuffers(const FrameBuffer &source, +bool PostProcessorYuv::isValidBuffers(const FrameBuffer *source, const CameraBuffer &destination) const { - if (source.planes().size() != 2) { + if (source->planes().size() != 2) { LOG(YUV, Error) << "Invalid number of source planes: " - << source.planes().size(); + << source->planes().size(); return false; } if (destination.numPlanes() != 2) { @@ -97,12 +97,12 @@ bool PostProcessorYuv::isValidBuffers(const FrameBuffer &source, return false; } - if (source.planes()[0].length < sourceLength_[0] || - source.planes()[1].length < sourceLength_[1]) { + if (source->planes()[0].length < sourceLength_[0] || + source->planes()[1].length < sourceLength_[1]) { LOG(YUV, Error) << "The source planes lengths are too small, actual size: {" - << source.planes()[0].length << ", " - << source.planes()[1].length + << source->planes()[0].length << ", " + << source->planes()[1].length << "}, expected size: {" << sourceLength_[0] << ", " << sourceLength_[1] << "}"; diff --git a/src/android/yuv/post_processor_yuv.h b/src/android/yuv/post_processor_yuv.h index f8b1ba23..44a04113 100644 --- a/src/android/yuv/post_processor_yuv.h +++ b/src/android/yuv/post_processor_yuv.h @@ -20,13 +20,13 @@ public: int configure(const libcamera::StreamConfiguration &incfg, const libcamera::StreamConfiguration &outcfg) override; - int process(const libcamera::FrameBuffer &source, + int process(const libcamera::FrameBuffer *source, CameraBuffer *destination, const CameraMetadata &requestMetadata, CameraMetadata *metadata) override; private: - bool isValidBuffers(const libcamera::FrameBuffer &source, + bool isValidBuffers(const libcamera::FrameBuffer *source, const CameraBuffer &destination) const; void calculateLengths(const libcamera::StreamConfiguration &inCfg, const libcamera::StreamConfiguration &outCfg); From patchwork Tue Sep 7 19:57:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 13751 X-Patchwork-Delegate: umang.jain@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 57FC4BDC71 for ; Tue, 7 Sep 2021 19:57:23 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 21A6769173; Tue, 7 Sep 2021 21:57:23 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="kfzRWD5+"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id F336F69173 for ; Tue, 7 Sep 2021 21:57:21 +0200 (CEST) Received: from perceval.ideasonboard.com (unknown [103.251.226.2]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id CD149499; Tue, 7 Sep 2021 21:57:20 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631044641; bh=3BplGfXY1FTGqvHh7eVzwe/p1KioPek2cTP9UzC7uEE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kfzRWD5+8wGYlM3Fxeb7zE0pO6usdYJ45rVWKp+Ld+1FgF/h9ii4z1DC3PR47ivWg 6yocLjccXhAtba+4H1Qavkw6596rnp2Y6tA7kKyruv0TBApNqiM+TNcta+AARNbzt1 8gh6k68NCYIZcCOesvm49QWC4S9CIKO7rReDlchg= From: Umang Jain To: libcamera-devel@lists.libcamera.org Date: Wed, 8 Sep 2021 01:27:00 +0530 Message-Id: <20210907195704.338079-3-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210907195704.338079-1-umang.jain@ideasonboard.com> References: <20210907195704.338079-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 2/6] android: post_processor: Notify post processing completion status X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We should be able to know if post-processing has been completed successfully or encountered some errors. This commit introduces a Signal<> which when connected, will notify that the post-processing has been completed. The status of PostProcessor::process() will be passed as a PostProcessor::Status argument. The signal will be required when the post-processor is meant to run asynchronously (in subsequent commits). Signed-off-by: Umang Jain --- src/android/jpeg/post_processor_jpeg.cpp | 7 ++++++- src/android/post_processor.h | 8 ++++++++ src/android/yuv/post_processor_yuv.cpp | 4 ++++ 3 files changed, 18 insertions(+), 1 deletion(-) diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp index fe41207a..69f3c5dd 100644 --- a/src/android/jpeg/post_processor_jpeg.cpp +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -87,8 +87,10 @@ int PostProcessorJpeg::process(const FrameBuffer *source, const CameraMetadata &requestMetadata, CameraMetadata *resultMetadata) { - if (!encoder_) + if (!encoder_) { + processComplete.emit(PostProcessor::Status::Failed); return 0; + } ASSERT(destination->numPlanes() == 1); @@ -181,6 +183,7 @@ int PostProcessorJpeg::process(const FrameBuffer *source, exif.data(), quality); if (jpeg_size < 0) { LOG(JPEG, Error) << "Failed to encode stream image"; + processComplete.emit(PostProcessor::Status::Failed); return jpeg_size; } @@ -195,5 +198,7 @@ int PostProcessorJpeg::process(const FrameBuffer *source, /* Update the JPEG result Metadata. */ resultMetadata->addEntry(ANDROID_JPEG_SIZE, jpeg_size); + processComplete.emit(PostProcessor::Status::Success); + return 0; } diff --git a/src/android/post_processor.h b/src/android/post_processor.h index 61dfb6d4..edf1a4bf 100644 --- a/src/android/post_processor.h +++ b/src/android/post_processor.h @@ -7,6 +7,8 @@ #ifndef __ANDROID_POST_PROCESSOR_H__ #define __ANDROID_POST_PROCESSOR_H__ +#include + #include #include @@ -25,6 +27,12 @@ public: CameraBuffer *destination, const CameraMetadata &requestMetadata, CameraMetadata *resultMetadata) = 0; + + enum Status { + Success, + Failed, + }; + libcamera::Signal processComplete; }; #endif /* __ANDROID_POST_PROCESSOR_H__ */ diff --git a/src/android/yuv/post_processor_yuv.cpp b/src/android/yuv/post_processor_yuv.cpp index 0a874886..d1db9216 100644 --- a/src/android/yuv/post_processor_yuv.cpp +++ b/src/android/yuv/post_processor_yuv.cpp @@ -60,6 +60,7 @@ int PostProcessorYuv::process(const FrameBuffer *source, const MappedFrameBuffer sourceMapped(source, MappedFrameBuffer::MapFlag::Read); if (!sourceMapped.isValid()) { LOG(YUV, Error) << "Failed to mmap camera frame buffer"; + processComplete.emit(PostProcessor::Status::Failed); return -EINVAL; } @@ -77,9 +78,12 @@ int PostProcessorYuv::process(const FrameBuffer *source, libyuv::FilterMode::kFilterBilinear); if (ret) { LOG(YUV, Error) << "Failed NV12 scaling: " << ret; + processComplete.emit(PostProcessor::Status::Failed); return -EINVAL; } + processComplete.emit(PostProcessor::Status::Success); + return 0; } From patchwork Tue Sep 7 19:57:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 13752 X-Patchwork-Delegate: umang.jain@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id DA058BDC71 for ; Tue, 7 Sep 2021 19:57:26 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A4B746916E; Tue, 7 Sep 2021 21:57:26 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="kFA1j+5w"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id ECD9C60252 for ; Tue, 7 Sep 2021 21:57:24 +0200 (CEST) Received: from perceval.ideasonboard.com (unknown [103.251.226.2]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id E36C9D88; Tue, 7 Sep 2021 21:57:23 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631044644; bh=LLo3KFcpy9fTufkprpbPc2J3sbizubI4b06W+NKLmnI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kFA1j+5wzAhossj3MuoBO0JKSUPZmNRx88dEXDm7Bdd+I1GNNKZlJcHcgQGkIFhBO eJ+I4DNjMTNbJXwQL0cuDuokAZ2oJe2gWzb++xEJDJ2gUC99yNzoCUgcuJATBcuYeK AKa98epRS2PK9PaXp0VLuD0EiR4RS955SMf06tUU= From: Umang Jain To: libcamera-devel@lists.libcamera.org Date: Wed, 8 Sep 2021 01:27:01 +0530 Message-Id: <20210907195704.338079-4-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210907195704.338079-1-umang.jain@ideasonboard.com> References: <20210907195704.338079-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 3/6] android: camera_stream: Notify process() status with a Signal X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" CameraStream takes care of any post-processing required via the CameraStream::process(). Since the post-processor will be moved to a separate thread (in subsequent commits), the caller of CameraStream::process() should be able to get notified about the status, in order to proceed further for completing the callbacks to android framework. Therefore, chain up to PostProcessor::processComplete signal in its signal handler. Emit the corresponding CameraStream::PostProcessing value according to the PostProcessor::status received in the signal handler. Signed-off-by: Umang Jain --- src/android/camera_stream.cpp | 10 ++++++++++ src/android/camera_stream.h | 13 ++++++++++++- 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/src/android/camera_stream.cpp b/src/android/camera_stream.cpp index 0fed5382..81af53af 100644 --- a/src/android/camera_stream.cpp +++ b/src/android/camera_stream.cpp @@ -81,6 +81,8 @@ int CameraStream::configure() int ret = postProcessor_->configure(configuration(), output); if (ret) return ret; + + postProcessor_->processComplete.connect(this, &CameraStream::handlePostProcessing); } if (allocator_) { @@ -121,6 +123,14 @@ int CameraStream::process(const FrameBuffer *source, return postProcessor_->process(source, &dest, requestMetadata, resultMetadata); } +void CameraStream::handlePostProcessing(PostProcessor::Status status) +{ + if (status == PostProcessor::Status::Success) + processComplete.emit(ProcessStatus::Success); + else + processComplete.emit(ProcessStatus::Failed); +} + FrameBuffer *CameraStream::getBuffer() { if (!allocator_) diff --git a/src/android/camera_stream.h b/src/android/camera_stream.h index 5c232cb6..f9b11e5d 100644 --- a/src/android/camera_stream.h +++ b/src/android/camera_stream.h @@ -13,15 +13,18 @@ #include +#include + #include #include #include #include #include +#include "post_processor.h" + class CameraDevice; class CameraMetadata; -class PostProcessor; class CameraStream { @@ -125,7 +128,15 @@ public: libcamera::FrameBuffer *getBuffer(); void putBuffer(libcamera::FrameBuffer *buffer); + enum ProcessStatus { + Failed, + Success, + }; + libcamera::Signal processComplete; + private: + void handlePostProcessing(PostProcessor::Status status); + CameraDevice *const cameraDevice_; const libcamera::CameraConfiguration *config_; const Type type_; From patchwork Tue Sep 7 19:57:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 13753 X-Patchwork-Delegate: umang.jain@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 6D796BDC71 for ; Tue, 7 Sep 2021 19:57:28 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3AB7A6916D; Tue, 7 Sep 2021 21:57:28 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="nqpNwoQl"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id CBFC369170 for ; Tue, 7 Sep 2021 21:57:26 +0200 (CEST) Received: from perceval.ideasonboard.com (unknown [103.251.226.2]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id BE507499; Tue, 7 Sep 2021 21:57:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631044646; bh=4MV9WVM3H+SvBCE2rQKm8NsrlZsuki7KrEWI8O02328=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nqpNwoQlVc9l9TIIB8xyQMZeecRHDwJyTErT8w1sFZPVwZzxfcw36PiBSPOA61G/p sRJ4I6DrPhKRk0QNVkwTZ73kguS7YegmA3HahrI/mzdepNRYtxdl91k0aJGZfk7zXY Cd5zM4u247QNI0xSJB3GTjxhmdUhZwkOh0x0bM2U= From: Umang Jain To: libcamera-devel@lists.libcamera.org Date: Wed, 8 Sep 2021 01:27:02 +0530 Message-Id: <20210907195704.338079-5-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210907195704.338079-1-umang.jain@ideasonboard.com> References: <20210907195704.338079-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 4/6] android: camera_device: Add a queue for sending capture results X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When a camera capture request completes, the next step is to send the capture results to the framework via process_capture_results(). All the capture associated result metadata is prepared and populated. If any post-processing is required, it will also take place in the same thread (which can be blocking for a subtaintial amount of time) before the results can be sent back to the framework. A follow up commit will move the post-processing to run in a separate thread. In order to do so, there is few bits of groundwork which this patch entails. Mainly, we need to preserve the order of sending the capture results back in which they were queued by the framework to the HAL (i.e. the order is sequential). Hence, we need to add a queue in which capture results can be queued with context. As per this patch, the post-processor still runs synchronously as before, but it will queue up the current capture results with context. Later down the line, the capture results are dequeud in order and sent back to the framework. The context is preserved using Camera3RequestDescriptor utility structure. It has been expanded accordingly to address the needs of the functionality. Signed-off-by: Umang Jain --- src/android/camera_device.cpp | 113 +++++++++++++++++++++++++++++++--- src/android/camera_device.h | 23 +++++++ 2 files changed, 126 insertions(+), 10 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index f2f36f32..9d4ec02e 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -240,6 +240,8 @@ CameraDevice::Camera3RequestDescriptor::Camera3RequestDescriptor( /* Clone the controls associated with the camera3 request. */ settings_ = CameraMetadata(camera3Request->settings); + internalBuffer_ = nullptr; + /* * Create the CaptureRequest, stored as a unique_ptr<> to tie its * lifetime to the descriptor. @@ -989,6 +991,7 @@ int CameraDevice::processCaptureRequest(camera3_capture_request_t *camera3Reques * once it has been processed. */ buffer = cameraStream->getBuffer(); + descriptor.internalBuffer_ = buffer; LOG(HAL, Debug) << ss.str() << " (internal)"; break; } @@ -1148,25 +1151,115 @@ void CameraDevice::requestComplete(Request *request) continue; } + /* + * Save the current context of capture result and queue the + * descriptor before processing the camera stream. + * + * When the processing completes, the descriptor will be + * dequeued and capture results will be sent to the framework. + */ + descriptor.status_ = Camera3RequestDescriptor::NOT_FINISHED; + descriptor.resultMetadata_ = std::move(resultMetadata); + descriptor.captureResult_ = captureResult; + + std::unique_ptr reqDescriptor = + std::make_unique(); + *reqDescriptor = std::move(descriptor); + queuedDescriptor_.push_back(std::move(reqDescriptor)); + Camera3RequestDescriptor *currentDescriptor = queuedDescriptor_.back().get(); + + CameraMetadata *metadata = currentDescriptor->resultMetadata_.get(); + cameraStream->processComplete.connect( + this, [=](CameraStream::ProcessStatus status) { + streamProcessingComplete(cameraStream, status, + metadata); + }); + int ret = cameraStream->process(src, *buffer.buffer, - descriptor.settings_, - resultMetadata.get()); + currentDescriptor->settings_, + metadata); + return; + } + + if (queuedDescriptor_.empty()) { + captureResult.result = resultMetadata->get(); + callbacks_->process_capture_result(callbacks_, &captureResult); + } else { + /* + * Previous capture results waiting to be sent to framework + * hence, queue the current capture results as well. After that, + * check if any results are ready to be sent back to the + * framework. + */ + descriptor.status_ = Camera3RequestDescriptor::FINISHED_SUCCESS; + descriptor.resultMetadata_ = std::move(resultMetadata); + descriptor.captureResult_ = captureResult; + std::unique_ptr reqDescriptor = + std::make_unique(); + *reqDescriptor = std::move(descriptor); + queuedDescriptor_.push_back(std::move(reqDescriptor)); + + sendQueuedCaptureResults(); + } +} + +void CameraDevice::streamProcessingComplete(CameraStream *cameraStream, + CameraStream::ProcessStatus status, + CameraMetadata *resultMetadata) +{ + for (auto &d : queuedDescriptor_) { + if (d->resultMetadata_.get() != resultMetadata) + continue; + /* * Return the FrameBuffer to the CameraStream now that we're * done processing it. */ if (cameraStream->type() == CameraStream::Type::Internal) - cameraStream->putBuffer(src); - - if (ret) { - buffer.status = CAMERA3_BUFFER_STATUS_ERROR; - notifyError(descriptor.frameNumber_, buffer.stream, - CAMERA3_MSG_ERROR_BUFFER); + cameraStream->putBuffer(d->internalBuffer_); + + if (status == CameraStream::ProcessStatus::Success) { + d->status_ = Camera3RequestDescriptor::FINISHED_SUCCESS; + } else { + d->status_ = Camera3RequestDescriptor::FINISHED_FAILED; + d->captureResult_.partial_result = 0; + for (camera3_stream_buffer_t &buffer : d->buffers_) { + CameraStream *cs = static_cast(buffer.stream->priv); + + if (cs->camera3Stream().format != HAL_PIXEL_FORMAT_BLOB) + continue; + + buffer.status = CAMERA3_BUFFER_STATUS_ERROR; + notifyError(d->frameNumber_, buffer.stream, + CAMERA3_MSG_ERROR_BUFFER); + } } + + break; } - captureResult.result = resultMetadata->get(); - callbacks_->process_capture_result(callbacks_, &captureResult); + /* + * Send back capture results to the framework by inspecting the queue. + * The framework can defer queueing further requests to the HAL (via + * process_capture_request) unless until it receives the capture results + * for already queued requests. + */ + sendQueuedCaptureResults(); +} + +void CameraDevice::sendQueuedCaptureResults() +{ + while (!queuedDescriptor_.empty()) { + std::unique_ptr &d = queuedDescriptor_.front(); + if (d->status_ != Camera3RequestDescriptor::NOT_FINISHED) { + d->captureResult_.result = d->resultMetadata_->get(); + callbacks_->process_capture_result(callbacks_, + &(d->captureResult_)); + queuedDescriptor_.pop_front(); + } else { + break; + } + } } std::string CameraDevice::logPrefix() const diff --git a/src/android/camera_device.h b/src/android/camera_device.h index a5576927..36425773 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -7,6 +7,7 @@ #ifndef __ANDROID_CAMERA_DEVICE_H__ #define __ANDROID_CAMERA_DEVICE_H__ +#include #include #include #include @@ -83,6 +84,21 @@ private: std::vector> frameBuffers_; CameraMetadata settings_; std::unique_ptr request_; + + /* + * Below are utility placeholders used when a capture result + * needs to be queued before completion via + * process_capture_result() callback. + */ + enum completionStatus { + NOT_FINISHED, + FINISHED_SUCCESS, + FINISHED_FAILED, + }; + std::unique_ptr resultMetadata_; + camera3_capture_result_t captureResult_; + libcamera::FrameBuffer *internalBuffer_; + completionStatus status_; }; enum class State { @@ -105,6 +121,11 @@ private: std::unique_ptr getResultMetadata( const Camera3RequestDescriptor &descriptor) const; + void sendQueuedCaptureResults(); + void streamProcessingComplete(CameraStream *stream, + CameraStream::ProcessStatus status, + CameraMetadata *resultMetadata); + unsigned int id_; camera3_device_t camera3Device_; @@ -125,6 +146,8 @@ private: libcamera::Mutex descriptorsMutex_; /* Protects descriptors_. */ std::map descriptors_; + std::deque> queuedDescriptor_; + std::string maker_; std::string model_; From patchwork Tue Sep 7 19:57:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 13754 X-Patchwork-Delegate: umang.jain@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 083B0BDC71 for ; Tue, 7 Sep 2021 19:57:31 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id BE4E56916D; Tue, 7 Sep 2021 21:57:30 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ZlP4LIbM"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C1E8C60251 for ; Tue, 7 Sep 2021 21:57:28 +0200 (CEST) Received: from perceval.ideasonboard.com (unknown [103.251.226.2]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id A92DE499; Tue, 7 Sep 2021 21:57:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631044648; bh=PFpwrkJ/ZQlOblUyt/bDW1RxI2/xWFWSUbL08yj5qUA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZlP4LIbMR1+3VtPs1ehmvcd1GWy+MYwP0GK651KXGPf4lg6ytRGZpqXNqL8ThPq/F F9KhJIlEMEQaav3aKNo2OhaOT4rCaJTE4jvIfI7utA5/nXXdGvcDlRdO9ovG+OBn1G tPe1L/g/Ky87uYH++jAQ6ya1vbyAFJDKQl+TNgNE= From: Umang Jain To: libcamera-devel@lists.libcamera.org Date: Wed, 8 Sep 2021 01:27:03 +0530 Message-Id: <20210907195704.338079-6-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210907195704.338079-1-umang.jain@ideasonboard.com> References: <20210907195704.338079-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 5/6] android: camera_device: Move buffer mapping for post-processing X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Buffer mapping currently happens in CameraStream::process() for post-processing. However, it can be easily be moved to CameraDevice just before the call to CameraStream::process(). The reason to do so is that we can hold the mapped destination buffer pointer as a part of Camera3RequestDescriptor. The Camera3RequestDescriptor already saves other post-processing related context, to enable async post processing. Signed-off-by: Umang Jain --- src/android/camera_device.cpp | 20 +++++++++++++++++++- src/android/camera_device.h | 1 + src/android/camera_stream.cpp | 16 ++-------------- src/android/camera_stream.h | 2 +- 4 files changed, 23 insertions(+), 16 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index 9d4ec02e..8741678d 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -1162,6 +1162,17 @@ void CameraDevice::requestComplete(Request *request) descriptor.resultMetadata_ = std::move(resultMetadata); descriptor.captureResult_ = captureResult; + /* \todo Buffer mapping should be moved to a separate thread? */ + const StreamConfiguration &output = cameraStream->configuration(); + descriptor.destBuffer_ = std::make_unique(*buffer.buffer, + formats::MJPEG, + output.size, + PROT_READ | PROT_WRITE); + if (!descriptor.destBuffer_->isValid()) { + LOG(HAL, Error) << "Failed to map android blob buffer"; + return; + } + std::unique_ptr reqDescriptor = std::make_unique(); *reqDescriptor = std::move(descriptor); @@ -1175,7 +1186,7 @@ void CameraDevice::requestComplete(Request *request) metadata); }); - int ret = cameraStream->process(src, *buffer.buffer, + int ret = cameraStream->process(src, currentDescriptor->destBuffer_.get(), currentDescriptor->settings_, metadata); return; @@ -1255,6 +1266,13 @@ void CameraDevice::sendQueuedCaptureResults() d->captureResult_.result = d->resultMetadata_->get(); callbacks_->process_capture_result(callbacks_, &(d->captureResult_)); + /* + * reset() here drops the error logs in + * https://paste.debian.net/1210728/ significantly. + * It still happens intermittently so comment the reset() + * out to see persistantly. + */ + d->destBuffer_.reset(); queuedDescriptor_.pop_front(); } else { break; diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 36425773..330c69a3 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -96,6 +96,7 @@ private: FINISHED_FAILED, }; std::unique_ptr resultMetadata_; + std::unique_ptr destBuffer_; camera3_capture_result_t captureResult_; libcamera::FrameBuffer *internalBuffer_; completionStatus status_; diff --git a/src/android/camera_stream.cpp b/src/android/camera_stream.cpp index 81af53af..1939cb9c 100644 --- a/src/android/camera_stream.cpp +++ b/src/android/camera_stream.cpp @@ -101,26 +101,14 @@ int CameraStream::configure() } int CameraStream::process(const FrameBuffer *source, - buffer_handle_t camera3Dest, + CameraBuffer *destBuffer, const CameraMetadata &requestMetadata, CameraMetadata *resultMetadata) { if (!postProcessor_) return 0; - /* - * \todo Buffer mapping and processing should be moved to a - * separate thread. - */ - const StreamConfiguration &output = configuration(); - CameraBuffer dest(camera3Dest, formats::MJPEG, output.size, - PROT_READ | PROT_WRITE); - if (!dest.isValid()) { - LOG(HAL, Error) << "Failed to map android blob buffer"; - return -EINVAL; - } - - return postProcessor_->process(source, &dest, requestMetadata, resultMetadata); + return postProcessor_->process(source, destBuffer, requestMetadata, resultMetadata); } void CameraStream::handlePostProcessing(PostProcessor::Status status) diff --git a/src/android/camera_stream.h b/src/android/camera_stream.h index f9b11e5d..ad03e4cf 100644 --- a/src/android/camera_stream.h +++ b/src/android/camera_stream.h @@ -122,7 +122,7 @@ public: int configure(); int process(const libcamera::FrameBuffer *source, - buffer_handle_t camera3Dest, + CameraBuffer *destBuffer, const CameraMetadata &requestMetadata, CameraMetadata *resultMetadata); libcamera::FrameBuffer *getBuffer(); From patchwork Tue Sep 7 19:57:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 13755 X-Patchwork-Delegate: umang.jain@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9F56EBDC71 for ; Tue, 7 Sep 2021 19:57:32 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 6F91B6916F; Tue, 7 Sep 2021 21:57:32 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="SwHjuvOC"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B4C646916C for ; Tue, 7 Sep 2021 21:57:30 +0200 (CEST) Received: from perceval.ideasonboard.com (unknown [103.251.226.2]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id A198E499; Tue, 7 Sep 2021 21:57:29 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1631044650; bh=izciIO8uRaoT5jeJakvr4co67CIHwdBvEhRVlTOIgYo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SwHjuvOCHqNEkxf52JoC7AdTrd5C2o1CLwjkOTJHd2bAO2ISNvQyke/IqU2P6H7us VZ56Umaeyy0dgJCXWiyx2VEG+J386geqB0Wbw7StqaVnUN20pPi+4RaCgzKumcL/yQ FIRHLQt9FtTXgSQqxOqMHnHyDomZrfcBL1pDLWds= From: Umang Jain To: libcamera-devel@lists.libcamera.org Date: Wed, 8 Sep 2021 01:27:04 +0530 Message-Id: <20210907195704.338079-7-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210907195704.338079-1-umang.jain@ideasonboard.com> References: <20210907195704.338079-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v1 6/6] android: camera_stream: Run post-processor in a separate thread X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" In CameraStream, introduce a new private PostProcessorWorker class derived from Object. A instance of PostProcessorWorker is moved to a separate thread instance which will be responsible to run the post-processor. Running PostProcessor asynchronously should entail that all the data context needed by the PostProcessor should remain valid for the entire duration of its run. Most of the context preserving part has been addressed inthe previous patch, we just need to ensure the source framebuffer data that comes via Camera::Request, should remain valid for the entire duration of post-processing running asynchronously. In order to so, we maintain a separate copy of the framebuffer data and add it to the Camera3RequestDescriptor in which we preserve rest of the context. Signed-off-by: Umang Jain --- src/android/camera_device.cpp | 18 +++++++++++++++++- src/android/camera_device.h | 4 ++++ src/android/camera_stream.cpp | 15 ++++++++++++++- src/android/camera_stream.h | 26 ++++++++++++++++++++++++++ 4 files changed, 61 insertions(+), 2 deletions(-) diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index 8741678d..5a442fb9 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -1173,6 +1173,21 @@ void CameraDevice::requestComplete(Request *request) return; } + FrameBuffer *source = src; + if (cameraStream->type() != CameraStream::Type::Internal) { + /* + * The source buffer is owned by Request object which + * can be reused for future capture request. Since + * post-processor will run asynchrnously, we need to + * copy the source buffer and use it as source buffer. + */ + for (const auto &plane : src->planes()) + descriptor.srcPlanes_.push_back(plane); + descriptor.srcFramebuffer_ = + std::make_unique(descriptor.srcPlanes_); + source = descriptor.srcFramebuffer_.get(); + } + std::unique_ptr reqDescriptor = std::make_unique(); *reqDescriptor = std::move(descriptor); @@ -1186,7 +1201,7 @@ void CameraDevice::requestComplete(Request *request) metadata); }); - int ret = cameraStream->process(src, currentDescriptor->destBuffer_.get(), + int ret = cameraStream->process(source, currentDescriptor->destBuffer_.get(), currentDescriptor->settings_, metadata); return; @@ -1260,6 +1275,7 @@ void CameraDevice::streamProcessingComplete(CameraStream *cameraStream, void CameraDevice::sendQueuedCaptureResults() { + MutexLocker lock(queuedDescriptorsMutex_); while (!queuedDescriptor_.empty()) { std::unique_ptr &d = queuedDescriptor_.front(); if (d->status_ != Camera3RequestDescriptor::NOT_FINISHED) { diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 330c69a3..87ae00ab 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -100,6 +100,9 @@ private: camera3_capture_result_t captureResult_; libcamera::FrameBuffer *internalBuffer_; completionStatus status_; + + std::unique_ptr srcFramebuffer_; + std::vector srcPlanes_; }; enum class State { @@ -147,6 +150,7 @@ private: libcamera::Mutex descriptorsMutex_; /* Protects descriptors_. */ std::map descriptors_; + libcamera::Mutex queuedDescriptorsMutex_; /* Protects queuedDescriptor_. */ std::deque> queuedDescriptor_; std::string maker_; diff --git a/src/android/camera_stream.cpp b/src/android/camera_stream.cpp index 1939cb9c..6a2c1f5b 100644 --- a/src/android/camera_stream.cpp +++ b/src/android/camera_stream.cpp @@ -55,6 +55,15 @@ CameraStream::CameraStream(CameraDevice *const cameraDevice, * is what we instantiate here. */ postProcessor_ = std::make_unique(cameraDevice_); + ppWorker_ = std::make_unique(postProcessor_.get()); + + thread_ = std::make_unique(); + ppWorker_->moveToThread(thread_.get()); + /* + * \todo: Class is MoveConstructible, so where to stop thread + * if we don't user-defined destructor? + */ + thread_->start(); } if (type == Type::Internal) { @@ -108,7 +117,11 @@ int CameraStream::process(const FrameBuffer *source, if (!postProcessor_) return 0; - return postProcessor_->process(source, destBuffer, requestMetadata, resultMetadata); + ppWorker_->invokeMethod(&PostProcessorWorker::process, + ConnectionTypeQueued, source, destBuffer, + requestMetadata, resultMetadata); + + return 0; } void CameraStream::handlePostProcessing(PostProcessor::Status status) diff --git a/src/android/camera_stream.h b/src/android/camera_stream.h index ad03e4cf..d8ce0972 100644 --- a/src/android/camera_stream.h +++ b/src/android/camera_stream.h @@ -13,7 +13,9 @@ #include +#include #include +#include #include #include @@ -23,6 +25,7 @@ #include "post_processor.h" +class CameraBuffer; class CameraDevice; class CameraMetadata; @@ -135,6 +138,27 @@ public: libcamera::Signal processComplete; private: + class PostProcessorWorker : public libcamera::Object + { + public: + PostProcessorWorker(PostProcessor *postProcessor) + { + postProcessor_ = postProcessor; + } + + void process(const libcamera::FrameBuffer *source, + CameraBuffer *destination, + const CameraMetadata &requestMetadata, + CameraMetadata *resultMetadata) + { + postProcessor_->process(source, destination, + requestMetadata, resultMetadata); + } + + private: + PostProcessor *postProcessor_; + }; + void handlePostProcessing(PostProcessor::Status status); CameraDevice *const cameraDevice_; @@ -151,6 +175,8 @@ private: */ std::unique_ptr mutex_; std::unique_ptr postProcessor_; + std::unique_ptr ppWorker_; + std::unique_ptr thread_; }; #endif /* __ANDROID_CAMERA_STREAM__ */