From patchwork Fri Oct 16 05:37:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 10074 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5FB63BE174 for ; Fri, 16 Oct 2020 05:38:06 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2D42460E5A; Fri, 16 Oct 2020 07:38:06 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=uajain.com header.i=@uajain.com header.b="HTDZhIxF"; dkim-atps=neutral Received: from mail.uajain.com (static.126.159.217.95.clients.your-server.de [95.217.159.126]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 531EB60354 for ; Fri, 16 Oct 2020 07:38:04 +0200 (CEST) From: Umang Jain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=uajain.com; s=mail; t=1602826683; bh=KTlKugmAmfObqwGs37aWbkUtA/JAOec/2eR28mqa27c=; h=From:To:Cc:Subject:In-Reply-To:References; b=HTDZhIxFxjzREwaQ8dNYPmzinKUObaXkrSQAKlpa8u82y5Q9tOPSCZGz5vrjERDfv ZBKiAg+6fNwH+flcGWhh3eZtSrb6ksxBpfdK4hPDIZykaff5SNd0awBaniVia/jhID IeuJayPzjdwpQXtbYK5pqyiTB/FhWzyDVphtbcKjRVOQ5RLFflmHLzNSQkYNdUZFIR jLN/wUm3gz1J1mpZ7rlultfWVV3tD/G6FHcInYsxexwdqOWnzjPU6JiYSk/E8m59je fy46LdmCjWWEBNBvEUrvawMUr9ScNYIps1dk+NYEgM3jtg2l99i5IkUodffgfHJa2b /orSq2curNyRA== To: libcamera-devel@lists.libcamera.org Date: Fri, 16 Oct 2020 11:07:53 +0530 Message-Id: <20201016053754.17251-2-email@uajain.com> In-Reply-To: <20201016053754.17251-1-email@uajain.com> References: <20201016053754.17251-1-email@uajain.com> Mime-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 1/2] android: post_processor: Introduce a PostProcessor interface X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Introduce a PostProcessor interface for the streams that require any kind of processing (refer to CameraStream::Type) for their consumption by the HAL layer. The PostProcessor interface can be configured via configure() and the actual processing can be initiated using process(). The post-processing layer can be extended to have multiple post processors for various stream configurations. As of now, we only have one post processor (JPEG), hence the subsequent commit will port its function to this interface. Signed-off-by: Umang Jain Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart Reviewed-by: Hirokazu Honda --- src/android/post_processor.h | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 src/android/post_processor.h diff --git a/src/android/post_processor.h b/src/android/post_processor.h new file mode 100644 index 0000000..a891c43 --- /dev/null +++ b/src/android/post_processor.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * post_processor.h - CameraStream Post Processing Interface + */ +#ifndef __ANDROID_POST_PROCESSOR_H__ +#define __ANDROID_POST_PROCESSOR_H__ + +#include +#include +#include + +class CameraMetadata; + +class PostProcessor +{ +public: + virtual ~PostProcessor() {} + + virtual int configure(const libcamera::StreamConfiguration &inCfg, + const libcamera::StreamConfiguration &outCfg) = 0; + virtual int process(const libcamera::FrameBuffer *source, + const libcamera::Span &destination, + CameraMetadata *metadata) = 0; +}; + +#endif /* __ANDROID_POST_PROCESSOR_H__ */ From patchwork Fri Oct 16 05:37:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 10075 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id B1E1BBE174 for ; Fri, 16 Oct 2020 05:38:07 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 803F2610A9; Fri, 16 Oct 2020 07:38:07 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=uajain.com header.i=@uajain.com header.b="SOmiCS0f"; dkim-atps=neutral Received: from mail.uajain.com (static.126.159.217.95.clients.your-server.de [95.217.159.126]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7AB2960E36 for ; Fri, 16 Oct 2020 07:38:06 +0200 (CEST) From: Umang Jain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=uajain.com; s=mail; t=1602826686; bh=oNfJQO434xfshRHxN2qQGUFtzj1ujbdxu2ASkjcpD6U=; h=From:To:Cc:Subject:In-Reply-To:References; b=SOmiCS0fau0nzAuGu1cIgBQINSZRkH6Ktf3ZPOUWa5Jmr+RogX8K2mCOy/rh9uyzV Vk53srARY5rSERlpIvzAl+olZRPUkgQ2coR8agMQd3zxWESejSf05haGcVzVe3AFJa bcTS7WpJeMOeXa0z2qghdJDbWAGWGvRbaNmaLSPT0roElc8wAVTBaG+3iO4pLbNhUh ifkvHZcsmt19/T2OVezaKbgQu2l1wAyOKlaKSqueGl/rhtrJej+YAlTkkPc1TbHYd+ dNLiXxnfHOOSUbnlIfjoIHNaumHjNUMwG9WYhKZO61WmVaPAGqsHGoq0ngI7J9YtMp 2CWfwMkjkDnsw== To: libcamera-devel@lists.libcamera.org Date: Fri, 16 Oct 2020 11:07:54 +0530 Message-Id: <20201016053754.17251-3-email@uajain.com> In-Reply-To: <20201016053754.17251-1-email@uajain.com> References: <20201016053754.17251-1-email@uajain.com> Mime-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 2/2] android: jpeg: Port to PostProcessor interface X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Port the CameraStream's JPEG-encoding bits to PostProcessorJpeg. This encapsulates the encoder and EXIF generation code into the PostProcessorJpeg layer and removes these specifics related to JPEG, from the CameraStream itself. Signed-off-by: Umang Jain Tested-by: Kieran Bingham Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/android/camera_device.cpp | 1 + src/android/camera_stream.cpp | 77 ++++------------- src/android/camera_stream.h | 4 +- src/android/jpeg/encoder_libjpeg.cpp | 2 +- src/android/jpeg/post_processor_jpeg.cpp | 105 +++++++++++++++++++++++ src/android/jpeg/post_processor_jpeg.h | 36 ++++++++ src/android/meson.build | 1 + 7 files changed, 164 insertions(+), 62 deletions(-) create mode 100644 src/android/jpeg/post_processor_jpeg.cpp create mode 100644 src/android/jpeg/post_processor_jpeg.h diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index 0983232..d706cf4 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -7,6 +7,7 @@ #include "camera_device.h" #include "camera_ops.h" +#include "post_processor.h" #include #include diff --git a/src/android/camera_stream.cpp b/src/android/camera_stream.cpp index d8e45c2..1b8afa8 100644 --- a/src/android/camera_stream.cpp +++ b/src/android/camera_stream.cpp @@ -9,9 +9,9 @@ #include "camera_device.h" #include "camera_metadata.h" -#include "jpeg/encoder.h" -#include "jpeg/encoder_libjpeg.h" -#include "jpeg/exif.h" +#include "jpeg/post_processor_jpeg.h" + +#include using namespace libcamera; @@ -45,8 +45,15 @@ CameraStream::CameraStream(CameraDevice *cameraDevice, Type type, { config_ = cameraDevice_->cameraConfiguration(); - if (type_ == Type::Internal || type_ == Type::Mapped) - encoder_ = std::make_unique(); + if (type_ == Type::Internal || type_ == Type::Mapped) { + /* + * \todo There might be multiple post-processors. The logic + * which should be instantiated here, is deferred for the + * future. For now, we only have PostProcessorJpeg and that + * is what we instantiate here. + */ + postProcessor_ = std::make_unique(cameraDevice_); + } if (type == Type::Internal) { allocator_ = std::make_unique(cameraDevice_->camera()); @@ -66,8 +73,10 @@ Stream *CameraStream::stream() const int CameraStream::configure() { - if (encoder_) { - int ret = encoder_->configure(configuration()); + if (postProcessor_) { + StreamConfiguration output = configuration(); + output.pixelFormat = formats::MJPEG; + int ret = postProcessor_->configure(configuration(), output); if (ret) return ret; } @@ -90,60 +99,10 @@ int CameraStream::configure() int CameraStream::process(const libcamera::FrameBuffer &source, MappedCamera3Buffer *dest, CameraMetadata *metadata) { - if (!encoder_) + if (!postProcessor_) return 0; - /* Set EXIF metadata for various tags. */ - Exif exif; - /* \todo Set Make and Model from external vendor tags. */ - exif.setMake("libcamera"); - exif.setModel("cameraModel"); - exif.setOrientation(cameraDevice_->orientation()); - exif.setSize(configuration().size); - /* - * We set the frame's EXIF timestamp as the time of encode. - * Since the precision we need for EXIF timestamp is only one - * second, it is good enough. - */ - exif.setTimestamp(std::time(nullptr)); - if (exif.generate() != 0) - LOG(HAL, Error) << "Failed to generate valid EXIF data"; - - int jpeg_size = encoder_->encode(&source, dest->maps()[0], exif.data()); - if (jpeg_size < 0) { - LOG(HAL, Error) << "Failed to encode stream image"; - return jpeg_size; - } - - /* - * Fill in the JPEG blob header. - * - * The mapped size of the buffer is being returned as - * substantially larger than the requested JPEG_MAX_SIZE - * (which is referenced from maxJpegBufferSize_). Utilise - * this static size to ensure the correct offset of the blob is - * determined. - * - * \todo Investigate if the buffer size mismatch is an issue or - * expected behaviour. - */ - uint8_t *resultPtr = dest->maps()[0].data() + - cameraDevice_->maxJpegBufferSize() - - sizeof(struct camera3_jpeg_blob); - auto *blob = reinterpret_cast(resultPtr); - blob->jpeg_blob_id = CAMERA3_JPEG_BLOB_ID; - blob->jpeg_size = jpeg_size; - - /* Update the JPEG result Metadata. */ - metadata->addEntry(ANDROID_JPEG_SIZE, &jpeg_size, 1); - - const uint32_t jpeg_quality = 95; - metadata->addEntry(ANDROID_JPEG_QUALITY, &jpeg_quality, 1); - - const uint32_t jpeg_orientation = 0; - metadata->addEntry(ANDROID_JPEG_ORIENTATION, &jpeg_orientation, 1); - - return 0; + return postProcessor_->process(&source, dest->maps()[0], metadata); } FrameBuffer *CameraStream::getBuffer() diff --git a/src/android/camera_stream.h b/src/android/camera_stream.h index d3925fb..c367a5f 100644 --- a/src/android/camera_stream.h +++ b/src/android/camera_stream.h @@ -19,10 +19,10 @@ #include #include -class Encoder; class CameraDevice; class CameraMetadata; class MappedCamera3Buffer; +class PostProcessor; class CameraStream { @@ -130,7 +130,6 @@ private: camera3_stream_t *camera3Stream_; unsigned int index_; - std::unique_ptr encoder_; std::unique_ptr allocator_; std::vector buffers_; /* @@ -138,6 +137,7 @@ private: * an std::vector in CameraDevice. */ std::unique_ptr mutex_; + std::unique_ptr postProcessor_; }; #endif /* __ANDROID_CAMERA_STREAM__ */ diff --git a/src/android/jpeg/encoder_libjpeg.cpp b/src/android/jpeg/encoder_libjpeg.cpp index a77f5b2..8995ba5 100644 --- a/src/android/jpeg/encoder_libjpeg.cpp +++ b/src/android/jpeg/encoder_libjpeg.cpp @@ -25,7 +25,7 @@ using namespace libcamera; -LOG_DEFINE_CATEGORY(JPEG) +LOG_DECLARE_CATEGORY(JPEG); namespace { diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp new file mode 100644 index 0000000..753c28e --- /dev/null +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -0,0 +1,105 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * post_processor_jpeg.cpp - JPEG Post Processor + */ + +#include "post_processor_jpeg.h" + +#include "../camera_device.h" +#include "../camera_metadata.h" +#include "encoder_libjpeg.h" +#include "exif.h" + +#include + +#include "libcamera/internal/log.h" + +using namespace libcamera; + +LOG_DEFINE_CATEGORY(JPEG); + +PostProcessorJpeg::PostProcessorJpeg(CameraDevice *device) + : cameraDevice_(device) +{ +} + +int PostProcessorJpeg::configure(const StreamConfiguration &inCfg, + const StreamConfiguration &outCfg) +{ + if (inCfg.size != outCfg.size) { + LOG(JPEG, Error) << "Mismatch of input and output stream sizes"; + return -EINVAL; + } + + if (outCfg.pixelFormat != formats::MJPEG) { + LOG(JPEG, Error) << "Output stream pixel format is not JPEG"; + return -EINVAL; + } + + streamSize_ = outCfg.size; + encoder_ = std::make_unique(); + + return encoder_->configure(inCfg); +} + +int PostProcessorJpeg::process(const libcamera::FrameBuffer *source, + const libcamera::Span &destination, + CameraMetadata *metadata) +{ + if (!encoder_) + return 0; + + /* Set EXIF metadata for various tags. */ + Exif exif; + /* \todo Set Make and Model from external vendor tags. */ + exif.setMake("libcamera"); + exif.setModel("cameraModel"); + exif.setOrientation(cameraDevice_->orientation()); + exif.setSize(streamSize_); + /* + * We set the frame's EXIF timestamp as the time of encode. + * Since the precision we need for EXIF timestamp is only one + * second, it is good enough. + */ + exif.setTimestamp(std::time(nullptr)); + if (exif.generate() != 0) + LOG(JPEG, Error) << "Failed to generate valid EXIF data"; + + int jpeg_size = encoder_->encode(source, destination, exif.data()); + if (jpeg_size < 0) { + LOG(JPEG, Error) << "Failed to encode stream image"; + return jpeg_size; + } + + /* + * Fill in the JPEG blob header. + * + * The mapped size of the buffer is being returned as + * substantially larger than the requested JPEG_MAX_SIZE + * (which is referenced from maxJpegBufferSize_). Utilise + * this static size to ensure the correct offset of the blob is + * determined. + * + * \todo Investigate if the buffer size mismatch is an issue or + * expected behaviour. + */ + uint8_t *resultPtr = destination.data() + + cameraDevice_->maxJpegBufferSize() - + sizeof(struct camera3_jpeg_blob); + auto *blob = reinterpret_cast(resultPtr); + blob->jpeg_blob_id = CAMERA3_JPEG_BLOB_ID; + blob->jpeg_size = jpeg_size; + + /* Update the JPEG result Metadata. */ + metadata->addEntry(ANDROID_JPEG_SIZE, &jpeg_size, 1); + + const uint32_t jpeg_quality = 95; + metadata->addEntry(ANDROID_JPEG_QUALITY, &jpeg_quality, 1); + + const uint32_t jpeg_orientation = 0; + metadata->addEntry(ANDROID_JPEG_ORIENTATION, &jpeg_orientation, 1); + + return 0; +} diff --git a/src/android/jpeg/post_processor_jpeg.h b/src/android/jpeg/post_processor_jpeg.h new file mode 100644 index 0000000..62c8650 --- /dev/null +++ b/src/android/jpeg/post_processor_jpeg.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * post_processor_jpeg.h - JPEG Post Processor + */ +#ifndef __ANDROID_POST_PROCESSOR_JPEG_H__ +#define __ANDROID_POST_PROCESSOR_JPEG_H__ + +#include "../post_processor.h" + +#include + +#include "libcamera/internal/buffer.h" + +class Encoder; +class CameraDevice; + +class PostProcessorJpeg : public PostProcessor +{ +public: + PostProcessorJpeg(CameraDevice *device); + + int configure(const libcamera::StreamConfiguration &incfg, + const libcamera::StreamConfiguration &outcfg) override; + int process(const libcamera::FrameBuffer *source, + const libcamera::Span &destination, + CameraMetadata *metadata) override; + +private: + CameraDevice *cameraDevice_; + std::unique_ptr encoder_; + libcamera::Size streamSize_; +}; + +#endif /* __ANDROID_POST_PROCESSOR_JPEG_H__ */ diff --git a/src/android/meson.build b/src/android/meson.build index b2b2293..5a01bea 100644 --- a/src/android/meson.build +++ b/src/android/meson.build @@ -24,6 +24,7 @@ android_hal_sources = files([ 'camera_worker.cpp', 'jpeg/encoder_libjpeg.cpp', 'jpeg/exif.cpp', + 'jpeg/post_processor_jpeg.cpp', ]) android_camera_metadata_sources = files([