From patchwork Thu Oct 15 17:14:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 10058 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 60ED3BE905 for ; Thu, 15 Oct 2020 17:15:08 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 30C2F60E8D; Thu, 15 Oct 2020 19:15:08 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=uajain.com header.i=@uajain.com header.b="JkujqYzL"; dkim-atps=neutral Received: from mail.uajain.com (static.126.159.217.95.clients.your-server.de [95.217.159.126]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4B90460E36 for ; Thu, 15 Oct 2020 19:15:06 +0200 (CEST) From: Umang Jain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=uajain.com; s=mail; t=1602782105; bh=U5d5s+uqvZ20yOYM5r4Z4ntu2jEJ9Hr/NgTrBYH88Js=; h=From:To:Cc:Subject:In-Reply-To:References; b=JkujqYzLB0qm3FI0s7YW85v120UV5rXWpSod1Hy/ia/m3uaDgtGOLwsNYEIrpjG4w BozfPKI902AEa86pyLsLvJgCwxaHiDhhm7jSR7iSfMrVjy0f9NPWHiD5x9/H59QjJP b6dFeoWi0+Qm2QDfvsmSAiZL9Gu9YNv+SAwHkh5jks6kcc2yTXLUOP3sVPs/1bTmXk JoVnjQtVpDYPKvX9BQke39A6RrlAft0oHZEyLzcWY12IJ6fkyZFRtxFx6cTg/iMncX 2hQbopGDlusWXDce+u8Mle6U5zSjpVp3h8E+2PR2afRWCxVTUOq4/5PZjRz6hsfVkz mVxGbZWkw7L7Q== To: libcamera-devel@lists.libcamera.org Date: Thu, 15 Oct 2020 22:44:55 +0530 Message-Id: <20201015171457.75678-2-email@uajain.com> In-Reply-To: <20201015171457.75678-1-email@uajain.com> References: <20201015171457.75678-1-email@uajain.com> Mime-Version: 1.0 Subject: [libcamera-devel] [PATCH 1/3] android: post_processor: Introduce a PostProcessor interface X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Introduce a PostProcessor interface for the streams that require any kind of processing (refer to CameraStream::Type) for their consumption by the HAL layer. The PostProcessor interface can be configured via configure() and the actual processing can be initiated using process(). The post-processing layer can be extended to have multiple post processors for various stream configurations. As of now, we only have one post processor (JPEG), hence the subsequent commit will port its function to this interface. Signed-off-by: Umang Jain Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/android/post_processor.h | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 src/android/post_processor.h diff --git a/src/android/post_processor.h b/src/android/post_processor.h new file mode 100644 index 0000000..a891c43 --- /dev/null +++ b/src/android/post_processor.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * post_processor.h - CameraStream Post Processing Interface + */ +#ifndef __ANDROID_POST_PROCESSOR_H__ +#define __ANDROID_POST_PROCESSOR_H__ + +#include +#include +#include + +class CameraMetadata; + +class PostProcessor +{ +public: + virtual ~PostProcessor() {} + + virtual int configure(const libcamera::StreamConfiguration &inCfg, + const libcamera::StreamConfiguration &outCfg) = 0; + virtual int process(const libcamera::FrameBuffer *source, + const libcamera::Span &destination, + CameraMetadata *metadata) = 0; +}; + +#endif /* __ANDROID_POST_PROCESSOR_H__ */ From patchwork Thu Oct 15 17:14:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 10059 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id BF5D6BE905 for ; Thu, 15 Oct 2020 17:15:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8443261075; Thu, 15 Oct 2020 19:15:09 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=uajain.com header.i=@uajain.com header.b="mKlDsauU"; dkim-atps=neutral Received: from mail.uajain.com (static.126.159.217.95.clients.your-server.de [95.217.159.126]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id ADDB060E5A for ; Thu, 15 Oct 2020 19:15:08 +0200 (CEST) From: Umang Jain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=uajain.com; s=mail; t=1602782108; bh=6q7kEJduKJ/0bhn70+8yYyh1yVDMiVD9Or1GarsLOcg=; h=From:To:Cc:Subject:In-Reply-To:References; b=mKlDsauU7VAhzW7ecykbwqeqwUCvZMH+dPBBIFKVwm5yIYabVZmWC5NveRCrnkZFG N5a5j9tKLifN+EspMDE8nMcpvLXlRrEL9fkghN2xJqnj+1ALa6rhyjlf1qvZZTjAUw zntcUL4lJov5DPur9k7WdDzlkHTcxxT0rUNX4i+Fsj44YUVIHAUNHymtg06vJfXFeU 7p2aHb7sd+JHw7coyAPDpgHOKSjoFT9mGg/3Q6XcfI5RNXt+ngXhlwj5Pm/cWWxQcY hJaefPfQmMjhp8yXTQ1BGTv8pC+Q+f3Z1cu9AHKDogEPxJtvuVI/Ig8JclCMlCDmvo riN9IwljR8LpQ== To: libcamera-devel@lists.libcamera.org Date: Thu, 15 Oct 2020 22:44:56 +0530 Message-Id: <20201015171457.75678-3-email@uajain.com> In-Reply-To: <20201015171457.75678-1-email@uajain.com> References: <20201015171457.75678-1-email@uajain.com> Mime-Version: 1.0 Subject: [libcamera-devel] [PATCH 2/3] android: jpeg: Port to PostProcessor interface X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Port the CameraStream's JPEG-encoding bits to PostProcessorJpeg. This encapsulates the encoder and EXIF generation code into the PostProcessorJpeg layer and removes these specifics related to JPEG, from the CameraStream itself. Signed-off-by: Umang Jain Reviewed-by: Kieran Bingham --- src/android/camera_device.cpp | 1 + src/android/camera_stream.cpp | 77 ++++------------ src/android/camera_stream.h | 4 +- src/android/jpeg/encoder_libjpeg.cpp | 2 +- src/android/jpeg/post_processor_jpeg.cpp | 110 +++++++++++++++++++++++ src/android/jpeg/post_processor_jpeg.h | 36 ++++++++ src/android/meson.build | 1 + 7 files changed, 169 insertions(+), 62 deletions(-) create mode 100644 src/android/jpeg/post_processor_jpeg.cpp create mode 100644 src/android/jpeg/post_processor_jpeg.h diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp index c29fcb4..d6a8acb 100644 --- a/src/android/camera_device.cpp +++ b/src/android/camera_device.cpp @@ -7,6 +7,7 @@ #include "camera_device.h" #include "camera_ops.h" +#include "post_processor.h" #include #include diff --git a/src/android/camera_stream.cpp b/src/android/camera_stream.cpp index eac1480..2a0ba16 100644 --- a/src/android/camera_stream.cpp +++ b/src/android/camera_stream.cpp @@ -9,9 +9,9 @@ #include "camera_device.h" #include "camera_metadata.h" -#include "jpeg/encoder.h" -#include "jpeg/encoder_libjpeg.h" -#include "jpeg/exif.h" +#include "jpeg/post_processor_jpeg.h" + +#include using namespace libcamera; @@ -24,8 +24,15 @@ CameraStream::CameraStream(CameraDevice *cameraDevice, Type type, { config_ = cameraDevice_->cameraConfiguration(); - if (type_ == Type::Internal || type_ == Type::Mapped) - encoder_ = std::make_unique(); + if (type_ == Type::Internal || type_ == Type::Mapped) { + /* + * \todo There might be multiple post-processors. The logic + * which should be instantiated here, is deferred for future. + * For now, we only have PostProcessorJpeg and that is what we + * will instantiate here. + */ + postProcessor_ = std::make_unique(cameraDevice_); + } if (type == Type::Internal) { allocator_ = std::make_unique(cameraDevice_->camera()); @@ -45,8 +52,10 @@ Stream *CameraStream::stream() const int CameraStream::configure() { - if (encoder_) { - int ret = encoder_->configure(configuration()); + if (postProcessor_) { + StreamConfiguration output = configuration(); + output.pixelFormat == formats::MJPEG; + int ret = postProcessor_->configure(configuration(), output); if (ret) return ret; } @@ -69,60 +78,10 @@ int CameraStream::configure() int CameraStream::process(const libcamera::FrameBuffer &source, MappedCamera3Buffer *dest, CameraMetadata *metadata) { - if (!encoder_) + if (!postProcessor_) return 0; - /* Set EXIF metadata for various tags. */ - Exif exif; - /* \todo Set Make and Model from external vendor tags. */ - exif.setMake("libcamera"); - exif.setModel("cameraModel"); - exif.setOrientation(cameraDevice_->orientation()); - exif.setSize(configuration().size); - /* - * We set the frame's EXIF timestamp as the time of encode. - * Since the precision we need for EXIF timestamp is only one - * second, it is good enough. - */ - exif.setTimestamp(std::time(nullptr)); - if (exif.generate() != 0) - LOG(HAL, Error) << "Failed to generate valid EXIF data"; - - int jpeg_size = encoder_->encode(&source, dest->maps()[0], exif.data()); - if (jpeg_size < 0) { - LOG(HAL, Error) << "Failed to encode stream image"; - return jpeg_size; - } - - /* - * Fill in the JPEG blob header. - * - * The mapped size of the buffer is being returned as - * substantially larger than the requested JPEG_MAX_SIZE - * (which is referenced from maxJpegBufferSize_). Utilise - * this static size to ensure the correct offset of the blob is - * determined. - * - * \todo Investigate if the buffer size mismatch is an issue or - * expected behaviour. - */ - uint8_t *resultPtr = dest->maps()[0].data() + - cameraDevice_->maxJpegBufferSize() - - sizeof(struct camera3_jpeg_blob); - auto *blob = reinterpret_cast(resultPtr); - blob->jpeg_blob_id = CAMERA3_JPEG_BLOB_ID; - blob->jpeg_size = jpeg_size; - - /* Update the JPEG result Metadata. */ - metadata->addEntry(ANDROID_JPEG_SIZE, &jpeg_size, 1); - - const uint32_t jpeg_quality = 95; - metadata->addEntry(ANDROID_JPEG_QUALITY, &jpeg_quality, 1); - - const uint32_t jpeg_orientation = 0; - metadata->addEntry(ANDROID_JPEG_ORIENTATION, &jpeg_orientation, 1); - - return 0; + return postProcessor_->process(&source, dest->maps()[0], metadata); } FrameBuffer *CameraStream::getBuffer() diff --git a/src/android/camera_stream.h b/src/android/camera_stream.h index 8df0101..c55d90b 100644 --- a/src/android/camera_stream.h +++ b/src/android/camera_stream.h @@ -19,10 +19,10 @@ #include #include -class Encoder; class CameraDevice; class CameraMetadata; class MappedCamera3Buffer; +class PostProcessor; class CameraStream { @@ -135,7 +135,6 @@ private: */ unsigned int index_; - std::unique_ptr encoder_; std::unique_ptr allocator_; std::vector buffers_; /* @@ -143,6 +142,7 @@ private: * an std::vector in CameraDevice. */ std::unique_ptr mutex_; + std::unique_ptr postProcessor_; }; #endif /* __ANDROID_CAMERA_STREAM__ */ diff --git a/src/android/jpeg/encoder_libjpeg.cpp b/src/android/jpeg/encoder_libjpeg.cpp index a77f5b2..8995ba5 100644 --- a/src/android/jpeg/encoder_libjpeg.cpp +++ b/src/android/jpeg/encoder_libjpeg.cpp @@ -25,7 +25,7 @@ using namespace libcamera; -LOG_DEFINE_CATEGORY(JPEG) +LOG_DECLARE_CATEGORY(JPEG); namespace { diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp new file mode 100644 index 0000000..d1ec95b --- /dev/null +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -0,0 +1,110 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * post_processor_jpeg.cpp - JPEG Post Processor + */ + +#include "post_processor_jpeg.h" + +#include "../camera_device.h" +#include "../camera_metadata.h" +#include "encoder_libjpeg.h" +#include "exif.h" + +#include + +#include "libcamera/internal/log.h" + +using namespace libcamera; + +LOG_DEFINE_CATEGORY(JPEG); + +PostProcessorJpeg::PostProcessorJpeg(CameraDevice *device) + : cameraDevice_(device) +{ +} + +int PostProcessorJpeg::configure(const StreamConfiguration &inCfg, + const StreamConfiguration &outCfg) +{ + int ret = 0; + + if (inCfg.size != outCfg.size) { + LOG(JPEG, Error) << "Mismatch of input and output stream sizes"; + return -1; + } + + if (outCfg.pixelFormat != formats::MJPEG) { + LOG(JPEG, Error) << "Output stream pixel format is not JPEG"; + return -1; + } + + streamSize_ = outCfg.size; + encoder_ = std::make_unique(); + + if (encoder_) + ret = encoder_->configure(inCfg); + + return ret; +} + +int PostProcessorJpeg::process(const libcamera::FrameBuffer *source, + const libcamera::Span &destination, + CameraMetadata *metadata) +{ + if (!encoder_) + return 0; + + /* Set EXIF metadata for various tags. */ + Exif exif; + /* \todo Set Make and Model from external vendor tags. */ + exif.setMake("libcamera"); + exif.setModel("cameraModel"); + exif.setOrientation(cameraDevice_->orientation()); + exif.setSize(streamSize_); + /* + * We set the frame's EXIF timestamp as the time of encode. + * Since the precision we need for EXIF timestamp is only one + * second, it is good enough. + */ + exif.setTimestamp(std::time(nullptr)); + if (exif.generate() != 0) + LOG(JPEG, Error) << "Failed to generate valid EXIF data"; + + int jpeg_size = encoder_->encode(source, destination, exif.data()); + if (jpeg_size < 0) { + LOG(JPEG, Error) << "Failed to encode stream image"; + return jpeg_size; + } + + /* + * Fill in the JPEG blob header. + * + * The mapped size of the buffer is being returned as + * substantially larger than the requested JPEG_MAX_SIZE + * (which is referenced from maxJpegBufferSize_). Utilise + * this static size to ensure the correct offset of the blob is + * determined. + * + * \todo Investigate if the buffer size mismatch is an issue or + * expected behaviour. + */ + uint8_t *resultPtr = destination.data() + + cameraDevice_->maxJpegBufferSize() - + sizeof(struct camera3_jpeg_blob); + auto *blob = reinterpret_cast(resultPtr); + blob->jpeg_blob_id = CAMERA3_JPEG_BLOB_ID; + blob->jpeg_size = jpeg_size; + + /* Update the JPEG result Metadata. */ + metadata->addEntry(ANDROID_JPEG_SIZE, &jpeg_size, 1); + + const uint32_t jpeg_quality = 95; + metadata->addEntry(ANDROID_JPEG_QUALITY, &jpeg_quality, 1); + + const uint32_t jpeg_orientation = 0; + metadata->addEntry(ANDROID_JPEG_ORIENTATION, &jpeg_orientation, 1); + + return 0; +} diff --git a/src/android/jpeg/post_processor_jpeg.h b/src/android/jpeg/post_processor_jpeg.h new file mode 100644 index 0000000..72aca9b --- /dev/null +++ b/src/android/jpeg/post_processor_jpeg.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * post_processor_jpeg.h - JPEG Post Processor + */ +#ifndef __ANDROID_POST_PROCESSOR_JPEG_H__ +#define __ANDROID_POST_PROCESSOR_JPEG_H__ + +#include "../post_processor.h" +#include "libcamera/internal/buffer.h" + +#include + +class Encoder; +class CameraDevice; + +class PostProcessorJpeg : public PostProcessor +{ +public: + PostProcessorJpeg(CameraDevice *device); + + int configure(const libcamera::StreamConfiguration &incfg, + const libcamera::StreamConfiguration &outcfg) override; + int process(const libcamera::FrameBuffer *source, + const libcamera::Span &destination, + CameraMetadata *metadata) override; + + +private: + CameraDevice *cameraDevice_; + std::unique_ptr encoder_; + libcamera::Size streamSize_; +}; + +#endif /* __ANDROID_POST_PROCESSOR_JPEG_H__ */ diff --git a/src/android/meson.build b/src/android/meson.build index 802bb89..eacd544 100644 --- a/src/android/meson.build +++ b/src/android/meson.build @@ -23,6 +23,7 @@ android_hal_sources = files([ 'camera_stream.cpp', 'jpeg/encoder_libjpeg.cpp', 'jpeg/exif.cpp', + 'jpeg/post_processor_jpeg.cpp', ]) android_camera_metadata_sources = files([ From patchwork Thu Oct 15 17:14:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 10060 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1345CBE905 for ; Thu, 15 Oct 2020 17:15:12 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D3E7B61027; Thu, 15 Oct 2020 19:15:11 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=uajain.com header.i=@uajain.com header.b="YbSWeDco"; dkim-atps=neutral Received: from mail.uajain.com (static.126.159.217.95.clients.your-server.de [95.217.159.126]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B71C460E5A for ; Thu, 15 Oct 2020 19:15:10 +0200 (CEST) From: Umang Jain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=uajain.com; s=mail; t=1602782110; bh=vl/GOXYa4qcngnArqex3I+btej7F6xWgng7PH/ZLHiY=; h=From:To:Cc:Subject:In-Reply-To:References; b=YbSWeDcowknw+KOyExuwUd977GHPQ8gqNCXqQJxCZ356ivujstA1YKvlWnfREKiPY WnW8ZaBghw0MoEPNgALsh683PbdtY64zvxJToTRLa1RcFqJkMuVEHQ+7m9k3rnOSAF 2NfTew1gvEeSpcWyKuAOPcCj8k9MjJE98hgtYxPY4U/c4fqpoHSuNTLT+mDAtVtFim 0ekraoa+ALSjcqXMBOqdILN/2em2q8G6s4PnAG4fhhqymnKm+h3KTNIfjQ1+AyUBAj 61zxeNvTIiOPN7++LNFKBEEqy+lW9WqayZdcGKsTRQHgYW98+4ONANkYU3jiozJDEV 7r5UAWU7TYDjw== To: libcamera-devel@lists.libcamera.org Date: Thu, 15 Oct 2020 22:44:57 +0530 Message-Id: <20201015171457.75678-4-email@uajain.com> In-Reply-To: <20201015171457.75678-1-email@uajain.com> References: <20201015171457.75678-1-email@uajain.com> Mime-Version: 1.0 Subject: [libcamera-devel] [PATCH 3/3] android: CameraStream: Fix nitpick indentation for mutex_ X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Signed-off-by: Umang Jain Reviewed-by: Kieran Bingham Reviewed-by: Laurent Pinchart --- src/android/camera_stream.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/android/camera_stream.h b/src/android/camera_stream.h index c55d90b..c2cb121 100644 --- a/src/android/camera_stream.h +++ b/src/android/camera_stream.h @@ -141,7 +141,7 @@ private: * The class has to be MoveConstructible as instances are stored in * an std::vector in CameraDevice. */ - std::unique_ptr mutex_; + std::unique_ptr mutex_; std::unique_ptr postProcessor_; };