From patchwork Thu Oct 8 14:10:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 10022 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 576E6BEEE0 for ; Thu, 8 Oct 2020 14:11:00 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 258E8605D1; Thu, 8 Oct 2020 16:11:00 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=uajain.com header.i=@uajain.com header.b="VOxFZ5c1"; dkim-atps=neutral Received: from mail.uajain.com (static.126.159.217.95.clients.your-server.de [95.217.159.126]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 932B26035D for ; Thu, 8 Oct 2020 16:10:58 +0200 (CEST) From: Umang Jain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=uajain.com; s=mail; t=1602166258; bh=73s4xjI1ABvJvhINL3UUIfGgLTKIalqMEcGIFiei4Jc=; h=From:To:Cc:Subject:In-Reply-To:References; b=VOxFZ5c1vnvOUKDkLdPq+6LD95iyz1ntDY1YfLB7UWONXUDeupgQDexG2ilSkNoU0 HQkVbQRAS/Awkz/uAiZddvouvDophD+xjZLVa/2UWi5orGHJzQxsqiL5tMtiUJL/0X eh/PQjqvUO4Gq4gfKTbF08ZDnA7RPs7dzLTcBFpZ8obK/MYbxnUGEqturyxpf+Id3U vtjSdM9sOObujGPsn+BqBlmkGZUYL/Pk4Er6jSpayXwv+ug9DdWONFPyJpThl0YeCT z594Ssqvqs6qqjGrIdhpQqgJXp+lBwRKIejvm6vr6bWFPkz+10N7D8bKy4zj++xtko u6YOAmG268MYQ== To: libcamera-devel@lists.libcamera.org Date: Thu, 8 Oct 2020 19:40:36 +0530 Message-Id: <20201008141038.83425-2-email@uajain.com> In-Reply-To: <20201008141038.83425-1-email@uajain.com> References: <20201008141038.83425-1-email@uajain.com> Mime-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH 1/3] android: post_processor: Introduce a PostProcessor interface X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Introduce a PostProcessor interface for the streams that require any kind of processing for their consumption by the HAL layer. The PostProcessor interface can be configured via configure() and the actual processing can be initiated using process(). The interface is similar to the Encoder interface. The PostProcessor is meant to replace the Encoder interface and introduce a more generic post-processing layer which can be extended to have multiple post processors for various stream configurations. As of now, we only have one post processor (JPEG), hence the subsequent commit will port its function to this interface. Signed-off-by: Umang Jain --- src/android/post_processor.h | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 src/android/post_processor.h diff --git a/src/android/post_processor.h b/src/android/post_processor.h new file mode 100644 index 0000000..fa676c9 --- /dev/null +++ b/src/android/post_processor.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.1-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * post_processor.h - CameraStream Post Processing Interface + */ +#ifndef __ANDROID_POST_PROCESSOR_H__ +#define __ANDROID_POST_PROCESSOR_H__ + +#include + +#include +#include +#include + +class CameraMetadata; + +class PostProcessor +{ +public: + virtual ~PostProcessor() {}; + + virtual int configure(const libcamera::StreamConfiguration &cfg) = 0; + virtual int process(const libcamera::FrameBuffer *source, + const libcamera::Span &destination, + CameraMetadata *metadata, + ...) = 0; +}; + +#endif /* __ANDROID_POST_PROCESSOR_H__ */ From patchwork Thu Oct 8 14:10:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 10023 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C4762BEEE0 for ; Thu, 8 Oct 2020 14:11:04 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 90625605C9; Thu, 8 Oct 2020 16:11:04 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=uajain.com header.i=@uajain.com header.b="QJzHx1xb"; dkim-atps=neutral Received: from mail.uajain.com (static.126.159.217.95.clients.your-server.de [95.217.159.126]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DD1496035D for ; Thu, 8 Oct 2020 16:11:02 +0200 (CEST) From: Umang Jain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=uajain.com; s=mail; t=1602166262; bh=7Y50rK2MDtOwPqLQIQhnr4SYGGAMG0W+e1fkCoMt0hw=; h=From:To:Cc:Subject:In-Reply-To:References; b=QJzHx1xbYG5j4dkHKE+9e6p6JHX9ce8F5d9fW8SC19+Ab0g3FeUkL2uLC8QZ9IwMo WNyBsfJL6+ID/N2TgfeT6mx/hbNK5gig6sJODnigxwZHgAiiFIdZk4Er4CRnf55RDG zn0T1uJ2Nu+eHJw4FB44aKSzTYHuc1rKmZ+qHokWQ/Wro+MNakS/RZFYyh2u326+Gb sFzyk4Bg8Ykgj61ICwmth+o9r69ULjuvqcVTNxoZL607O84DOO/0E7XQd1soLdx/lw HIqIaJP0z6T4uoaD/dk9gctCFEYqLzcutmIJ4qJZHteJjPs09N0u/HRLSSo8bDD3xb HDpHDKfzc5CiQ== To: libcamera-devel@lists.libcamera.org Date: Thu, 8 Oct 2020 19:40:37 +0530 Message-Id: <20201008141038.83425-3-email@uajain.com> In-Reply-To: <20201008141038.83425-1-email@uajain.com> References: <20201008141038.83425-1-email@uajain.com> Mime-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH 2/3] android: jpeg: Port to PostProcessor interface X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Remove the existing Encoder interface completely and use the PostProcessor interface instead. Now the ::encode() function will be called by PostProcessor::process() internally and will not be directly exposed in CameraStream. Similarly, other operations can be introduced internally in the PostProcessorJpeg, and can be called through its process() interface. Signed-off-by: Umang Jain --- src/android/camera_device.h | 1 - src/android/camera_stream.cpp | 74 +++------------ src/android/camera_stream.h | 9 +- src/android/jpeg/encoder.h | 25 ----- ...er_libjpeg.cpp => post_processor_jpeg.cpp} | 92 +++++++++++++++++-- ...ncoder_libjpeg.h => post_processor_jpeg.h} | 27 ++++-- src/android/meson.build | 2 +- 7 files changed, 122 insertions(+), 108 deletions(-) delete mode 100644 src/android/jpeg/encoder.h rename src/android/jpeg/{encoder_libjpeg.cpp => post_processor_jpeg.cpp} (67%) rename src/android/jpeg/{encoder_libjpeg.h => post_processor_jpeg.h} (55%) diff --git a/src/android/camera_device.h b/src/android/camera_device.h index 777d1a3..25de12e 100644 --- a/src/android/camera_device.h +++ b/src/android/camera_device.h @@ -25,7 +25,6 @@ #include "libcamera/internal/message.h" #include "camera_stream.h" -#include "jpeg/encoder.h" class CameraMetadata; diff --git a/src/android/camera_stream.cpp b/src/android/camera_stream.cpp index eac1480..ed3bb41 100644 --- a/src/android/camera_stream.cpp +++ b/src/android/camera_stream.cpp @@ -9,9 +9,7 @@ #include "camera_device.h" #include "camera_metadata.h" -#include "jpeg/encoder.h" -#include "jpeg/encoder_libjpeg.h" -#include "jpeg/exif.h" +#include "jpeg/post_processor_jpeg.h" using namespace libcamera; @@ -24,8 +22,15 @@ CameraStream::CameraStream(CameraDevice *cameraDevice, Type type, { config_ = cameraDevice_->cameraConfiguration(); - if (type_ == Type::Internal || type_ == Type::Mapped) - encoder_ = std::make_unique(); + if (type_ == Type::Internal || type_ == Type::Mapped) { + /* + * \todo There might be multiple post-processors. The logic + * which should be instantiated here, is deferred for future. + * For now, we only have PostProcessJpeg and that is what we + * will instantiate here. + */ + postProcessor_ = std::make_unique(); + } if (type == Type::Internal) { allocator_ = std::make_unique(cameraDevice_->camera()); @@ -45,8 +50,8 @@ Stream *CameraStream::stream() const int CameraStream::configure() { - if (encoder_) { - int ret = encoder_->configure(configuration()); + if (postProcessor_) { + int ret = postProcessor_->configure(configuration()); if (ret) return ret; } @@ -69,60 +74,11 @@ int CameraStream::configure() int CameraStream::process(const libcamera::FrameBuffer &source, MappedCamera3Buffer *dest, CameraMetadata *metadata) { - if (!encoder_) + if (!postProcessor_) return 0; - /* Set EXIF metadata for various tags. */ - Exif exif; - /* \todo Set Make and Model from external vendor tags. */ - exif.setMake("libcamera"); - exif.setModel("cameraModel"); - exif.setOrientation(cameraDevice_->orientation()); - exif.setSize(configuration().size); - /* - * We set the frame's EXIF timestamp as the time of encode. - * Since the precision we need for EXIF timestamp is only one - * second, it is good enough. - */ - exif.setTimestamp(std::time(nullptr)); - if (exif.generate() != 0) - LOG(HAL, Error) << "Failed to generate valid EXIF data"; - - int jpeg_size = encoder_->encode(&source, dest->maps()[0], exif.data()); - if (jpeg_size < 0) { - LOG(HAL, Error) << "Failed to encode stream image"; - return jpeg_size; - } - - /* - * Fill in the JPEG blob header. - * - * The mapped size of the buffer is being returned as - * substantially larger than the requested JPEG_MAX_SIZE - * (which is referenced from maxJpegBufferSize_). Utilise - * this static size to ensure the correct offset of the blob is - * determined. - * - * \todo Investigate if the buffer size mismatch is an issue or - * expected behaviour. - */ - uint8_t *resultPtr = dest->maps()[0].data() + - cameraDevice_->maxJpegBufferSize() - - sizeof(struct camera3_jpeg_blob); - auto *blob = reinterpret_cast(resultPtr); - blob->jpeg_blob_id = CAMERA3_JPEG_BLOB_ID; - blob->jpeg_size = jpeg_size; - - /* Update the JPEG result Metadata. */ - metadata->addEntry(ANDROID_JPEG_SIZE, &jpeg_size, 1); - - const uint32_t jpeg_quality = 95; - metadata->addEntry(ANDROID_JPEG_QUALITY, &jpeg_quality, 1); - - const uint32_t jpeg_orientation = 0; - metadata->addEntry(ANDROID_JPEG_ORIENTATION, &jpeg_orientation, 1); - - return 0; + return postProcessor_->process(&source, dest->maps()[0], + metadata, cameraDevice_); } FrameBuffer *CameraStream::getBuffer() diff --git a/src/android/camera_stream.h b/src/android/camera_stream.h index 8df0101..8d0f2e3 100644 --- a/src/android/camera_stream.h +++ b/src/android/camera_stream.h @@ -19,7 +19,12 @@ #include #include -class Encoder; +/* + * \todo Ideally we want to replace this include with forward-declaration. + * If we do that. currently we get a compile error. + */ +#include "post_processor.h" + class CameraDevice; class CameraMetadata; class MappedCamera3Buffer; @@ -135,7 +140,7 @@ private: */ unsigned int index_; - std::unique_ptr encoder_; + std::unique_ptr postProcessor_; std::unique_ptr allocator_; std::vector buffers_; /* diff --git a/src/android/jpeg/encoder.h b/src/android/jpeg/encoder.h deleted file mode 100644 index cf26d67..0000000 --- a/src/android/jpeg/encoder.h +++ /dev/null @@ -1,25 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * Copyright (C) 2020, Google Inc. - * - * encoder.h - Image encoding interface - */ -#ifndef __ANDROID_JPEG_ENCODER_H__ -#define __ANDROID_JPEG_ENCODER_H__ - -#include -#include -#include - -class Encoder -{ -public: - virtual ~Encoder() {}; - - virtual int configure(const libcamera::StreamConfiguration &cfg) = 0; - virtual int encode(const libcamera::FrameBuffer *source, - const libcamera::Span &destination, - const libcamera::Span &exifData) = 0; -}; - -#endif /* __ANDROID_JPEG_ENCODER_H__ */ diff --git a/src/android/jpeg/encoder_libjpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp similarity index 67% rename from src/android/jpeg/encoder_libjpeg.cpp rename to src/android/jpeg/post_processor_jpeg.cpp index 510613c..eeb4e95 100644 --- a/src/android/jpeg/encoder_libjpeg.cpp +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -2,10 +2,14 @@ /* * Copyright (C) 2020, Google Inc. * - * encoder_libjpeg.cpp - JPEG encoding using libjpeg native API + * post_processor_jpeg.cpp - JPEG Post Processor */ -#include "encoder_libjpeg.h" +#include "post_processor_jpeg.h" + +#include "exif.h" + +#include "../camera_device.h" #include #include @@ -25,6 +29,14 @@ using namespace libcamera; +#define extract_va_arg(type, arg, last) \ +{ \ + va_list ap; \ + va_start(ap, last); \ + arg = va_arg(ap, type); \ + va_end(ap); \ +} + LOG_DEFINE_CATEGORY(JPEG) namespace { @@ -67,7 +79,7 @@ const struct JPEGPixelFormatInfo &findPixelInfo(const PixelFormat &format) } /* namespace */ -EncoderLibJpeg::EncoderLibJpeg() +PostProcessorJpeg::PostProcessorJpeg() : quality_(95) { /* \todo Expand error handling coverage with a custom handler. */ @@ -76,12 +88,12 @@ EncoderLibJpeg::EncoderLibJpeg() jpeg_create_compress(&compress_); } -EncoderLibJpeg::~EncoderLibJpeg() +PostProcessorJpeg::~PostProcessorJpeg() { jpeg_destroy_compress(&compress_); } -int EncoderLibJpeg::configure(const StreamConfiguration &cfg) +int PostProcessorJpeg::configure(const StreamConfiguration &cfg) { const struct JPEGPixelFormatInfo info = findPixelInfo(cfg.pixelFormat); if (info.colorSpace == JCS_UNKNOWN) @@ -104,7 +116,67 @@ int EncoderLibJpeg::configure(const StreamConfiguration &cfg) return 0; } -void EncoderLibJpeg::compressRGB(const libcamera::MappedBuffer *frame) +int PostProcessorJpeg::process(const libcamera::FrameBuffer *source, + const libcamera::Span &destination, + CameraMetadata *metadata, ...) +{ + CameraDevice *device = nullptr; + extract_va_arg(CameraDevice *, device, metadata); + + /* Set EXIF metadata for various tags. */ + Exif exif; + /* \todo Set Make and Model from external vendor tags. */ + exif.setMake("libcamera"); + exif.setModel("cameraModel"); + exif.setOrientation(device->orientation()); + exif.setSize(Size {compress_.image_width, compress_.image_height}); + /* + * We set the frame's EXIF timestamp as the time of encode. + * Since the precision we need for EXIF timestamp is only one + * second, it is good enough. + */ + exif.setTimestamp(std::time(nullptr)); + if (exif.generate() != 0) + LOG(JPEG, Error) << "Failed to generate valid EXIF data"; + + int jpeg_size = encode(source, destination, exif.data()); + if (jpeg_size < 0) { + LOG(JPEG, Error) << "Failed to encode stream image"; + return jpeg_size; + } + + /* + * Fill in the JPEG blob header. + * + * The mapped size of the buffer is being returned as + * substantially larger than the requested JPEG_MAX_SIZE + * (which is referenced from maxJpegBufferSize_). Utilise + * this static size to ensure the correct offset of the blob is + * determined. + * + * \todo Investigate if the buffer size mismatch is an issue or + * expected behaviour. + */ + uint8_t *resultPtr = destination.data() + + device->maxJpegBufferSize() - + sizeof(struct camera3_jpeg_blob); + auto *blob = reinterpret_cast(resultPtr); + blob->jpeg_blob_id = CAMERA3_JPEG_BLOB_ID; + blob->jpeg_size = jpeg_size; + + /* Update the JPEG result Metadata. */ + metadata->addEntry(ANDROID_JPEG_SIZE, &jpeg_size, 1); + + const uint32_t jpeg_quality = 95; + metadata->addEntry(ANDROID_JPEG_QUALITY, &jpeg_quality, 1); + + const uint32_t jpeg_orientation = 0; + metadata->addEntry(ANDROID_JPEG_ORIENTATION, &jpeg_orientation, 1); + + return 0; +} + +void PostProcessorJpeg::compressRGB(const libcamera::MappedBuffer *frame) { unsigned char *src = static_cast(frame->maps()[0].data()); /* \todo Stride information should come from buffer configuration. */ @@ -122,7 +194,7 @@ void EncoderLibJpeg::compressRGB(const libcamera::MappedBuffer *frame) * Compress the incoming buffer from a supported NV format. * This naively unpacks the semi-planar NV12 to a YUV888 format for libjpeg. */ -void EncoderLibJpeg::compressNV(const libcamera::MappedBuffer *frame) +void PostProcessorJpeg::compressNV(const libcamera::MappedBuffer *frame) { uint8_t tmprowbuf[compress_.image_width * 3]; @@ -179,9 +251,9 @@ void EncoderLibJpeg::compressNV(const libcamera::MappedBuffer *frame) } } -int EncoderLibJpeg::encode(const FrameBuffer *source, - const libcamera::Span &dest, - const libcamera::Span &exifData) +int PostProcessorJpeg::encode(const FrameBuffer *source, + const libcamera::Span &dest, + const libcamera::Span &exifData) { MappedFrameBuffer frame(source, PROT_READ); if (!frame.isValid()) { diff --git a/src/android/jpeg/encoder_libjpeg.h b/src/android/jpeg/post_processor_jpeg.h similarity index 55% rename from src/android/jpeg/encoder_libjpeg.h rename to src/android/jpeg/post_processor_jpeg.h index 1e8df05..7f9ce70 100644 --- a/src/android/jpeg/encoder_libjpeg.h +++ b/src/android/jpeg/post_processor_jpeg.h @@ -2,30 +2,37 @@ /* * Copyright (C) 2020, Google Inc. * - * encoder_libjpeg.h - JPEG encoding using libjpeg + * post_processor_jpeg.h - JPEG Post Processor */ -#ifndef __ANDROID_JPEG_ENCODER_LIBJPEG_H__ -#define __ANDROID_JPEG_ENCODER_LIBJPEG_H__ +#ifndef __ANDROID_POST_PROCESSOR_JPEG_H__ +#define __ANDROID_POST_PROCESSOR_JPEG_H__ -#include "encoder.h" +#include "../post_processor.h" +#include "../camera_metadata.h" #include "libcamera/internal/buffer.h" #include "libcamera/internal/formats.h" #include -class EncoderLibJpeg : public Encoder +class PostProcessorJpeg : public PostProcessor { public: - EncoderLibJpeg(); - ~EncoderLibJpeg(); + PostProcessorJpeg(); + ~PostProcessorJpeg(); int configure(const libcamera::StreamConfiguration &cfg) override; + int process(const libcamera::FrameBuffer *source, + const libcamera::Span &destination, + CameraMetadata *metadata, + ...) override; + + +private: int encode(const libcamera::FrameBuffer *source, const libcamera::Span &destination, - const libcamera::Span &exifData) override; + const libcamera::Span &exifData); -private: void compressRGB(const libcamera::MappedBuffer *frame); void compressNV(const libcamera::MappedBuffer *frame); @@ -40,4 +47,4 @@ private: bool nvSwap_; }; -#endif /* __ANDROID_JPEG_ENCODER_LIBJPEG_H__ */ +#endif /* __ANDROID_POST_PROCESSOR_JPEG_H__ */ diff --git a/src/android/meson.build b/src/android/meson.build index 802bb89..02b3b47 100644 --- a/src/android/meson.build +++ b/src/android/meson.build @@ -21,8 +21,8 @@ android_hal_sources = files([ 'camera_metadata.cpp', 'camera_ops.cpp', 'camera_stream.cpp', - 'jpeg/encoder_libjpeg.cpp', 'jpeg/exif.cpp', + 'jpeg/post_processor_jpeg.cpp', ]) android_camera_metadata_sources = files([ From patchwork Thu Oct 8 14:10:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 10024 X-Patchwork-Delegate: umang.jain@ideasonboard.com Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 21510BEEE0 for ; Thu, 8 Oct 2020 14:11:06 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E30B6605BD; Thu, 8 Oct 2020 16:11:05 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=uajain.com header.i=@uajain.com header.b="QDloVlxu"; dkim-atps=neutral Received: from mail.uajain.com (static.126.159.217.95.clients.your-server.de [95.217.159.126]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 48710605C6 for ; Thu, 8 Oct 2020 16:11:05 +0200 (CEST) From: Umang Jain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=uajain.com; s=mail; t=1602166264; bh=+pNVRCEjylQXJmnshAK8AfP4E9dazoNgS0yyDsIketU=; h=From:To:Cc:Subject:In-Reply-To:References; b=QDloVlxuCY2EdyFg6KlCG1l3TYaEleDMyUVz+E0RDFQ5mOuK+78yrgbmx/aH48ZNj fLvfnmN3tf1spA6/zSlQ1VDaTMDjMRLmPpzky14YR7dUaT68VDLZTm/KTW3FsaBE6E qaUUx5di9qSFoR4S7MkB53LKb1U3ebximFiQRHphySk5t2efFSDMkU7tPLtMZJG2Gg 2BJHeNgA8n2SqGmKheBQ+b3t8e3O7jlStAIe5ycXknwsyt7KtV8w+U+XEkq9D7AvnD 2HCqIRLDL8SEDltmJhISL4nMxOmGDL/b3ir8onCSbDPjfuPB7pTzMx43OqgDkyqUyd EmM6qhMCsVbsw== To: libcamera-devel@lists.libcamera.org Date: Thu, 8 Oct 2020 19:40:38 +0530 Message-Id: <20201008141038.83425-4-email@uajain.com> In-Reply-To: <20201008141038.83425-1-email@uajain.com> References: <20201008141038.83425-1-email@uajain.com> Mime-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH 3/3] android: jpeg: Add a basic NV12 image thumbnailer X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a basic image thumbnailer for NV12 frames being captured. It shall generate a thumbnail image to be embedded as a part of EXIF metadata of the frame. The output of the thumbnail will still be NV12. Signed-off-by: Umang Jain --- src/android/jpeg/post_processor_jpeg.cpp | 11 ++ src/android/jpeg/thumbnailer.cpp | 134 +++++++++++++++++++++++ src/android/jpeg/thumbnailer.h | 34 ++++++ src/android/meson.build | 1 + 4 files changed, 180 insertions(+) create mode 100644 src/android/jpeg/thumbnailer.cpp create mode 100644 src/android/jpeg/thumbnailer.h diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp index eeb4e95..9076d04 100644 --- a/src/android/jpeg/post_processor_jpeg.cpp +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -8,6 +8,7 @@ #include "post_processor_jpeg.h" #include "exif.h" +#include "thumbnailer.h" #include "../camera_device.h" @@ -286,6 +287,16 @@ int PostProcessorJpeg::encode(const FrameBuffer *source, LOG(JPEG, Debug) << "JPEG Encode Starting:" << compress_.image_width << "x" << compress_.image_height; + Thumbnailer th; + libcamera::Span thumbnail; + th.configure(Size (compress_.image_width, compress_.image_height), + pixelFormatInfo_->format); + th.scaleBuffer(source, thumbnail); + /* + * \todo: Compress the thumbnail again encode() and set it in the + * respective EXIF field. + */ + if (nv_) compressNV(&frame); else diff --git a/src/android/jpeg/thumbnailer.cpp b/src/android/jpeg/thumbnailer.cpp new file mode 100644 index 0000000..d01b4af --- /dev/null +++ b/src/android/jpeg/thumbnailer.cpp @@ -0,0 +1,134 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * thumbnailer.cpp - Basic image thumbnailer from NV12 + */ + +#include "thumbnailer.h" + +#include + +#include "libcamera/internal/file.h" +#include "libcamera/internal/log.h" + +using namespace libcamera; + +LOG_DEFINE_CATEGORY(Thumbnailer) + +Thumbnailer::Thumbnailer() + : validConfiguration_(false) +{ +} + +void Thumbnailer::configure(const Size &sourceSize, PixelFormat pixelFormat) +{ + sourceSize_ = sourceSize; + pixelFormat_ = pixelFormat; + + if (pixelFormat_ != formats::NV12) { + LOG (Thumbnailer, Error) << "Failed to configure: Pixel Format " + << pixelFormat_.toString() << " unsupported."; + return; + } + + validConfiguration_ = true; +} + +static std::string datetime() +{ + time_t rawtime; + struct tm *timeinfo; + char buffer[80]; + static unsigned int milliseconds = 0; + + time(&rawtime); + timeinfo = localtime(&rawtime); + + strftime(buffer, 80, "%d-%m-%Y.%H-%M-%S.", timeinfo); + + /* milliseconds is just a fast hack to ensure unique filenames */ + return std::string(buffer) + std::to_string(milliseconds++); +} + +/* + * The Exif specification recommends the width of the thumbnail to be a + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding height + * keeping the aspect ratio same as of the source. + */ +Size Thumbnailer::computeThumbnailSize() +{ + unsigned int targetHeight; + unsigned int targetWidth = 160; + + targetHeight = targetWidth * sourceSize_.height / sourceSize_.width; + + if (targetHeight & 1) + targetHeight++; + + return Size(targetWidth, targetHeight); +} + +int Thumbnailer::scaleBuffer(const FrameBuffer *source, Span &dest) +{ + MappedFrameBuffer frame(source, PROT_READ); + if (!frame.isValid()) { + LOG(Thumbnailer, Error) << "Failed to map FrameBuffer : " + << strerror(frame.error()); + return frame.error(); + } + + if (!validConfiguration_) { + LOG(Thumbnailer, Error) << "config is unconfigured or invalid."; + return -1; + } + + targetSize_ = computeThumbnailSize(); + + const unsigned int sw = sourceSize_.width; + const unsigned int sh = sourceSize_.height; + const unsigned int tw = targetSize_.width; + const unsigned int th = targetSize_.height; + + /* Image scaling block implementing nearest-neighbour algorithm. */ + unsigned char *src = static_cast(frame.maps()[0].data()); + unsigned char *src_c = src + sh * sw; + unsigned int cb_pos = 0; + unsigned int cr_pos = 1; + unsigned char *src_cb, *src_cr; + + size_t dstSize = (th * tw) + ((th/2) * tw); + unsigned char *destination = static_cast(malloc(dstSize)); + unsigned char *dst = destination; + unsigned char *dst_c = destination + th * tw; + + for (unsigned int y = 0; y < th; y+=2) { + unsigned int sourceY = (sh*y + th/2) / th; + + src_cb = src_c + (sourceY/2) * sw + cb_pos; + src_cr = src_c + (sourceY/2) * sw + cr_pos; + + for (unsigned int x = 0; x < tw; x+=2) { + unsigned int sourceX = (sw*x + tw/2) / tw; + + dst[y * tw + x] = src[sw * sourceY + sourceX]; + dst[(y+1) * tw + x] = src[sw * (sourceY+1) + sourceX]; + dst[y * tw + (x+1)] = src[sw * sourceY + (sourceX+1)]; + dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) + (sourceX+1)]; + + dst_c[(y/2) * tw + x + cb_pos] = src_cb[(sourceX/2) * 2]; + dst_c[(y/2) * tw + x + cr_pos] = src_cr[(sourceX/2) * 2]; + } + } + + /* Helper code: Write the output pixels to a file so we can inspect */ + File file("/tmp/" + datetime() + ".raw"); + int32_t ret = file.open(File::WriteOnly); + ret = file.write({ destination, dstSize }); + LOG(Thumbnailer, Info) << "Wrote " << ret << " bytes: " << targetSize_.width << "x" << targetSize_.height; + + /* Write scaled pixels to dest */ + dest = { destination, dstSize }; + + return 0; +} diff --git a/src/android/jpeg/thumbnailer.h b/src/android/jpeg/thumbnailer.h new file mode 100644 index 0000000..af3a194 --- /dev/null +++ b/src/android/jpeg/thumbnailer.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * thumbnailer.h - Basic image thumbnailer from NV12 + */ +#ifndef __ANDROID_JPEG_THUMBNAILER_H__ +#define __ANDROID_JPEG_THUMBNAILER_H__ + +#include + +#include "libcamera/internal/buffer.h" +#include "libcamera/internal/formats.h" + +class Thumbnailer +{ +public: + Thumbnailer(); + + void configure(const libcamera::Size &sourceSize, + libcamera::PixelFormat pixelFormat); + int scaleBuffer(const libcamera::FrameBuffer *source, libcamera::Span &dest); + +private: + libcamera::Size computeThumbnailSize(); + + libcamera::PixelFormat pixelFormat_; + libcamera::Size sourceSize_; + libcamera::Size targetSize_; + + bool validConfiguration_; +}; + +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */ diff --git a/src/android/meson.build b/src/android/meson.build index 02b3b47..854005e 100644 --- a/src/android/meson.build +++ b/src/android/meson.build @@ -23,6 +23,7 @@ android_hal_sources = files([ 'camera_stream.cpp', 'jpeg/exif.cpp', 'jpeg/post_processor_jpeg.cpp', + 'jpeg/thumbnailer.cpp', ]) android_camera_metadata_sources = files([