{"id":10023,"url":"https://patchwork.libcamera.org/api/1.1/patches/10023/?format=json","web_url":"https://patchwork.libcamera.org/patch/10023/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20201008141038.83425-3-email@uajain.com>","date":"2020-10-08T14:10:37","name":"[libcamera-devel,RFC,2/3] android: jpeg: Port to PostProcessor interface","commit_ref":null,"pull_url":null,"state":"accepted","archived":false,"hash":"56b1c0c7807123a885f8338c3ad63bc48d8c2201","submitter":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/people/1/?format=json","name":"Umang Jain","email":"email@uajain.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/10023/mbox/","series":[{"id":1367,"url":"https://patchwork.libcamera.org/api/1.1/series/1367/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=1367","date":"2020-10-08T14:10:35","name":"Introduce PostProcessor Interface for CameraStream","version":1,"mbox":"https://patchwork.libcamera.org/series/1367/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/10023/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/10023/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C4762BEEE0\n\tfor ;\n\tThu, 8 Oct 2020 14:11:04 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 90625605C9;\n\tThu, 8 Oct 2020 16:11:04 +0200 (CEST)","from mail.uajain.com (static.126.159.217.95.clients.your-server.de\n\t[95.217.159.126])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id DD1496035D\n\tfor ;\n\tThu, 8 Oct 2020 16:11:02 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=uajain.com header.i=@uajain.com\n\theader.b=\"QJzHx1xb\"; dkim-atps=neutral","From":"Umang Jain ","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=uajain.com; s=mail;\n\tt=1602166262; bh=7Y50rK2MDtOwPqLQIQhnr4SYGGAMG0W+e1fkCoMt0hw=;\n\th=From:To:Cc:Subject:In-Reply-To:References;\n\tb=QJzHx1xbYG5j4dkHKE+9e6p6JHX9ce8F5d9fW8SC19+Ab0g3FeUkL2uLC8QZ9IwMo\n\tWNyBsfJL6+ID/N2TgfeT6mx/hbNK5gig6sJODnigxwZHgAiiFIdZk4Er4CRnf55RDG\n\tzn0T1uJ2Nu+eHJw4FB44aKSzTYHuc1rKmZ+qHokWQ/Wro+MNakS/RZFYyh2u326+Gb\n\tsFzyk4Bg8Ykgj61ICwmth+o9r69ULjuvqcVTNxoZL607O84DOO/0E7XQd1soLdx/lw\n\tHIqIaJP0z6T4uoaD/dk9gctCFEYqLzcutmIJ4qJZHteJjPs09N0u/HRLSSo8bDD3xb\n\tHDpHDKfzc5CiQ==","To":"libcamera-devel@lists.libcamera.org","Date":"Thu, 8 Oct 2020 19:40:37 +0530","Message-Id":"<20201008141038.83425-3-email@uajain.com>","In-Reply-To":"<20201008141038.83425-1-email@uajain.com>","References":"<20201008141038.83425-1-email@uajain.com>","Mime-Version":"1.0","Subject":"[libcamera-devel] [RFC PATCH 2/3] android: jpeg: Port to\n\tPostProcessor interface","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Remove the existing Encoder interface completely and use the\nPostProcessor interface instead.\n\nNow the ::encode() function will be called by PostProcessor::process()\ninternally and will not be directly exposed in CameraStream. Similarly,\nother operations can be introduced internally in the PostProcessorJpeg,\nand can be called through its process() interface.\n\nSigned-off-by: Umang Jain \n---\n src/android/camera_device.h | 1 -\n src/android/camera_stream.cpp | 74 +++------------\n src/android/camera_stream.h | 9 +-\n src/android/jpeg/encoder.h | 25 -----\n ...er_libjpeg.cpp => post_processor_jpeg.cpp} | 92 +++++++++++++++++--\n ...ncoder_libjpeg.h => post_processor_jpeg.h} | 27 ++++--\n src/android/meson.build | 2 +-\n 7 files changed, 122 insertions(+), 108 deletions(-)\n delete mode 100644 src/android/jpeg/encoder.h\n rename src/android/jpeg/{encoder_libjpeg.cpp => post_processor_jpeg.cpp} (67%)\n rename src/android/jpeg/{encoder_libjpeg.h => post_processor_jpeg.h} (55%)","diff":"diff --git a/src/android/camera_device.h b/src/android/camera_device.h\nindex 777d1a3..25de12e 100644\n--- a/src/android/camera_device.h\n+++ b/src/android/camera_device.h\n@@ -25,7 +25,6 @@\n #include \"libcamera/internal/message.h\"\n \n #include \"camera_stream.h\"\n-#include \"jpeg/encoder.h\"\n \n class CameraMetadata;\n \ndiff --git a/src/android/camera_stream.cpp b/src/android/camera_stream.cpp\nindex eac1480..ed3bb41 100644\n--- a/src/android/camera_stream.cpp\n+++ b/src/android/camera_stream.cpp\n@@ -9,9 +9,7 @@\n \n #include \"camera_device.h\"\n #include \"camera_metadata.h\"\n-#include \"jpeg/encoder.h\"\n-#include \"jpeg/encoder_libjpeg.h\"\n-#include \"jpeg/exif.h\"\n+#include \"jpeg/post_processor_jpeg.h\"\n \n using namespace libcamera;\n \n@@ -24,8 +22,15 @@ CameraStream::CameraStream(CameraDevice *cameraDevice, Type type,\n {\n \tconfig_ = cameraDevice_->cameraConfiguration();\n \n-\tif (type_ == Type::Internal || type_ == Type::Mapped)\n-\t\tencoder_ = std::make_unique();\n+\tif (type_ == Type::Internal || type_ == Type::Mapped) {\n+\t\t/*\n+\t\t * \\todo There might be multiple post-processors. The logic\n+\t\t * which should be instantiated here, is deferred for future.\n+\t\t * For now, we only have PostProcessJpeg and that is what we\n+\t\t * will instantiate here.\n+\t\t */\n+\t\tpostProcessor_ = std::make_unique();\n+\t}\n \n \tif (type == Type::Internal) {\n \t\tallocator_ = std::make_unique(cameraDevice_->camera());\n@@ -45,8 +50,8 @@ Stream *CameraStream::stream() const\n \n int CameraStream::configure()\n {\n-\tif (encoder_) {\n-\t\tint ret = encoder_->configure(configuration());\n+\tif (postProcessor_) {\n+\t\tint ret = postProcessor_->configure(configuration());\n \t\tif (ret)\n \t\t\treturn ret;\n \t}\n@@ -69,60 +74,11 @@ int CameraStream::configure()\n int CameraStream::process(const libcamera::FrameBuffer &source,\n \t\t\t MappedCamera3Buffer *dest, CameraMetadata *metadata)\n {\n-\tif (!encoder_)\n+\tif (!postProcessor_)\n \t\treturn 0;\n \n-\t/* Set EXIF metadata for various tags. */\n-\tExif exif;\n-\t/* \\todo Set Make and Model from external vendor tags. */\n-\texif.setMake(\"libcamera\");\n-\texif.setModel(\"cameraModel\");\n-\texif.setOrientation(cameraDevice_->orientation());\n-\texif.setSize(configuration().size);\n-\t/*\n-\t * We set the frame's EXIF timestamp as the time of encode.\n-\t * Since the precision we need for EXIF timestamp is only one\n-\t * second, it is good enough.\n-\t */\n-\texif.setTimestamp(std::time(nullptr));\n-\tif (exif.generate() != 0)\n-\t\tLOG(HAL, Error) << \"Failed to generate valid EXIF data\";\n-\n-\tint jpeg_size = encoder_->encode(&source, dest->maps()[0], exif.data());\n-\tif (jpeg_size < 0) {\n-\t\tLOG(HAL, Error) << \"Failed to encode stream image\";\n-\t\treturn jpeg_size;\n-\t}\n-\n-\t/*\n-\t * Fill in the JPEG blob header.\n-\t *\n-\t * The mapped size of the buffer is being returned as\n-\t * substantially larger than the requested JPEG_MAX_SIZE\n-\t * (which is referenced from maxJpegBufferSize_). Utilise\n-\t * this static size to ensure the correct offset of the blob is\n-\t * determined.\n-\t *\n-\t * \\todo Investigate if the buffer size mismatch is an issue or\n-\t * expected behaviour.\n-\t */\n-\tuint8_t *resultPtr = dest->maps()[0].data() +\n-\t\t\t cameraDevice_->maxJpegBufferSize() -\n-\t\t\t sizeof(struct camera3_jpeg_blob);\n-\tauto *blob = reinterpret_cast(resultPtr);\n-\tblob->jpeg_blob_id = CAMERA3_JPEG_BLOB_ID;\n-\tblob->jpeg_size = jpeg_size;\n-\n-\t/* Update the JPEG result Metadata. */\n-\tmetadata->addEntry(ANDROID_JPEG_SIZE, &jpeg_size, 1);\n-\n-\tconst uint32_t jpeg_quality = 95;\n-\tmetadata->addEntry(ANDROID_JPEG_QUALITY, &jpeg_quality, 1);\n-\n-\tconst uint32_t jpeg_orientation = 0;\n-\tmetadata->addEntry(ANDROID_JPEG_ORIENTATION, &jpeg_orientation, 1);\n-\n-\treturn 0;\n+\treturn postProcessor_->process(&source, dest->maps()[0],\n+\t\t\t\t metadata, cameraDevice_);\n }\n \n FrameBuffer *CameraStream::getBuffer()\ndiff --git a/src/android/camera_stream.h b/src/android/camera_stream.h\nindex 8df0101..8d0f2e3 100644\n--- a/src/android/camera_stream.h\n+++ b/src/android/camera_stream.h\n@@ -19,7 +19,12 @@\n #include \n #include \n \n-class Encoder;\n+/*\n+ * \\todo Ideally we want to replace this include with forward-declaration.\n+ * If we do that. currently we get a compile error.\n+ */\n+#include \"post_processor.h\"\n+\n class CameraDevice;\n class CameraMetadata;\n class MappedCamera3Buffer;\n@@ -135,7 +140,7 @@ private:\n \t */\n \tunsigned int index_;\n \n-\tstd::unique_ptr encoder_;\n+\tstd::unique_ptr postProcessor_;\n \tstd::unique_ptr allocator_;\n \tstd::vector buffers_;\n \t/*\ndiff --git a/src/android/jpeg/encoder.h b/src/android/jpeg/encoder.h\ndeleted file mode 100644\nindex cf26d67..0000000\n--- a/src/android/jpeg/encoder.h\n+++ /dev/null\n@@ -1,25 +0,0 @@\n-/* SPDX-License-Identifier: GPL-2.0-or-later */\n-/*\n- * Copyright (C) 2020, Google Inc.\n- *\n- * encoder.h - Image encoding interface\n- */\n-#ifndef __ANDROID_JPEG_ENCODER_H__\n-#define __ANDROID_JPEG_ENCODER_H__\n-\n-#include \n-#include \n-#include \n-\n-class Encoder\n-{\n-public:\n-\tvirtual ~Encoder() {};\n-\n-\tvirtual int configure(const libcamera::StreamConfiguration &cfg) = 0;\n-\tvirtual int encode(const libcamera::FrameBuffer *source,\n-\t\t\t const libcamera::Span &destination,\n-\t\t\t const libcamera::Span &exifData) = 0;\n-};\n-\n-#endif /* __ANDROID_JPEG_ENCODER_H__ */\ndiff --git a/src/android/jpeg/encoder_libjpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp\nsimilarity index 67%\nrename from src/android/jpeg/encoder_libjpeg.cpp\nrename to src/android/jpeg/post_processor_jpeg.cpp\nindex 510613c..eeb4e95 100644\n--- a/src/android/jpeg/encoder_libjpeg.cpp\n+++ b/src/android/jpeg/post_processor_jpeg.cpp\n@@ -2,10 +2,14 @@\n /*\n * Copyright (C) 2020, Google Inc.\n *\n- * encoder_libjpeg.cpp - JPEG encoding using libjpeg native API\n+ * post_processor_jpeg.cpp - JPEG Post Processor\n */\n \n-#include \"encoder_libjpeg.h\"\n+#include \"post_processor_jpeg.h\"\n+\n+#include \"exif.h\"\n+\n+#include \"../camera_device.h\"\n \n #include \n #include \n@@ -25,6 +29,14 @@\n \n using namespace libcamera;\n \n+#define extract_va_arg(type, arg, last) \\\n+{ \\\n+ va_list ap; \\\n+ va_start(ap, last); \\\n+ arg = va_arg(ap, type); \\\n+ va_end(ap); \\\n+}\n+\n LOG_DEFINE_CATEGORY(JPEG)\n \n namespace {\n@@ -67,7 +79,7 @@ const struct JPEGPixelFormatInfo &findPixelInfo(const PixelFormat &format)\n \n } /* namespace */\n \n-EncoderLibJpeg::EncoderLibJpeg()\n+PostProcessorJpeg::PostProcessorJpeg()\n \t: quality_(95)\n {\n \t/* \\todo Expand error handling coverage with a custom handler. */\n@@ -76,12 +88,12 @@ EncoderLibJpeg::EncoderLibJpeg()\n \tjpeg_create_compress(&compress_);\n }\n \n-EncoderLibJpeg::~EncoderLibJpeg()\n+PostProcessorJpeg::~PostProcessorJpeg()\n {\n \tjpeg_destroy_compress(&compress_);\n }\n \n-int EncoderLibJpeg::configure(const StreamConfiguration &cfg)\n+int PostProcessorJpeg::configure(const StreamConfiguration &cfg)\n {\n \tconst struct JPEGPixelFormatInfo info = findPixelInfo(cfg.pixelFormat);\n \tif (info.colorSpace == JCS_UNKNOWN)\n@@ -104,7 +116,67 @@ int EncoderLibJpeg::configure(const StreamConfiguration &cfg)\n \treturn 0;\n }\n \n-void EncoderLibJpeg::compressRGB(const libcamera::MappedBuffer *frame)\n+int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n+\t\t\t const libcamera::Span &destination,\n+\t\t\t CameraMetadata *metadata, ...)\n+{\n+\tCameraDevice *device = nullptr;\n+\textract_va_arg(CameraDevice *, device, metadata);\n+\n+\t/* Set EXIF metadata for various tags. */\n+\tExif exif;\n+\t/* \\todo Set Make and Model from external vendor tags. */\n+\texif.setMake(\"libcamera\");\n+\texif.setModel(\"cameraModel\");\n+\texif.setOrientation(device->orientation());\n+\texif.setSize(Size {compress_.image_width, compress_.image_height});\n+\t/*\n+\t * We set the frame's EXIF timestamp as the time of encode.\n+\t * Since the precision we need for EXIF timestamp is only one\n+\t * second, it is good enough.\n+\t */\n+\texif.setTimestamp(std::time(nullptr));\n+\tif (exif.generate() != 0)\n+\t\tLOG(JPEG, Error) << \"Failed to generate valid EXIF data\";\n+\n+\tint jpeg_size = encode(source, destination, exif.data());\n+\tif (jpeg_size < 0) {\n+\t\tLOG(JPEG, Error) << \"Failed to encode stream image\";\n+\t\treturn jpeg_size;\n+\t}\n+\n+\t/*\n+\t * Fill in the JPEG blob header.\n+\t *\n+\t * The mapped size of the buffer is being returned as\n+\t * substantially larger than the requested JPEG_MAX_SIZE\n+\t * (which is referenced from maxJpegBufferSize_). Utilise\n+\t * this static size to ensure the correct offset of the blob is\n+\t * determined.\n+\t *\n+\t * \\todo Investigate if the buffer size mismatch is an issue or\n+\t * expected behaviour.\n+\t */\n+\tuint8_t *resultPtr = destination.data() +\n+\t\t\t device->maxJpegBufferSize() -\n+\t\t\t sizeof(struct camera3_jpeg_blob);\n+\tauto *blob = reinterpret_cast(resultPtr);\n+\tblob->jpeg_blob_id = CAMERA3_JPEG_BLOB_ID;\n+\tblob->jpeg_size = jpeg_size;\n+\n+\t/* Update the JPEG result Metadata. */\n+\tmetadata->addEntry(ANDROID_JPEG_SIZE, &jpeg_size, 1);\n+\n+\tconst uint32_t jpeg_quality = 95;\n+\tmetadata->addEntry(ANDROID_JPEG_QUALITY, &jpeg_quality, 1);\n+\n+\tconst uint32_t jpeg_orientation = 0;\n+\tmetadata->addEntry(ANDROID_JPEG_ORIENTATION, &jpeg_orientation, 1);\n+\n+\treturn 0;\n+}\n+\n+void PostProcessorJpeg::compressRGB(const libcamera::MappedBuffer *frame)\n {\n \tunsigned char *src = static_cast(frame->maps()[0].data());\n \t/* \\todo Stride information should come from buffer configuration. */\n@@ -122,7 +194,7 @@ void EncoderLibJpeg::compressRGB(const libcamera::MappedBuffer *frame)\n * Compress the incoming buffer from a supported NV format.\n * This naively unpacks the semi-planar NV12 to a YUV888 format for libjpeg.\n */\n-void EncoderLibJpeg::compressNV(const libcamera::MappedBuffer *frame)\n+void PostProcessorJpeg::compressNV(const libcamera::MappedBuffer *frame)\n {\n \tuint8_t tmprowbuf[compress_.image_width * 3];\n \n@@ -179,9 +251,9 @@ void EncoderLibJpeg::compressNV(const libcamera::MappedBuffer *frame)\n \t}\n }\n \n-int EncoderLibJpeg::encode(const FrameBuffer *source,\n-\t\t\t const libcamera::Span &dest,\n-\t\t\t const libcamera::Span &exifData)\n+int PostProcessorJpeg::encode(const FrameBuffer *source,\n+\t\t\t const libcamera::Span &dest,\n+\t\t\t const libcamera::Span &exifData)\n {\n \tMappedFrameBuffer frame(source, PROT_READ);\n \tif (!frame.isValid()) {\ndiff --git a/src/android/jpeg/encoder_libjpeg.h b/src/android/jpeg/post_processor_jpeg.h\nsimilarity index 55%\nrename from src/android/jpeg/encoder_libjpeg.h\nrename to src/android/jpeg/post_processor_jpeg.h\nindex 1e8df05..7f9ce70 100644\n--- a/src/android/jpeg/encoder_libjpeg.h\n+++ b/src/android/jpeg/post_processor_jpeg.h\n@@ -2,30 +2,37 @@\n /*\n * Copyright (C) 2020, Google Inc.\n *\n- * encoder_libjpeg.h - JPEG encoding using libjpeg\n+ * post_processor_jpeg.h - JPEG Post Processor\n */\n-#ifndef __ANDROID_JPEG_ENCODER_LIBJPEG_H__\n-#define __ANDROID_JPEG_ENCODER_LIBJPEG_H__\n+#ifndef __ANDROID_POST_PROCESSOR_JPEG_H__\n+#define __ANDROID_POST_PROCESSOR_JPEG_H__\n \n-#include \"encoder.h\"\n+#include \"../post_processor.h\"\n+#include \"../camera_metadata.h\"\n \n #include \"libcamera/internal/buffer.h\"\n #include \"libcamera/internal/formats.h\"\n \n #include \n \n-class EncoderLibJpeg : public Encoder\n+class PostProcessorJpeg : public PostProcessor\n {\n public:\n-\tEncoderLibJpeg();\n-\t~EncoderLibJpeg();\n+\tPostProcessorJpeg();\n+\t~PostProcessorJpeg();\n \n \tint configure(const libcamera::StreamConfiguration &cfg) override;\n+\tint process(const libcamera::FrameBuffer *source,\n+\t\t const libcamera::Span &destination,\n+\t\t CameraMetadata *metadata,\n+\t\t ...) override;\n+\n+\n+private:\n \tint encode(const libcamera::FrameBuffer *source,\n \t\t const libcamera::Span &destination,\n-\t\t const libcamera::Span &exifData) override;\n+\t\t const libcamera::Span &exifData);\n \n-private:\n \tvoid compressRGB(const libcamera::MappedBuffer *frame);\n \tvoid compressNV(const libcamera::MappedBuffer *frame);\n \n@@ -40,4 +47,4 @@ private:\n \tbool nvSwap_;\n };\n \n-#endif /* __ANDROID_JPEG_ENCODER_LIBJPEG_H__ */\n+#endif /* __ANDROID_POST_PROCESSOR_JPEG_H__ */\ndiff --git a/src/android/meson.build b/src/android/meson.build\nindex 802bb89..02b3b47 100644\n--- a/src/android/meson.build\n+++ b/src/android/meson.build\n@@ -21,8 +21,8 @@ android_hal_sources = files([\n 'camera_metadata.cpp',\n 'camera_ops.cpp',\n 'camera_stream.cpp',\n- 'jpeg/encoder_libjpeg.cpp',\n 'jpeg/exif.cpp',\n+ 'jpeg/post_processor_jpeg.cpp',\n ])\n \n android_camera_metadata_sources = files([\n","prefixes":["libcamera-devel","RFC","2/3"]}