{"id":10255,"url":"https://patchwork.libcamera.org/api/1.1/patches/10255/?format=json","web_url":"https://patchwork.libcamera.org/patch/10255/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20201026140134.44166-4-email@uajain.com>","date":"2020-10-26T14:01:34","name":"[libcamera-devel,3/3] android: jpeg: exif: Embed a JPEG-encoded thumbnail","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"2e4ac98b9e3bce3924d1c87bae5224ab5ef8a9df","submitter":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/people/1/?format=json","name":"Umang Jain","email":"email@uajain.com"},"delegate":{"id":12,"url":"https://patchwork.libcamera.org/api/1.1/users/12/?format=json","username":"uajain","first_name":"Umang","last_name":"Jain","email":"umang.jain@ideasonboard.com"},"mbox":"https://patchwork.libcamera.org/patch/10255/mbox/","series":[{"id":1416,"url":"https://patchwork.libcamera.org/api/1.1/series/1416/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=1416","date":"2020-10-26T14:01:31","name":"android: jpeg: exif: Embed a JPEG-encoded thumbnail","version":1,"mbox":"https://patchwork.libcamera.org/series/1416/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/10255/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/10255/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 2EDC5BDB13\n\tfor ;\n\tMon, 26 Oct 2020 14:02:42 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id EE77D61E91;\n\tMon, 26 Oct 2020 15:02:41 +0100 (CET)","from mail.uajain.com (static.126.159.217.95.clients.your-server.de\n\t[95.217.159.126])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id B263461E71\n\tfor ;\n\tMon, 26 Oct 2020 15:02:40 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=uajain.com header.i=@uajain.com\n\theader.b=\"L2jLFVYD\"; dkim-atps=neutral","From":"Umang Jain ","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=uajain.com; s=mail;\n\tt=1603720960; bh=QmYKKfebxTENrzsoOI3L5vBR4EfpOxRpBW+hX3xGy6g=;\n\th=From:To:Cc:Subject:In-Reply-To:References;\n\tb=L2jLFVYDY92+INuUlS+Ga0IkXrxSBuGWy/eAGCGslLtUT0Qvxbz91zDhuiElUXwec\n\tuCGASryRghRFkjtQVsfvU71T1ObsAbAe8p4NMBWOjdQksSoKnKSKY45hb8wow/Klnv\n\tFlrk3G5M1fLtCcGtahugYKau3qdFjiZB9x5Hi1st+adcJIdYftQj/DuBGZS+3xauT5\n\tHe4dxCfg6NuufLsSomYdayHEV7sEbg/2D8KgcSXB8v4qEVlYedcq0++3MBTxIIfdFk\n\tP2x9MQ7LAQvPwJYBZDFrRWUPzJB/+trJ9MZB6z2/1XLXbe7+iri1npX1uGuyTHeL5q\n\t2hRcoNU/tBVqw==","To":"libcamera-devel@lists.libcamera.org","Date":"Mon, 26 Oct 2020 19:31:34 +0530","Message-Id":"<20201026140134.44166-4-email@uajain.com>","In-Reply-To":"<20201026140134.44166-1-email@uajain.com>","References":"<20201026140134.44166-1-email@uajain.com>","Mime-Version":"1.0","Subject":"[libcamera-devel] [PATCH 3/3] android: jpeg: exif: Embed a\n\tJPEG-encoded thumbnail","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Add a basic image thumbnailer for NV12 frames being captured.\nIt shall generate a thumbnail image to be embedded as a part of\nEXIF metadata of the frame. The output of the thumbnail will still\nbe NV12.\n\nSigned-off-by: Umang Jain \n---\n src/android/jpeg/exif.cpp | 16 +++-\n src/android/jpeg/exif.h | 1 +\n src/android/jpeg/post_processor_jpeg.cpp | 35 +++++++-\n src/android/jpeg/post_processor_jpeg.h | 8 +-\n src/android/jpeg/thumbnailer.cpp | 109 +++++++++++++++++++++++\n src/android/jpeg/thumbnailer.h | 37 ++++++++\n src/android/meson.build | 1 +\n 7 files changed, 204 insertions(+), 3 deletions(-)\n create mode 100644 src/android/jpeg/thumbnailer.cpp\n create mode 100644 src/android/jpeg/thumbnailer.h","diff":"diff --git a/src/android/jpeg/exif.cpp b/src/android/jpeg/exif.cpp\nindex d21534a..24197bd 100644\n--- a/src/android/jpeg/exif.cpp\n+++ b/src/android/jpeg/exif.cpp\n@@ -75,8 +75,16 @@ Exif::~Exif()\n \tif (exifData_)\n \t\tfree(exifData_);\n \n-\tif (data_)\n+\tif (data_) {\n+\t\t/*\n+\t\t * Reset thumbnail data to avoid getting double-freed by\n+\t\t * libexif. It is owned by the caller (i.e. PostProcessorJpeg).\n+\t\t */\n+\t\tdata_->data = nullptr;\n+\t\tdata_->size = 0;\n+\n \t\texif_data_unref(data_);\n+\t}\n \n \tif (mem_)\n \t\texif_mem_unref(mem_);\n@@ -268,6 +276,12 @@ void Exif::setOrientation(int orientation)\n \tsetShort(EXIF_IFD_0, EXIF_TAG_ORIENTATION, value);\n }\n \n+void Exif::setThumbnail(std::vector &thumbnail)\n+{\n+\tdata_->data = thumbnail.data();\n+\tdata_->size = thumbnail.size();\n+}\n+\n [[nodiscard]] int Exif::generate()\n {\n \tif (exifData_) {\ndiff --git a/src/android/jpeg/exif.h b/src/android/jpeg/exif.h\nindex 12c27b6..bd54a31 100644\n--- a/src/android/jpeg/exif.h\n+++ b/src/android/jpeg/exif.h\n@@ -26,6 +26,7 @@ public:\n \n \tvoid setOrientation(int orientation);\n \tvoid setSize(const libcamera::Size &size);\n+\tvoid setThumbnail(std::vector &thumbnail);\n \tvoid setTimestamp(time_t timestamp);\n \n \tlibcamera::Span data() const { return { exifData_, size_ }; }\ndiff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp\nindex c56f1b2..416e831 100644\n--- a/src/android/jpeg/post_processor_jpeg.cpp\n+++ b/src/android/jpeg/post_processor_jpeg.cpp\n@@ -9,7 +9,6 @@\n \n #include \"../camera_device.h\"\n #include \"../camera_metadata.h\"\n-#include \"encoder_libjpeg.h\"\n #include \"exif.h\"\n \n #include \n@@ -39,11 +38,42 @@ int PostProcessorJpeg::configure(const StreamConfiguration &inCfg,\n \t}\n \n \tstreamSize_ = outCfg.size;\n+\n+\tthumbnailer_.configure(inCfg.size, inCfg.pixelFormat);\n+\tStreamConfiguration thCfg = inCfg;\n+\tthCfg.size = thumbnailer_.size();\n+\tif (thumbnailEncoder_.configure(thCfg) != 0) {\n+\t\tLOG(JPEG, Error) << \"Failed to configure thumbnail encoder\";\n+\t\treturn -EINVAL;\n+\t}\n+\n \tencoder_ = std::make_unique();\n \n \treturn encoder_->configure(inCfg);\n }\n \n+void PostProcessorJpeg::generateThumbnail(const FrameBuffer &source,\n+\t\t\t\t\t std::vector &thumbnail)\n+{\n+\t/* Stores the raw scaled-down thumbnail bytes. */\n+\tstd::vector rawThumbnail;\n+\n+\tthumbnailer_.scaleBuffer(source, rawThumbnail);\n+\n+\tif (rawThumbnail.data()) {\n+\t\tthumbnail.reserve(rawThumbnail.capacity());\n+\n+\t\tint jpeg_size = thumbnailEncoder_.encode(\n+\t\t\t\t{ rawThumbnail.data(), rawThumbnail.capacity() },\n+\t\t\t\t{ thumbnail.data(), thumbnail.capacity() },\n+\t\t\t\t{ });\n+\t\tthumbnail.resize(jpeg_size);\n+\n+\t\tLOG(JPEG, Info) << \"Thumbnail compress returned \"\n+\t\t\t\t<< jpeg_size << \" bytes\";\n+\t}\n+}\n+\n int PostProcessorJpeg::process(const FrameBuffer &source,\n \t\t\t Span destination,\n \t\t\t CameraMetadata *metadata)\n@@ -64,6 +94,9 @@ int PostProcessorJpeg::process(const FrameBuffer &source,\n \t * second, it is good enough.\n \t */\n \texif.setTimestamp(std::time(nullptr));\n+\tstd::vector thumbnail;\n+\tgenerateThumbnail(source, thumbnail);\n+\texif.setThumbnail(thumbnail);\n \tif (exif.generate() != 0)\n \t\tLOG(JPEG, Error) << \"Failed to generate valid EXIF data\";\n \ndiff --git a/src/android/jpeg/post_processor_jpeg.h b/src/android/jpeg/post_processor_jpeg.h\nindex 3706cec..3894231 100644\n--- a/src/android/jpeg/post_processor_jpeg.h\n+++ b/src/android/jpeg/post_processor_jpeg.h\n@@ -8,12 +8,13 @@\n #define __ANDROID_POST_PROCESSOR_JPEG_H__\n \n #include \"../post_processor.h\"\n+#include \"encoder_libjpeg.h\"\n+#include \"thumbnailer.h\"\n \n #include \n \n #include \"libcamera/internal/buffer.h\"\n \n-class Encoder;\n class CameraDevice;\n \n class PostProcessorJpeg : public PostProcessor\n@@ -28,9 +29,14 @@ public:\n \t\t CameraMetadata *metadata) override;\n \n private:\n+\tvoid generateThumbnail(const libcamera::FrameBuffer &source,\n+\t\t\t std::vector &thumbnail);\n+\n \tCameraDevice *const cameraDevice_;\n \tstd::unique_ptr encoder_;\n \tlibcamera::Size streamSize_;\n+\tEncoderLibJpeg thumbnailEncoder_;\n+\tThumbnailer thumbnailer_;\n };\n \n #endif /* __ANDROID_POST_PROCESSOR_JPEG_H__ */\ndiff --git a/src/android/jpeg/thumbnailer.cpp b/src/android/jpeg/thumbnailer.cpp\nnew file mode 100644\nindex 0000000..f880ffb\n--- /dev/null\n+++ b/src/android/jpeg/thumbnailer.cpp\n@@ -0,0 +1,109 @@\n+/* SPDX-License-Identifier: GPL-2.0-or-later */\n+/*\n+ * Copyright (C) 2020, Google Inc.\n+ *\n+ * thumbnailer.cpp - Simple image thumbnailer\n+ */\n+\n+#include \"thumbnailer.h\"\n+\n+#include \n+\n+#include \"libcamera/internal/file.h\"\n+#include \"libcamera/internal/log.h\"\n+\n+using namespace libcamera;\n+\n+LOG_DEFINE_CATEGORY(Thumbnailer)\n+\n+Thumbnailer::Thumbnailer() : valid_(false)\n+{\n+}\n+\n+void Thumbnailer::configure(const Size &sourceSize, PixelFormat pixelFormat)\n+{\n+\tsourceSize_ = sourceSize;\n+\tpixelFormat_ = pixelFormat;\n+\n+\tif (pixelFormat_ != formats::NV12) {\n+\t\tLOG (Thumbnailer, Error) << \"Failed to configure: Pixel Format \"\n+\t\t\t\t << pixelFormat_.toString() << \" unsupported.\";\n+\t\treturn;\n+\t}\n+\n+\ttargetSize_ = computeThumbnailSize();\n+\n+\tvalid_ = true;\n+}\n+\n+/*\n+ * The Exif specification recommends the width of the thumbnail to be a\n+ * mutiple of 16 (section 4.8.1). Hence, compute the corresponding height\n+ * keeping the aspect ratio same as of the source.\n+ */\n+Size Thumbnailer::computeThumbnailSize()\n+{\n+\tunsigned int targetHeight;\n+\tunsigned int targetWidth = 160;\n+\n+\ttargetHeight = targetWidth * sourceSize_.height / sourceSize_.width;\n+\n+\tif (targetHeight & 1)\n+\t\ttargetHeight++;\n+\n+\treturn Size(targetWidth, targetHeight);\n+}\n+\n+void\n+Thumbnailer::scaleBuffer(const FrameBuffer &source,\n+\t\t\t std::vector &destination)\n+{\n+\tMappedFrameBuffer frame(&source, PROT_READ);\n+\tif (!frame.isValid()) {\n+\t\tLOG(Thumbnailer, Error) << \"Failed to map FrameBuffer : \"\n+\t\t\t\t << strerror(frame.error());\n+\t\treturn;\n+\t}\n+\n+\tif (!valid_) {\n+\t\tLOG(Thumbnailer, Error) << \"config is unconfigured or invalid.\";\n+\t\treturn;\n+\t}\n+\n+\tconst unsigned int sw = sourceSize_.width;\n+\tconst unsigned int sh = sourceSize_.height;\n+\tconst unsigned int tw = targetSize_.width;\n+\tconst unsigned int th = targetSize_.height;\n+\n+\t/* Image scaling block implementing nearest-neighbour algorithm. */\n+\tunsigned char *src = static_cast(frame.maps()[0].data());\n+\tunsigned char *src_c = src + sh * sw;\n+\tunsigned char *src_cb, *src_cr;\n+\tunsigned char *dst_y, *src_y;\n+\n+\tsize_t dstSize = (th * tw) + ((th / 2) * tw);\n+\tdestination.reserve(dstSize);\n+\tunsigned char *dst = destination.data();\n+\tunsigned char *dst_c = dst + th * tw;\n+\n+\tfor (unsigned int y = 0; y < th; y += 2) {\n+\t\tunsigned int sourceY = (sh * y + th / 2) / th;\n+\n+\t\tdst_y = dst + y * tw;\n+\t\tsrc_y = src + sw * sourceY;\n+\t\tsrc_cb = src_c + (sourceY / 2) * sw + 0;\n+\t\tsrc_cr = src_c + (sourceY / 2) * sw + 1;\n+\n+\t\tfor (unsigned int x = 0; x < tw; x += 2) {\n+\t\t\tunsigned int sourceX = (sw * x + tw / 2) / tw;\n+\n+\t\t\tdst_y[x] = src_y[sourceX];\n+\t\t\tdst_y[tw + x] = src_y[sw + sourceX];\n+\t\t\tdst_y[x + 1] = src_y[sourceX + 1];\n+\t\t\tdst_y[tw + x + 1] = src_y[sw + sourceX + 1];\n+\n+\t\t\tdst_c[(y / 2) * tw + x + 0] = src_cb[(sourceX / 2) * 2];\n+\t\t\tdst_c[(y / 2) * tw + x + 1] = src_cr[(sourceX / 2) * 2];\n+\t\t}\n+\t}\n+ }\ndiff --git a/src/android/jpeg/thumbnailer.h b/src/android/jpeg/thumbnailer.h\nnew file mode 100644\nindex 0000000..b769619\n--- /dev/null\n+++ b/src/android/jpeg/thumbnailer.h\n@@ -0,0 +1,37 @@\n+/* SPDX-License-Identifier: GPL-2.0-or-later */\n+/*\n+ * Copyright (C) 2020, Google Inc.\n+ *\n+ * thumbnailer.h - Simple image thumbnailer\n+ */\n+#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n+#define __ANDROID_JPEG_THUMBNAILER_H__\n+\n+#include \n+\n+#include \"libcamera/internal/buffer.h\"\n+#include \"libcamera/internal/formats.h\"\n+\n+class Thumbnailer\n+{\n+public:\n+\tThumbnailer();\n+\n+\tvoid configure(const libcamera::Size &sourceSize,\n+\t\t libcamera::PixelFormat pixelFormat);\n+\tvoid scaleBuffer(const libcamera::FrameBuffer &source,\n+\t\t\t std::vector &dest);\n+\tlibcamera::Size size() const { return targetSize_; }\n+\n+private:\n+\tlibcamera::Size computeThumbnailSize();\n+\n+\tlibcamera::PixelFormat pixelFormat_;\n+\tlibcamera::Size sourceSize_;\n+\tlibcamera::Size targetSize_;\n+\n+\tbool valid_;\n+\n+};\n+\n+#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\ndiff --git a/src/android/meson.build b/src/android/meson.build\nindex f72376a..3d4d3be 100644\n--- a/src/android/meson.build\n+++ b/src/android/meson.build\n@@ -25,6 +25,7 @@ android_hal_sources = files([\n 'jpeg/encoder_libjpeg.cpp',\n 'jpeg/exif.cpp',\n 'jpeg/post_processor_jpeg.cpp',\n+ 'jpeg/thumbnailer.cpp',\n ])\n \n android_camera_metadata_sources = files([\n","prefixes":["libcamera-devel","3/3"]}