[{"id":13316,"web_url":"https://patchwork.libcamera.org/comment/13316/","msgid":"","date":"2020-10-21T09:06:03","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi Umang,\n\nOn 21/10/2020 09:08, Umang Jain wrote:\n> Add a basic image thumbnailer for NV12 frames being captured.\n> It shall generate a thumbnail image to be embedded as a part of\n> EXIF metadata of the frame. The output of the thumbnail will still\n> be NV12.\n> \n> Signed-off-by: Umang Jain \n> ---\n> src/android/jpeg/post_processor_jpeg.cpp | 39 +++++++++\n> src/android/jpeg/post_processor_jpeg.h | 3 +\n> src/android/jpeg/thumbnailer.cpp | 100 +++++++++++++++++++++++\n> src/android/jpeg/thumbnailer.h | 40 +++++++++\n> src/android/meson.build | 1 +\n> 5 files changed, 183 insertions(+)\n> create mode 100644 src/android/jpeg/thumbnailer.cpp\n> create mode 100644 src/android/jpeg/thumbnailer.h\n> \n> diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp\n> index 9d452b7..f5f1f78 100644\n> --- a/src/android/jpeg/post_processor_jpeg.cpp\n> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n> @@ -11,6 +11,7 @@\n> #include \"../camera_metadata.h\"\n> #include \"encoder_libjpeg.h\"\n> #include \"exif.h\"\n> +#include \"thumbnailer.h\"\n> \n> #include \n> \n> @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const StreamConfiguration &inCfg,\n> \treturn encoder_->configure(inCfg);\n> }\n> \n> +void PostProcessorJpeg::generateThumbnail(const libcamera::Span &source,\n> +\t\t\t\t\t std::vector &thumbnail)\n> +{\n> +\tlibcamera::Span destination;\n> +\tThumbnailer thumbnailer;\n> +\n> +\tthumbnailer.configure(streamSize_, formats::NV12);\n\nIt would be nicer to see this format set from the input configuration,\nso that if we ever get given a differnet format the thumbnailer will\nknow (and complain accordingly, until it supports that format).\n\n\n\n> +\tlibcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n> +\tthumbnailer.scaleBuffer(source, thumbnail);\n> +\n> +\tif (thumbnail.data()) {\n> +\t\tStreamConfiguration thumbnailCfg;\n> +\n> +\t\tstd::unique_ptr encoder =\n> +\t\t\t\tstd::make_unique();\n\n\nNo need to make this a unique pointer. It can just be constructed as a\nlocal object:\n\t\tEncoderLibJpeg encoder;\n\nThe only reason the PostProcessorJpeg::encoder is a unique_ptr is\nbecause it might be assigned to different types of encoder:\n\n\n\n\nOr perhaps rather than creating a new one each time, you could put it as\na private member in the PostProcessorJpeg, and configure it once during\nPostProcessorJpeg::configure();\n\nclass PostProcessorJpeg : public PostProcessor\n{\nprivate\n\t\tEncoderLibJpeg thumbnailEncoder_;\n}\n\nint PostProcessorJpeg::configure(incfg, outcfg)\n{\n....\n\tthumbnailEncoder_.configure(...);\n....\n}\n\n\nint PostProcessorJpeg::generateThumbnail(....)\n{\n\t...\n\t\n\tthumbnailEncoder_.encode(...);\n\t...\n\n}\n\n> +\n> +\t\tthumbnailCfg.pixelFormat = formats::NV12;\n> +\t\tthumbnailCfg.size = targetSize;\n> +\t\tencoder->configure(thumbnailCfg);\n> +\t\tint jpeg_size = encoder->encode({ thumbnail.data(), thumbnail.capacity() },\n> +\t\t\t\tdestination, { });\n> +\t\tLOG(JPEG, Info) << \"Thumbnail compress returned \"\n> +\t\t\t\t<< jpeg_size << \" bytes\";\n> +\t}\n> +}\n> +\n> int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n> \t\t\t const libcamera::Span &destination,\n> \t\t\t CameraMetadata *metadata)\n> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n> \t\treturn jpeg_size;\n> \t}\n> \n> +\tstd::vector thumbnail;\n> +\tgenerateThumbnail(destination, thumbnail);\n> +\t/*\n> +\t * \\todo: Write the compressed thumbnail to a file for inspection.\n> +\t * (I) Check if we can still write the thumbnail to EXIF here.\n> +\t * If not, we might need to move the thumbnailer logic to encoder.\n> +\t * And if we do that, first we need to make sure we get can\n> +\t * compressed data written to destination first before calling\n> +\t * jpeg_finish_compress operation somehow. Thumbnailing will\n> +\t * only occur if we have compressed data available first.\n\nI don't see how thumbnailing will help in the encoder. The exif is\nconstructed in PostProcessorJpeg::process().\n\nPerhaps it's about getting the thumbnail into the encoder/exif.\nFor that I'd add a helper to our Exif class to be able to\n\n exif.setThumbnail(span &thumbnail);\n\nThen the exif.generate() should ideally make sure the exif is fully\ncreated correctly. (including the thumbnail) before it goes into the\nencapsulating jpeg.\n\n> +\t */\n> +\n> \t/*\n> \t * Fill in the JPEG blob header.\n> \t *\n> diff --git a/src/android/jpeg/post_processor_jpeg.h b/src/android/jpeg/post_processor_jpeg.h\n> index 62c8650..05601ee 100644\n> --- a/src/android/jpeg/post_processor_jpeg.h\n> +++ b/src/android/jpeg/post_processor_jpeg.h\n> @@ -28,6 +28,9 @@ public:\n> \t\t CameraMetadata *metadata) override;\n> \n> private:\n> +\tvoid generateThumbnail(const libcamera::Span &source,\n> +\t\t\t std::vector &thumbnail);\n> +\n> \tCameraDevice *cameraDevice_;\n> \tstd::unique_ptr encoder_;\n> \tlibcamera::Size streamSize_;\n> diff --git a/src/android/jpeg/thumbnailer.cpp b/src/android/jpeg/thumbnailer.cpp\n> new file mode 100644\n> index 0000000..3163576\n> --- /dev/null\n> +++ b/src/android/jpeg/thumbnailer.cpp\n> @@ -0,0 +1,100 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n\nI'd call it a 'Simple image thumbnailer'\n\nNo need to specify the format in the description. It might need\nextending later anyway.\n\n\n> + */\n> +\n> +#include \"thumbnailer.h\"\n> +\n> +#include \n> +\n> +#include \"libcamera/internal/file.h\"\n> +#include \"libcamera/internal/log.h\"\n> +\n> +using namespace libcamera;\n> +\n> +LOG_DEFINE_CATEGORY(Thumbnailer)\n> +\n> +Thumbnailer::Thumbnailer()\n> +\t: validConfiguration_(false)\n\nthis could simply be valid_ if you want it to be shorter, but this is fine.\n\n\n> +{\n> +}\n> +\n> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat pixelFormat)\n> +{\n> +\tsourceSize_ = sourceSize;\n> +\tpixelFormat_ = pixelFormat;\n> +\n> +\tif (pixelFormat_ != formats::NV12) {\n> +\t\tLOG (Thumbnailer, Error) << \"Failed to configure: Pixel Format \"\n> +\t\t\t\t << pixelFormat_.toString() << \" unsupported.\";\n> +\t\treturn;\n> +\t}\n> +\n> +\tvalidConfiguration_ = true;\n> +}\n> +\n> +/*\n> + * The Exif specification recommends the width of the thumbnail to be a\n> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding height\n> + * keeping the aspect ratio same as of the source.\n> + */\n> +Size Thumbnailer::computeThumbnailSize()\n> +{\n> +\tunsigned int targetHeight;\n> +\tunsigned int targetWidth = 160;\n> +\n> +\ttargetHeight = targetWidth * sourceSize_.height / sourceSize_.width;\n> +\n> +\tif (targetHeight & 1)\n> +\t\ttargetHeight++;\n> +\n> +\treturn Size(targetWidth, targetHeight);\n> +}\n\nI would make this a private function, and call it from configure().\nOr inline it in configure (It can be kept separate if that helps/makes\nsense).\n\nThen have a public function:\n\n Thumbnailer::size() { return outputSize_; } const\n\n> +\n> +void\n> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n> +\t\t\t std::vector &destination)\n> +{\n> +\tif (!validConfiguration_) {\n> +\t\tLOG(Thumbnailer, Error) << \"config is unconfigured or invalid.\";\n\nPerhaps:\n\n\"Thumbnailer has an invalid configuration.\";\n\n\n> +\t\treturn;\n> +\t}\n> +\n> +\ttargetSize_ = computeThumbnailSize();\n\nThis should be in configure();\n\n> +\n> +\tconst unsigned int sw = sourceSize_.width;\n> +\tconst unsigned int sh = sourceSize_.height;\n> +\tconst unsigned int tw = targetSize_.width;\n> +\tconst unsigned int th = targetSize_.height;\n> +\n> +\t/* Image scaling block implementing nearest-neighbour algorithm. */\n> +\tunsigned char *src = static_cast(source.data());\n> +\tunsigned char *src_c = src + sh * sw;\n> +\tunsigned char *src_cb, *src_cr;\n> +\n> +\tsize_t dstSize = (th * tw) + ((th/2) * tw);\n> +\tdestination.reserve(dstSize);\n> +\tunsigned char *dst = destination.data();\n> +\tunsigned char *dst_c = dst + th * tw;\n> +\n> +\tfor (unsigned int y = 0; y < th; y+=2) {\n> +\t\tunsigned int sourceY = (sh*y + th/2) / th;\n> +\n> +\t\tsrc_cb = src_c + (sourceY/2) * sw + 0;\n> +\t\tsrc_cr = src_c + (sourceY/2) * sw + 1;\n> +\n> +\t\tfor (unsigned int x = 0; x < tw; x+=2) {\n> +\t\t\tunsigned int sourceX = (sw*x + tw/2) / tw;\n> +\n> +\t\t\tdst[y * tw + x] = src[sw * sourceY + sourceX];\n> +\t\t\tdst[(y+1) * tw + x] = src[sw * (sourceY+1) + sourceX];\n> +\t\t\tdst[y * tw + (x+1)] = src[sw * sourceY + (sourceX+1)];\n> +\t\t\tdst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) + (sourceX+1)];\n> +\n> +\t\t\tdst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n> +\t\t\tdst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n> +\t\t}\n> +\t}\n> +}\n> diff --git a/src/android/jpeg/thumbnailer.h b/src/android/jpeg/thumbnailer.h\n> new file mode 100644\n> index 0000000..bab9855\n> --- /dev/null\n> +++ b/src/android/jpeg/thumbnailer.h\n> @@ -0,0 +1,40 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * thumbnailer.h - Basic image thumbnailer from NV12\n> + */\n> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n> +#define __ANDROID_JPEG_THUMBNAILER_H__\n> +\n> +#include \n> +\n> +#include \"libcamera/internal/buffer.h\"\n> +#include \"libcamera/internal/formats.h\"\n> +\n> +class Thumbnailer\n> +{\n> +public:\n> +\tThumbnailer();\n> +\n> +\tvoid configure(const libcamera::Size &sourceSize,\n> +\t\t libcamera::PixelFormat pixelFormat);\n\n\nShouldn't configure return an int to return if the configuration is\nunsupported ?\n\n(which in this current case would only be if someone supplies an\nincorrect pixelformat I expect).\n\nAlternatively, we could simplify the thumbnailer down to just a single\nprocess function which passes in the source size,\n\n> +\n> +\t/*\n> +\t * \\todo: Discuss if we can return targetSize_ via configure() or\n> +\t * scaleBuffer(). We need targetSize_ to re-encode the scaled buffer\n> +\t * via encoder in PostProcssorJpeg::writeThumbnail().\n\ns/PostProcssorJpeg/PostProcessorJpeg/\n\n> +\t */\n> +\tlibcamera::Size computeThumbnailSize();\n> +\tvoid scaleBuffer(const libcamera::Span &source,\n> +\t\t\t std::vector &dest);\n> +\n> +private:\n> +\tlibcamera::PixelFormat pixelFormat_;\n\nUntil this supports something that isn't NV12, you don't actually need\nto store the pixelFormat_;\n\n\n> +\tlibcamera::Size sourceSize_;\n> +\tlibcamera::Size targetSize_;\n> +\n> +\tbool validConfiguration_;\n> +};\n> +\n> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n> diff --git a/src/android/meson.build b/src/android/meson.build\n> index 5a01bea..3905e2f 100644\n> --- a/src/android/meson.build\n> +++ b/src/android/meson.build\n> @@ -25,6 +25,7 @@ android_hal_sources = files([\n> 'jpeg/encoder_libjpeg.cpp',\n> 'jpeg/exif.cpp',\n> 'jpeg/post_processor_jpeg.cpp',\n> + 'jpeg/thumbnailer.cpp',\n> ])\n> \n> android_camera_metadata_sources = files([\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 8AE32C3D3C\n\tfor ;\n\tWed, 21 Oct 2020 09:06:09 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 0738961D98;\n\tWed, 21 Oct 2020 11:06:09 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 3022B60352\n\tfor ;\n\tWed, 21 Oct 2020 11:06:07 +0200 (CEST)","from [192.168.0.20]\n\t(cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id A2CDD92;\n\tWed, 21 Oct 2020 11:06:06 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"Qyx2KOup\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1603271166;\n\tbh=FHhic2FBsIv40vI0CGVbdEClGTP09glns0tcAQih9Y4=;\n\th=Reply-To:Subject:To:References:From:Date:In-Reply-To:From;\n\tb=Qyx2KOupN3iVt74v1ustDDjw+DpMvy922O5L9bG9HakLR3RS44VjR8b7YAe5zqJR+\n\tnQDCKFti2gXqYEWXbhrBPUu5GGMHRfkj9hZ2r9tbOk2RAe52vI0SvK0ItJR5Wwjl4F\n\t3b/ZqItluYPudUUG+jmlip1ri/Jy1pC8RsfZHLSE=","To":"Umang Jain , libcamera-devel@lists.libcamera.org","References":"<20201021080806.46636-1-email@uajain.com>\n\t<20201021080806.46636-3-email@uajain.com>","From":"Kieran Bingham ","Autocrypt":"addr=kieran.bingham@ideasonboard.com; keydata=\n\tmQINBFYE/WYBEACs1PwjMD9rgCu1hlIiUA1AXR4rv2v+BCLUq//vrX5S5bjzxKAryRf0uHat\n\tV/zwz6hiDrZuHUACDB7X8OaQcwhLaVlq6byfoBr25+hbZG7G3+5EUl9cQ7dQEdvNj6V6y/SC\n\trRanWfelwQThCHckbobWiQJfK9n7rYNcPMq9B8e9F020LFH7Kj6YmO95ewJGgLm+idg1Kb3C\n\tpotzWkXc1xmPzcQ1fvQMOfMwdS+4SNw4rY9f07Xb2K99rjMwZVDgESKIzhsDB5GY465sCsiQ\n\tcSAZRxqE49RTBq2+EQsbrQpIc8XiffAB8qexh5/QPzCmR4kJgCGeHIXBtgRj+nIkCJPZvZtf\n\tKr2EAbc6tgg6DkAEHJb+1okosV09+0+TXywYvtEop/WUOWQ+zo+Y/OBd+8Ptgt1pDRyOBzL8\n\tRXa8ZqRf0Mwg75D+dKntZeJHzPRJyrlfQokngAAs4PaFt6UfS+ypMAF37T6CeDArQC41V3ko\n\tlPn1yMsVD0p+6i3DPvA/GPIksDC4owjnzVX9kM8Zc5Cx+XoAN0w5Eqo4t6qEVbuettxx55gq\n\t8K8FieAjgjMSxngo/HST8TpFeqI5nVeq0/lqtBRQKumuIqDg+Bkr4L1V/PSB6XgQcOdhtd36\n\tOe9X9dXB8YSNt7VjOcO7BTmFn/Z8r92mSAfHXpb07YJWJosQOQARAQABtDBLaWVyYW4gQmlu\n\tZ2hhbSA8a2llcmFuLmJpbmdoYW1AaWRlYXNvbmJvYXJkLmNvbT6JAlcEEwEKAEECGwMFCwkI\n\tBwIGFQgJCgsCBBYCAwECHgECF4ACGQEWIQSQLdeYP70o/eNy1HqhHkZyEKRh/QUCXWTtygUJ\n\tCyJXZAAKCRChHkZyEKRh/f8dEACTDsbLN2nioNZMwyLuQRUAFcXNolDX48xcUXsWS2QjxaPm\n\tVsJx8Uy8aYkS85mdPBh0C83OovQR/OVbr8AxhGvYqBs3nQvbWuTl/+4od7DfK2VZOoKBAu5S\n\tQK2FYuUcikDqYcFWJ8DQnubxfE8dvzojHEkXw0sA4igINHDDFX3HJGZtLio+WpEFQtCbfTAG\n\tYZslasz1YZRbwEdSsmO3/kqy5eMnczlm8a21A3fKUo3g8oAZEFM+f4DUNzqIltg31OAB/kZS\n\tenKZQ/SWC8PmLg/ZXBrReYakxXtkP6w3FwMlzOlhGxqhIRNiAJfXJBaRhuUWzPOpEDE9q5YJ\n\tBmqQL2WJm1VSNNVxbXJHpaWMH1sA2R00vmvRrPXGwyIO0IPYeUYQa3gsy6k+En/aMQJd27dp\n\taScf9am9PFICPY5T4ppneeJLif2lyLojo0mcHOV+uyrds9XkLpp14GfTkeKPdPMrLLTsHRfH\n\tfA4I4OBpRrEPiGIZB/0im98MkGY/Mu6qxeZmYLCcgD6qz4idOvfgVOrNh+aA8HzIVR+RMW8H\n\tQGBN9f0E3kfwxuhl3omo6V7lDw8XOdmuWZNC9zPq1UfryVHANYbLGz9KJ4Aw6M+OgBC2JpkD\n\thXMdHUkC+d20dwXrwHTlrJi1YNp6rBc+xald3wsUPOZ5z8moTHUX/uPA/qhGsbkCDQRWBP1m\n\tARAAzijkb+Sau4hAncr1JjOY+KyFEdUNxRy+hqTJdJfaYihxyaj0Ee0P0zEi35CbE6lgU0Uz\n\ttih9fiUbSV3wfsWqg1Ut3/5rTKu7kLFp15kF7eqvV4uezXRD3Qu4yjv/rMmEJbbD4cTvGCYI\n\td6MDC417f7vK3hCbCVIZSp3GXxyC1LU+UQr3fFcOyCwmP9vDUR9JV0BSqHHxRDdpUXE26Dk6\n\tmhf0V1YkspE5St814ETXpEus2urZE5yJIUROlWPIL+hm3NEWfAP06vsQUyLvr/GtbOT79vXl\n\tEn1aulcYyu20dRRxhkQ6iILaURcxIAVJJKPi8dsoMnS8pB0QW12AHWuirPF0g6DiuUfPmrA5\n\tPKe56IGlpkjc8cO51lIxHkWTpCMWigRdPDexKX+Sb+W9QWK/0JjIc4t3KBaiG8O4yRX8ml2R\n\t+rxfAVKM6V769P/hWoRGdgUMgYHFpHGSgEt80OKK5HeUPy2cngDUXzwrqiM5Sz6Od0qw5pCk\n\tNlXqI0W/who0iSVM+8+RmyY0OEkxEcci7rRLsGnM15B5PjLJjh1f2ULYkv8s4SnDwMZ/kE04\n\t/UqCMK/KnX8pwXEMCjz0h6qWNpGwJ0/tYIgQJZh6bqkvBrDogAvuhf60Sogw+mH8b+PBlx1L\n\toeTK396wc+4c3BfiC6pNtUS5GpsPMMjYMk7kVvEAEQEAAYkCPAQYAQoAJgIbDBYhBJAt15g/\n\tvSj943LUeqEeRnIQpGH9BQJdizzIBQkLSKZiAAoJEKEeRnIQpGH9eYgQAJpjaWNgqNOnMTmD\n\tMJggbwjIotypzIXfhHNCeTkG7+qCDlSaBPclcPGYrTwCt0YWPU2TgGgJrVhYT20ierN8LUvj\n\t6qOPTd+Uk7NFzL65qkh80ZKNBFddx1AabQpSVQKbdcLb8OFs85kuSvFdgqZwgxA1vl4TFhNz\n\tPZ79NAmXLackAx3sOVFhk4WQaKRshCB7cSl+RIng5S/ThOBlwNlcKG7j7W2MC06BlTbdEkUp\n\tECzuuRBv8wX4OQl+hbWbB/VKIx5HKlLu1eypen/5lNVzSqMMIYkkZcjV2SWQyUGxSwq0O/sx\n\tS0A8/atCHUXOboUsn54qdxrVDaK+6jIAuo8JiRWctP16KjzUM7MO0/+4zllM8EY57rXrj48j\n\tsbEYX0YQnzaj+jO6kJtoZsIaYR7rMMq9aUAjyiaEZpmP1qF/2sYenDx0Fg2BSlLvLvXM0vU8\n\tpQk3kgDu7kb/7PRYrZvBsr21EIQoIjXbZxDz/o7z95frkP71EaICttZ6k9q5oxxA5WC6sTXc\n\tMW8zs8avFNuA9VpXt0YupJd2ijtZy2mpZNG02fFVXhIn4G807G7+9mhuC4XG5rKlBBUXTvPU\n\tAfYnB4JBDLmLzBFavQfvonSfbitgXwCG3vS+9HEwAjU30Bar1PEOmIbiAoMzuKeRm2LVpmq4\n\tWZw01QYHU/GUV/zHJSFk","Organization":"Ideas on Board","Message-ID":"","Date":"Wed, 21 Oct 2020 10:06:03 +0100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101\n\tThunderbird/68.10.0","MIME-Version":"1.0","In-Reply-To":"<20201021080806.46636-3-email@uajain.com>","Content-Language":"en-GB","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Reply-To":"kieran.bingham@ideasonboard.com","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13317,"web_url":"https://patchwork.libcamera.org/comment/13317/","msgid":"","date":"2020-10-21T09:19:15","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi Umang,\n\nOn 21/10/2020 09:08, Umang Jain wrote:\n> Add a basic image thumbnailer for NV12 frames being captured.\n> It shall generate a thumbnail image to be embedded as a part of\n> EXIF metadata of the frame. The output of the thumbnail will still\n> be NV12.\n> \n> Signed-off-by: Umang Jain \n> ---\n> src/android/jpeg/post_processor_jpeg.cpp | 39 +++++++++\n> src/android/jpeg/post_processor_jpeg.h | 3 +\n> src/android/jpeg/thumbnailer.cpp | 100 +++++++++++++++++++++++\n> src/android/jpeg/thumbnailer.h | 40 +++++++++\n> src/android/meson.build | 1 +\n> 5 files changed, 183 insertions(+)\n> create mode 100644 src/android/jpeg/thumbnailer.cpp\n> create mode 100644 src/android/jpeg/thumbnailer.h\n> \n> diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp\n> index 9d452b7..f5f1f78 100644\n> --- a/src/android/jpeg/post_processor_jpeg.cpp\n> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n> @@ -11,6 +11,7 @@\n> #include \"../camera_metadata.h\"\n> #include \"encoder_libjpeg.h\"\n> #include \"exif.h\"\n> +#include \"thumbnailer.h\"\n> \n> #include \n> \n> @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const StreamConfiguration &inCfg,\n> \treturn encoder_->configure(inCfg);\n> }\n> \n> +void PostProcessorJpeg::generateThumbnail(const libcamera::Span &source,\n> +\t\t\t\t\t std::vector &thumbnail)\n> +{\n> +\tlibcamera::Span destination;\n> +\tThumbnailer thumbnailer;\n> +\n> +\tthumbnailer.configure(streamSize_, formats::NV12);\n> +\tlibcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n> +\tthumbnailer.scaleBuffer(source, thumbnail);\n> +\n> +\tif (thumbnail.data()) {\n> +\t\tStreamConfiguration thumbnailCfg;\n> +\n> +\t\tstd::unique_ptr encoder =\n> +\t\t\t\tstd::make_unique();\n> +\n> +\t\tthumbnailCfg.pixelFormat = formats::NV12;\n> +\t\tthumbnailCfg.size = targetSize;\n> +\t\tencoder->configure(thumbnailCfg);\n\nthumbnail.capacity() might be quite low here.\nWe need to make sure the vector is big enough at this point, you might\nneed to do something like:\n\n\tthumbnail.resize(targetSize.width * targetSize.height * 4);\n\n\nReally we should obtain that size from the encoder. I thought we had a\nhelper to do that already, but it seems we don't.\n\nWe could/should add Encoder::maxOutput(); which would ask the encoder\n\"For your given configuration, what is the maximum number of bytes you\nmight output\".\n\nThen of course we'd do:\n\tthumbnail.resize(encoder.maxOutput());\n\n\n> +\t\tint jpeg_size = encoder->encode({ thumbnail.data(), thumbnail.capacity() },\n> +\t\t\t\tdestination, { });\n> +\t\tLOG(JPEG, Info) << \"Thumbnail compress returned \"\n> +\t\t\t\t<< jpeg_size << \" bytes\";\n\nAnd I presume we could then do an:\n\t\tthumbnail.resize(jpeg_size);\n\nhere to update the thumbnail with the correct size. I'd be weary of the\nresize operations doing lots of re-allocations though, so perhaps we\nwant to minimize that. But lets get to something that works first before\nworrying about optimising.\n\n\n> +\t}\n> +}\n> +\n> int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n> \t\t\t const libcamera::Span &destination,\n> \t\t\t CameraMetadata *metadata)\n> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n> \t\treturn jpeg_size;\n> \t}\n> \n> +\tstd::vector thumbnail;\n\nYou need to resize this somewhere.\nEdit: Now seen a better place above ;-)\n\n> +\tgenerateThumbnail(destination, thumbnail);\n> +\t/*\n> +\t * \\todo: Write the compressed thumbnail to a file for inspection.\n> +\t * (I) Check if we can still write the thumbnail to EXIF here.\n> +\t * If not, we might need to move the thumbnailer logic to encoder.\n> +\t * And if we do that, first we need to make sure we get can\n> +\t * compressed data written to destination first before calling\n> +\t * jpeg_finish_compress operation somehow. Thumbnailing will\n> +\t * only occur if we have compressed data available first.\n> +\t */\n> +\n> \t/*\n> \t * Fill in the JPEG blob header.\n> \t *\n> diff --git a/src/android/jpeg/post_processor_jpeg.h b/src/android/jpeg/post_processor_jpeg.h\n> index 62c8650..05601ee 100644\n> --- a/src/android/jpeg/post_processor_jpeg.h\n> +++ b/src/android/jpeg/post_processor_jpeg.h\n> @@ -28,6 +28,9 @@ public:\n> \t\t CameraMetadata *metadata) override;\n> \n> private:\n> +\tvoid generateThumbnail(const libcamera::Span &source,\n> +\t\t\t std::vector &thumbnail);\n> +\n> \tCameraDevice *cameraDevice_;\n> \tstd::unique_ptr encoder_;\n> \tlibcamera::Size streamSize_;\n> diff --git a/src/android/jpeg/thumbnailer.cpp b/src/android/jpeg/thumbnailer.cpp\n> new file mode 100644\n> index 0000000..3163576\n> --- /dev/null\n> +++ b/src/android/jpeg/thumbnailer.cpp\n> @@ -0,0 +1,100 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n> + */\n> +\n> +#include \"thumbnailer.h\"\n> +\n> +#include \n> +\n> +#include \"libcamera/internal/file.h\"\n> +#include \"libcamera/internal/log.h\"\n> +\n> +using namespace libcamera;\n> +\n> +LOG_DEFINE_CATEGORY(Thumbnailer)\n> +\n> +Thumbnailer::Thumbnailer()\n> +\t: validConfiguration_(false)\n> +{\n> +}\n> +\n> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat pixelFormat)\n> +{\n> +\tsourceSize_ = sourceSize;\n> +\tpixelFormat_ = pixelFormat;\n> +\n> +\tif (pixelFormat_ != formats::NV12) {\n> +\t\tLOG (Thumbnailer, Error) << \"Failed to configure: Pixel Format \"\n> +\t\t\t\t << pixelFormat_.toString() << \" unsupported.\";\n> +\t\treturn;\n> +\t}\n> +\n> +\tvalidConfiguration_ = true;\n> +}\n> +\n> +/*\n> + * The Exif specification recommends the width of the thumbnail to be a\n> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding height\n> + * keeping the aspect ratio same as of the source.\n> + */\n> +Size Thumbnailer::computeThumbnailSize()\n> +{\n> +\tunsigned int targetHeight;\n> +\tunsigned int targetWidth = 160;\n> +\n> +\ttargetHeight = targetWidth * sourceSize_.height / sourceSize_.width;\n> +\n> +\tif (targetHeight & 1)\n> +\t\ttargetHeight++;\n> +\n> +\treturn Size(targetWidth, targetHeight);\n> +}\n> +\n> +void\n> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n> +\t\t\t std::vector &destination)\n> +{\n> +\tif (!validConfiguration_) {\n> +\t\tLOG(Thumbnailer, Error) << \"config is unconfigured or invalid.\";\n> +\t\treturn;\n> +\t}\n> +\n> +\ttargetSize_ = computeThumbnailSize();\n> +\n> +\tconst unsigned int sw = sourceSize_.width;\n> +\tconst unsigned int sh = sourceSize_.height;\n> +\tconst unsigned int tw = targetSize_.width;\n> +\tconst unsigned int th = targetSize_.height;\n> +\n> +\t/* Image scaling block implementing nearest-neighbour algorithm. */\n> +\tunsigned char *src = static_cast(source.data());\n> +\tunsigned char *src_c = src + sh * sw;\n> +\tunsigned char *src_cb, *src_cr;\n> +\n> +\tsize_t dstSize = (th * tw) + ((th/2) * tw);\n> +\tdestination.reserve(dstSize);\n> +\tunsigned char *dst = destination.data();\n> +\tunsigned char *dst_c = dst + th * tw;\n> +\n> +\tfor (unsigned int y = 0; y < th; y+=2) {\n> +\t\tunsigned int sourceY = (sh*y + th/2) / th;\n> +\n> +\t\tsrc_cb = src_c + (sourceY/2) * sw + 0;\n> +\t\tsrc_cr = src_c + (sourceY/2) * sw + 1;\n> +\n> +\t\tfor (unsigned int x = 0; x < tw; x+=2) {\n> +\t\t\tunsigned int sourceX = (sw*x + tw/2) / tw;\n> +\n> +\t\t\tdst[y * tw + x] = src[sw * sourceY + sourceX];\n> +\t\t\tdst[(y+1) * tw + x] = src[sw * (sourceY+1) + sourceX];\n> +\t\t\tdst[y * tw + (x+1)] = src[sw * sourceY + (sourceX+1)];\n> +\t\t\tdst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) + (sourceX+1)];\n> +\n> +\t\t\tdst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n> +\t\t\tdst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n> +\t\t}\n> +\t}\n> +}\n> diff --git a/src/android/jpeg/thumbnailer.h b/src/android/jpeg/thumbnailer.h\n> new file mode 100644\n> index 0000000..bab9855\n> --- /dev/null\n> +++ b/src/android/jpeg/thumbnailer.h\n> @@ -0,0 +1,40 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2020, Google Inc.\n> + *\n> + * thumbnailer.h - Basic image thumbnailer from NV12\n> + */\n> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n> +#define __ANDROID_JPEG_THUMBNAILER_H__\n> +\n> +#include \n> +\n> +#include \"libcamera/internal/buffer.h\"\n> +#include \"libcamera/internal/formats.h\"\n> +\n> +class Thumbnailer\n> +{\n> +public:\n> +\tThumbnailer();\n> +\n> +\tvoid configure(const libcamera::Size &sourceSize,\n> +\t\t libcamera::PixelFormat pixelFormat);\n> +\n> +\t/*\n> +\t * \\todo: Discuss if we can return targetSize_ via configure() or\n> +\t * scaleBuffer(). We need targetSize_ to re-encode the scaled buffer\n> +\t * via encoder in PostProcssorJpeg::writeThumbnail().\n> +\t */\n> +\tlibcamera::Size computeThumbnailSize();\n> +\tvoid scaleBuffer(const libcamera::Span &source,\n> +\t\t\t std::vector &dest);\n> +\n> +private:\n> +\tlibcamera::PixelFormat pixelFormat_;\n> +\tlibcamera::Size sourceSize_;\n> +\tlibcamera::Size targetSize_;\n> +\n> +\tbool validConfiguration_;\n> +};\n> +\n> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n> diff --git a/src/android/meson.build b/src/android/meson.build\n> index 5a01bea..3905e2f 100644\n> --- a/src/android/meson.build\n> +++ b/src/android/meson.build\n> @@ -25,6 +25,7 @@ android_hal_sources = files([\n> 'jpeg/encoder_libjpeg.cpp',\n> 'jpeg/exif.cpp',\n> 'jpeg/post_processor_jpeg.cpp',\n> + 'jpeg/thumbnailer.cpp',\n> ])\n> \n> android_camera_metadata_sources = files([\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 1564DBDB13\n\tfor ;\n\tWed, 21 Oct 2020 09:19:20 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 7BCE261D98;\n\tWed, 21 Oct 2020 11:19:19 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 72EE560352\n\tfor ;\n\tWed, 21 Oct 2020 11:19:18 +0200 (CEST)","from [192.168.0.20]\n\t(cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id D6D2592;\n\tWed, 21 Oct 2020 11:19:17 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"Xe9EVst5\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1603271958;\n\tbh=D4RoLWxL2D6Epf1Tj7KeY3IMqGb1tEDXzfRQ/xLN3KI=;\n\th=Reply-To:Subject:To:References:From:Date:In-Reply-To:From;\n\tb=Xe9EVst5KiLNgp9cbfFHMnpL3e+TYCQxSvbGKNzz0zMdOCzYSQ4ZJ5qhGLGXJ279M\n\tAaD3dysjnLJQWTBQ72rbug94zzFGXtJOWi4undVrfurISvxzQADjTRWTxnoPKK/C65\n\tkpSBjiYGixthRaKpXc9/+7yPGE2bLcuXuOuLaeXs=","To":"Umang Jain , libcamera-devel@lists.libcamera.org","References":"<20201021080806.46636-1-email@uajain.com>\n\t<20201021080806.46636-3-email@uajain.com>","From":"Kieran Bingham ","Autocrypt":"addr=kieran.bingham@ideasonboard.com; keydata=\n\tmQINBFYE/WYBEACs1PwjMD9rgCu1hlIiUA1AXR4rv2v+BCLUq//vrX5S5bjzxKAryRf0uHat\n\tV/zwz6hiDrZuHUACDB7X8OaQcwhLaVlq6byfoBr25+hbZG7G3+5EUl9cQ7dQEdvNj6V6y/SC\n\trRanWfelwQThCHckbobWiQJfK9n7rYNcPMq9B8e9F020LFH7Kj6YmO95ewJGgLm+idg1Kb3C\n\tpotzWkXc1xmPzcQ1fvQMOfMwdS+4SNw4rY9f07Xb2K99rjMwZVDgESKIzhsDB5GY465sCsiQ\n\tcSAZRxqE49RTBq2+EQsbrQpIc8XiffAB8qexh5/QPzCmR4kJgCGeHIXBtgRj+nIkCJPZvZtf\n\tKr2EAbc6tgg6DkAEHJb+1okosV09+0+TXywYvtEop/WUOWQ+zo+Y/OBd+8Ptgt1pDRyOBzL8\n\tRXa8ZqRf0Mwg75D+dKntZeJHzPRJyrlfQokngAAs4PaFt6UfS+ypMAF37T6CeDArQC41V3ko\n\tlPn1yMsVD0p+6i3DPvA/GPIksDC4owjnzVX9kM8Zc5Cx+XoAN0w5Eqo4t6qEVbuettxx55gq\n\t8K8FieAjgjMSxngo/HST8TpFeqI5nVeq0/lqtBRQKumuIqDg+Bkr4L1V/PSB6XgQcOdhtd36\n\tOe9X9dXB8YSNt7VjOcO7BTmFn/Z8r92mSAfHXpb07YJWJosQOQARAQABtDBLaWVyYW4gQmlu\n\tZ2hhbSA8a2llcmFuLmJpbmdoYW1AaWRlYXNvbmJvYXJkLmNvbT6JAlcEEwEKAEECGwMFCwkI\n\tBwIGFQgJCgsCBBYCAwECHgECF4ACGQEWIQSQLdeYP70o/eNy1HqhHkZyEKRh/QUCXWTtygUJ\n\tCyJXZAAKCRChHkZyEKRh/f8dEACTDsbLN2nioNZMwyLuQRUAFcXNolDX48xcUXsWS2QjxaPm\n\tVsJx8Uy8aYkS85mdPBh0C83OovQR/OVbr8AxhGvYqBs3nQvbWuTl/+4od7DfK2VZOoKBAu5S\n\tQK2FYuUcikDqYcFWJ8DQnubxfE8dvzojHEkXw0sA4igINHDDFX3HJGZtLio+WpEFQtCbfTAG\n\tYZslasz1YZRbwEdSsmO3/kqy5eMnczlm8a21A3fKUo3g8oAZEFM+f4DUNzqIltg31OAB/kZS\n\tenKZQ/SWC8PmLg/ZXBrReYakxXtkP6w3FwMlzOlhGxqhIRNiAJfXJBaRhuUWzPOpEDE9q5YJ\n\tBmqQL2WJm1VSNNVxbXJHpaWMH1sA2R00vmvRrPXGwyIO0IPYeUYQa3gsy6k+En/aMQJd27dp\n\taScf9am9PFICPY5T4ppneeJLif2lyLojo0mcHOV+uyrds9XkLpp14GfTkeKPdPMrLLTsHRfH\n\tfA4I4OBpRrEPiGIZB/0im98MkGY/Mu6qxeZmYLCcgD6qz4idOvfgVOrNh+aA8HzIVR+RMW8H\n\tQGBN9f0E3kfwxuhl3omo6V7lDw8XOdmuWZNC9zPq1UfryVHANYbLGz9KJ4Aw6M+OgBC2JpkD\n\thXMdHUkC+d20dwXrwHTlrJi1YNp6rBc+xald3wsUPOZ5z8moTHUX/uPA/qhGsbkCDQRWBP1m\n\tARAAzijkb+Sau4hAncr1JjOY+KyFEdUNxRy+hqTJdJfaYihxyaj0Ee0P0zEi35CbE6lgU0Uz\n\ttih9fiUbSV3wfsWqg1Ut3/5rTKu7kLFp15kF7eqvV4uezXRD3Qu4yjv/rMmEJbbD4cTvGCYI\n\td6MDC417f7vK3hCbCVIZSp3GXxyC1LU+UQr3fFcOyCwmP9vDUR9JV0BSqHHxRDdpUXE26Dk6\n\tmhf0V1YkspE5St814ETXpEus2urZE5yJIUROlWPIL+hm3NEWfAP06vsQUyLvr/GtbOT79vXl\n\tEn1aulcYyu20dRRxhkQ6iILaURcxIAVJJKPi8dsoMnS8pB0QW12AHWuirPF0g6DiuUfPmrA5\n\tPKe56IGlpkjc8cO51lIxHkWTpCMWigRdPDexKX+Sb+W9QWK/0JjIc4t3KBaiG8O4yRX8ml2R\n\t+rxfAVKM6V769P/hWoRGdgUMgYHFpHGSgEt80OKK5HeUPy2cngDUXzwrqiM5Sz6Od0qw5pCk\n\tNlXqI0W/who0iSVM+8+RmyY0OEkxEcci7rRLsGnM15B5PjLJjh1f2ULYkv8s4SnDwMZ/kE04\n\t/UqCMK/KnX8pwXEMCjz0h6qWNpGwJ0/tYIgQJZh6bqkvBrDogAvuhf60Sogw+mH8b+PBlx1L\n\toeTK396wc+4c3BfiC6pNtUS5GpsPMMjYMk7kVvEAEQEAAYkCPAQYAQoAJgIbDBYhBJAt15g/\n\tvSj943LUeqEeRnIQpGH9BQJdizzIBQkLSKZiAAoJEKEeRnIQpGH9eYgQAJpjaWNgqNOnMTmD\n\tMJggbwjIotypzIXfhHNCeTkG7+qCDlSaBPclcPGYrTwCt0YWPU2TgGgJrVhYT20ierN8LUvj\n\t6qOPTd+Uk7NFzL65qkh80ZKNBFddx1AabQpSVQKbdcLb8OFs85kuSvFdgqZwgxA1vl4TFhNz\n\tPZ79NAmXLackAx3sOVFhk4WQaKRshCB7cSl+RIng5S/ThOBlwNlcKG7j7W2MC06BlTbdEkUp\n\tECzuuRBv8wX4OQl+hbWbB/VKIx5HKlLu1eypen/5lNVzSqMMIYkkZcjV2SWQyUGxSwq0O/sx\n\tS0A8/atCHUXOboUsn54qdxrVDaK+6jIAuo8JiRWctP16KjzUM7MO0/+4zllM8EY57rXrj48j\n\tsbEYX0YQnzaj+jO6kJtoZsIaYR7rMMq9aUAjyiaEZpmP1qF/2sYenDx0Fg2BSlLvLvXM0vU8\n\tpQk3kgDu7kb/7PRYrZvBsr21EIQoIjXbZxDz/o7z95frkP71EaICttZ6k9q5oxxA5WC6sTXc\n\tMW8zs8avFNuA9VpXt0YupJd2ijtZy2mpZNG02fFVXhIn4G807G7+9mhuC4XG5rKlBBUXTvPU\n\tAfYnB4JBDLmLzBFavQfvonSfbitgXwCG3vS+9HEwAjU30Bar1PEOmIbiAoMzuKeRm2LVpmq4\n\tWZw01QYHU/GUV/zHJSFk","Organization":"Ideas on Board","Message-ID":"","Date":"Wed, 21 Oct 2020 10:19:15 +0100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101\n\tThunderbird/68.10.0","MIME-Version":"1.0","In-Reply-To":"<20201021080806.46636-3-email@uajain.com>","Content-Language":"en-GB","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Reply-To":"kieran.bingham@ideasonboard.com","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13325,"web_url":"https://patchwork.libcamera.org/comment/13325/","msgid":"","date":"2020-10-21T10:51:20","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":1,"url":"https://patchwork.libcamera.org/api/people/1/","name":"Umang Jain","email":"email@uajain.com"},"content":"Hi Kieran,\n\nThanks for the comments.\n\nOn 10/21/20 2:49 PM, Kieran Bingham wrote:\n> Hi Umang,\n>\n> On 21/10/2020 09:08, Umang Jain wrote:\n>> Add a basic image thumbnailer for NV12 frames being captured.\n>> It shall generate a thumbnail image to be embedded as a part of\n>> EXIF metadata of the frame. The output of the thumbnail will still\n>> be NV12.\n>>\n>> Signed-off-by: Umang Jain \n>> ---\n>> src/android/jpeg/post_processor_jpeg.cpp | 39 +++++++++\n>> src/android/jpeg/post_processor_jpeg.h | 3 +\n>> src/android/jpeg/thumbnailer.cpp | 100 +++++++++++++++++++++++\n>> src/android/jpeg/thumbnailer.h | 40 +++++++++\n>> src/android/meson.build | 1 +\n>> 5 files changed, 183 insertions(+)\n>> create mode 100644 src/android/jpeg/thumbnailer.cpp\n>> create mode 100644 src/android/jpeg/thumbnailer.h\n>>\n>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp\n>> index 9d452b7..f5f1f78 100644\n>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n>> @@ -11,6 +11,7 @@\n>> #include \"../camera_metadata.h\"\n>> #include \"encoder_libjpeg.h\"\n>> #include \"exif.h\"\n>> +#include \"thumbnailer.h\"\n>> \n>> #include \n>> \n>> @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const StreamConfiguration &inCfg,\n>> \treturn encoder_->configure(inCfg);\n>> }\n>> \n>> +void PostProcessorJpeg::generateThumbnail(const libcamera::Span &source,\n>> +\t\t\t\t\t std::vector &thumbnail)\n>> +{\n>> +\tlibcamera::Span destination;\n>> +\tThumbnailer thumbnailer;\n>> +\n>> +\tthumbnailer.configure(streamSize_, formats::NV12);\n>> +\tlibcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n>> +\tthumbnailer.scaleBuffer(source, thumbnail);\n>> +\n>> +\tif (thumbnail.data()) {\n>> +\t\tStreamConfiguration thumbnailCfg;\n>> +\n>> +\t\tstd::unique_ptr encoder =\n>> +\t\t\t\tstd::make_unique();\n>> +\n>> +\t\tthumbnailCfg.pixelFormat = formats::NV12;\n>> +\t\tthumbnailCfg.size = targetSize;\n>> +\t\tencoder->configure(thumbnailCfg);\n> thumbnail.capacity() might be quite low here.\n> We need to make sure the vector is big enough at this point, you might\n> need to do something like:\n>\n> \tthumbnail.resize(targetSize.width * targetSize.height * 4);\nI am not sure I follow. This is compressing the thumbnail part right? So \nthumbnail is the \"source\" for the encoder here (Please refer to \n->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n>\n>\n> Really we should obtain that size from the encoder. I thought we had a\n> helper to do that already, but it seems we don't.\n> We could/should add Encoder::maxOutput(); which would ask the encoder\n> \"For your given configuration, what is the maximum number of bytes you\n> might output\".\n>\n> Then of course we'd do:\n> \tthumbnail.resize(encoder.maxOutput());\n>\n>\n>> +\t\tint jpeg_size = encoder->encode({ thumbnail.data(), thumbnail.capacity() },\n>> +\t\t\t\tdestination, { });\nAs said above, thumbnail is the source here. The compressed thumbnail \noutput is carried in destination. And, destination is a local span<> \nhere. Does it makes sense?\n>> +\t\tLOG(JPEG, Info) << \"Thumbnail compress returned \"\n>> +\t\t\t\t<< jpeg_size << \" bytes\";\n> And I presume we could then do an:\n> \t\tthumbnail.resize(jpeg_size);\n>\n> here to update the thumbnail with the correct size. I'd be weary of the\n> resize operations doing lots of re-allocations though, so perhaps we\n> want to minimize that. But lets get to something that works first before\n> worrying about optimising.\n>\n>\n>> +\t}\n>> +}\n>> +\n>> int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n>> \t\t\t const libcamera::Span &destination,\n>> \t\t\t CameraMetadata *metadata)\n>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n>> \t\treturn jpeg_size;\n>> \t}\n>> \n>> +\tstd::vector thumbnail;\n> You need to resize this somewhere.\n> Edit: Now seen a better place above ;-)\nThe thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\nWe just pass in to the thumbnailer, while keeping its ownership in \nPostProcessorJpeg.\n>\n>> +\tgenerateThumbnail(destination, thumbnail);\n>> +\t/*\n>> +\t * \\todo: Write the compressed thumbnail to a file for inspection.\n>> +\t * (I) Check if we can still write the thumbnail to EXIF here.\n>> +\t * If not, we might need to move the thumbnailer logic to encoder.\n>> +\t * And if we do that, first we need to make sure we get can\n>> +\t * compressed data written to destination first before calling\n>> +\t * jpeg_finish_compress operation somehow. Thumbnailing will\n>> +\t * only occur if we have compressed data available first.\n>> +\t */\n>> +\n>> \t/*\n>> \t * Fill in the JPEG blob header.\n>> \t *\n>> diff --git a/src/android/jpeg/post_processor_jpeg.h b/src/android/jpeg/post_processor_jpeg.h\n>> index 62c8650..05601ee 100644\n>> --- a/src/android/jpeg/post_processor_jpeg.h\n>> +++ b/src/android/jpeg/post_processor_jpeg.h\n>> @@ -28,6 +28,9 @@ public:\n>> \t\t CameraMetadata *metadata) override;\n>> \n>> private:\n>> +\tvoid generateThumbnail(const libcamera::Span &source,\n>> +\t\t\t std::vector &thumbnail);\n>> +\n>> \tCameraDevice *cameraDevice_;\n>> \tstd::unique_ptr encoder_;\n>> \tlibcamera::Size streamSize_;\n>> diff --git a/src/android/jpeg/thumbnailer.cpp b/src/android/jpeg/thumbnailer.cpp\n>> new file mode 100644\n>> index 0000000..3163576\n>> --- /dev/null\n>> +++ b/src/android/jpeg/thumbnailer.cpp\n>> @@ -0,0 +1,100 @@\n>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n>> +/*\n>> + * Copyright (C) 2020, Google Inc.\n>> + *\n>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n>> + */\n>> +\n>> +#include \"thumbnailer.h\"\n>> +\n>> +#include \n>> +\n>> +#include \"libcamera/internal/file.h\"\n>> +#include \"libcamera/internal/log.h\"\n>> +\n>> +using namespace libcamera;\n>> +\n>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n>> +\n>> +Thumbnailer::Thumbnailer()\n>> +\t: validConfiguration_(false)\n>> +{\n>> +}\n>> +\n>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat pixelFormat)\n>> +{\n>> +\tsourceSize_ = sourceSize;\n>> +\tpixelFormat_ = pixelFormat;\n>> +\n>> +\tif (pixelFormat_ != formats::NV12) {\n>> +\t\tLOG (Thumbnailer, Error) << \"Failed to configure: Pixel Format \"\n>> +\t\t\t\t << pixelFormat_.toString() << \" unsupported.\";\n>> +\t\treturn;\n>> +\t}\n>> +\n>> +\tvalidConfiguration_ = true;\n>> +}\n>> +\n>> +/*\n>> + * The Exif specification recommends the width of the thumbnail to be a\n>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding height\n>> + * keeping the aspect ratio same as of the source.\n>> + */\n>> +Size Thumbnailer::computeThumbnailSize()\n>> +{\n>> +\tunsigned int targetHeight;\n>> +\tunsigned int targetWidth = 160;\n>> +\n>> +\ttargetHeight = targetWidth * sourceSize_.height / sourceSize_.width;\n>> +\n>> +\tif (targetHeight & 1)\n>> +\t\ttargetHeight++;\n>> +\n>> +\treturn Size(targetWidth, targetHeight);\n>> +}\n>> +\n>> +void\n>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n>> +\t\t\t std::vector &destination)\n>> +{\n>> +\tif (!validConfiguration_) {\n>> +\t\tLOG(Thumbnailer, Error) << \"config is unconfigured or invalid.\";\n>> +\t\treturn;\n>> +\t}\n>> +\n>> +\ttargetSize_ = computeThumbnailSize();\n>> +\n>> +\tconst unsigned int sw = sourceSize_.width;\n>> +\tconst unsigned int sh = sourceSize_.height;\n>> +\tconst unsigned int tw = targetSize_.width;\n>> +\tconst unsigned int th = targetSize_.height;\n>> +\n>> +\t/* Image scaling block implementing nearest-neighbour algorithm. */\n>> +\tunsigned char *src = static_cast(source.data());\n>> +\tunsigned char *src_c = src + sh * sw;\n>> +\tunsigned char *src_cb, *src_cr;\n>> +\n>> +\tsize_t dstSize = (th * tw) + ((th/2) * tw);\n>> +\tdestination.reserve(dstSize);\n>> +\tunsigned char *dst = destination.data();\n>> +\tunsigned char *dst_c = dst + th * tw;\n>> +\n>> +\tfor (unsigned int y = 0; y < th; y+=2) {\n>> +\t\tunsigned int sourceY = (sh*y + th/2) / th;\n>> +\n>> +\t\tsrc_cb = src_c + (sourceY/2) * sw + 0;\n>> +\t\tsrc_cr = src_c + (sourceY/2) * sw + 1;\n>> +\n>> +\t\tfor (unsigned int x = 0; x < tw; x+=2) {\n>> +\t\t\tunsigned int sourceX = (sw*x + tw/2) / tw;\n>> +\n>> +\t\t\tdst[y * tw + x] = src[sw * sourceY + sourceX];\n>> +\t\t\tdst[(y+1) * tw + x] = src[sw * (sourceY+1) + sourceX];\n>> +\t\t\tdst[y * tw + (x+1)] = src[sw * sourceY + (sourceX+1)];\n>> +\t\t\tdst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) + (sourceX+1)];\n>> +\n>> +\t\t\tdst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n>> +\t\t\tdst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n>> +\t\t}\n>> +\t}\n>> +}\n>> diff --git a/src/android/jpeg/thumbnailer.h b/src/android/jpeg/thumbnailer.h\n>> new file mode 100644\n>> index 0000000..bab9855\n>> --- /dev/null\n>> +++ b/src/android/jpeg/thumbnailer.h\n>> @@ -0,0 +1,40 @@\n>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n>> +/*\n>> + * Copyright (C) 2020, Google Inc.\n>> + *\n>> + * thumbnailer.h - Basic image thumbnailer from NV12\n>> + */\n>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n>> +\n>> +#include \n>> +\n>> +#include \"libcamera/internal/buffer.h\"\n>> +#include \"libcamera/internal/formats.h\"\n>> +\n>> +class Thumbnailer\n>> +{\n>> +public:\n>> +\tThumbnailer();\n>> +\n>> +\tvoid configure(const libcamera::Size &sourceSize,\n>> +\t\t libcamera::PixelFormat pixelFormat);\n>> +\n>> +\t/*\n>> +\t * \\todo: Discuss if we can return targetSize_ via configure() or\n>> +\t * scaleBuffer(). We need targetSize_ to re-encode the scaled buffer\n>> +\t * via encoder in PostProcssorJpeg::writeThumbnail().\n>> +\t */\n>> +\tlibcamera::Size computeThumbnailSize();\n>> +\tvoid scaleBuffer(const libcamera::Span &source,\n>> +\t\t\t std::vector &dest);\n>> +\n>> +private:\n>> +\tlibcamera::PixelFormat pixelFormat_;\n>> +\tlibcamera::Size sourceSize_;\n>> +\tlibcamera::Size targetSize_;\n>> +\n>> +\tbool validConfiguration_;\n>> +};\n>> +\n>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n>> diff --git a/src/android/meson.build b/src/android/meson.build\n>> index 5a01bea..3905e2f 100644\n>> --- a/src/android/meson.build\n>> +++ b/src/android/meson.build\n>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n>> 'jpeg/encoder_libjpeg.cpp',\n>> 'jpeg/exif.cpp',\n>> 'jpeg/post_processor_jpeg.cpp',\n>> + 'jpeg/thumbnailer.cpp',\n>> ])\n>> \n>> android_camera_metadata_sources = files([\n>>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 64176C3D3C\n\tfor ;\n\tWed, 21 Oct 2020 10:51:27 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id E0E6960361;\n\tWed, 21 Oct 2020 12:51:26 +0200 (CEST)","from mail.uajain.com (static.126.159.217.95.clients.your-server.de\n\t[95.217.159.126])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 128DD60353\n\tfor ;\n\tWed, 21 Oct 2020 12:51:25 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=uajain.com header.i=@uajain.com\n\theader.b=\"dWGPsiDZ\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=uajain.com; s=mail;\n\tt=1603277484; bh=8s7TI8paLoU2f1nN/sQVc0Ti2AwBblcVBia39JNMnKs=;\n\th=Subject:To:References:From:In-Reply-To;\n\tb=dWGPsiDZA6G12UghU8U8Q5+ZlkD8TSCsXlT4LDi+akAYhwrGoOcNxWdel2scHm9kG\n\tYGzlM5R8hzOnWUAY36+vZ9gYjGV1p9yqerfc3+Yc4Owsfn+cEOrL9VeV8oiUktQHK7\n\tbwqCgVZ0yTp+5/8bAF6X+eWUj16bs16GKQPa4ZirkMDHJGb6GKycuJJ6S5UQjTATRr\n\ttH6Jv5wGNhM6I2DVKn4WnkjyUwL7RqG1eBz/Fovv+Iteq4VimYmAvkbPad0CRyQOMz\n\tdn5KsEM7nUOKN/EYOCi37PbIFhboPVU4kBWvwXXZnANZq0kgoqfa7mlLRUhh3bLLBe\n\tfLNUjyXBmBxHg==","To":"kieran.bingham@ideasonboard.com, libcamera-devel@lists.libcamera.org","References":"<20201021080806.46636-1-email@uajain.com>\n\t<20201021080806.46636-3-email@uajain.com>\n\t","From":"Umang Jain ","Message-ID":"","Date":"Wed, 21 Oct 2020 16:21:20 +0530","Mime-Version":"1.0","In-Reply-To":"","Content-Language":"en-US","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Transfer-Encoding":"7bit","Content-Type":"text/plain; charset=\"us-ascii\"; Format=\"flowed\"","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13326,"web_url":"https://patchwork.libcamera.org/comment/13326/","msgid":"<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>","date":"2020-10-21T10:55:22","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi Umang,\n\n\nOn 21/10/2020 11:51, Umang Jain wrote:\n> Hi Kieran,\n> \n> Thanks for the comments.\n> \n> On 10/21/20 2:49 PM, Kieran Bingham wrote:\n>> Hi Umang,\n>>\n>> On 21/10/2020 09:08, Umang Jain wrote:\n>>> Add a basic image thumbnailer for NV12 frames being captured.\n>>> It shall generate a thumbnail image to be embedded as a part of\n>>> EXIF metadata of the frame. The output of the thumbnail will still\n>>> be NV12.\n>>>\n>>> Signed-off-by: Umang Jain \n>>> ---\n>>>   src/android/jpeg/post_processor_jpeg.cpp |  39 +++++++++\n>>>   src/android/jpeg/post_processor_jpeg.h   |   3 +\n>>>   src/android/jpeg/thumbnailer.cpp         | 100 +++++++++++++++++++++++\n>>>   src/android/jpeg/thumbnailer.h           |  40 +++++++++\n>>>   src/android/meson.build                  |   1 +\n>>>   5 files changed, 183 insertions(+)\n>>>   create mode 100644 src/android/jpeg/thumbnailer.cpp\n>>>   create mode 100644 src/android/jpeg/thumbnailer.h\n>>>\n>>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp\n>>> b/src/android/jpeg/post_processor_jpeg.cpp\n>>> index 9d452b7..f5f1f78 100644\n>>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n>>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n>>> @@ -11,6 +11,7 @@\n>>>   #include \"../camera_metadata.h\"\n>>>   #include \"encoder_libjpeg.h\"\n>>>   #include \"exif.h\"\n>>> +#include \"thumbnailer.h\"\n>>>     #include \n>>>   @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const\n>>> StreamConfiguration &inCfg,\n>>>       return encoder_->configure(inCfg);\n>>>   }\n>>>   +void PostProcessorJpeg::generateThumbnail(const\n>>> libcamera::Span &source,\n>>> +                      std::vector &thumbnail)\n>>> +{\n>>> +    libcamera::Span destination;\n>>> +    Thumbnailer thumbnailer;\n>>> +\n>>> +    thumbnailer.configure(streamSize_, formats::NV12);\n>>> +    libcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n>>> +    thumbnailer.scaleBuffer(source, thumbnail);\n>>> +\n>>> +    if (thumbnail.data()) {\n>>> +        StreamConfiguration thumbnailCfg;\n>>> +\n>>> +        std::unique_ptr encoder =\n>>> +                std::make_unique();\n>>> +\n>>> +        thumbnailCfg.pixelFormat = formats::NV12;\n>>> +        thumbnailCfg.size = targetSize;\n>>> +        encoder->configure(thumbnailCfg);\n>> thumbnail.capacity() might be quite low here.\n>> We need to make sure the vector is big enough at this point, you might\n>> need to do something like:\n>>\n>>     thumbnail.resize(targetSize.width * targetSize.height * 4);\n> I am not sure I follow. This is compressing the thumbnail part right? So\n> thumbnail is the \"source\" for the encoder here (Please refer to\n> ->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n>>\n\nUgh, sorry - tired eyes indeed. Yes, please ignore those comments.\n\n\n\n>>\n>> Really we should obtain that size from the encoder. I thought we had a\n>> helper to do that already, but it seems we don't.\n>> We could/should add Encoder::maxOutput(); which would ask the encoder\n>> \"For your given configuration, what is the maximum number of bytes you\n>> might output\".\n>>\n>> Then of course we'd do:\n>>     thumbnail.resize(encoder.maxOutput());\n>>\n>>\n>>> +        int jpeg_size = encoder->encode({ thumbnail.data(),\n>>> thumbnail.capacity() },\n>>> +                destination, { });\n> As said above, thumbnail is the source here. The compressed thumbnail\n> output is carried in destination. And, destination is a local span<>\n> here. Does it makes sense?\n\nYes, I had mixed up source and destination I'm sorry.\n\nSo who/where is allocating the destination?\n\n\n\n>>> +        LOG(JPEG, Info) << \"Thumbnail compress returned \"\n>>> +                << jpeg_size << \" bytes\";\n>> And I presume we could then do an:\n>>         thumbnail.resize(jpeg_size);\n>>\n>> here to update the thumbnail with the correct size. I'd be weary of the\n>> resize operations doing lots of re-allocations though, so perhaps we\n>> want to minimize that. But lets get to something that works first before\n>> worrying about optimising.\n>>\n>>\n>>> +    }\n>>> +}\n>>> +\n>>>   int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n>>>                      const libcamera::Span &destination,\n>>>                      CameraMetadata *metadata)\n>>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const\n>>> libcamera::FrameBuffer *source,\n>>>           return jpeg_size;\n>>>       }\n>>>   +    std::vector thumbnail;\n>> You need to resize this somewhere.\n>> Edit: Now seen a better place above ;-)\n> The thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\n> We just pass in to the thumbnailer, while keeping its ownership in\n> PostProcessorJpeg.\n>>\n>>> +    generateThumbnail(destination, thumbnail);\n>>> +    /*\n>>> +     * \\todo: Write the compressed thumbnail to a file for inspection.\n>>> +     * (I) Check if we can still write the thumbnail to EXIF here.\n>>> +     *     If not, we might need to move the thumbnailer logic to\n>>> encoder.\n>>> +     *     And if we do that, first we need to make sure we get can\n>>> +     *     compressed data written to destination first before calling\n>>> +     *     jpeg_finish_compress operation somehow. Thumbnailing will\n>>> +     *     only occur if we have compressed data available first.\n>>> +     */\n>>> +\n>>>       /*\n>>>        * Fill in the JPEG blob header.\n>>>        *\n>>> diff --git a/src/android/jpeg/post_processor_jpeg.h\n>>> b/src/android/jpeg/post_processor_jpeg.h\n>>> index 62c8650..05601ee 100644\n>>> --- a/src/android/jpeg/post_processor_jpeg.h\n>>> +++ b/src/android/jpeg/post_processor_jpeg.h\n>>> @@ -28,6 +28,9 @@ public:\n>>>               CameraMetadata *metadata) override;\n>>>     private:\n>>> +    void generateThumbnail(const libcamera::Span &source,\n>>> +                   std::vector &thumbnail);\n>>> +\n>>>       CameraDevice *cameraDevice_;\n>>>       std::unique_ptr encoder_;\n>>>       libcamera::Size streamSize_;\n>>> diff --git a/src/android/jpeg/thumbnailer.cpp\n>>> b/src/android/jpeg/thumbnailer.cpp\n>>> new file mode 100644\n>>> index 0000000..3163576\n>>> --- /dev/null\n>>> +++ b/src/android/jpeg/thumbnailer.cpp\n>>> @@ -0,0 +1,100 @@\n>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n>>> +/*\n>>> + * Copyright (C) 2020, Google Inc.\n>>> + *\n>>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n>>> + */\n>>> +\n>>> +#include \"thumbnailer.h\"\n>>> +\n>>> +#include \n>>> +\n>>> +#include \"libcamera/internal/file.h\"\n>>> +#include \"libcamera/internal/log.h\"\n>>> +\n>>> +using namespace libcamera;\n>>> +\n>>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n>>> +\n>>> +Thumbnailer::Thumbnailer()\n>>> +    : validConfiguration_(false)\n>>> +{\n>>> +}\n>>> +\n>>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat\n>>> pixelFormat)\n>>> +{\n>>> +    sourceSize_ = sourceSize;\n>>> +    pixelFormat_ = pixelFormat;\n>>> +\n>>> +    if (pixelFormat_ != formats::NV12) {\n>>> +        LOG (Thumbnailer, Error) << \"Failed to configure: Pixel\n>>> Format \"\n>>> +                    << pixelFormat_.toString() << \" unsupported.\";\n>>> +        return;\n>>> +    }\n>>> +\n>>> +    validConfiguration_ = true;\n>>> +}\n>>> +\n>>> +/*\n>>> + * The Exif specification recommends the width of the thumbnail to be a\n>>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding\n>>> height\n>>> + * keeping the aspect ratio same as of the source.\n>>> + */\n>>> +Size Thumbnailer::computeThumbnailSize()\n>>> +{\n>>> +    unsigned int targetHeight;\n>>> +    unsigned int targetWidth = 160;\n>>> +\n>>> +    targetHeight = targetWidth * sourceSize_.height /\n>>> sourceSize_.width;\n>>> +\n>>> +    if (targetHeight & 1)\n>>> +        targetHeight++;\n>>> +\n>>> +    return Size(targetWidth, targetHeight);\n>>> +}\n>>> +\n>>> +void\n>>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n>>> +             std::vector &destination)\n>>> +{\n>>> +    if (!validConfiguration_) {\n>>> +        LOG(Thumbnailer, Error) << \"config is unconfigured or\n>>> invalid.\";\n>>> +        return;\n>>> +    }\n>>> +\n>>> +    targetSize_ = computeThumbnailSize();\n>>> +\n>>> +    const unsigned int sw = sourceSize_.width;\n>>> +    const unsigned int sh = sourceSize_.height;\n>>> +    const unsigned int tw = targetSize_.width;\n>>> +    const unsigned int th = targetSize_.height;\n>>> +\n>>> +    /* Image scaling block implementing nearest-neighbour algorithm. */\n>>> +    unsigned char *src = static_cast(source.data());\n>>> +    unsigned char *src_c = src + sh * sw;\n>>> +    unsigned char *src_cb, *src_cr;\n>>> +\n>>> +    size_t dstSize = (th * tw) + ((th/2) * tw);\n>>> +    destination.reserve(dstSize);\n>>> +    unsigned char *dst = destination.data();\n>>> +    unsigned char *dst_c = dst + th * tw;\n>>> +\n>>> +    for (unsigned int y = 0; y < th; y+=2) {\n>>> +        unsigned int sourceY = (sh*y + th/2) / th;\n>>> +\n>>> +        src_cb = src_c + (sourceY/2) * sw + 0;\n>>> +        src_cr = src_c + (sourceY/2) * sw + 1;\n>>> +\n>>> +        for (unsigned int x = 0; x < tw; x+=2) {\n>>> +            unsigned int sourceX = (sw*x + tw/2) / tw;\n>>> +\n>>> +            dst[y     * tw + x]     = src[sw * sourceY     + sourceX];\n>>> +            dst[(y+1) * tw + x]     = src[sw * (sourceY+1) + sourceX];\n>>> +            dst[y     * tw + (x+1)] = src[sw * sourceY     +\n>>> (sourceX+1)];\n>>> +            dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) +\n>>> (sourceX+1)];\n>>> +\n>>> +            dst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n>>> +            dst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n>>> +        }\n>>> +    }\n>>> +}\n>>> diff --git a/src/android/jpeg/thumbnailer.h\n>>> b/src/android/jpeg/thumbnailer.h\n>>> new file mode 100644\n>>> index 0000000..bab9855\n>>> --- /dev/null\n>>> +++ b/src/android/jpeg/thumbnailer.h\n>>> @@ -0,0 +1,40 @@\n>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n>>> +/*\n>>> + * Copyright (C) 2020, Google Inc.\n>>> + *\n>>> + * thumbnailer.h - Basic image thumbnailer from NV12\n>>> + */\n>>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n>>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n>>> +\n>>> +#include \n>>> +\n>>> +#include \"libcamera/internal/buffer.h\"\n>>> +#include \"libcamera/internal/formats.h\"\n>>> +\n>>> +class Thumbnailer\n>>> +{\n>>> +public:\n>>> +    Thumbnailer();\n>>> +\n>>> +    void configure(const libcamera::Size &sourceSize,\n>>> +               libcamera::PixelFormat pixelFormat);\n>>> +\n>>> +    /*\n>>> +     * \\todo: Discuss if we can return targetSize_ via configure() or\n>>> +     * scaleBuffer(). We need targetSize_ to re-encode the scaled\n>>> buffer\n>>> +     * via encoder in PostProcssorJpeg::writeThumbnail().\n>>> +     */\n>>> +    libcamera::Size computeThumbnailSize();\n>>> +    void scaleBuffer(const libcamera::Span &source,\n>>> +             std::vector &dest);\n>>> +\n>>> +private:\n>>> +    libcamera::PixelFormat pixelFormat_;\n>>> +    libcamera::Size sourceSize_;\n>>> +    libcamera::Size targetSize_;\n>>> +\n>>> +    bool validConfiguration_;\n>>> +};\n>>> +\n>>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n>>> diff --git a/src/android/meson.build b/src/android/meson.build\n>>> index 5a01bea..3905e2f 100644\n>>> --- a/src/android/meson.build\n>>> +++ b/src/android/meson.build\n>>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n>>>       'jpeg/encoder_libjpeg.cpp',\n>>>       'jpeg/exif.cpp',\n>>>       'jpeg/post_processor_jpeg.cpp',\n>>> +    'jpeg/thumbnailer.cpp',\n>>>   ])\n>>>     android_camera_metadata_sources = files([\n>>>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 30E0EBDB13\n\tfor ;\n\tWed, 21 Oct 2020 10:55:28 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 8EAC761DA6;\n\tWed, 21 Oct 2020 12:55:27 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 79C0960353\n\tfor ;\n\tWed, 21 Oct 2020 12:55:26 +0200 (CEST)","from [192.168.0.20]\n\t(cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id E357092;\n\tWed, 21 Oct 2020 12:55:25 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"bKb7XZSL\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1603277726;\n\tbh=/uZQRJ065uZepbjA8BhhPhboIH9uluByj5wevJiNZdM=;\n\th=Reply-To:Subject:To:References:From:Date:In-Reply-To:From;\n\tb=bKb7XZSLrJnbeSxeMwmIni+ZvXC9kMGxKLDr162p85J0ni4LiWI2EwZKA51sqCsOA\n\tnEQtA05yLpJ3LCr9VRekBDJDuXMdPBfbrX8botPaR8DYG0wo02Wn/GDlFMQlT7ucyX\n\tAYbY9eTGkViPa2yda81hxlnxUaj8+ZaUxjIHYe2c=","To":"Umang Jain , libcamera-devel@lists.libcamera.org","References":"<20201021080806.46636-1-email@uajain.com>\n\t<20201021080806.46636-3-email@uajain.com>\n\t\n\t","From":"Kieran Bingham ","Autocrypt":"addr=kieran.bingham@ideasonboard.com; keydata=\n\tmQINBFYE/WYBEACs1PwjMD9rgCu1hlIiUA1AXR4rv2v+BCLUq//vrX5S5bjzxKAryRf0uHat\n\tV/zwz6hiDrZuHUACDB7X8OaQcwhLaVlq6byfoBr25+hbZG7G3+5EUl9cQ7dQEdvNj6V6y/SC\n\trRanWfelwQThCHckbobWiQJfK9n7rYNcPMq9B8e9F020LFH7Kj6YmO95ewJGgLm+idg1Kb3C\n\tpotzWkXc1xmPzcQ1fvQMOfMwdS+4SNw4rY9f07Xb2K99rjMwZVDgESKIzhsDB5GY465sCsiQ\n\tcSAZRxqE49RTBq2+EQsbrQpIc8XiffAB8qexh5/QPzCmR4kJgCGeHIXBtgRj+nIkCJPZvZtf\n\tKr2EAbc6tgg6DkAEHJb+1okosV09+0+TXywYvtEop/WUOWQ+zo+Y/OBd+8Ptgt1pDRyOBzL8\n\tRXa8ZqRf0Mwg75D+dKntZeJHzPRJyrlfQokngAAs4PaFt6UfS+ypMAF37T6CeDArQC41V3ko\n\tlPn1yMsVD0p+6i3DPvA/GPIksDC4owjnzVX9kM8Zc5Cx+XoAN0w5Eqo4t6qEVbuettxx55gq\n\t8K8FieAjgjMSxngo/HST8TpFeqI5nVeq0/lqtBRQKumuIqDg+Bkr4L1V/PSB6XgQcOdhtd36\n\tOe9X9dXB8YSNt7VjOcO7BTmFn/Z8r92mSAfHXpb07YJWJosQOQARAQABtDBLaWVyYW4gQmlu\n\tZ2hhbSA8a2llcmFuLmJpbmdoYW1AaWRlYXNvbmJvYXJkLmNvbT6JAlcEEwEKAEECGwMFCwkI\n\tBwIGFQgJCgsCBBYCAwECHgECF4ACGQEWIQSQLdeYP70o/eNy1HqhHkZyEKRh/QUCXWTtygUJ\n\tCyJXZAAKCRChHkZyEKRh/f8dEACTDsbLN2nioNZMwyLuQRUAFcXNolDX48xcUXsWS2QjxaPm\n\tVsJx8Uy8aYkS85mdPBh0C83OovQR/OVbr8AxhGvYqBs3nQvbWuTl/+4od7DfK2VZOoKBAu5S\n\tQK2FYuUcikDqYcFWJ8DQnubxfE8dvzojHEkXw0sA4igINHDDFX3HJGZtLio+WpEFQtCbfTAG\n\tYZslasz1YZRbwEdSsmO3/kqy5eMnczlm8a21A3fKUo3g8oAZEFM+f4DUNzqIltg31OAB/kZS\n\tenKZQ/SWC8PmLg/ZXBrReYakxXtkP6w3FwMlzOlhGxqhIRNiAJfXJBaRhuUWzPOpEDE9q5YJ\n\tBmqQL2WJm1VSNNVxbXJHpaWMH1sA2R00vmvRrPXGwyIO0IPYeUYQa3gsy6k+En/aMQJd27dp\n\taScf9am9PFICPY5T4ppneeJLif2lyLojo0mcHOV+uyrds9XkLpp14GfTkeKPdPMrLLTsHRfH\n\tfA4I4OBpRrEPiGIZB/0im98MkGY/Mu6qxeZmYLCcgD6qz4idOvfgVOrNh+aA8HzIVR+RMW8H\n\tQGBN9f0E3kfwxuhl3omo6V7lDw8XOdmuWZNC9zPq1UfryVHANYbLGz9KJ4Aw6M+OgBC2JpkD\n\thXMdHUkC+d20dwXrwHTlrJi1YNp6rBc+xald3wsUPOZ5z8moTHUX/uPA/qhGsbkCDQRWBP1m\n\tARAAzijkb+Sau4hAncr1JjOY+KyFEdUNxRy+hqTJdJfaYihxyaj0Ee0P0zEi35CbE6lgU0Uz\n\ttih9fiUbSV3wfsWqg1Ut3/5rTKu7kLFp15kF7eqvV4uezXRD3Qu4yjv/rMmEJbbD4cTvGCYI\n\td6MDC417f7vK3hCbCVIZSp3GXxyC1LU+UQr3fFcOyCwmP9vDUR9JV0BSqHHxRDdpUXE26Dk6\n\tmhf0V1YkspE5St814ETXpEus2urZE5yJIUROlWPIL+hm3NEWfAP06vsQUyLvr/GtbOT79vXl\n\tEn1aulcYyu20dRRxhkQ6iILaURcxIAVJJKPi8dsoMnS8pB0QW12AHWuirPF0g6DiuUfPmrA5\n\tPKe56IGlpkjc8cO51lIxHkWTpCMWigRdPDexKX+Sb+W9QWK/0JjIc4t3KBaiG8O4yRX8ml2R\n\t+rxfAVKM6V769P/hWoRGdgUMgYHFpHGSgEt80OKK5HeUPy2cngDUXzwrqiM5Sz6Od0qw5pCk\n\tNlXqI0W/who0iSVM+8+RmyY0OEkxEcci7rRLsGnM15B5PjLJjh1f2ULYkv8s4SnDwMZ/kE04\n\t/UqCMK/KnX8pwXEMCjz0h6qWNpGwJ0/tYIgQJZh6bqkvBrDogAvuhf60Sogw+mH8b+PBlx1L\n\toeTK396wc+4c3BfiC6pNtUS5GpsPMMjYMk7kVvEAEQEAAYkCPAQYAQoAJgIbDBYhBJAt15g/\n\tvSj943LUeqEeRnIQpGH9BQJdizzIBQkLSKZiAAoJEKEeRnIQpGH9eYgQAJpjaWNgqNOnMTmD\n\tMJggbwjIotypzIXfhHNCeTkG7+qCDlSaBPclcPGYrTwCt0YWPU2TgGgJrVhYT20ierN8LUvj\n\t6qOPTd+Uk7NFzL65qkh80ZKNBFddx1AabQpSVQKbdcLb8OFs85kuSvFdgqZwgxA1vl4TFhNz\n\tPZ79NAmXLackAx3sOVFhk4WQaKRshCB7cSl+RIng5S/ThOBlwNlcKG7j7W2MC06BlTbdEkUp\n\tECzuuRBv8wX4OQl+hbWbB/VKIx5HKlLu1eypen/5lNVzSqMMIYkkZcjV2SWQyUGxSwq0O/sx\n\tS0A8/atCHUXOboUsn54qdxrVDaK+6jIAuo8JiRWctP16KjzUM7MO0/+4zllM8EY57rXrj48j\n\tsbEYX0YQnzaj+jO6kJtoZsIaYR7rMMq9aUAjyiaEZpmP1qF/2sYenDx0Fg2BSlLvLvXM0vU8\n\tpQk3kgDu7kb/7PRYrZvBsr21EIQoIjXbZxDz/o7z95frkP71EaICttZ6k9q5oxxA5WC6sTXc\n\tMW8zs8avFNuA9VpXt0YupJd2ijtZy2mpZNG02fFVXhIn4G807G7+9mhuC4XG5rKlBBUXTvPU\n\tAfYnB4JBDLmLzBFavQfvonSfbitgXwCG3vS+9HEwAjU30Bar1PEOmIbiAoMzuKeRm2LVpmq4\n\tWZw01QYHU/GUV/zHJSFk","Organization":"Ideas on Board","Message-ID":"<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>","Date":"Wed, 21 Oct 2020 11:55:22 +0100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101\n\tThunderbird/68.10.0","MIME-Version":"1.0","In-Reply-To":"","Content-Language":"en-GB","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Reply-To":"kieran.bingham@ideasonboard.com","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13327,"web_url":"https://patchwork.libcamera.org/comment/13327/","msgid":"<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>","date":"2020-10-21T11:03:40","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":1,"url":"https://patchwork.libcamera.org/api/people/1/","name":"Umang Jain","email":"email@uajain.com"},"content":"Hi Kieran,\n\nOn 10/21/20 4:25 PM, Kieran Bingham wrote:\n> Hi Umang,\n>\n>\n> On 21/10/2020 11:51, Umang Jain wrote:\n>> Hi Kieran,\n>>\n>> Thanks for the comments.\n>>\n>> On 10/21/20 2:49 PM, Kieran Bingham wrote:\n>>> Hi Umang,\n>>>\n>>> On 21/10/2020 09:08, Umang Jain wrote:\n>>>> Add a basic image thumbnailer for NV12 frames being captured.\n>>>> It shall generate a thumbnail image to be embedded as a part of\n>>>> EXIF metadata of the frame. The output of the thumbnail will still\n>>>> be NV12.\n>>>>\n>>>> Signed-off-by: Umang Jain \n>>>> ---\n>>>>   src/android/jpeg/post_processor_jpeg.cpp |  39 +++++++++\n>>>>   src/android/jpeg/post_processor_jpeg.h   |   3 +\n>>>>   src/android/jpeg/thumbnailer.cpp         | 100 +++++++++++++++++++++++\n>>>>   src/android/jpeg/thumbnailer.h           |  40 +++++++++\n>>>>   src/android/meson.build                  |   1 +\n>>>>   5 files changed, 183 insertions(+)\n>>>>   create mode 100644 src/android/jpeg/thumbnailer.cpp\n>>>>   create mode 100644 src/android/jpeg/thumbnailer.h\n>>>>\n>>>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp\n>>>> b/src/android/jpeg/post_processor_jpeg.cpp\n>>>> index 9d452b7..f5f1f78 100644\n>>>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n>>>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n>>>> @@ -11,6 +11,7 @@\n>>>>   #include \"../camera_metadata.h\"\n>>>>   #include \"encoder_libjpeg.h\"\n>>>>   #include \"exif.h\"\n>>>> +#include \"thumbnailer.h\"\n>>>>     #include \n>>>>   @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const\n>>>> StreamConfiguration &inCfg,\n>>>>       return encoder_->configure(inCfg);\n>>>>   }\n>>>>   +void PostProcessorJpeg::generateThumbnail(const\n>>>> libcamera::Span &source,\n>>>> +                      std::vector &thumbnail)\n>>>> +{\n>>>> +    libcamera::Span destination;\n>>>> +    Thumbnailer thumbnailer;\n>>>> +\n>>>> +    thumbnailer.configure(streamSize_, formats::NV12);\n>>>> +    libcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n>>>> +    thumbnailer.scaleBuffer(source, thumbnail);\n>>>> +\n>>>> +    if (thumbnail.data()) {\n>>>> +        StreamConfiguration thumbnailCfg;\n>>>> +\n>>>> +        std::unique_ptr encoder =\n>>>> +                std::make_unique();\n>>>> +\n>>>> +        thumbnailCfg.pixelFormat = formats::NV12;\n>>>> +        thumbnailCfg.size = targetSize;\n>>>> +        encoder->configure(thumbnailCfg);\n>>> thumbnail.capacity() might be quite low here.\n>>> We need to make sure the vector is big enough at this point, you might\n>>> need to do something like:\n>>>\n>>>     thumbnail.resize(targetSize.width * targetSize.height * 4);\n>> I am not sure I follow. This is compressing the thumbnail part right? So\n>> thumbnail is the \"source\" for the encoder here (Please refer to\n>> ->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n> Ugh, sorry - tired eyes indeed. Yes, please ignore those comments.\n>\n>\n>\n>>> Really we should obtain that size from the encoder. I thought we had a\n>>> helper to do that already, but it seems we don't.\n>>> We could/should add Encoder::maxOutput(); which would ask the encoder\n>>> \"For your given configuration, what is the maximum number of bytes you\n>>> might output\".\n>>>\n>>> Then of course we'd do:\n>>>     thumbnail.resize(encoder.maxOutput());\n>>>\n>>>\n>>>> +        int jpeg_size = encoder->encode({ thumbnail.data(),\n>>>> thumbnail.capacity() },\n>>>> +                destination, { });\n>> As said above, thumbnail is the source here. The compressed thumbnail\n>> output is carried in destination. And, destination is a local span<>\n>> here. Does it makes sense?\n> Yes, I had mixed up source and destination I'm sorry.\n>\n> So who/where is allocating the destination?\nAh, that's a good question and I think I just found a bug. I currently \npass on a locally allocated Span<> 'destination' to the encoder. But \nthere is no explicit allocation here. I think I need to point / create \nthe Span with the output size / bytes and then pass to the encode().\n>\n>\n>>>> +        LOG(JPEG, Info) << \"Thumbnail compress returned \"\n>>>> +                << jpeg_size << \" bytes\";\n>>> And I presume we could then do an:\n>>>         thumbnail.resize(jpeg_size);\n>>>\n>>> here to update the thumbnail with the correct size. I'd be weary of the\n>>> resize operations doing lots of re-allocations though, so perhaps we\n>>> want to minimize that. But lets get to something that works first before\n>>> worrying about optimising.\n>>>\n>>>\n>>>> +    }\n>>>> +}\n>>>> +\n>>>>   int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n>>>>                      const libcamera::Span &destination,\n>>>>                      CameraMetadata *metadata)\n>>>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const\n>>>> libcamera::FrameBuffer *source,\n>>>>           return jpeg_size;\n>>>>       }\n>>>>   +    std::vector thumbnail;\n>>> You need to resize this somewhere.\n>>> Edit: Now seen a better place above ;-)\n>> The thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\n>> We just pass in to the thumbnailer, while keeping its ownership in\n>> PostProcessorJpeg.\n>>>> +    generateThumbnail(destination, thumbnail);\n>>>> +    /*\n>>>> +     * \\todo: Write the compressed thumbnail to a file for inspection.\n>>>> +     * (I) Check if we can still write the thumbnail to EXIF here.\n>>>> +     *     If not, we might need to move the thumbnailer logic to\n>>>> encoder.\n>>>> +     *     And if we do that, first we need to make sure we get can\n>>>> +     *     compressed data written to destination first before calling\n>>>> +     *     jpeg_finish_compress operation somehow. Thumbnailing will\n>>>> +     *     only occur if we have compressed data available first.\n>>>> +     */\n>>>> +\n>>>>       /*\n>>>>        * Fill in the JPEG blob header.\n>>>>        *\n>>>> diff --git a/src/android/jpeg/post_processor_jpeg.h\n>>>> b/src/android/jpeg/post_processor_jpeg.h\n>>>> index 62c8650..05601ee 100644\n>>>> --- a/src/android/jpeg/post_processor_jpeg.h\n>>>> +++ b/src/android/jpeg/post_processor_jpeg.h\n>>>> @@ -28,6 +28,9 @@ public:\n>>>>               CameraMetadata *metadata) override;\n>>>>     private:\n>>>> +    void generateThumbnail(const libcamera::Span &source,\n>>>> +                   std::vector &thumbnail);\n>>>> +\n>>>>       CameraDevice *cameraDevice_;\n>>>>       std::unique_ptr encoder_;\n>>>>       libcamera::Size streamSize_;\n>>>> diff --git a/src/android/jpeg/thumbnailer.cpp\n>>>> b/src/android/jpeg/thumbnailer.cpp\n>>>> new file mode 100644\n>>>> index 0000000..3163576\n>>>> --- /dev/null\n>>>> +++ b/src/android/jpeg/thumbnailer.cpp\n>>>> @@ -0,0 +1,100 @@\n>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n>>>> +/*\n>>>> + * Copyright (C) 2020, Google Inc.\n>>>> + *\n>>>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n>>>> + */\n>>>> +\n>>>> +#include \"thumbnailer.h\"\n>>>> +\n>>>> +#include \n>>>> +\n>>>> +#include \"libcamera/internal/file.h\"\n>>>> +#include \"libcamera/internal/log.h\"\n>>>> +\n>>>> +using namespace libcamera;\n>>>> +\n>>>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n>>>> +\n>>>> +Thumbnailer::Thumbnailer()\n>>>> +    : validConfiguration_(false)\n>>>> +{\n>>>> +}\n>>>> +\n>>>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat\n>>>> pixelFormat)\n>>>> +{\n>>>> +    sourceSize_ = sourceSize;\n>>>> +    pixelFormat_ = pixelFormat;\n>>>> +\n>>>> +    if (pixelFormat_ != formats::NV12) {\n>>>> +        LOG (Thumbnailer, Error) << \"Failed to configure: Pixel\n>>>> Format \"\n>>>> +                    << pixelFormat_.toString() << \" unsupported.\";\n>>>> +        return;\n>>>> +    }\n>>>> +\n>>>> +    validConfiguration_ = true;\n>>>> +}\n>>>> +\n>>>> +/*\n>>>> + * The Exif specification recommends the width of the thumbnail to be a\n>>>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding\n>>>> height\n>>>> + * keeping the aspect ratio same as of the source.\n>>>> + */\n>>>> +Size Thumbnailer::computeThumbnailSize()\n>>>> +{\n>>>> +    unsigned int targetHeight;\n>>>> +    unsigned int targetWidth = 160;\n>>>> +\n>>>> +    targetHeight = targetWidth * sourceSize_.height /\n>>>> sourceSize_.width;\n>>>> +\n>>>> +    if (targetHeight & 1)\n>>>> +        targetHeight++;\n>>>> +\n>>>> +    return Size(targetWidth, targetHeight);\n>>>> +}\n>>>> +\n>>>> +void\n>>>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n>>>> +             std::vector &destination)\n>>>> +{\n>>>> +    if (!validConfiguration_) {\n>>>> +        LOG(Thumbnailer, Error) << \"config is unconfigured or\n>>>> invalid.\";\n>>>> +        return;\n>>>> +    }\n>>>> +\n>>>> +    targetSize_ = computeThumbnailSize();\n>>>> +\n>>>> +    const unsigned int sw = sourceSize_.width;\n>>>> +    const unsigned int sh = sourceSize_.height;\n>>>> +    const unsigned int tw = targetSize_.width;\n>>>> +    const unsigned int th = targetSize_.height;\n>>>> +\n>>>> +    /* Image scaling block implementing nearest-neighbour algorithm. */\n>>>> +    unsigned char *src = static_cast(source.data());\n>>>> +    unsigned char *src_c = src + sh * sw;\n>>>> +    unsigned char *src_cb, *src_cr;\n>>>> +\n>>>> +    size_t dstSize = (th * tw) + ((th/2) * tw);\n>>>> +    destination.reserve(dstSize);\n>>>> +    unsigned char *dst = destination.data();\n>>>> +    unsigned char *dst_c = dst + th * tw;\n>>>> +\n>>>> +    for (unsigned int y = 0; y < th; y+=2) {\n>>>> +        unsigned int sourceY = (sh*y + th/2) / th;\n>>>> +\n>>>> +        src_cb = src_c + (sourceY/2) * sw + 0;\n>>>> +        src_cr = src_c + (sourceY/2) * sw + 1;\n>>>> +\n>>>> +        for (unsigned int x = 0; x < tw; x+=2) {\n>>>> +            unsigned int sourceX = (sw*x + tw/2) / tw;\n>>>> +\n>>>> +            dst[y     * tw + x]     = src[sw * sourceY     + sourceX];\n>>>> +            dst[(y+1) * tw + x]     = src[sw * (sourceY+1) + sourceX];\n>>>> +            dst[y     * tw + (x+1)] = src[sw * sourceY     +\n>>>> (sourceX+1)];\n>>>> +            dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) +\n>>>> (sourceX+1)];\n>>>> +\n>>>> +            dst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n>>>> +            dst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n>>>> +        }\n>>>> +    }\n>>>> +}\n>>>> diff --git a/src/android/jpeg/thumbnailer.h\n>>>> b/src/android/jpeg/thumbnailer.h\n>>>> new file mode 100644\n>>>> index 0000000..bab9855\n>>>> --- /dev/null\n>>>> +++ b/src/android/jpeg/thumbnailer.h\n>>>> @@ -0,0 +1,40 @@\n>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n>>>> +/*\n>>>> + * Copyright (C) 2020, Google Inc.\n>>>> + *\n>>>> + * thumbnailer.h - Basic image thumbnailer from NV12\n>>>> + */\n>>>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n>>>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n>>>> +\n>>>> +#include \n>>>> +\n>>>> +#include \"libcamera/internal/buffer.h\"\n>>>> +#include \"libcamera/internal/formats.h\"\n>>>> +\n>>>> +class Thumbnailer\n>>>> +{\n>>>> +public:\n>>>> +    Thumbnailer();\n>>>> +\n>>>> +    void configure(const libcamera::Size &sourceSize,\n>>>> +               libcamera::PixelFormat pixelFormat);\n>>>> +\n>>>> +    /*\n>>>> +     * \\todo: Discuss if we can return targetSize_ via configure() or\n>>>> +     * scaleBuffer(). We need targetSize_ to re-encode the scaled\n>>>> buffer\n>>>> +     * via encoder in PostProcssorJpeg::writeThumbnail().\n>>>> +     */\n>>>> +    libcamera::Size computeThumbnailSize();\n>>>> +    void scaleBuffer(const libcamera::Span &source,\n>>>> +             std::vector &dest);\n>>>> +\n>>>> +private:\n>>>> +    libcamera::PixelFormat pixelFormat_;\n>>>> +    libcamera::Size sourceSize_;\n>>>> +    libcamera::Size targetSize_;\n>>>> +\n>>>> +    bool validConfiguration_;\n>>>> +};\n>>>> +\n>>>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n>>>> diff --git a/src/android/meson.build b/src/android/meson.build\n>>>> index 5a01bea..3905e2f 100644\n>>>> --- a/src/android/meson.build\n>>>> +++ b/src/android/meson.build\n>>>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n>>>>       'jpeg/encoder_libjpeg.cpp',\n>>>>       'jpeg/exif.cpp',\n>>>>       'jpeg/post_processor_jpeg.cpp',\n>>>> +    'jpeg/thumbnailer.cpp',\n>>>>   ])\n>>>>     android_camera_metadata_sources = files([\n>>>>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 5F0F8C3D3C\n\tfor ;\n\tWed, 21 Oct 2020 11:03:46 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id EB4FC60361;\n\tWed, 21 Oct 2020 13:03:45 +0200 (CEST)","from mail.uajain.com (static.126.159.217.95.clients.your-server.de\n\t[95.217.159.126])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C4F6360353\n\tfor ;\n\tWed, 21 Oct 2020 13:03:43 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=uajain.com header.i=@uajain.com\n\theader.b=\"XWXDYuKv\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=uajain.com; s=mail;\n\tt=1603278223; bh=jLDaraVK6hOtnCuTIQIqU4Co//Zu3c8GM5GoaKt8ug4=;\n\th=Subject:To:References:From:In-Reply-To;\n\tb=XWXDYuKvp7QJ1cCXMyAjSzdpm3QC6QN/240ZY7lXkYfBtib8VUYfS9ruMI/ra15YI\n\tlvHto/l4/5o1ZWYXBp8HFBdtWDz6jnD+LuoeuWbYA+wrcueJzX5olbcWPNaarTNipZ\n\t/QSW5h1a/vB9Guau7NtSaQWGIaK60e+9aRwS03rmRBfuk7QrUsocPUEu0q9tsTwOuU\n\t/eOzzN5+Xac9ZE2dKVUnaeFU0ahmGlS9hTCB3x7omhkhCOX0MUnL+BDX7hlgS5TtUf\n\tGHVJCP5dwz5tF8sEozpqlnYsUEMWwgpB9Dctor8OaG6qjPvbABpFHRDTTaz7H71dN8\n\t9MCYZeGEunVug==","To":"kieran.bingham@ideasonboard.com, libcamera-devel@lists.libcamera.org","References":"<20201021080806.46636-1-email@uajain.com>\n\t<20201021080806.46636-3-email@uajain.com>\n\t\n\t\n\t<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>","From":"Umang Jain ","Message-ID":"<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>","Date":"Wed, 21 Oct 2020 16:33:40 +0530","Mime-Version":"1.0","In-Reply-To":"<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>","Content-Language":"en-US","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Transfer-Encoding":"base64","Content-Type":"text/plain; charset=\"utf-8\"; Format=\"flowed\"","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13328,"web_url":"https://patchwork.libcamera.org/comment/13328/","msgid":"<9ca3b492-8468-e9cf-b103-b4350be472b9@ideasonboard.com>","date":"2020-10-21T11:09:23","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi Umang,\n\nOn 21/10/2020 12:03, Umang Jain wrote:\n> Hi Kieran,\n> \n> On 10/21/20 4:25 PM, Kieran Bingham wrote:\n>> Hi Umang,\n>>\n>>\n>> On 21/10/2020 11:51, Umang Jain wrote:\n>>> Hi Kieran,\n>>>\n>>> Thanks for the comments.\n>>>\n>>> On 10/21/20 2:49 PM, Kieran Bingham wrote:\n>>>> Hi Umang,\n>>>>\n>>>> On 21/10/2020 09:08, Umang Jain wrote:\n>>>>> Add a basic image thumbnailer for NV12 frames being captured.\n>>>>> It shall generate a thumbnail image to be embedded as a part of\n>>>>> EXIF metadata of the frame. The output of the thumbnail will still\n>>>>> be NV12.\n>>>>>\n>>>>> Signed-off-by: Umang Jain \n>>>>> ---\n>>>>>    src/android/jpeg/post_processor_jpeg.cpp |  39 +++++++++\n>>>>>    src/android/jpeg/post_processor_jpeg.h   |   3 +\n>>>>>    src/android/jpeg/thumbnailer.cpp         | 100\n>>>>> +++++++++++++++++++++++\n>>>>>    src/android/jpeg/thumbnailer.h           |  40 +++++++++\n>>>>>    src/android/meson.build                  |   1 +\n>>>>>    5 files changed, 183 insertions(+)\n>>>>>    create mode 100644 src/android/jpeg/thumbnailer.cpp\n>>>>>    create mode 100644 src/android/jpeg/thumbnailer.h\n>>>>>\n>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp\n>>>>> b/src/android/jpeg/post_processor_jpeg.cpp\n>>>>> index 9d452b7..f5f1f78 100644\n>>>>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n>>>>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n>>>>> @@ -11,6 +11,7 @@\n>>>>>    #include \"../camera_metadata.h\"\n>>>>>    #include \"encoder_libjpeg.h\"\n>>>>>    #include \"exif.h\"\n>>>>> +#include \"thumbnailer.h\"\n>>>>>      #include \n>>>>>    @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const\n>>>>> StreamConfiguration &inCfg,\n>>>>>        return encoder_->configure(inCfg);\n>>>>>    }\n>>>>>    +void PostProcessorJpeg::generateThumbnail(const\n>>>>> libcamera::Span &source,\n>>>>> +                      std::vector &thumbnail)\n>>>>> +{\n>>>>> +    libcamera::Span destination;\n>>>>> +    Thumbnailer thumbnailer;\n>>>>> +\n>>>>> +    thumbnailer.configure(streamSize_, formats::NV12);\n>>>>> +    libcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n>>>>> +    thumbnailer.scaleBuffer(source, thumbnail);\n>>>>> +\n>>>>> +    if (thumbnail.data()) {\n>>>>> +        StreamConfiguration thumbnailCfg;\n>>>>> +\n>>>>> +        std::unique_ptr encoder =\n>>>>> +                std::make_unique();\n>>>>> +\n>>>>> +        thumbnailCfg.pixelFormat = formats::NV12;\n>>>>> +        thumbnailCfg.size = targetSize;\n>>>>> +        encoder->configure(thumbnailCfg);\n>>>> thumbnail.capacity() might be quite low here.\n>>>> We need to make sure the vector is big enough at this point, you might\n>>>> need to do something like:\n>>>>\n>>>>      thumbnail.resize(targetSize.width * targetSize.height * 4);\n>>> I am not sure I follow. This is compressing the thumbnail part right? So\n>>> thumbnail is the \"source\" for the encoder here (Please refer to\n>>> ->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n>> Ugh, sorry - tired eyes indeed. Yes, please ignore those comments.\n>>\n>>\n>>\n>>>> Really we should obtain that size from the encoder. I thought we had a\n>>>> helper to do that already, but it seems we don't.\n>>>> We could/should add Encoder::maxOutput(); which would ask the encoder\n>>>> \"For your given configuration, what is the maximum number of bytes you\n>>>> might output\".\n>>>>\n>>>> Then of course we'd do:\n>>>>      thumbnail.resize(encoder.maxOutput());\n>>>>\n>>>>\n>>>>> +        int jpeg_size = encoder->encode({ thumbnail.data(),\n>>>>> thumbnail.capacity() },\n>>>>> +                destination, { });\n>>> As said above, thumbnail is the source here. The compressed thumbnail\n>>> output is carried in destination. And, destination is a local span<>\n>>> here. Does it makes sense?\n>> Yes, I had mixed up source and destination I'm sorry.\n>>\n>> So who/where is allocating the destination?\n> Ah, that's a good question and I think I just found a bug. I currently\n> pass on a locally allocated Span<> 'destination' to the encoder. But\n> there is no explicit allocation here. I think I need to point / create\n> the Span with the output size / bytes and then pass to the encode().\n\n:-) Ok - then please apply all of my earlier comments to that part\ninstead. ;-)\n\nWe certainly need somewhere to store the compressed image, to be able to\npass it into the exif :-D\n\n--\nKieran\n\n\n>>>>> +        LOG(JPEG, Info) << \"Thumbnail compress returned \"\n>>>>> +                << jpeg_size << \" bytes\";\n>>>> And I presume we could then do an:\n>>>>          thumbnail.resize(jpeg_size);\n>>>>\n>>>> here to update the thumbnail with the correct size. I'd be weary of the\n>>>> resize operations doing lots of re-allocations though, so perhaps we\n>>>> want to minimize that. But lets get to something that works first\n>>>> before\n>>>> worrying about optimising.\n>>>>\n>>>>\n>>>>> +    }\n>>>>> +}\n>>>>> +\n>>>>>    int PostProcessorJpeg::process(const libcamera::FrameBuffer\n>>>>> *source,\n>>>>>                       const libcamera::Span &destination,\n>>>>>                       CameraMetadata *metadata)\n>>>>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const\n>>>>> libcamera::FrameBuffer *source,\n>>>>>            return jpeg_size;\n>>>>>        }\n>>>>>    +    std::vector thumbnail;\n>>>> You need to resize this somewhere.\n>>>> Edit: Now seen a better place above ;-)\n>>> The thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\n>>> We just pass in to the thumbnailer, while keeping its ownership in\n>>> PostProcessorJpeg.\n>>>>> +    generateThumbnail(destination, thumbnail);\n>>>>> +    /*\n>>>>> +     * \\todo: Write the compressed thumbnail to a file for\n>>>>> inspection.\n>>>>> +     * (I) Check if we can still write the thumbnail to EXIF here.\n>>>>> +     *     If not, we might need to move the thumbnailer logic to\n>>>>> encoder.\n>>>>> +     *     And if we do that, first we need to make sure we get can\n>>>>> +     *     compressed data written to destination first before\n>>>>> calling\n>>>>> +     *     jpeg_finish_compress operation somehow. Thumbnailing will\n>>>>> +     *     only occur if we have compressed data available first.\n>>>>> +     */\n>>>>> +\n>>>>>        /*\n>>>>>         * Fill in the JPEG blob header.\n>>>>>         *\n>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.h\n>>>>> b/src/android/jpeg/post_processor_jpeg.h\n>>>>> index 62c8650..05601ee 100644\n>>>>> --- a/src/android/jpeg/post_processor_jpeg.h\n>>>>> +++ b/src/android/jpeg/post_processor_jpeg.h\n>>>>> @@ -28,6 +28,9 @@ public:\n>>>>>                CameraMetadata *metadata) override;\n>>>>>      private:\n>>>>> +    void generateThumbnail(const libcamera::Span &source,\n>>>>> +                   std::vector &thumbnail);\n>>>>> +\n>>>>>        CameraDevice *cameraDevice_;\n>>>>>        std::unique_ptr encoder_;\n>>>>>        libcamera::Size streamSize_;\n>>>>> diff --git a/src/android/jpeg/thumbnailer.cpp\n>>>>> b/src/android/jpeg/thumbnailer.cpp\n>>>>> new file mode 100644\n>>>>> index 0000000..3163576\n>>>>> --- /dev/null\n>>>>> +++ b/src/android/jpeg/thumbnailer.cpp\n>>>>> @@ -0,0 +1,100 @@\n>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n>>>>> +/*\n>>>>> + * Copyright (C) 2020, Google Inc.\n>>>>> + *\n>>>>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n>>>>> + */\n>>>>> +\n>>>>> +#include \"thumbnailer.h\"\n>>>>> +\n>>>>> +#include \n>>>>> +\n>>>>> +#include \"libcamera/internal/file.h\"\n>>>>> +#include \"libcamera/internal/log.h\"\n>>>>> +\n>>>>> +using namespace libcamera;\n>>>>> +\n>>>>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n>>>>> +\n>>>>> +Thumbnailer::Thumbnailer()\n>>>>> +    : validConfiguration_(false)\n>>>>> +{\n>>>>> +}\n>>>>> +\n>>>>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat\n>>>>> pixelFormat)\n>>>>> +{\n>>>>> +    sourceSize_ = sourceSize;\n>>>>> +    pixelFormat_ = pixelFormat;\n>>>>> +\n>>>>> +    if (pixelFormat_ != formats::NV12) {\n>>>>> +        LOG (Thumbnailer, Error) << \"Failed to configure: Pixel\n>>>>> Format \"\n>>>>> +                    << pixelFormat_.toString() << \" unsupported.\";\n>>>>> +        return;\n>>>>> +    }\n>>>>> +\n>>>>> +    validConfiguration_ = true;\n>>>>> +}\n>>>>> +\n>>>>> +/*\n>>>>> + * The Exif specification recommends the width of the thumbnail to\n>>>>> be a\n>>>>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding\n>>>>> height\n>>>>> + * keeping the aspect ratio same as of the source.\n>>>>> + */\n>>>>> +Size Thumbnailer::computeThumbnailSize()\n>>>>> +{\n>>>>> +    unsigned int targetHeight;\n>>>>> +    unsigned int targetWidth = 160;\n>>>>> +\n>>>>> +    targetHeight = targetWidth * sourceSize_.height /\n>>>>> sourceSize_.width;\n>>>>> +\n>>>>> +    if (targetHeight & 1)\n>>>>> +        targetHeight++;\n>>>>> +\n>>>>> +    return Size(targetWidth, targetHeight);\n>>>>> +}\n>>>>> +\n>>>>> +void\n>>>>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n>>>>> +             std::vector &destination)\n>>>>> +{\n>>>>> +    if (!validConfiguration_) {\n>>>>> +        LOG(Thumbnailer, Error) << \"config is unconfigured or\n>>>>> invalid.\";\n>>>>> +        return;\n>>>>> +    }\n>>>>> +\n>>>>> +    targetSize_ = computeThumbnailSize();\n>>>>> +\n>>>>> +    const unsigned int sw = sourceSize_.width;\n>>>>> +    const unsigned int sh = sourceSize_.height;\n>>>>> +    const unsigned int tw = targetSize_.width;\n>>>>> +    const unsigned int th = targetSize_.height;\n>>>>> +\n>>>>> +    /* Image scaling block implementing nearest-neighbour\n>>>>> algorithm. */\n>>>>> +    unsigned char *src = static_cast(source.data());\n>>>>> +    unsigned char *src_c = src + sh * sw;\n>>>>> +    unsigned char *src_cb, *src_cr;\n>>>>> +\n>>>>> +    size_t dstSize = (th * tw) + ((th/2) * tw);\n>>>>> +    destination.reserve(dstSize);\n>>>>> +    unsigned char *dst = destination.data();\n>>>>> +    unsigned char *dst_c = dst + th * tw;\n>>>>> +\n>>>>> +    for (unsigned int y = 0; y < th; y+=2) {\n>>>>> +        unsigned int sourceY = (sh*y + th/2) / th;\n>>>>> +\n>>>>> +        src_cb = src_c + (sourceY/2) * sw + 0;\n>>>>> +        src_cr = src_c + (sourceY/2) * sw + 1;\n>>>>> +\n>>>>> +        for (unsigned int x = 0; x < tw; x+=2) {\n>>>>> +            unsigned int sourceX = (sw*x + tw/2) / tw;\n>>>>> +\n>>>>> +            dst[y     * tw + x]     = src[sw * sourceY     +\n>>>>> sourceX];\n>>>>> +            dst[(y+1) * tw + x]     = src[sw * (sourceY+1) +\n>>>>> sourceX];\n>>>>> +            dst[y     * tw + (x+1)] = src[sw * sourceY     +\n>>>>> (sourceX+1)];\n>>>>> +            dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) +\n>>>>> (sourceX+1)];\n>>>>> +\n>>>>> +            dst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n>>>>> +            dst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n>>>>> +        }\n>>>>> +    }\n>>>>> +}\n>>>>> diff --git a/src/android/jpeg/thumbnailer.h\n>>>>> b/src/android/jpeg/thumbnailer.h\n>>>>> new file mode 100644\n>>>>> index 0000000..bab9855\n>>>>> --- /dev/null\n>>>>> +++ b/src/android/jpeg/thumbnailer.h\n>>>>> @@ -0,0 +1,40 @@\n>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n>>>>> +/*\n>>>>> + * Copyright (C) 2020, Google Inc.\n>>>>> + *\n>>>>> + * thumbnailer.h - Basic image thumbnailer from NV12\n>>>>> + */\n>>>>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n>>>>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n>>>>> +\n>>>>> +#include \n>>>>> +\n>>>>> +#include \"libcamera/internal/buffer.h\"\n>>>>> +#include \"libcamera/internal/formats.h\"\n>>>>> +\n>>>>> +class Thumbnailer\n>>>>> +{\n>>>>> +public:\n>>>>> +    Thumbnailer();\n>>>>> +\n>>>>> +    void configure(const libcamera::Size &sourceSize,\n>>>>> +               libcamera::PixelFormat pixelFormat);\n>>>>> +\n>>>>> +    /*\n>>>>> +     * \\todo: Discuss if we can return targetSize_ via configure() or\n>>>>> +     * scaleBuffer(). We need targetSize_ to re-encode the scaled\n>>>>> buffer\n>>>>> +     * via encoder in PostProcssorJpeg::writeThumbnail().\n>>>>> +     */\n>>>>> +    libcamera::Size computeThumbnailSize();\n>>>>> +    void scaleBuffer(const libcamera::Span &source,\n>>>>> +             std::vector &dest);\n>>>>> +\n>>>>> +private:\n>>>>> +    libcamera::PixelFormat pixelFormat_;\n>>>>> +    libcamera::Size sourceSize_;\n>>>>> +    libcamera::Size targetSize_;\n>>>>> +\n>>>>> +    bool validConfiguration_;\n>>>>> +};\n>>>>> +\n>>>>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n>>>>> diff --git a/src/android/meson.build b/src/android/meson.build\n>>>>> index 5a01bea..3905e2f 100644\n>>>>> --- a/src/android/meson.build\n>>>>> +++ b/src/android/meson.build\n>>>>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n>>>>>        'jpeg/encoder_libjpeg.cpp',\n>>>>>        'jpeg/exif.cpp',\n>>>>>        'jpeg/post_processor_jpeg.cpp',\n>>>>> +    'jpeg/thumbnailer.cpp',\n>>>>>    ])\n>>>>>      android_camera_metadata_sources = files([\n>>>>>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 2CF6BBDB13\n\tfor ;\n\tWed, 21 Oct 2020 11:09:28 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id A8BA161DA6;\n\tWed, 21 Oct 2020 13:09:27 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id DE53F60353\n\tfor ;\n\tWed, 21 Oct 2020 13:09:26 +0200 (CEST)","from [192.168.0.20]\n\t(cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 3C80992;\n\tWed, 21 Oct 2020 13:09:26 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"DLmwFLKL\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1603278566;\n\tbh=5m1KTKvMDDG7XOixoWAbK098LWHIs3Leu+OWYZzBsHs=;\n\th=Reply-To:Subject:To:References:From:Date:In-Reply-To:From;\n\tb=DLmwFLKLJvUJXubB8siAZ4fV5DGUh6xh141qgX/6k8nHPWphq84XDo9prLF20/XT1\n\te0krpgnty1MIkLfTz5b+meW9QdVf72YlMV1tpj1x0b8Zr5f5eiQfkqRO/aa0fewbdX\n\tl7P54Cx+7+8kxj+/CP8k3XTlqzV+58D3g9NhxRKo=","To":"Umang Jain , libcamera-devel@lists.libcamera.org","References":"<20201021080806.46636-1-email@uajain.com>\n\t<20201021080806.46636-3-email@uajain.com>\n\t\n\t\n\t<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>\n\t<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>","From":"Kieran Bingham ","Autocrypt":"addr=kieran.bingham@ideasonboard.com; keydata=\n\tmQINBFYE/WYBEACs1PwjMD9rgCu1hlIiUA1AXR4rv2v+BCLUq//vrX5S5bjzxKAryRf0uHat\n\tV/zwz6hiDrZuHUACDB7X8OaQcwhLaVlq6byfoBr25+hbZG7G3+5EUl9cQ7dQEdvNj6V6y/SC\n\trRanWfelwQThCHckbobWiQJfK9n7rYNcPMq9B8e9F020LFH7Kj6YmO95ewJGgLm+idg1Kb3C\n\tpotzWkXc1xmPzcQ1fvQMOfMwdS+4SNw4rY9f07Xb2K99rjMwZVDgESKIzhsDB5GY465sCsiQ\n\tcSAZRxqE49RTBq2+EQsbrQpIc8XiffAB8qexh5/QPzCmR4kJgCGeHIXBtgRj+nIkCJPZvZtf\n\tKr2EAbc6tgg6DkAEHJb+1okosV09+0+TXywYvtEop/WUOWQ+zo+Y/OBd+8Ptgt1pDRyOBzL8\n\tRXa8ZqRf0Mwg75D+dKntZeJHzPRJyrlfQokngAAs4PaFt6UfS+ypMAF37T6CeDArQC41V3ko\n\tlPn1yMsVD0p+6i3DPvA/GPIksDC4owjnzVX9kM8Zc5Cx+XoAN0w5Eqo4t6qEVbuettxx55gq\n\t8K8FieAjgjMSxngo/HST8TpFeqI5nVeq0/lqtBRQKumuIqDg+Bkr4L1V/PSB6XgQcOdhtd36\n\tOe9X9dXB8YSNt7VjOcO7BTmFn/Z8r92mSAfHXpb07YJWJosQOQARAQABtDBLaWVyYW4gQmlu\n\tZ2hhbSA8a2llcmFuLmJpbmdoYW1AaWRlYXNvbmJvYXJkLmNvbT6JAlcEEwEKAEECGwMFCwkI\n\tBwIGFQgJCgsCBBYCAwECHgECF4ACGQEWIQSQLdeYP70o/eNy1HqhHkZyEKRh/QUCXWTtygUJ\n\tCyJXZAAKCRChHkZyEKRh/f8dEACTDsbLN2nioNZMwyLuQRUAFcXNolDX48xcUXsWS2QjxaPm\n\tVsJx8Uy8aYkS85mdPBh0C83OovQR/OVbr8AxhGvYqBs3nQvbWuTl/+4od7DfK2VZOoKBAu5S\n\tQK2FYuUcikDqYcFWJ8DQnubxfE8dvzojHEkXw0sA4igINHDDFX3HJGZtLio+WpEFQtCbfTAG\n\tYZslasz1YZRbwEdSsmO3/kqy5eMnczlm8a21A3fKUo3g8oAZEFM+f4DUNzqIltg31OAB/kZS\n\tenKZQ/SWC8PmLg/ZXBrReYakxXtkP6w3FwMlzOlhGxqhIRNiAJfXJBaRhuUWzPOpEDE9q5YJ\n\tBmqQL2WJm1VSNNVxbXJHpaWMH1sA2R00vmvRrPXGwyIO0IPYeUYQa3gsy6k+En/aMQJd27dp\n\taScf9am9PFICPY5T4ppneeJLif2lyLojo0mcHOV+uyrds9XkLpp14GfTkeKPdPMrLLTsHRfH\n\tfA4I4OBpRrEPiGIZB/0im98MkGY/Mu6qxeZmYLCcgD6qz4idOvfgVOrNh+aA8HzIVR+RMW8H\n\tQGBN9f0E3kfwxuhl3omo6V7lDw8XOdmuWZNC9zPq1UfryVHANYbLGz9KJ4Aw6M+OgBC2JpkD\n\thXMdHUkC+d20dwXrwHTlrJi1YNp6rBc+xald3wsUPOZ5z8moTHUX/uPA/qhGsbkCDQRWBP1m\n\tARAAzijkb+Sau4hAncr1JjOY+KyFEdUNxRy+hqTJdJfaYihxyaj0Ee0P0zEi35CbE6lgU0Uz\n\ttih9fiUbSV3wfsWqg1Ut3/5rTKu7kLFp15kF7eqvV4uezXRD3Qu4yjv/rMmEJbbD4cTvGCYI\n\td6MDC417f7vK3hCbCVIZSp3GXxyC1LU+UQr3fFcOyCwmP9vDUR9JV0BSqHHxRDdpUXE26Dk6\n\tmhf0V1YkspE5St814ETXpEus2urZE5yJIUROlWPIL+hm3NEWfAP06vsQUyLvr/GtbOT79vXl\n\tEn1aulcYyu20dRRxhkQ6iILaURcxIAVJJKPi8dsoMnS8pB0QW12AHWuirPF0g6DiuUfPmrA5\n\tPKe56IGlpkjc8cO51lIxHkWTpCMWigRdPDexKX+Sb+W9QWK/0JjIc4t3KBaiG8O4yRX8ml2R\n\t+rxfAVKM6V769P/hWoRGdgUMgYHFpHGSgEt80OKK5HeUPy2cngDUXzwrqiM5Sz6Od0qw5pCk\n\tNlXqI0W/who0iSVM+8+RmyY0OEkxEcci7rRLsGnM15B5PjLJjh1f2ULYkv8s4SnDwMZ/kE04\n\t/UqCMK/KnX8pwXEMCjz0h6qWNpGwJ0/tYIgQJZh6bqkvBrDogAvuhf60Sogw+mH8b+PBlx1L\n\toeTK396wc+4c3BfiC6pNtUS5GpsPMMjYMk7kVvEAEQEAAYkCPAQYAQoAJgIbDBYhBJAt15g/\n\tvSj943LUeqEeRnIQpGH9BQJdizzIBQkLSKZiAAoJEKEeRnIQpGH9eYgQAJpjaWNgqNOnMTmD\n\tMJggbwjIotypzIXfhHNCeTkG7+qCDlSaBPclcPGYrTwCt0YWPU2TgGgJrVhYT20ierN8LUvj\n\t6qOPTd+Uk7NFzL65qkh80ZKNBFddx1AabQpSVQKbdcLb8OFs85kuSvFdgqZwgxA1vl4TFhNz\n\tPZ79NAmXLackAx3sOVFhk4WQaKRshCB7cSl+RIng5S/ThOBlwNlcKG7j7W2MC06BlTbdEkUp\n\tECzuuRBv8wX4OQl+hbWbB/VKIx5HKlLu1eypen/5lNVzSqMMIYkkZcjV2SWQyUGxSwq0O/sx\n\tS0A8/atCHUXOboUsn54qdxrVDaK+6jIAuo8JiRWctP16KjzUM7MO0/+4zllM8EY57rXrj48j\n\tsbEYX0YQnzaj+jO6kJtoZsIaYR7rMMq9aUAjyiaEZpmP1qF/2sYenDx0Fg2BSlLvLvXM0vU8\n\tpQk3kgDu7kb/7PRYrZvBsr21EIQoIjXbZxDz/o7z95frkP71EaICttZ6k9q5oxxA5WC6sTXc\n\tMW8zs8avFNuA9VpXt0YupJd2ijtZy2mpZNG02fFVXhIn4G807G7+9mhuC4XG5rKlBBUXTvPU\n\tAfYnB4JBDLmLzBFavQfvonSfbitgXwCG3vS+9HEwAjU30Bar1PEOmIbiAoMzuKeRm2LVpmq4\n\tWZw01QYHU/GUV/zHJSFk","Organization":"Ideas on Board","Message-ID":"<9ca3b492-8468-e9cf-b103-b4350be472b9@ideasonboard.com>","Date":"Wed, 21 Oct 2020 12:09:23 +0100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101\n\tThunderbird/68.10.0","MIME-Version":"1.0","In-Reply-To":"<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>","Content-Language":"en-GB","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Reply-To":"kieran.bingham@ideasonboard.com","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13412,"web_url":"https://patchwork.libcamera.org/comment/13412/","msgid":"","date":"2020-10-22T11:42:01","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":63,"url":"https://patchwork.libcamera.org/api/people/63/","name":"Hirokazu Honda","email":"hiroh@chromium.org"},"content":"Hi Umang,\nThanks for the work. I couldn't have time to review these today and\nwill review tomorrow.\nIn my understanding, the thumbnail class is basically to scale frames\nto the given destination.\nCan we generalize it as PostProcessor interface so that we will be\nable to make use of it for down-scaling regardless of whether jpeg\nencoding is required.\nIt will be necessary when Android HAL adaptation layer has to output\nmultiple different NV12 streams while the native camera can produce a\nsingle NV12 stream simultaneously.\n\nBest Regards,\n-Hiro\n\nOn Wed, Oct 21, 2020 at 8:09 PM Kieran Bingham\n wrote:\n>\n> Hi Umang,\n>\n> On 21/10/2020 12:03, Umang Jain wrote:\n> > Hi Kieran,\n> >\n> > On 10/21/20 4:25 PM, Kieran Bingham wrote:\n> >> Hi Umang,\n> >>\n> >>\n> >> On 21/10/2020 11:51, Umang Jain wrote:\n> >>> Hi Kieran,\n> >>>\n> >>> Thanks for the comments.\n> >>>\n> >>> On 10/21/20 2:49 PM, Kieran Bingham wrote:\n> >>>> Hi Umang,\n> >>>>\n> >>>> On 21/10/2020 09:08, Umang Jain wrote:\n> >>>>> Add a basic image thumbnailer for NV12 frames being captured.\n> >>>>> It shall generate a thumbnail image to be embedded as a part of\n> >>>>> EXIF metadata of the frame. The output of the thumbnail will still\n> >>>>> be NV12.\n> >>>>>\n> >>>>> Signed-off-by: Umang Jain \n> >>>>> ---\n> >>>>> src/android/jpeg/post_processor_jpeg.cpp | 39 +++++++++\n> >>>>> src/android/jpeg/post_processor_jpeg.h | 3 +\n> >>>>> src/android/jpeg/thumbnailer.cpp | 100\n> >>>>> +++++++++++++++++++++++\n> >>>>> src/android/jpeg/thumbnailer.h | 40 +++++++++\n> >>>>> src/android/meson.build | 1 +\n> >>>>> 5 files changed, 183 insertions(+)\n> >>>>> create mode 100644 src/android/jpeg/thumbnailer.cpp\n> >>>>> create mode 100644 src/android/jpeg/thumbnailer.h\n> >>>>>\n> >>>>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp\n> >>>>> b/src/android/jpeg/post_processor_jpeg.cpp\n> >>>>> index 9d452b7..f5f1f78 100644\n> >>>>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n> >>>>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n> >>>>> @@ -11,6 +11,7 @@\n> >>>>> #include \"../camera_metadata.h\"\n> >>>>> #include \"encoder_libjpeg.h\"\n> >>>>> #include \"exif.h\"\n> >>>>> +#include \"thumbnailer.h\"\n> >>>>> #include \n> >>>>> @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const\n> >>>>> StreamConfiguration &inCfg,\n> >>>>> return encoder_->configure(inCfg);\n> >>>>> }\n> >>>>> +void PostProcessorJpeg::generateThumbnail(const\n> >>>>> libcamera::Span &source,\n> >>>>> + std::vector &thumbnail)\n> >>>>> +{\n> >>>>> + libcamera::Span destination;\n> >>>>> + Thumbnailer thumbnailer;\n> >>>>> +\n> >>>>> + thumbnailer.configure(streamSize_, formats::NV12);\n> >>>>> + libcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n> >>>>> + thumbnailer.scaleBuffer(source, thumbnail);\n> >>>>> +\n> >>>>> + if (thumbnail.data()) {\n> >>>>> + StreamConfiguration thumbnailCfg;\n> >>>>> +\n> >>>>> + std::unique_ptr encoder =\n> >>>>> + std::make_unique();\n> >>>>> +\n> >>>>> + thumbnailCfg.pixelFormat = formats::NV12;\n> >>>>> + thumbnailCfg.size = targetSize;\n> >>>>> + encoder->configure(thumbnailCfg);\n> >>>> thumbnail.capacity() might be quite low here.\n> >>>> We need to make sure the vector is big enough at this point, you might\n> >>>> need to do something like:\n> >>>>\n> >>>> thumbnail.resize(targetSize.width * targetSize.height * 4);\n> >>> I am not sure I follow. This is compressing the thumbnail part right? So\n> >>> thumbnail is the \"source\" for the encoder here (Please refer to\n> >>> ->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n> >> Ugh, sorry - tired eyes indeed. Yes, please ignore those comments.\n> >>\n> >>\n> >>\n> >>>> Really we should obtain that size from the encoder. I thought we had a\n> >>>> helper to do that already, but it seems we don't.\n> >>>> We could/should add Encoder::maxOutput(); which would ask the encoder\n> >>>> \"For your given configuration, what is the maximum number of bytes you\n> >>>> might output\".\n> >>>>\n> >>>> Then of course we'd do:\n> >>>> thumbnail.resize(encoder.maxOutput());\n> >>>>\n> >>>>\n> >>>>> + int jpeg_size = encoder->encode({ thumbnail.data(),\n> >>>>> thumbnail.capacity() },\n> >>>>> + destination, { });\n> >>> As said above, thumbnail is the source here. The compressed thumbnail\n> >>> output is carried in destination. And, destination is a local span<>\n> >>> here. Does it makes sense?\n> >> Yes, I had mixed up source and destination I'm sorry.\n> >>\n> >> So who/where is allocating the destination?\n> > Ah, that's a good question and I think I just found a bug. I currently\n> > pass on a locally allocated Span<> 'destination' to the encoder. But\n> > there is no explicit allocation here. I think I need to point / create\n> > the Span with the output size / bytes and then pass to the encode().\n>\n> :-) Ok - then please apply all of my earlier comments to that part\n> instead. ;-)\n>\n> We certainly need somewhere to store the compressed image, to be able to\n> pass it into the exif :-D\n>\n> --\n> Kieran\n>\n>\n> >>>>> + LOG(JPEG, Info) << \"Thumbnail compress returned \"\n> >>>>> + << jpeg_size << \" bytes\";\n> >>>> And I presume we could then do an:\n> >>>> thumbnail.resize(jpeg_size);\n> >>>>\n> >>>> here to update the thumbnail with the correct size. I'd be weary of the\n> >>>> resize operations doing lots of re-allocations though, so perhaps we\n> >>>> want to minimize that. But lets get to something that works first\n> >>>> before\n> >>>> worrying about optimising.\n> >>>>\n> >>>>\n> >>>>> + }\n> >>>>> +}\n> >>>>> +\n> >>>>> int PostProcessorJpeg::process(const libcamera::FrameBuffer\n> >>>>> *source,\n> >>>>> const libcamera::Span &destination,\n> >>>>> CameraMetadata *metadata)\n> >>>>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const\n> >>>>> libcamera::FrameBuffer *source,\n> >>>>> return jpeg_size;\n> >>>>> }\n> >>>>> + std::vector thumbnail;\n> >>>> You need to resize this somewhere.\n> >>>> Edit: Now seen a better place above ;-)\n> >>> The thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\n> >>> We just pass in to the thumbnailer, while keeping its ownership in\n> >>> PostProcessorJpeg.\n> >>>>> + generateThumbnail(destination, thumbnail);\n> >>>>> + /*\n> >>>>> + * \\todo: Write the compressed thumbnail to a file for\n> >>>>> inspection.\n> >>>>> + * (I) Check if we can still write the thumbnail to EXIF here.\n> >>>>> + * If not, we might need to move the thumbnailer logic to\n> >>>>> encoder.\n> >>>>> + * And if we do that, first we need to make sure we get can\n> >>>>> + * compressed data written to destination first before\n> >>>>> calling\n> >>>>> + * jpeg_finish_compress operation somehow. Thumbnailing will\n> >>>>> + * only occur if we have compressed data available first.\n> >>>>> + */\n> >>>>> +\n> >>>>> /*\n> >>>>> * Fill in the JPEG blob header.\n> >>>>> *\n> >>>>> diff --git a/src/android/jpeg/post_processor_jpeg.h\n> >>>>> b/src/android/jpeg/post_processor_jpeg.h\n> >>>>> index 62c8650..05601ee 100644\n> >>>>> --- a/src/android/jpeg/post_processor_jpeg.h\n> >>>>> +++ b/src/android/jpeg/post_processor_jpeg.h\n> >>>>> @@ -28,6 +28,9 @@ public:\n> >>>>> CameraMetadata *metadata) override;\n> >>>>> private:\n> >>>>> + void generateThumbnail(const libcamera::Span &source,\n> >>>>> + std::vector &thumbnail);\n> >>>>> +\n> >>>>> CameraDevice *cameraDevice_;\n> >>>>> std::unique_ptr encoder_;\n> >>>>> libcamera::Size streamSize_;\n> >>>>> diff --git a/src/android/jpeg/thumbnailer.cpp\n> >>>>> b/src/android/jpeg/thumbnailer.cpp\n> >>>>> new file mode 100644\n> >>>>> index 0000000..3163576\n> >>>>> --- /dev/null\n> >>>>> +++ b/src/android/jpeg/thumbnailer.cpp\n> >>>>> @@ -0,0 +1,100 @@\n> >>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> >>>>> +/*\n> >>>>> + * Copyright (C) 2020, Google Inc.\n> >>>>> + *\n> >>>>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n> >>>>> + */\n> >>>>> +\n> >>>>> +#include \"thumbnailer.h\"\n> >>>>> +\n> >>>>> +#include \n> >>>>> +\n> >>>>> +#include \"libcamera/internal/file.h\"\n> >>>>> +#include \"libcamera/internal/log.h\"\n> >>>>> +\n> >>>>> +using namespace libcamera;\n> >>>>> +\n> >>>>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n> >>>>> +\n> >>>>> +Thumbnailer::Thumbnailer()\n> >>>>> + : validConfiguration_(false)\n> >>>>> +{\n> >>>>> +}\n> >>>>> +\n> >>>>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat\n> >>>>> pixelFormat)\n> >>>>> +{\n> >>>>> + sourceSize_ = sourceSize;\n> >>>>> + pixelFormat_ = pixelFormat;\n> >>>>> +\n> >>>>> + if (pixelFormat_ != formats::NV12) {\n> >>>>> + LOG (Thumbnailer, Error) << \"Failed to configure: Pixel\n> >>>>> Format \"\n> >>>>> + << pixelFormat_.toString() << \" unsupported.\";\n> >>>>> + return;\n> >>>>> + }\n> >>>>> +\n> >>>>> + validConfiguration_ = true;\n> >>>>> +}\n> >>>>> +\n> >>>>> +/*\n> >>>>> + * The Exif specification recommends the width of the thumbnail to\n> >>>>> be a\n> >>>>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding\n> >>>>> height\n> >>>>> + * keeping the aspect ratio same as of the source.\n> >>>>> + */\n> >>>>> +Size Thumbnailer::computeThumbnailSize()\n> >>>>> +{\n> >>>>> + unsigned int targetHeight;\n> >>>>> + unsigned int targetWidth = 160;\n> >>>>> +\n> >>>>> + targetHeight = targetWidth * sourceSize_.height /\n> >>>>> sourceSize_.width;\n> >>>>> +\n> >>>>> + if (targetHeight & 1)\n> >>>>> + targetHeight++;\n> >>>>> +\n> >>>>> + return Size(targetWidth, targetHeight);\n> >>>>> +}\n> >>>>> +\n> >>>>> +void\n> >>>>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n> >>>>> + std::vector &destination)\n> >>>>> +{\n> >>>>> + if (!validConfiguration_) {\n> >>>>> + LOG(Thumbnailer, Error) << \"config is unconfigured or\n> >>>>> invalid.\";\n> >>>>> + return;\n> >>>>> + }\n> >>>>> +\n> >>>>> + targetSize_ = computeThumbnailSize();\n> >>>>> +\n> >>>>> + const unsigned int sw = sourceSize_.width;\n> >>>>> + const unsigned int sh = sourceSize_.height;\n> >>>>> + const unsigned int tw = targetSize_.width;\n> >>>>> + const unsigned int th = targetSize_.height;\n> >>>>> +\n> >>>>> + /* Image scaling block implementing nearest-neighbour\n> >>>>> algorithm. */\n> >>>>> + unsigned char *src = static_cast(source.data());\n> >>>>> + unsigned char *src_c = src + sh * sw;\n> >>>>> + unsigned char *src_cb, *src_cr;\n> >>>>> +\n> >>>>> + size_t dstSize = (th * tw) + ((th/2) * tw);\n> >>>>> + destination.reserve(dstSize);\n> >>>>> + unsigned char *dst = destination.data();\n> >>>>> + unsigned char *dst_c = dst + th * tw;\n> >>>>> +\n> >>>>> + for (unsigned int y = 0; y < th; y+=2) {\n> >>>>> + unsigned int sourceY = (sh*y + th/2) / th;\n> >>>>> +\n> >>>>> + src_cb = src_c + (sourceY/2) * sw + 0;\n> >>>>> + src_cr = src_c + (sourceY/2) * sw + 1;\n> >>>>> +\n> >>>>> + for (unsigned int x = 0; x < tw; x+=2) {\n> >>>>> + unsigned int sourceX = (sw*x + tw/2) / tw;\n> >>>>> +\n> >>>>> + dst[y * tw + x] = src[sw * sourceY +\n> >>>>> sourceX];\n> >>>>> + dst[(y+1) * tw + x] = src[sw * (sourceY+1) +\n> >>>>> sourceX];\n> >>>>> + dst[y * tw + (x+1)] = src[sw * sourceY +\n> >>>>> (sourceX+1)];\n> >>>>> + dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) +\n> >>>>> (sourceX+1)];\n> >>>>> +\n> >>>>> + dst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n> >>>>> + dst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n> >>>>> + }\n> >>>>> + }\n> >>>>> +}\n> >>>>> diff --git a/src/android/jpeg/thumbnailer.h\n> >>>>> b/src/android/jpeg/thumbnailer.h\n> >>>>> new file mode 100644\n> >>>>> index 0000000..bab9855\n> >>>>> --- /dev/null\n> >>>>> +++ b/src/android/jpeg/thumbnailer.h\n> >>>>> @@ -0,0 +1,40 @@\n> >>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> >>>>> +/*\n> >>>>> + * Copyright (C) 2020, Google Inc.\n> >>>>> + *\n> >>>>> + * thumbnailer.h - Basic image thumbnailer from NV12\n> >>>>> + */\n> >>>>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n> >>>>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n> >>>>> +\n> >>>>> +#include \n> >>>>> +\n> >>>>> +#include \"libcamera/internal/buffer.h\"\n> >>>>> +#include \"libcamera/internal/formats.h\"\n> >>>>> +\n> >>>>> +class Thumbnailer\n> >>>>> +{\n> >>>>> +public:\n> >>>>> + Thumbnailer();\n> >>>>> +\n> >>>>> + void configure(const libcamera::Size &sourceSize,\n> >>>>> + libcamera::PixelFormat pixelFormat);\n> >>>>> +\n> >>>>> + /*\n> >>>>> + * \\todo: Discuss if we can return targetSize_ via configure() or\n> >>>>> + * scaleBuffer(). We need targetSize_ to re-encode the scaled\n> >>>>> buffer\n> >>>>> + * via encoder in PostProcssorJpeg::writeThumbnail().\n> >>>>> + */\n> >>>>> + libcamera::Size computeThumbnailSize();\n> >>>>> + void scaleBuffer(const libcamera::Span &source,\n> >>>>> + std::vector &dest);\n> >>>>> +\n> >>>>> +private:\n> >>>>> + libcamera::PixelFormat pixelFormat_;\n> >>>>> + libcamera::Size sourceSize_;\n> >>>>> + libcamera::Size targetSize_;\n> >>>>> +\n> >>>>> + bool validConfiguration_;\n> >>>>> +};\n> >>>>> +\n> >>>>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n> >>>>> diff --git a/src/android/meson.build b/src/android/meson.build\n> >>>>> index 5a01bea..3905e2f 100644\n> >>>>> --- a/src/android/meson.build\n> >>>>> +++ b/src/android/meson.build\n> >>>>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n> >>>>> 'jpeg/encoder_libjpeg.cpp',\n> >>>>> 'jpeg/exif.cpp',\n> >>>>> 'jpeg/post_processor_jpeg.cpp',\n> >>>>> + 'jpeg/thumbnailer.cpp',\n> >>>>> ])\n> >>>>> android_camera_metadata_sources = files([\n> >>>>>\n> >\n>\n> --\n> Regards\n> --\n> Kieran\n> _______________________________________________\n> libcamera-devel mailing list\n> libcamera-devel@lists.libcamera.org\n> https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id ED393C3D3C\n\tfor ;\n\tThu, 22 Oct 2020 11:42:13 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 7F841613E6;\n\tThu, 22 Oct 2020 13:42:13 +0200 (CEST)","from mail-ed1-x530.google.com (mail-ed1-x530.google.com\n\t[IPv6:2a00:1450:4864:20::530])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 7D24D60352\n\tfor ;\n\tThu, 22 Oct 2020 13:42:12 +0200 (CEST)","by mail-ed1-x530.google.com with SMTP id bc23so1427196edb.5\n\tfor ;\n\tThu, 22 Oct 2020 04:42:12 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=chromium.org header.i=@chromium.org\n\theader.b=\"mxj4c5JV\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org;\n\ts=google; \n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=8gT/AvRyLADwogIuXpDagxiEYD1Nj+K1KBVJFrtpyOg=;\n\tb=mxj4c5JV2JVuoidHNpVOmalzQDbPby3cyOGyvFauTIeX0PHYsUXnV8WWUrw5b9fVEv\n\ty5XuYzD9bTg92dW5wgNe9IB7JCyEC89DRWP+PhiG/KTbYUISHQvw4hqnWvoiP5Vnv+gd\n\tG/YYZxfiCJGlcZ2CldQMbMLhugwY2U5GFE6po=","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=8gT/AvRyLADwogIuXpDagxiEYD1Nj+K1KBVJFrtpyOg=;\n\tb=A4jersUa+t/VL9AjXcFjYnaSt9117OWq+oG8F923LanFa7pEYaM7ouw3yGkr8eDes5\n\t6NQyGvJPe9UDMMP1osoxOYvkmTffhZ8v1Gd5+CGO9mPKj7ui09K4gxbd/La0y2d13Kyl\n\tDhLwkL+mHN1yrTpAYyo1M0/l0UixZxMoSH7Uw0LEo5qR4pkVXMFn9022wowvtNkwHJxg\n\tM3n4GqFopFng9XlNCiY/NDQwfpwyucZBUf7VF1/6dy7cB1jSugeZYFdn2smvQkjT/Wwh\n\tmAJtaQD0UTvK0nDpH07tyuv2KOVzqUZWNN/hdmRXe5Wh2r+pDGXmYwFZAyVN67i9y6K4\n\toQ+Q==","X-Gm-Message-State":"AOAM533jbYIRuj5H2XCte+7JfTCWTL22CvDDsMWENVm2+4Hah39Z+y9g\n\ta7SVGl6AqyyZg8Ah9Tj6d1wN8ejplu/Sno2iKClgxB3HKus=","X-Google-Smtp-Source":"ABdhPJwaLs2yMXHBQ7zV/d8unHpcvBwWcu+xZ6QBpDjDHbvw5FuDQFssVzSO4/Wftj6cjT7XKpYZzl18Iy6S/vFrZk8=","X-Received":"by 2002:a05:6402:1446:: with SMTP id\n\td6mr1829739edx.244.1603366931895; \n\tThu, 22 Oct 2020 04:42:11 -0700 (PDT)","MIME-Version":"1.0","References":"<20201021080806.46636-1-email@uajain.com>\n\t<20201021080806.46636-3-email@uajain.com>\n\t\n\t\n\t<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>\n\t<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>\n\t<9ca3b492-8468-e9cf-b103-b4350be472b9@ideasonboard.com>","In-Reply-To":"<9ca3b492-8468-e9cf-b103-b4350be472b9@ideasonboard.com>","From":"Hirokazu Honda ","Date":"Thu, 22 Oct 2020 20:42:01 +0900","Message-ID":"","To":"Kieran Bingham ","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13414,"web_url":"https://patchwork.libcamera.org/comment/13414/","msgid":"","date":"2020-10-22T12:14:09","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":1,"url":"https://patchwork.libcamera.org/api/people/1/","name":"Umang Jain","email":"email@uajain.com"},"content":"Hi Hiro,\n\nOn 10/22/20 5:12 PM, Hirokazu Honda wrote:\n> Hi Umang,\n> Thanks for the work. I couldn't have time to review these today and\n> will review tomorrow.\nThanks, but I think as this is still in a bit of flux, you can delay the \nreviewing when I submit actual patches for merging :) I got what I \nwanted from this RFC version, so, please use your discretion :)\n> In my understanding, the thumbnail class is basically to scale frames\n> to the given destination.\n> Can we generalize it as PostProcessor interface so that we will be\n> able to make use of it for down-scaling regardless of whether jpeg\n> encoding is required.\n> It will be necessary when Android HAL adaptation layer has to output\n> multiple different NV12 streams while the native camera can produce a\n> single NV12 stream simultaneously.\nThis sounds good. However, I would prefer if it can be done on top \nmaybe? That way, I would have a bit more context and can evaluate what \n(new) use cases we need to satisfy when we reach there. Since, I am \nalready fatigued a bit with this series (it dragged for a while now), I \nam keen on getting this merged to support our Exif work. Anyways,  I am \nup for the discussion if you want to start a thread on the list.\n>\n> Best Regards,\n> -Hiro\n>\n> On Wed, Oct 21, 2020 at 8:09 PM Kieran Bingham\n> wrote:\n>> Hi Umang,\n>>\n>> On 21/10/2020 12:03, Umang Jain wrote:\n>>> Hi Kieran,\n>>>\n>>> On 10/21/20 4:25 PM, Kieran Bingham wrote:\n>>>> Hi Umang,\n>>>>\n>>>>\n>>>> On 21/10/2020 11:51, Umang Jain wrote:\n>>>>> Hi Kieran,\n>>>>>\n>>>>> Thanks for the comments.\n>>>>>\n>>>>> On 10/21/20 2:49 PM, Kieran Bingham wrote:\n>>>>>> Hi Umang,\n>>>>>>\n>>>>>> On 21/10/2020 09:08, Umang Jain wrote:\n>>>>>>> Add a basic image thumbnailer for NV12 frames being captured.\n>>>>>>> It shall generate a thumbnail image to be embedded as a part of\n>>>>>>> EXIF metadata of the frame. The output of the thumbnail will still\n>>>>>>> be NV12.\n>>>>>>>\n>>>>>>> Signed-off-by: Umang Jain \n>>>>>>> ---\n>>>>>>> src/android/jpeg/post_processor_jpeg.cpp | 39 +++++++++\n>>>>>>> src/android/jpeg/post_processor_jpeg.h | 3 +\n>>>>>>> src/android/jpeg/thumbnailer.cpp | 100\n>>>>>>> +++++++++++++++++++++++\n>>>>>>> src/android/jpeg/thumbnailer.h | 40 +++++++++\n>>>>>>> src/android/meson.build | 1 +\n>>>>>>> 5 files changed, 183 insertions(+)\n>>>>>>> create mode 100644 src/android/jpeg/thumbnailer.cpp\n>>>>>>> create mode 100644 src/android/jpeg/thumbnailer.h\n>>>>>>>\n>>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp\n>>>>>>> b/src/android/jpeg/post_processor_jpeg.cpp\n>>>>>>> index 9d452b7..f5f1f78 100644\n>>>>>>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n>>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n>>>>>>> @@ -11,6 +11,7 @@\n>>>>>>> #include \"../camera_metadata.h\"\n>>>>>>> #include \"encoder_libjpeg.h\"\n>>>>>>> #include \"exif.h\"\n>>>>>>> +#include \"thumbnailer.h\"\n>>>>>>> #include \n>>>>>>> @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const\n>>>>>>> StreamConfiguration &inCfg,\n>>>>>>> return encoder_->configure(inCfg);\n>>>>>>> }\n>>>>>>> +void PostProcessorJpeg::generateThumbnail(const\n>>>>>>> libcamera::Span &source,\n>>>>>>> + std::vector &thumbnail)\n>>>>>>> +{\n>>>>>>> + libcamera::Span destination;\n>>>>>>> + Thumbnailer thumbnailer;\n>>>>>>> +\n>>>>>>> + thumbnailer.configure(streamSize_, formats::NV12);\n>>>>>>> + libcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n>>>>>>> + thumbnailer.scaleBuffer(source, thumbnail);\n>>>>>>> +\n>>>>>>> + if (thumbnail.data()) {\n>>>>>>> + StreamConfiguration thumbnailCfg;\n>>>>>>> +\n>>>>>>> + std::unique_ptr encoder =\n>>>>>>> + std::make_unique();\n>>>>>>> +\n>>>>>>> + thumbnailCfg.pixelFormat = formats::NV12;\n>>>>>>> + thumbnailCfg.size = targetSize;\n>>>>>>> + encoder->configure(thumbnailCfg);\n>>>>>> thumbnail.capacity() might be quite low here.\n>>>>>> We need to make sure the vector is big enough at this point, you might\n>>>>>> need to do something like:\n>>>>>>\n>>>>>> thumbnail.resize(targetSize.width * targetSize.height * 4);\n>>>>> I am not sure I follow. This is compressing the thumbnail part right? So\n>>>>> thumbnail is the \"source\" for the encoder here (Please refer to\n>>>>> ->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n>>>> Ugh, sorry - tired eyes indeed. Yes, please ignore those comments.\n>>>>\n>>>>\n>>>>\n>>>>>> Really we should obtain that size from the encoder. I thought we had a\n>>>>>> helper to do that already, but it seems we don't.\n>>>>>> We could/should add Encoder::maxOutput(); which would ask the encoder\n>>>>>> \"For your given configuration, what is the maximum number of bytes you\n>>>>>> might output\".\n>>>>>>\n>>>>>> Then of course we'd do:\n>>>>>> thumbnail.resize(encoder.maxOutput());\n>>>>>>\n>>>>>>\n>>>>>>> + int jpeg_size = encoder->encode({ thumbnail.data(),\n>>>>>>> thumbnail.capacity() },\n>>>>>>> + destination, { });\n>>>>> As said above, thumbnail is the source here. The compressed thumbnail\n>>>>> output is carried in destination. And, destination is a local span<>\n>>>>> here. Does it makes sense?\n>>>> Yes, I had mixed up source and destination I'm sorry.\n>>>>\n>>>> So who/where is allocating the destination?\n>>> Ah, that's a good question and I think I just found a bug. I currently\n>>> pass on a locally allocated Span<> 'destination' to the encoder. But\n>>> there is no explicit allocation here. I think I need to point / create\n>>> the Span with the output size / bytes and then pass to the encode().\n>> :-) Ok - then please apply all of my earlier comments to that part\n>> instead. ;-)\n>>\n>> We certainly need somewhere to store the compressed image, to be able to\n>> pass it into the exif :-D\n>>\n>> --\n>> Kieran\n>>\n>>\n>>>>>>> + LOG(JPEG, Info) << \"Thumbnail compress returned \"\n>>>>>>> + << jpeg_size << \" bytes\";\n>>>>>> And I presume we could then do an:\n>>>>>> thumbnail.resize(jpeg_size);\n>>>>>>\n>>>>>> here to update the thumbnail with the correct size. I'd be weary of the\n>>>>>> resize operations doing lots of re-allocations though, so perhaps we\n>>>>>> want to minimize that. But lets get to something that works first\n>>>>>> before\n>>>>>> worrying about optimising.\n>>>>>>\n>>>>>>\n>>>>>>> + }\n>>>>>>> +}\n>>>>>>> +\n>>>>>>> int PostProcessorJpeg::process(const libcamera::FrameBuffer\n>>>>>>> *source,\n>>>>>>> const libcamera::Span &destination,\n>>>>>>> CameraMetadata *metadata)\n>>>>>>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const\n>>>>>>> libcamera::FrameBuffer *source,\n>>>>>>> return jpeg_size;\n>>>>>>> }\n>>>>>>> + std::vector thumbnail;\n>>>>>> You need to resize this somewhere.\n>>>>>> Edit: Now seen a better place above ;-)\n>>>>> The thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\n>>>>> We just pass in to the thumbnailer, while keeping its ownership in\n>>>>> PostProcessorJpeg.\n>>>>>>> + generateThumbnail(destination, thumbnail);\n>>>>>>> + /*\n>>>>>>> + * \\todo: Write the compressed thumbnail to a file for\n>>>>>>> inspection.\n>>>>>>> + * (I) Check if we can still write the thumbnail to EXIF here.\n>>>>>>> + * If not, we might need to move the thumbnailer logic to\n>>>>>>> encoder.\n>>>>>>> + * And if we do that, first we need to make sure we get can\n>>>>>>> + * compressed data written to destination first before\n>>>>>>> calling\n>>>>>>> + * jpeg_finish_compress operation somehow. Thumbnailing will\n>>>>>>> + * only occur if we have compressed data available first.\n>>>>>>> + */\n>>>>>>> +\n>>>>>>> /*\n>>>>>>> * Fill in the JPEG blob header.\n>>>>>>> *\n>>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.h\n>>>>>>> b/src/android/jpeg/post_processor_jpeg.h\n>>>>>>> index 62c8650..05601ee 100644\n>>>>>>> --- a/src/android/jpeg/post_processor_jpeg.h\n>>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.h\n>>>>>>> @@ -28,6 +28,9 @@ public:\n>>>>>>> CameraMetadata *metadata) override;\n>>>>>>> private:\n>>>>>>> + void generateThumbnail(const libcamera::Span &source,\n>>>>>>> + std::vector &thumbnail);\n>>>>>>> +\n>>>>>>> CameraDevice *cameraDevice_;\n>>>>>>> std::unique_ptr encoder_;\n>>>>>>> libcamera::Size streamSize_;\n>>>>>>> diff --git a/src/android/jpeg/thumbnailer.cpp\n>>>>>>> b/src/android/jpeg/thumbnailer.cpp\n>>>>>>> new file mode 100644\n>>>>>>> index 0000000..3163576\n>>>>>>> --- /dev/null\n>>>>>>> +++ b/src/android/jpeg/thumbnailer.cpp\n>>>>>>> @@ -0,0 +1,100 @@\n>>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n>>>>>>> +/*\n>>>>>>> + * Copyright (C) 2020, Google Inc.\n>>>>>>> + *\n>>>>>>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n>>>>>>> + */\n>>>>>>> +\n>>>>>>> +#include \"thumbnailer.h\"\n>>>>>>> +\n>>>>>>> +#include \n>>>>>>> +\n>>>>>>> +#include \"libcamera/internal/file.h\"\n>>>>>>> +#include \"libcamera/internal/log.h\"\n>>>>>>> +\n>>>>>>> +using namespace libcamera;\n>>>>>>> +\n>>>>>>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n>>>>>>> +\n>>>>>>> +Thumbnailer::Thumbnailer()\n>>>>>>> + : validConfiguration_(false)\n>>>>>>> +{\n>>>>>>> +}\n>>>>>>> +\n>>>>>>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat\n>>>>>>> pixelFormat)\n>>>>>>> +{\n>>>>>>> + sourceSize_ = sourceSize;\n>>>>>>> + pixelFormat_ = pixelFormat;\n>>>>>>> +\n>>>>>>> + if (pixelFormat_ != formats::NV12) {\n>>>>>>> + LOG (Thumbnailer, Error) << \"Failed to configure: Pixel\n>>>>>>> Format \"\n>>>>>>> + << pixelFormat_.toString() << \" unsupported.\";\n>>>>>>> + return;\n>>>>>>> + }\n>>>>>>> +\n>>>>>>> + validConfiguration_ = true;\n>>>>>>> +}\n>>>>>>> +\n>>>>>>> +/*\n>>>>>>> + * The Exif specification recommends the width of the thumbnail to\n>>>>>>> be a\n>>>>>>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding\n>>>>>>> height\n>>>>>>> + * keeping the aspect ratio same as of the source.\n>>>>>>> + */\n>>>>>>> +Size Thumbnailer::computeThumbnailSize()\n>>>>>>> +{\n>>>>>>> + unsigned int targetHeight;\n>>>>>>> + unsigned int targetWidth = 160;\n>>>>>>> +\n>>>>>>> + targetHeight = targetWidth * sourceSize_.height /\n>>>>>>> sourceSize_.width;\n>>>>>>> +\n>>>>>>> + if (targetHeight & 1)\n>>>>>>> + targetHeight++;\n>>>>>>> +\n>>>>>>> + return Size(targetWidth, targetHeight);\n>>>>>>> +}\n>>>>>>> +\n>>>>>>> +void\n>>>>>>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n>>>>>>> + std::vector &destination)\n>>>>>>> +{\n>>>>>>> + if (!validConfiguration_) {\n>>>>>>> + LOG(Thumbnailer, Error) << \"config is unconfigured or\n>>>>>>> invalid.\";\n>>>>>>> + return;\n>>>>>>> + }\n>>>>>>> +\n>>>>>>> + targetSize_ = computeThumbnailSize();\n>>>>>>> +\n>>>>>>> + const unsigned int sw = sourceSize_.width;\n>>>>>>> + const unsigned int sh = sourceSize_.height;\n>>>>>>> + const unsigned int tw = targetSize_.width;\n>>>>>>> + const unsigned int th = targetSize_.height;\n>>>>>>> +\n>>>>>>> + /* Image scaling block implementing nearest-neighbour\n>>>>>>> algorithm. */\n>>>>>>> + unsigned char *src = static_cast(source.data());\n>>>>>>> + unsigned char *src_c = src + sh * sw;\n>>>>>>> + unsigned char *src_cb, *src_cr;\n>>>>>>> +\n>>>>>>> + size_t dstSize = (th * tw) + ((th/2) * tw);\n>>>>>>> + destination.reserve(dstSize);\n>>>>>>> + unsigned char *dst = destination.data();\n>>>>>>> + unsigned char *dst_c = dst + th * tw;\n>>>>>>> +\n>>>>>>> + for (unsigned int y = 0; y < th; y+=2) {\n>>>>>>> + unsigned int sourceY = (sh*y + th/2) / th;\n>>>>>>> +\n>>>>>>> + src_cb = src_c + (sourceY/2) * sw + 0;\n>>>>>>> + src_cr = src_c + (sourceY/2) * sw + 1;\n>>>>>>> +\n>>>>>>> + for (unsigned int x = 0; x < tw; x+=2) {\n>>>>>>> + unsigned int sourceX = (sw*x + tw/2) / tw;\n>>>>>>> +\n>>>>>>> + dst[y * tw + x] = src[sw * sourceY +\n>>>>>>> sourceX];\n>>>>>>> + dst[(y+1) * tw + x] = src[sw * (sourceY+1) +\n>>>>>>> sourceX];\n>>>>>>> + dst[y * tw + (x+1)] = src[sw * sourceY +\n>>>>>>> (sourceX+1)];\n>>>>>>> + dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) +\n>>>>>>> (sourceX+1)];\n>>>>>>> +\n>>>>>>> + dst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n>>>>>>> + dst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n>>>>>>> + }\n>>>>>>> + }\n>>>>>>> +}\n>>>>>>> diff --git a/src/android/jpeg/thumbnailer.h\n>>>>>>> b/src/android/jpeg/thumbnailer.h\n>>>>>>> new file mode 100644\n>>>>>>> index 0000000..bab9855\n>>>>>>> --- /dev/null\n>>>>>>> +++ b/src/android/jpeg/thumbnailer.h\n>>>>>>> @@ -0,0 +1,40 @@\n>>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n>>>>>>> +/*\n>>>>>>> + * Copyright (C) 2020, Google Inc.\n>>>>>>> + *\n>>>>>>> + * thumbnailer.h - Basic image thumbnailer from NV12\n>>>>>>> + */\n>>>>>>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n>>>>>>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n>>>>>>> +\n>>>>>>> +#include \n>>>>>>> +\n>>>>>>> +#include \"libcamera/internal/buffer.h\"\n>>>>>>> +#include \"libcamera/internal/formats.h\"\n>>>>>>> +\n>>>>>>> +class Thumbnailer\n>>>>>>> +{\n>>>>>>> +public:\n>>>>>>> + Thumbnailer();\n>>>>>>> +\n>>>>>>> + void configure(const libcamera::Size &sourceSize,\n>>>>>>> + libcamera::PixelFormat pixelFormat);\n>>>>>>> +\n>>>>>>> + /*\n>>>>>>> + * \\todo: Discuss if we can return targetSize_ via configure() or\n>>>>>>> + * scaleBuffer(). We need targetSize_ to re-encode the scaled\n>>>>>>> buffer\n>>>>>>> + * via encoder in PostProcssorJpeg::writeThumbnail().\n>>>>>>> + */\n>>>>>>> + libcamera::Size computeThumbnailSize();\n>>>>>>> + void scaleBuffer(const libcamera::Span &source,\n>>>>>>> + std::vector &dest);\n>>>>>>> +\n>>>>>>> +private:\n>>>>>>> + libcamera::PixelFormat pixelFormat_;\n>>>>>>> + libcamera::Size sourceSize_;\n>>>>>>> + libcamera::Size targetSize_;\n>>>>>>> +\n>>>>>>> + bool validConfiguration_;\n>>>>>>> +};\n>>>>>>> +\n>>>>>>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n>>>>>>> diff --git a/src/android/meson.build b/src/android/meson.build\n>>>>>>> index 5a01bea..3905e2f 100644\n>>>>>>> --- a/src/android/meson.build\n>>>>>>> +++ b/src/android/meson.build\n>>>>>>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n>>>>>>> 'jpeg/encoder_libjpeg.cpp',\n>>>>>>> 'jpeg/exif.cpp',\n>>>>>>> 'jpeg/post_processor_jpeg.cpp',\n>>>>>>> + 'jpeg/thumbnailer.cpp',\n>>>>>>> ])\n>>>>>>> android_camera_metadata_sources = files([\n>>>>>>>\n>> --\n>> Regards\n>> --\n>> Kieran\n>> _______________________________________________\n>> libcamera-devel mailing list\n>> libcamera-devel@lists.libcamera.org\n>> https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 68AA0C3D3C\n\tfor ;\n\tThu, 22 Oct 2020 12:14:15 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id EB76F613E6;\n\tThu, 22 Oct 2020 14:14:14 +0200 (CEST)","from mail.uajain.com (static.126.159.217.95.clients.your-server.de\n\t[95.217.159.126])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id B405D60352\n\tfor ;\n\tThu, 22 Oct 2020 14:14:13 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=uajain.com header.i=@uajain.com\n\theader.b=\"tp7QLunB\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=uajain.com; s=mail;\n\tt=1603368853; bh=1R9s1B1ZI6t/nexJme6ajC+yztiJeL7pMCeVT26F/OI=;\n\th=Subject:To:Cc:References:From:In-Reply-To;\n\tb=tp7QLunBK6KAzlMVLlS6kQr0ENvJvBtMRjbF/932TnG6lRi/RRFdaHAHR4d5khI3v\n\tYhkt1aKlPoNiAxBQdhRdCTbMtUOlM8EP4HLRVz8tUx42XtV/sGNFsbtBtpAnznZbsM\n\tp7TQ2JTl85/I80NJ0tGmyYyMNlp6BzlY8hqPC0vdb00jwTT/RM4SNGyQ9+/YSp+qGS\n\tX0ISiZqffNiW0MrNOU1OAQZKB+DXWJ+pJe+7aLquN8GX8fScj/K9wZ74EPrdnKZ7As\n\tmb/HN6gTt8leH/9AUo1W5IWuZNbLp7Cy4b8VaY1zenyAWpYC5cJkyVDlnQ78vBnYJM\n\toGoVxuQWX+oqQ==","To":"Hirokazu Honda ,\n\tKieran Bingham ","References":"<20201021080806.46636-1-email@uajain.com>\n\t<20201021080806.46636-3-email@uajain.com>\n\t\n\t\n\t<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>\n\t<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>\n\t<9ca3b492-8468-e9cf-b103-b4350be472b9@ideasonboard.com>\n\t","From":"Umang Jain ","Message-ID":"","Date":"Thu, 22 Oct 2020 17:44:09 +0530","Mime-Version":"1.0","In-Reply-To":"","Content-Language":"en-US","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Transfer-Encoding":"base64","Content-Type":"text/plain; charset=\"utf-8\"; Format=\"flowed\"","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13415,"web_url":"https://patchwork.libcamera.org/comment/13415/","msgid":"","date":"2020-10-22T13:39:00","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":63,"url":"https://patchwork.libcamera.org/api/people/63/","name":"Hirokazu Honda","email":"hiroh@chromium.org"},"content":"On Thu, Oct 22, 2020 at 9:14 PM Umang Jain wrote:\n>\n> Hi Hiro,\n>\n> On 10/22/20 5:12 PM, Hirokazu Honda wrote:\n> > Hi Umang,\n> > Thanks for the work. I couldn't have time to review these today and\n> > will review tomorrow.\n> Thanks, but I think as this is still in a bit of flux, you can delay the\n> reviewing when I submit actual patches for merging :) I got what I\n> wanted from this RFC version, so, please use your discretion :)\n> > In my understanding, the thumbnail class is basically to scale frames\n> > to the given destination.\n> > Can we generalize it as PostProcessor interface so that we will be\n> > able to make use of it for down-scaling regardless of whether jpeg\n> > encoding is required.\n> > It will be necessary when Android HAL adaptation layer has to output\n> > multiple different NV12 streams while the native camera can produce a\n> > single NV12 stream simultaneously.\n> This sounds good. However, I would prefer if it can be done on top\n> maybe? That way, I would have a bit more context and can evaluate what\n> (new) use cases we need to satisfy when we reach there. Since, I am\n> already fatigued a bit with this series (it dragged for a while now), I\n> am keen on getting this merged to support our Exif work. Anyways, I am\n> up for the discussion if you want to start a thread on the list.\n\nI am ok with your idea.\nAnother question, you scale with your own code.\nHow about using libyuv [1]?\nThe code is optimized for various platforms and the libyuv is widely\nused on a lot of products.\nFaster and less buggy.\n[1] https://chromium.googlesource.com/libyuv/libyuv/+/refs/heads/master/include/libyuv/scale.h#159\n\nBest Regards,\n-Hiro\n> >\n> > Best Regards,\n> > -Hiro\n> >\n> > On Wed, Oct 21, 2020 at 8:09 PM Kieran Bingham\n> > wrote:\n> >> Hi Umang,\n> >>\n> >> On 21/10/2020 12:03, Umang Jain wrote:\n> >>> Hi Kieran,\n> >>>\n> >>> On 10/21/20 4:25 PM, Kieran Bingham wrote:\n> >>>> Hi Umang,\n> >>>>\n> >>>>\n> >>>> On 21/10/2020 11:51, Umang Jain wrote:\n> >>>>> Hi Kieran,\n> >>>>>\n> >>>>> Thanks for the comments.\n> >>>>>\n> >>>>> On 10/21/20 2:49 PM, Kieran Bingham wrote:\n> >>>>>> Hi Umang,\n> >>>>>>\n> >>>>>> On 21/10/2020 09:08, Umang Jain wrote:\n> >>>>>>> Add a basic image thumbnailer for NV12 frames being captured.\n> >>>>>>> It shall generate a thumbnail image to be embedded as a part of\n> >>>>>>> EXIF metadata of the frame. The output of the thumbnail will still\n> >>>>>>> be NV12.\n> >>>>>>>\n> >>>>>>> Signed-off-by: Umang Jain \n> >>>>>>> ---\n> >>>>>>> src/android/jpeg/post_processor_jpeg.cpp | 39 +++++++++\n> >>>>>>> src/android/jpeg/post_processor_jpeg.h | 3 +\n> >>>>>>> src/android/jpeg/thumbnailer.cpp | 100\n> >>>>>>> +++++++++++++++++++++++\n> >>>>>>> src/android/jpeg/thumbnailer.h | 40 +++++++++\n> >>>>>>> src/android/meson.build | 1 +\n> >>>>>>> 5 files changed, 183 insertions(+)\n> >>>>>>> create mode 100644 src/android/jpeg/thumbnailer.cpp\n> >>>>>>> create mode 100644 src/android/jpeg/thumbnailer.h\n> >>>>>>>\n> >>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp\n> >>>>>>> b/src/android/jpeg/post_processor_jpeg.cpp\n> >>>>>>> index 9d452b7..f5f1f78 100644\n> >>>>>>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n> >>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n> >>>>>>> @@ -11,6 +11,7 @@\n> >>>>>>> #include \"../camera_metadata.h\"\n> >>>>>>> #include \"encoder_libjpeg.h\"\n> >>>>>>> #include \"exif.h\"\n> >>>>>>> +#include \"thumbnailer.h\"\n> >>>>>>> #include \n> >>>>>>> @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const\n> >>>>>>> StreamConfiguration &inCfg,\n> >>>>>>> return encoder_->configure(inCfg);\n> >>>>>>> }\n> >>>>>>> +void PostProcessorJpeg::generateThumbnail(const\n> >>>>>>> libcamera::Span &source,\n> >>>>>>> + std::vector &thumbnail)\n> >>>>>>> +{\n> >>>>>>> + libcamera::Span destination;\n> >>>>>>> + Thumbnailer thumbnailer;\n> >>>>>>> +\n> >>>>>>> + thumbnailer.configure(streamSize_, formats::NV12);\n> >>>>>>> + libcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n> >>>>>>> + thumbnailer.scaleBuffer(source, thumbnail);\n> >>>>>>> +\n> >>>>>>> + if (thumbnail.data()) {\n> >>>>>>> + StreamConfiguration thumbnailCfg;\n> >>>>>>> +\n> >>>>>>> + std::unique_ptr encoder =\n> >>>>>>> + std::make_unique();\n> >>>>>>> +\n> >>>>>>> + thumbnailCfg.pixelFormat = formats::NV12;\n> >>>>>>> + thumbnailCfg.size = targetSize;\n> >>>>>>> + encoder->configure(thumbnailCfg);\n> >>>>>> thumbnail.capacity() might be quite low here.\n> >>>>>> We need to make sure the vector is big enough at this point, you might\n> >>>>>> need to do something like:\n> >>>>>>\n> >>>>>> thumbnail.resize(targetSize.width * targetSize.height * 4);\n> >>>>> I am not sure I follow. This is compressing the thumbnail part right? So\n> >>>>> thumbnail is the \"source\" for the encoder here (Please refer to\n> >>>>> ->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n> >>>> Ugh, sorry - tired eyes indeed. Yes, please ignore those comments.\n> >>>>\n> >>>>\n> >>>>\n> >>>>>> Really we should obtain that size from the encoder. I thought we had a\n> >>>>>> helper to do that already, but it seems we don't.\n> >>>>>> We could/should add Encoder::maxOutput(); which would ask the encoder\n> >>>>>> \"For your given configuration, what is the maximum number of bytes you\n> >>>>>> might output\".\n> >>>>>>\n> >>>>>> Then of course we'd do:\n> >>>>>> thumbnail.resize(encoder.maxOutput());\n> >>>>>>\n> >>>>>>\n> >>>>>>> + int jpeg_size = encoder->encode({ thumbnail.data(),\n> >>>>>>> thumbnail.capacity() },\n> >>>>>>> + destination, { });\n> >>>>> As said above, thumbnail is the source here. The compressed thumbnail\n> >>>>> output is carried in destination. And, destination is a local span<>\n> >>>>> here. Does it makes sense?\n> >>>> Yes, I had mixed up source and destination I'm sorry.\n> >>>>\n> >>>> So who/where is allocating the destination?\n> >>> Ah, that's a good question and I think I just found a bug. I currently\n> >>> pass on a locally allocated Span<> 'destination' to the encoder. But\n> >>> there is no explicit allocation here. I think I need to point / create\n> >>> the Span with the output size / bytes and then pass to the encode().\n> >> :-) Ok - then please apply all of my earlier comments to that part\n> >> instead. ;-)\n> >>\n> >> We certainly need somewhere to store the compressed image, to be able to\n> >> pass it into the exif :-D\n> >>\n> >> --\n> >> Kieran\n> >>\n> >>\n> >>>>>>> + LOG(JPEG, Info) << \"Thumbnail compress returned \"\n> >>>>>>> + << jpeg_size << \" bytes\";\n> >>>>>> And I presume we could then do an:\n> >>>>>> thumbnail.resize(jpeg_size);\n> >>>>>>\n> >>>>>> here to update the thumbnail with the correct size. I'd be weary of the\n> >>>>>> resize operations doing lots of re-allocations though, so perhaps we\n> >>>>>> want to minimize that. But lets get to something that works first\n> >>>>>> before\n> >>>>>> worrying about optimising.\n> >>>>>>\n> >>>>>>\n> >>>>>>> + }\n> >>>>>>> +}\n> >>>>>>> +\n> >>>>>>> int PostProcessorJpeg::process(const libcamera::FrameBuffer\n> >>>>>>> *source,\n> >>>>>>> const libcamera::Span &destination,\n> >>>>>>> CameraMetadata *metadata)\n> >>>>>>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const\n> >>>>>>> libcamera::FrameBuffer *source,\n> >>>>>>> return jpeg_size;\n> >>>>>>> }\n> >>>>>>> + std::vector thumbnail;\n> >>>>>> You need to resize this somewhere.\n> >>>>>> Edit: Now seen a better place above ;-)\n> >>>>> The thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\n> >>>>> We just pass in to the thumbnailer, while keeping its ownership in\n> >>>>> PostProcessorJpeg.\n> >>>>>>> + generateThumbnail(destination, thumbnail);\n> >>>>>>> + /*\n> >>>>>>> + * \\todo: Write the compressed thumbnail to a file for\n> >>>>>>> inspection.\n> >>>>>>> + * (I) Check if we can still write the thumbnail to EXIF here.\n> >>>>>>> + * If not, we might need to move the thumbnailer logic to\n> >>>>>>> encoder.\n> >>>>>>> + * And if we do that, first we need to make sure we get can\n> >>>>>>> + * compressed data written to destination first before\n> >>>>>>> calling\n> >>>>>>> + * jpeg_finish_compress operation somehow. Thumbnailing will\n> >>>>>>> + * only occur if we have compressed data available first.\n> >>>>>>> + */\n> >>>>>>> +\n> >>>>>>> /*\n> >>>>>>> * Fill in the JPEG blob header.\n> >>>>>>> *\n> >>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.h\n> >>>>>>> b/src/android/jpeg/post_processor_jpeg.h\n> >>>>>>> index 62c8650..05601ee 100644\n> >>>>>>> --- a/src/android/jpeg/post_processor_jpeg.h\n> >>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.h\n> >>>>>>> @@ -28,6 +28,9 @@ public:\n> >>>>>>> CameraMetadata *metadata) override;\n> >>>>>>> private:\n> >>>>>>> + void generateThumbnail(const libcamera::Span &source,\n> >>>>>>> + std::vector &thumbnail);\n> >>>>>>> +\n> >>>>>>> CameraDevice *cameraDevice_;\n> >>>>>>> std::unique_ptr encoder_;\n> >>>>>>> libcamera::Size streamSize_;\n> >>>>>>> diff --git a/src/android/jpeg/thumbnailer.cpp\n> >>>>>>> b/src/android/jpeg/thumbnailer.cpp\n> >>>>>>> new file mode 100644\n> >>>>>>> index 0000000..3163576\n> >>>>>>> --- /dev/null\n> >>>>>>> +++ b/src/android/jpeg/thumbnailer.cpp\n> >>>>>>> @@ -0,0 +1,100 @@\n> >>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> >>>>>>> +/*\n> >>>>>>> + * Copyright (C) 2020, Google Inc.\n> >>>>>>> + *\n> >>>>>>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n> >>>>>>> + */\n> >>>>>>> +\n> >>>>>>> +#include \"thumbnailer.h\"\n> >>>>>>> +\n> >>>>>>> +#include \n> >>>>>>> +\n> >>>>>>> +#include \"libcamera/internal/file.h\"\n> >>>>>>> +#include \"libcamera/internal/log.h\"\n> >>>>>>> +\n> >>>>>>> +using namespace libcamera;\n> >>>>>>> +\n> >>>>>>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n> >>>>>>> +\n> >>>>>>> +Thumbnailer::Thumbnailer()\n> >>>>>>> + : validConfiguration_(false)\n> >>>>>>> +{\n> >>>>>>> +}\n> >>>>>>> +\n> >>>>>>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat\n> >>>>>>> pixelFormat)\n> >>>>>>> +{\n> >>>>>>> + sourceSize_ = sourceSize;\n> >>>>>>> + pixelFormat_ = pixelFormat;\n> >>>>>>> +\n> >>>>>>> + if (pixelFormat_ != formats::NV12) {\n> >>>>>>> + LOG (Thumbnailer, Error) << \"Failed to configure: Pixel\n> >>>>>>> Format \"\n> >>>>>>> + << pixelFormat_.toString() << \" unsupported.\";\n> >>>>>>> + return;\n> >>>>>>> + }\n> >>>>>>> +\n> >>>>>>> + validConfiguration_ = true;\n> >>>>>>> +}\n> >>>>>>> +\n> >>>>>>> +/*\n> >>>>>>> + * The Exif specification recommends the width of the thumbnail to\n> >>>>>>> be a\n> >>>>>>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding\n> >>>>>>> height\n> >>>>>>> + * keeping the aspect ratio same as of the source.\n> >>>>>>> + */\n> >>>>>>> +Size Thumbnailer::computeThumbnailSize()\n> >>>>>>> +{\n> >>>>>>> + unsigned int targetHeight;\n> >>>>>>> + unsigned int targetWidth = 160;\n> >>>>>>> +\n> >>>>>>> + targetHeight = targetWidth * sourceSize_.height /\n> >>>>>>> sourceSize_.width;\n> >>>>>>> +\n> >>>>>>> + if (targetHeight & 1)\n> >>>>>>> + targetHeight++;\n> >>>>>>> +\n> >>>>>>> + return Size(targetWidth, targetHeight);\n> >>>>>>> +}\n> >>>>>>> +\n> >>>>>>> +void\n> >>>>>>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n> >>>>>>> + std::vector &destination)\n> >>>>>>> +{\n> >>>>>>> + if (!validConfiguration_) {\n> >>>>>>> + LOG(Thumbnailer, Error) << \"config is unconfigured or\n> >>>>>>> invalid.\";\n> >>>>>>> + return;\n> >>>>>>> + }\n> >>>>>>> +\n> >>>>>>> + targetSize_ = computeThumbnailSize();\n> >>>>>>> +\n> >>>>>>> + const unsigned int sw = sourceSize_.width;\n> >>>>>>> + const unsigned int sh = sourceSize_.height;\n> >>>>>>> + const unsigned int tw = targetSize_.width;\n> >>>>>>> + const unsigned int th = targetSize_.height;\n> >>>>>>> +\n> >>>>>>> + /* Image scaling block implementing nearest-neighbour\n> >>>>>>> algorithm. */\n> >>>>>>> + unsigned char *src = static_cast(source.data());\n> >>>>>>> + unsigned char *src_c = src + sh * sw;\n> >>>>>>> + unsigned char *src_cb, *src_cr;\n> >>>>>>> +\n> >>>>>>> + size_t dstSize = (th * tw) + ((th/2) * tw);\n> >>>>>>> + destination.reserve(dstSize);\n> >>>>>>> + unsigned char *dst = destination.data();\n> >>>>>>> + unsigned char *dst_c = dst + th * tw;\n> >>>>>>> +\n> >>>>>>> + for (unsigned int y = 0; y < th; y+=2) {\n> >>>>>>> + unsigned int sourceY = (sh*y + th/2) / th;\n> >>>>>>> +\n> >>>>>>> + src_cb = src_c + (sourceY/2) * sw + 0;\n> >>>>>>> + src_cr = src_c + (sourceY/2) * sw + 1;\n> >>>>>>> +\n> >>>>>>> + for (unsigned int x = 0; x < tw; x+=2) {\n> >>>>>>> + unsigned int sourceX = (sw*x + tw/2) / tw;\n> >>>>>>> +\n> >>>>>>> + dst[y * tw + x] = src[sw * sourceY +\n> >>>>>>> sourceX];\n> >>>>>>> + dst[(y+1) * tw + x] = src[sw * (sourceY+1) +\n> >>>>>>> sourceX];\n> >>>>>>> + dst[y * tw + (x+1)] = src[sw * sourceY +\n> >>>>>>> (sourceX+1)];\n> >>>>>>> + dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) +\n> >>>>>>> (sourceX+1)];\n> >>>>>>> +\n> >>>>>>> + dst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n> >>>>>>> + dst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n> >>>>>>> + }\n> >>>>>>> + }\n> >>>>>>> +}\n> >>>>>>> diff --git a/src/android/jpeg/thumbnailer.h\n> >>>>>>> b/src/android/jpeg/thumbnailer.h\n> >>>>>>> new file mode 100644\n> >>>>>>> index 0000000..bab9855\n> >>>>>>> --- /dev/null\n> >>>>>>> +++ b/src/android/jpeg/thumbnailer.h\n> >>>>>>> @@ -0,0 +1,40 @@\n> >>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> >>>>>>> +/*\n> >>>>>>> + * Copyright (C) 2020, Google Inc.\n> >>>>>>> + *\n> >>>>>>> + * thumbnailer.h - Basic image thumbnailer from NV12\n> >>>>>>> + */\n> >>>>>>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n> >>>>>>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n> >>>>>>> +\n> >>>>>>> +#include \n> >>>>>>> +\n> >>>>>>> +#include \"libcamera/internal/buffer.h\"\n> >>>>>>> +#include \"libcamera/internal/formats.h\"\n> >>>>>>> +\n> >>>>>>> +class Thumbnailer\n> >>>>>>> +{\n> >>>>>>> +public:\n> >>>>>>> + Thumbnailer();\n> >>>>>>> +\n> >>>>>>> + void configure(const libcamera::Size &sourceSize,\n> >>>>>>> + libcamera::PixelFormat pixelFormat);\n> >>>>>>> +\n> >>>>>>> + /*\n> >>>>>>> + * \\todo: Discuss if we can return targetSize_ via configure() or\n> >>>>>>> + * scaleBuffer(). We need targetSize_ to re-encode the scaled\n> >>>>>>> buffer\n> >>>>>>> + * via encoder in PostProcssorJpeg::writeThumbnail().\n> >>>>>>> + */\n> >>>>>>> + libcamera::Size computeThumbnailSize();\n> >>>>>>> + void scaleBuffer(const libcamera::Span &source,\n> >>>>>>> + std::vector &dest);\n> >>>>>>> +\n> >>>>>>> +private:\n> >>>>>>> + libcamera::PixelFormat pixelFormat_;\n> >>>>>>> + libcamera::Size sourceSize_;\n> >>>>>>> + libcamera::Size targetSize_;\n> >>>>>>> +\n> >>>>>>> + bool validConfiguration_;\n> >>>>>>> +};\n> >>>>>>> +\n> >>>>>>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n> >>>>>>> diff --git a/src/android/meson.build b/src/android/meson.build\n> >>>>>>> index 5a01bea..3905e2f 100644\n> >>>>>>> --- a/src/android/meson.build\n> >>>>>>> +++ b/src/android/meson.build\n> >>>>>>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n> >>>>>>> 'jpeg/encoder_libjpeg.cpp',\n> >>>>>>> 'jpeg/exif.cpp',\n> >>>>>>> 'jpeg/post_processor_jpeg.cpp',\n> >>>>>>> + 'jpeg/thumbnailer.cpp',\n> >>>>>>> ])\n> >>>>>>> android_camera_metadata_sources = files([\n> >>>>>>>\n> >> --\n> >> Regards\n> >> --\n> >> Kieran\n> >> _______________________________________________\n> >> libcamera-devel mailing list\n> >> libcamera-devel@lists.libcamera.org\n> >> https://lists.libcamera.org/listinfo/libcamera-devel\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 13536BDB13\n\tfor ;\n\tThu, 22 Oct 2020 13:39:15 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 7B3FF61439;\n\tThu, 22 Oct 2020 15:39:14 +0200 (CEST)","from mail-ed1-x531.google.com (mail-ed1-x531.google.com\n\t[IPv6:2a00:1450:4864:20::531])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id DE5DB60352\n\tfor ;\n\tThu, 22 Oct 2020 15:39:12 +0200 (CEST)","by mail-ed1-x531.google.com with SMTP id x1so1812722eds.1\n\tfor ;\n\tThu, 22 Oct 2020 06:39:12 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=chromium.org header.i=@chromium.org\n\theader.b=\"LFUo1h3v\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org;\n\ts=google; \n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=te1wwf37E/F6uCF7McVkT5AyFiDJ7Xy/tcCrRzodV6E=;\n\tb=LFUo1h3vIeql8xz49p8yXHl3Qx2rQRy1+TdjZr/MQJXlianlhMcbeWdBWSBoXhIKGe\n\t/0ax3KfMTU4nRCDjOXQeBP01732EqQz0ZZLGcMU5UFhdvvdQvzG1KNHGs01bXPidmAGG\n\tHMsXjKHj88czoXCg6ZzewHTVpI9sT2LhzfCXg=","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=te1wwf37E/F6uCF7McVkT5AyFiDJ7Xy/tcCrRzodV6E=;\n\tb=mRyR+oT+gFdTBxu9dhUS6LDcBQAQDyNnwwnher1uWCYBivDPgYBGRKmdhBU5x/hTiL\n\tEU00C3qC77Ovm8PUgRhYiSV19TQZNvBExAY2ttst3pZ+qzOGG+xprOcOtgRnUNCtswis\n\t+E44fUxusjOkGypI8GsTR4JinMlyVbV7m+tfbB9VgsewvsAsZOa617j5kiViEObjDA09\n\t1ZhMRgKaIjshIwPSG7DtvdJGM3adQUuaxYPvEc0PtuG0jj6gVM4gJT12et2l/MQCitJu\n\tlanjWSySTbprYR1SkeVPaiUmFKbsqXPQBoZTU840xXCoVd1Pce0jjMO5olivALOHicra\n\tuuSA==","X-Gm-Message-State":"AOAM532o0QvaelL0O2cTzNMddsktsj+NYYng2hQqjK8rZjnlagoAW5yU\n\tu7p4mXLQG+X6gSv0XFHnYYQEhZZHd/pK8cgE/0N9oC+7T/EDZg==","X-Google-Smtp-Source":"ABdhPJx16goH6ZrD/mqT21lHGGct73WeJLMrix2nUuJfe02ByFvHZ2sZ1vOy61cj/FBBFGSvulYtjJ3bw3xgw/zZV2c=","X-Received":"by 2002:a05:6402:22ae:: with SMTP id\n\tcx14mr2308980edb.73.1603373952216; \n\tThu, 22 Oct 2020 06:39:12 -0700 (PDT)","MIME-Version":"1.0","References":"<20201021080806.46636-1-email@uajain.com>\n\t<20201021080806.46636-3-email@uajain.com>\n\t\n\t\n\t<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>\n\t<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>\n\t<9ca3b492-8468-e9cf-b103-b4350be472b9@ideasonboard.com>\n\t\n\t","In-Reply-To":"","From":"Hirokazu Honda ","Date":"Thu, 22 Oct 2020 22:39:00 +0900","Message-ID":"","To":"Umang Jain ","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13417,"web_url":"https://patchwork.libcamera.org/comment/13417/","msgid":"<20201022135914.GA3938@pendragon.ideasonboard.com>","date":"2020-10-22T13:59:14","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Hiro-san,\n\nOn Thu, Oct 22, 2020 at 08:42:01PM +0900, Hirokazu Honda wrote:\n> Hi Umang,\n> Thanks for the work. I couldn't have time to review these today and\n> will review tomorrow.\n> In my understanding, the thumbnail class is basically to scale frames\n> to the given destination.\n> Can we generalize it as PostProcessor interface so that we will be\n> able to make use of it for down-scaling regardless of whether jpeg\n> encoding is required.\n> It will be necessary when Android HAL adaptation layer has to output\n> multiple different NV12 streams while the native camera can produce a\n> single NV12 stream simultaneously.\n\nOne point to consider is that the thumbnailer is a down-scaler using a\nnearest-neighbour algorithm. It's good enough for thumbnails, but will\nlead to awful quality when scaling streams. For that, I think we need a\nbetter scaler implementation, and we could then replace the thumbnailer\nclass with it.\n\n> On Wed, Oct 21, 2020 at 8:09 PM Kieran Bingham wrote:\n> > On 21/10/2020 12:03, Umang Jain wrote:\n> > > On 10/21/20 4:25 PM, Kieran Bingham wrote:\n> > >> On 21/10/2020 11:51, Umang Jain wrote:\n> > >>> On 10/21/20 2:49 PM, Kieran Bingham wrote:\n> > >>>> On 21/10/2020 09:08, Umang Jain wrote:\n> > >>>>> Add a basic image thumbnailer for NV12 frames being captured.\n> > >>>>> It shall generate a thumbnail image to be embedded as a part of\n> > >>>>> EXIF metadata of the frame. The output of the thumbnail will still\n> > >>>>> be NV12.\n> > >>>>>\n> > >>>>> Signed-off-by: Umang Jain \n> > >>>>> ---\n> > >>>>> src/android/jpeg/post_processor_jpeg.cpp | 39 +++++++++\n> > >>>>> src/android/jpeg/post_processor_jpeg.h | 3 +\n> > >>>>> src/android/jpeg/thumbnailer.cpp | 100\n> > >>>>> +++++++++++++++++++++++\n> > >>>>> src/android/jpeg/thumbnailer.h | 40 +++++++++\n> > >>>>> src/android/meson.build | 1 +\n> > >>>>> 5 files changed, 183 insertions(+)\n> > >>>>> create mode 100644 src/android/jpeg/thumbnailer.cpp\n> > >>>>> create mode 100644 src/android/jpeg/thumbnailer.h\n> > >>>>>\n> > >>>>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp\n> > >>>>> b/src/android/jpeg/post_processor_jpeg.cpp\n> > >>>>> index 9d452b7..f5f1f78 100644\n> > >>>>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n> > >>>>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n> > >>>>> @@ -11,6 +11,7 @@\n> > >>>>> #include \"../camera_metadata.h\"\n> > >>>>> #include \"encoder_libjpeg.h\"\n> > >>>>> #include \"exif.h\"\n> > >>>>> +#include \"thumbnailer.h\"\n> > >>>>> #include \n> > >>>>> @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const\n> > >>>>> StreamConfiguration &inCfg,\n> > >>>>> return encoder_->configure(inCfg);\n> > >>>>> }\n> > >>>>> +void PostProcessorJpeg::generateThumbnail(const\n> > >>>>> libcamera::Span &source,\n> > >>>>> + std::vector &thumbnail)\n> > >>>>> +{\n> > >>>>> + libcamera::Span destination;\n> > >>>>> + Thumbnailer thumbnailer;\n> > >>>>> +\n> > >>>>> + thumbnailer.configure(streamSize_, formats::NV12);\n> > >>>>> + libcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n> > >>>>> + thumbnailer.scaleBuffer(source, thumbnail);\n> > >>>>> +\n> > >>>>> + if (thumbnail.data()) {\n> > >>>>> + StreamConfiguration thumbnailCfg;\n> > >>>>> +\n> > >>>>> + std::unique_ptr encoder =\n> > >>>>> + std::make_unique();\n> > >>>>> +\n> > >>>>> + thumbnailCfg.pixelFormat = formats::NV12;\n> > >>>>> + thumbnailCfg.size = targetSize;\n> > >>>>> + encoder->configure(thumbnailCfg);\n> > >>>> thumbnail.capacity() might be quite low here.\n> > >>>> We need to make sure the vector is big enough at this point, you might\n> > >>>> need to do something like:\n> > >>>>\n> > >>>> thumbnail.resize(targetSize.width * targetSize.height * 4);\n> > >>> I am not sure I follow. This is compressing the thumbnail part right? So\n> > >>> thumbnail is the \"source\" for the encoder here (Please refer to\n> > >>> ->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n> > >> Ugh, sorry - tired eyes indeed. Yes, please ignore those comments.\n> > >>\n> > >>\n> > >>\n> > >>>> Really we should obtain that size from the encoder. I thought we had a\n> > >>>> helper to do that already, but it seems we don't.\n> > >>>> We could/should add Encoder::maxOutput(); which would ask the encoder\n> > >>>> \"For your given configuration, what is the maximum number of bytes you\n> > >>>> might output\".\n> > >>>>\n> > >>>> Then of course we'd do:\n> > >>>> thumbnail.resize(encoder.maxOutput());\n> > >>>>\n> > >>>>\n> > >>>>> + int jpeg_size = encoder->encode({ thumbnail.data(),\n> > >>>>> thumbnail.capacity() },\n> > >>>>> + destination, { });\n> > >>> As said above, thumbnail is the source here. The compressed thumbnail\n> > >>> output is carried in destination. And, destination is a local span<>\n> > >>> here. Does it makes sense?\n> > >> Yes, I had mixed up source and destination I'm sorry.\n> > >>\n> > >> So who/where is allocating the destination?\n> > > Ah, that's a good question and I think I just found a bug. I currently\n> > > pass on a locally allocated Span<> 'destination' to the encoder. But\n> > > there is no explicit allocation here. I think I need to point / create\n> > > the Span with the output size / bytes and then pass to the encode().\n> >\n> > :-) Ok - then please apply all of my earlier comments to that part\n> > instead. ;-)\n> >\n> > We certainly need somewhere to store the compressed image, to be able to\n> > pass it into the exif :-D\n> >\n> > --\n> > Kieran\n> >\n> >\n> > >>>>> + LOG(JPEG, Info) << \"Thumbnail compress returned \"\n> > >>>>> + << jpeg_size << \" bytes\";\n> > >>>> And I presume we could then do an:\n> > >>>> thumbnail.resize(jpeg_size);\n> > >>>>\n> > >>>> here to update the thumbnail with the correct size. I'd be weary of the\n> > >>>> resize operations doing lots of re-allocations though, so perhaps we\n> > >>>> want to minimize that. But lets get to something that works first\n> > >>>> before\n> > >>>> worrying about optimising.\n> > >>>>\n> > >>>>\n> > >>>>> + }\n> > >>>>> +}\n> > >>>>> +\n> > >>>>> int PostProcessorJpeg::process(const libcamera::FrameBuffer\n> > >>>>> *source,\n> > >>>>> const libcamera::Span &destination,\n> > >>>>> CameraMetadata *metadata)\n> > >>>>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const\n> > >>>>> libcamera::FrameBuffer *source,\n> > >>>>> return jpeg_size;\n> > >>>>> }\n> > >>>>> + std::vector thumbnail;\n> > >>>> You need to resize this somewhere.\n> > >>>> Edit: Now seen a better place above ;-)\n> > >>> The thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\n> > >>> We just pass in to the thumbnailer, while keeping its ownership in\n> > >>> PostProcessorJpeg.\n> > >>>>> + generateThumbnail(destination, thumbnail);\n> > >>>>> + /*\n> > >>>>> + * \\todo: Write the compressed thumbnail to a file for\n> > >>>>> inspection.\n> > >>>>> + * (I) Check if we can still write the thumbnail to EXIF here.\n> > >>>>> + * If not, we might need to move the thumbnailer logic to\n> > >>>>> encoder.\n> > >>>>> + * And if we do that, first we need to make sure we get can\n> > >>>>> + * compressed data written to destination first before\n> > >>>>> calling\n> > >>>>> + * jpeg_finish_compress operation somehow. Thumbnailing will\n> > >>>>> + * only occur if we have compressed data available first.\n> > >>>>> + */\n> > >>>>> +\n> > >>>>> /*\n> > >>>>> * Fill in the JPEG blob header.\n> > >>>>> *\n> > >>>>> diff --git a/src/android/jpeg/post_processor_jpeg.h\n> > >>>>> b/src/android/jpeg/post_processor_jpeg.h\n> > >>>>> index 62c8650..05601ee 100644\n> > >>>>> --- a/src/android/jpeg/post_processor_jpeg.h\n> > >>>>> +++ b/src/android/jpeg/post_processor_jpeg.h\n> > >>>>> @@ -28,6 +28,9 @@ public:\n> > >>>>> CameraMetadata *metadata) override;\n> > >>>>> private:\n> > >>>>> + void generateThumbnail(const libcamera::Span &source,\n> > >>>>> + std::vector &thumbnail);\n> > >>>>> +\n> > >>>>> CameraDevice *cameraDevice_;\n> > >>>>> std::unique_ptr encoder_;\n> > >>>>> libcamera::Size streamSize_;\n> > >>>>> diff --git a/src/android/jpeg/thumbnailer.cpp\n> > >>>>> b/src/android/jpeg/thumbnailer.cpp\n> > >>>>> new file mode 100644\n> > >>>>> index 0000000..3163576\n> > >>>>> --- /dev/null\n> > >>>>> +++ b/src/android/jpeg/thumbnailer.cpp\n> > >>>>> @@ -0,0 +1,100 @@\n> > >>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > >>>>> +/*\n> > >>>>> + * Copyright (C) 2020, Google Inc.\n> > >>>>> + *\n> > >>>>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n> > >>>>> + */\n> > >>>>> +\n> > >>>>> +#include \"thumbnailer.h\"\n> > >>>>> +\n> > >>>>> +#include \n> > >>>>> +\n> > >>>>> +#include \"libcamera/internal/file.h\"\n> > >>>>> +#include \"libcamera/internal/log.h\"\n> > >>>>> +\n> > >>>>> +using namespace libcamera;\n> > >>>>> +\n> > >>>>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n> > >>>>> +\n> > >>>>> +Thumbnailer::Thumbnailer()\n> > >>>>> + : validConfiguration_(false)\n> > >>>>> +{\n> > >>>>> +}\n> > >>>>> +\n> > >>>>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat\n> > >>>>> pixelFormat)\n> > >>>>> +{\n> > >>>>> + sourceSize_ = sourceSize;\n> > >>>>> + pixelFormat_ = pixelFormat;\n> > >>>>> +\n> > >>>>> + if (pixelFormat_ != formats::NV12) {\n> > >>>>> + LOG (Thumbnailer, Error) << \"Failed to configure: Pixel\n> > >>>>> Format \"\n> > >>>>> + << pixelFormat_.toString() << \" unsupported.\";\n> > >>>>> + return;\n> > >>>>> + }\n> > >>>>> +\n> > >>>>> + validConfiguration_ = true;\n> > >>>>> +}\n> > >>>>> +\n> > >>>>> +/*\n> > >>>>> + * The Exif specification recommends the width of the thumbnail to\n> > >>>>> be a\n> > >>>>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding\n> > >>>>> height\n> > >>>>> + * keeping the aspect ratio same as of the source.\n> > >>>>> + */\n> > >>>>> +Size Thumbnailer::computeThumbnailSize()\n> > >>>>> +{\n> > >>>>> + unsigned int targetHeight;\n> > >>>>> + unsigned int targetWidth = 160;\n> > >>>>> +\n> > >>>>> + targetHeight = targetWidth * sourceSize_.height /\n> > >>>>> sourceSize_.width;\n> > >>>>> +\n> > >>>>> + if (targetHeight & 1)\n> > >>>>> + targetHeight++;\n> > >>>>> +\n> > >>>>> + return Size(targetWidth, targetHeight);\n> > >>>>> +}\n> > >>>>> +\n> > >>>>> +void\n> > >>>>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n> > >>>>> + std::vector &destination)\n> > >>>>> +{\n> > >>>>> + if (!validConfiguration_) {\n> > >>>>> + LOG(Thumbnailer, Error) << \"config is unconfigured or\n> > >>>>> invalid.\";\n> > >>>>> + return;\n> > >>>>> + }\n> > >>>>> +\n> > >>>>> + targetSize_ = computeThumbnailSize();\n> > >>>>> +\n> > >>>>> + const unsigned int sw = sourceSize_.width;\n> > >>>>> + const unsigned int sh = sourceSize_.height;\n> > >>>>> + const unsigned int tw = targetSize_.width;\n> > >>>>> + const unsigned int th = targetSize_.height;\n> > >>>>> +\n> > >>>>> + /* Image scaling block implementing nearest-neighbour\n> > >>>>> algorithm. */\n> > >>>>> + unsigned char *src = static_cast(source.data());\n> > >>>>> + unsigned char *src_c = src + sh * sw;\n> > >>>>> + unsigned char *src_cb, *src_cr;\n> > >>>>> +\n> > >>>>> + size_t dstSize = (th * tw) + ((th/2) * tw);\n> > >>>>> + destination.reserve(dstSize);\n> > >>>>> + unsigned char *dst = destination.data();\n> > >>>>> + unsigned char *dst_c = dst + th * tw;\n> > >>>>> +\n> > >>>>> + for (unsigned int y = 0; y < th; y+=2) {\n> > >>>>> + unsigned int sourceY = (sh*y + th/2) / th;\n> > >>>>> +\n> > >>>>> + src_cb = src_c + (sourceY/2) * sw + 0;\n> > >>>>> + src_cr = src_c + (sourceY/2) * sw + 1;\n> > >>>>> +\n> > >>>>> + for (unsigned int x = 0; x < tw; x+=2) {\n> > >>>>> + unsigned int sourceX = (sw*x + tw/2) / tw;\n> > >>>>> +\n> > >>>>> + dst[y * tw + x] = src[sw * sourceY +\n> > >>>>> sourceX];\n> > >>>>> + dst[(y+1) * tw + x] = src[sw * (sourceY+1) +\n> > >>>>> sourceX];\n> > >>>>> + dst[y * tw + (x+1)] = src[sw * sourceY +\n> > >>>>> (sourceX+1)];\n> > >>>>> + dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) +\n> > >>>>> (sourceX+1)];\n> > >>>>> +\n> > >>>>> + dst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n> > >>>>> + dst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n> > >>>>> + }\n> > >>>>> + }\n> > >>>>> +}\n> > >>>>> diff --git a/src/android/jpeg/thumbnailer.h\n> > >>>>> b/src/android/jpeg/thumbnailer.h\n> > >>>>> new file mode 100644\n> > >>>>> index 0000000..bab9855\n> > >>>>> --- /dev/null\n> > >>>>> +++ b/src/android/jpeg/thumbnailer.h\n> > >>>>> @@ -0,0 +1,40 @@\n> > >>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > >>>>> +/*\n> > >>>>> + * Copyright (C) 2020, Google Inc.\n> > >>>>> + *\n> > >>>>> + * thumbnailer.h - Basic image thumbnailer from NV12\n> > >>>>> + */\n> > >>>>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n> > >>>>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n> > >>>>> +\n> > >>>>> +#include \n> > >>>>> +\n> > >>>>> +#include \"libcamera/internal/buffer.h\"\n> > >>>>> +#include \"libcamera/internal/formats.h\"\n> > >>>>> +\n> > >>>>> +class Thumbnailer\n> > >>>>> +{\n> > >>>>> +public:\n> > >>>>> + Thumbnailer();\n> > >>>>> +\n> > >>>>> + void configure(const libcamera::Size &sourceSize,\n> > >>>>> + libcamera::PixelFormat pixelFormat);\n> > >>>>> +\n> > >>>>> + /*\n> > >>>>> + * \\todo: Discuss if we can return targetSize_ via configure() or\n> > >>>>> + * scaleBuffer(). We need targetSize_ to re-encode the scaled\n> > >>>>> buffer\n> > >>>>> + * via encoder in PostProcssorJpeg::writeThumbnail().\n> > >>>>> + */\n> > >>>>> + libcamera::Size computeThumbnailSize();\n> > >>>>> + void scaleBuffer(const libcamera::Span &source,\n> > >>>>> + std::vector &dest);\n> > >>>>> +\n> > >>>>> +private:\n> > >>>>> + libcamera::PixelFormat pixelFormat_;\n> > >>>>> + libcamera::Size sourceSize_;\n> > >>>>> + libcamera::Size targetSize_;\n> > >>>>> +\n> > >>>>> + bool validConfiguration_;\n> > >>>>> +};\n> > >>>>> +\n> > >>>>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n> > >>>>> diff --git a/src/android/meson.build b/src/android/meson.build\n> > >>>>> index 5a01bea..3905e2f 100644\n> > >>>>> --- a/src/android/meson.build\n> > >>>>> +++ b/src/android/meson.build\n> > >>>>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n> > >>>>> 'jpeg/encoder_libjpeg.cpp',\n> > >>>>> 'jpeg/exif.cpp',\n> > >>>>> 'jpeg/post_processor_jpeg.cpp',\n> > >>>>> + 'jpeg/thumbnailer.cpp',\n> > >>>>> ])\n> > >>>>> android_camera_metadata_sources = files([\n> > >>>>>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 9022BBDB13\n\tfor ;\n\tThu, 22 Oct 2020 14:00:02 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 0611C613E6;\n\tThu, 22 Oct 2020 16:00:02 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 7F31460352\n\tfor ;\n\tThu, 22 Oct 2020 16:00:00 +0200 (CEST)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id DF70553;\n\tThu, 22 Oct 2020 15:59:59 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"iNTC8y9S\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1603375200;\n\tbh=M7V2QnHAqSK3m9GL858cG7lB+Id2Aw1X7iEyJE4BK04=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=iNTC8y9SR9MFxANQxsaAcUNu4oofTe52BJBwBXI69hbGPY7cVBOSY66ZQkWhRN1Qa\n\tscWuWDsRtFHVCPHiiVXxcooCu/ufMXSiuP98m+Gn1pme/4o6xo6lRowpV9lDRt/xk3\n\tvKrfJ7tkPrScutOpbScdv5zPkpv20/wB+V0vbqxM=","Date":"Thu, 22 Oct 2020 16:59:14 +0300","From":"Laurent Pinchart ","To":"Hirokazu Honda ","Message-ID":"<20201022135914.GA3938@pendragon.ideasonboard.com>","References":"<20201021080806.46636-1-email@uajain.com>\n\t<20201021080806.46636-3-email@uajain.com>\n\t\n\t\n\t<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>\n\t<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>\n\t<9ca3b492-8468-e9cf-b103-b4350be472b9@ideasonboard.com>\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13421,"web_url":"https://patchwork.libcamera.org/comment/13421/","msgid":"","date":"2020-10-23T00:37:09","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":63,"url":"https://patchwork.libcamera.org/api/people/63/","name":"Hirokazu Honda","email":"hiroh@chromium.org"},"content":"Hi Laurent,\n\nOn Thu, Oct 22, 2020 at 11:00 PM Laurent Pinchart\n wrote:\n>\n> Hi Hiro-san,\n>\n> On Thu, Oct 22, 2020 at 08:42:01PM +0900, Hirokazu Honda wrote:\n> > Hi Umang,\n> > Thanks for the work. I couldn't have time to review these today and\n> > will review tomorrow.\n> > In my understanding, the thumbnail class is basically to scale frames\n> > to the given destination.\n> > Can we generalize it as PostProcessor interface so that we will be\n> > able to make use of it for down-scaling regardless of whether jpeg\n> > encoding is required.\n> > It will be necessary when Android HAL adaptation layer has to output\n> > multiple different NV12 streams while the native camera can produce a\n> > single NV12 stream simultaneously.\n>\n> One point to consider is that the thumbnailer is a down-scaler using a\n> nearest-neighbour algorithm. It's good enough for thumbnails, but will\n> lead to awful quality when scaling streams. For that, I think we need a\n> better scaler implementation, and we could then replace the thumbnailer\n> class with it.\n>\n\nWe can select nearest-neighbor and bilinear algorithms in\nlibyuv::NV12Scale with the filtering mode argument[1].\nWe may want to specify the scale algorithm on a class construction.\nI would use libyuv here for better performance and less buggy (+ less\nlines to be reviewed).\nWhat do you'all think?\n\n[1] https://chromium.googlesource.com/libyuv/libyuv/+/refs/heads/master/include/libyuv/scale.h#148\n\nBest Regards,\n-Hiro\n\n> > On Wed, Oct 21, 2020 at 8:09 PM Kieran Bingham wrote:\n> > > On 21/10/2020 12:03, Umang Jain wrote:\n> > > > On 10/21/20 4:25 PM, Kieran Bingham wrote:\n> > > >> On 21/10/2020 11:51, Umang Jain wrote:\n> > > >>> On 10/21/20 2:49 PM, Kieran Bingham wrote:\n> > > >>>> On 21/10/2020 09:08, Umang Jain wrote:\n> > > >>>>> Add a basic image thumbnailer for NV12 frames being captured.\n> > > >>>>> It shall generate a thumbnail image to be embedded as a part of\n> > > >>>>> EXIF metadata of the frame. The output of the thumbnail will still\n> > > >>>>> be NV12.\n> > > >>>>>\n> > > >>>>> Signed-off-by: Umang Jain \n> > > >>>>> ---\n> > > >>>>> src/android/jpeg/post_processor_jpeg.cpp | 39 +++++++++\n> > > >>>>> src/android/jpeg/post_processor_jpeg.h | 3 +\n> > > >>>>> src/android/jpeg/thumbnailer.cpp | 100\n> > > >>>>> +++++++++++++++++++++++\n> > > >>>>> src/android/jpeg/thumbnailer.h | 40 +++++++++\n> > > >>>>> src/android/meson.build | 1 +\n> > > >>>>> 5 files changed, 183 insertions(+)\n> > > >>>>> create mode 100644 src/android/jpeg/thumbnailer.cpp\n> > > >>>>> create mode 100644 src/android/jpeg/thumbnailer.h\n> > > >>>>>\n> > > >>>>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp\n> > > >>>>> b/src/android/jpeg/post_processor_jpeg.cpp\n> > > >>>>> index 9d452b7..f5f1f78 100644\n> > > >>>>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n> > > >>>>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n> > > >>>>> @@ -11,6 +11,7 @@\n> > > >>>>> #include \"../camera_metadata.h\"\n> > > >>>>> #include \"encoder_libjpeg.h\"\n> > > >>>>> #include \"exif.h\"\n> > > >>>>> +#include \"thumbnailer.h\"\n> > > >>>>> #include \n> > > >>>>> @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const\n> > > >>>>> StreamConfiguration &inCfg,\n> > > >>>>> return encoder_->configure(inCfg);\n> > > >>>>> }\n> > > >>>>> +void PostProcessorJpeg::generateThumbnail(const\n> > > >>>>> libcamera::Span &source,\n> > > >>>>> + std::vector &thumbnail)\n> > > >>>>> +{\n> > > >>>>> + libcamera::Span destination;\n> > > >>>>> + Thumbnailer thumbnailer;\n> > > >>>>> +\n> > > >>>>> + thumbnailer.configure(streamSize_, formats::NV12);\n> > > >>>>> + libcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n> > > >>>>> + thumbnailer.scaleBuffer(source, thumbnail);\n> > > >>>>> +\n> > > >>>>> + if (thumbnail.data()) {\n> > > >>>>> + StreamConfiguration thumbnailCfg;\n> > > >>>>> +\n> > > >>>>> + std::unique_ptr encoder =\n> > > >>>>> + std::make_unique();\n> > > >>>>> +\n> > > >>>>> + thumbnailCfg.pixelFormat = formats::NV12;\n> > > >>>>> + thumbnailCfg.size = targetSize;\n> > > >>>>> + encoder->configure(thumbnailCfg);\n> > > >>>> thumbnail.capacity() might be quite low here.\n> > > >>>> We need to make sure the vector is big enough at this point, you might\n> > > >>>> need to do something like:\n> > > >>>>\n> > > >>>> thumbnail.resize(targetSize.width * targetSize.height * 4);\n> > > >>> I am not sure I follow. This is compressing the thumbnail part right? So\n> > > >>> thumbnail is the \"source\" for the encoder here (Please refer to\n> > > >>> ->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n> > > >> Ugh, sorry - tired eyes indeed. Yes, please ignore those comments.\n> > > >>\n> > > >>\n> > > >>\n> > > >>>> Really we should obtain that size from the encoder. I thought we had a\n> > > >>>> helper to do that already, but it seems we don't.\n> > > >>>> We could/should add Encoder::maxOutput(); which would ask the encoder\n> > > >>>> \"For your given configuration, what is the maximum number of bytes you\n> > > >>>> might output\".\n> > > >>>>\n> > > >>>> Then of course we'd do:\n> > > >>>> thumbnail.resize(encoder.maxOutput());\n> > > >>>>\n> > > >>>>\n> > > >>>>> + int jpeg_size = encoder->encode({ thumbnail.data(),\n> > > >>>>> thumbnail.capacity() },\n> > > >>>>> + destination, { });\n> > > >>> As said above, thumbnail is the source here. The compressed thumbnail\n> > > >>> output is carried in destination. And, destination is a local span<>\n> > > >>> here. Does it makes sense?\n> > > >> Yes, I had mixed up source and destination I'm sorry.\n> > > >>\n> > > >> So who/where is allocating the destination?\n> > > > Ah, that's a good question and I think I just found a bug. I currently\n> > > > pass on a locally allocated Span<> 'destination' to the encoder. But\n> > > > there is no explicit allocation here. I think I need to point / create\n> > > > the Span with the output size / bytes and then pass to the encode().\n> > >\n> > > :-) Ok - then please apply all of my earlier comments to that part\n> > > instead. ;-)\n> > >\n> > > We certainly need somewhere to store the compressed image, to be able to\n> > > pass it into the exif :-D\n> > >\n> > > --\n> > > Kieran\n> > >\n> > >\n> > > >>>>> + LOG(JPEG, Info) << \"Thumbnail compress returned \"\n> > > >>>>> + << jpeg_size << \" bytes\";\n> > > >>>> And I presume we could then do an:\n> > > >>>> thumbnail.resize(jpeg_size);\n> > > >>>>\n> > > >>>> here to update the thumbnail with the correct size. I'd be weary of the\n> > > >>>> resize operations doing lots of re-allocations though, so perhaps we\n> > > >>>> want to minimize that. But lets get to something that works first\n> > > >>>> before\n> > > >>>> worrying about optimising.\n> > > >>>>\n> > > >>>>\n> > > >>>>> + }\n> > > >>>>> +}\n> > > >>>>> +\n> > > >>>>> int PostProcessorJpeg::process(const libcamera::FrameBuffer\n> > > >>>>> *source,\n> > > >>>>> const libcamera::Span &destination,\n> > > >>>>> CameraMetadata *metadata)\n> > > >>>>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const\n> > > >>>>> libcamera::FrameBuffer *source,\n> > > >>>>> return jpeg_size;\n> > > >>>>> }\n> > > >>>>> + std::vector thumbnail;\n> > > >>>> You need to resize this somewhere.\n> > > >>>> Edit: Now seen a better place above ;-)\n> > > >>> The thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\n> > > >>> We just pass in to the thumbnailer, while keeping its ownership in\n> > > >>> PostProcessorJpeg.\n> > > >>>>> + generateThumbnail(destination, thumbnail);\n> > > >>>>> + /*\n> > > >>>>> + * \\todo: Write the compressed thumbnail to a file for\n> > > >>>>> inspection.\n> > > >>>>> + * (I) Check if we can still write the thumbnail to EXIF here.\n> > > >>>>> + * If not, we might need to move the thumbnailer logic to\n> > > >>>>> encoder.\n> > > >>>>> + * And if we do that, first we need to make sure we get can\n> > > >>>>> + * compressed data written to destination first before\n> > > >>>>> calling\n> > > >>>>> + * jpeg_finish_compress operation somehow. Thumbnailing will\n> > > >>>>> + * only occur if we have compressed data available first.\n> > > >>>>> + */\n> > > >>>>> +\n> > > >>>>> /*\n> > > >>>>> * Fill in the JPEG blob header.\n> > > >>>>> *\n> > > >>>>> diff --git a/src/android/jpeg/post_processor_jpeg.h\n> > > >>>>> b/src/android/jpeg/post_processor_jpeg.h\n> > > >>>>> index 62c8650..05601ee 100644\n> > > >>>>> --- a/src/android/jpeg/post_processor_jpeg.h\n> > > >>>>> +++ b/src/android/jpeg/post_processor_jpeg.h\n> > > >>>>> @@ -28,6 +28,9 @@ public:\n> > > >>>>> CameraMetadata *metadata) override;\n> > > >>>>> private:\n> > > >>>>> + void generateThumbnail(const libcamera::Span &source,\n> > > >>>>> + std::vector &thumbnail);\n> > > >>>>> +\n> > > >>>>> CameraDevice *cameraDevice_;\n> > > >>>>> std::unique_ptr encoder_;\n> > > >>>>> libcamera::Size streamSize_;\n> > > >>>>> diff --git a/src/android/jpeg/thumbnailer.cpp\n> > > >>>>> b/src/android/jpeg/thumbnailer.cpp\n> > > >>>>> new file mode 100644\n> > > >>>>> index 0000000..3163576\n> > > >>>>> --- /dev/null\n> > > >>>>> +++ b/src/android/jpeg/thumbnailer.cpp\n> > > >>>>> @@ -0,0 +1,100 @@\n> > > >>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > >>>>> +/*\n> > > >>>>> + * Copyright (C) 2020, Google Inc.\n> > > >>>>> + *\n> > > >>>>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n> > > >>>>> + */\n> > > >>>>> +\n> > > >>>>> +#include \"thumbnailer.h\"\n> > > >>>>> +\n> > > >>>>> +#include \n> > > >>>>> +\n> > > >>>>> +#include \"libcamera/internal/file.h\"\n> > > >>>>> +#include \"libcamera/internal/log.h\"\n> > > >>>>> +\n> > > >>>>> +using namespace libcamera;\n> > > >>>>> +\n> > > >>>>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n> > > >>>>> +\n> > > >>>>> +Thumbnailer::Thumbnailer()\n> > > >>>>> + : validConfiguration_(false)\n> > > >>>>> +{\n> > > >>>>> +}\n> > > >>>>> +\n> > > >>>>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat\n> > > >>>>> pixelFormat)\n> > > >>>>> +{\n> > > >>>>> + sourceSize_ = sourceSize;\n> > > >>>>> + pixelFormat_ = pixelFormat;\n> > > >>>>> +\n> > > >>>>> + if (pixelFormat_ != formats::NV12) {\n> > > >>>>> + LOG (Thumbnailer, Error) << \"Failed to configure: Pixel\n> > > >>>>> Format \"\n> > > >>>>> + << pixelFormat_.toString() << \" unsupported.\";\n> > > >>>>> + return;\n> > > >>>>> + }\n> > > >>>>> +\n> > > >>>>> + validConfiguration_ = true;\n> > > >>>>> +}\n> > > >>>>> +\n> > > >>>>> +/*\n> > > >>>>> + * The Exif specification recommends the width of the thumbnail to\n> > > >>>>> be a\n> > > >>>>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding\n> > > >>>>> height\n> > > >>>>> + * keeping the aspect ratio same as of the source.\n> > > >>>>> + */\n> > > >>>>> +Size Thumbnailer::computeThumbnailSize()\n> > > >>>>> +{\n> > > >>>>> + unsigned int targetHeight;\n> > > >>>>> + unsigned int targetWidth = 160;\n> > > >>>>> +\n> > > >>>>> + targetHeight = targetWidth * sourceSize_.height /\n> > > >>>>> sourceSize_.width;\n> > > >>>>> +\n> > > >>>>> + if (targetHeight & 1)\n> > > >>>>> + targetHeight++;\n> > > >>>>> +\n> > > >>>>> + return Size(targetWidth, targetHeight);\n> > > >>>>> +}\n> > > >>>>> +\n> > > >>>>> +void\n> > > >>>>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n> > > >>>>> + std::vector &destination)\n> > > >>>>> +{\n> > > >>>>> + if (!validConfiguration_) {\n> > > >>>>> + LOG(Thumbnailer, Error) << \"config is unconfigured or\n> > > >>>>> invalid.\";\n> > > >>>>> + return;\n> > > >>>>> + }\n> > > >>>>> +\n> > > >>>>> + targetSize_ = computeThumbnailSize();\n> > > >>>>> +\n> > > >>>>> + const unsigned int sw = sourceSize_.width;\n> > > >>>>> + const unsigned int sh = sourceSize_.height;\n> > > >>>>> + const unsigned int tw = targetSize_.width;\n> > > >>>>> + const unsigned int th = targetSize_.height;\n> > > >>>>> +\n> > > >>>>> + /* Image scaling block implementing nearest-neighbour\n> > > >>>>> algorithm. */\n> > > >>>>> + unsigned char *src = static_cast(source.data());\n> > > >>>>> + unsigned char *src_c = src + sh * sw;\n> > > >>>>> + unsigned char *src_cb, *src_cr;\n> > > >>>>> +\n> > > >>>>> + size_t dstSize = (th * tw) + ((th/2) * tw);\n> > > >>>>> + destination.reserve(dstSize);\n> > > >>>>> + unsigned char *dst = destination.data();\n> > > >>>>> + unsigned char *dst_c = dst + th * tw;\n> > > >>>>> +\n> > > >>>>> + for (unsigned int y = 0; y < th; y+=2) {\n> > > >>>>> + unsigned int sourceY = (sh*y + th/2) / th;\n> > > >>>>> +\n> > > >>>>> + src_cb = src_c + (sourceY/2) * sw + 0;\n> > > >>>>> + src_cr = src_c + (sourceY/2) * sw + 1;\n> > > >>>>> +\n> > > >>>>> + for (unsigned int x = 0; x < tw; x+=2) {\n> > > >>>>> + unsigned int sourceX = (sw*x + tw/2) / tw;\n> > > >>>>> +\n> > > >>>>> + dst[y * tw + x] = src[sw * sourceY +\n> > > >>>>> sourceX];\n> > > >>>>> + dst[(y+1) * tw + x] = src[sw * (sourceY+1) +\n> > > >>>>> sourceX];\n> > > >>>>> + dst[y * tw + (x+1)] = src[sw * sourceY +\n> > > >>>>> (sourceX+1)];\n> > > >>>>> + dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) +\n> > > >>>>> (sourceX+1)];\n> > > >>>>> +\n> > > >>>>> + dst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n> > > >>>>> + dst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n> > > >>>>> + }\n> > > >>>>> + }\n> > > >>>>> +}\n> > > >>>>> diff --git a/src/android/jpeg/thumbnailer.h\n> > > >>>>> b/src/android/jpeg/thumbnailer.h\n> > > >>>>> new file mode 100644\n> > > >>>>> index 0000000..bab9855\n> > > >>>>> --- /dev/null\n> > > >>>>> +++ b/src/android/jpeg/thumbnailer.h\n> > > >>>>> @@ -0,0 +1,40 @@\n> > > >>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > >>>>> +/*\n> > > >>>>> + * Copyright (C) 2020, Google Inc.\n> > > >>>>> + *\n> > > >>>>> + * thumbnailer.h - Basic image thumbnailer from NV12\n> > > >>>>> + */\n> > > >>>>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n> > > >>>>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n> > > >>>>> +\n> > > >>>>> +#include \n> > > >>>>> +\n> > > >>>>> +#include \"libcamera/internal/buffer.h\"\n> > > >>>>> +#include \"libcamera/internal/formats.h\"\n> > > >>>>> +\n> > > >>>>> +class Thumbnailer\n> > > >>>>> +{\n> > > >>>>> +public:\n> > > >>>>> + Thumbnailer();\n> > > >>>>> +\n> > > >>>>> + void configure(const libcamera::Size &sourceSize,\n> > > >>>>> + libcamera::PixelFormat pixelFormat);\n> > > >>>>> +\n> > > >>>>> + /*\n> > > >>>>> + * \\todo: Discuss if we can return targetSize_ via configure() or\n> > > >>>>> + * scaleBuffer(). We need targetSize_ to re-encode the scaled\n> > > >>>>> buffer\n> > > >>>>> + * via encoder in PostProcssorJpeg::writeThumbnail().\n> > > >>>>> + */\n> > > >>>>> + libcamera::Size computeThumbnailSize();\n> > > >>>>> + void scaleBuffer(const libcamera::Span &source,\n> > > >>>>> + std::vector &dest);\n> > > >>>>> +\n> > > >>>>> +private:\n> > > >>>>> + libcamera::PixelFormat pixelFormat_;\n> > > >>>>> + libcamera::Size sourceSize_;\n> > > >>>>> + libcamera::Size targetSize_;\n> > > >>>>> +\n> > > >>>>> + bool validConfiguration_;\n> > > >>>>> +};\n> > > >>>>> +\n> > > >>>>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n> > > >>>>> diff --git a/src/android/meson.build b/src/android/meson.build\n> > > >>>>> index 5a01bea..3905e2f 100644\n> > > >>>>> --- a/src/android/meson.build\n> > > >>>>> +++ b/src/android/meson.build\n> > > >>>>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n> > > >>>>> 'jpeg/encoder_libjpeg.cpp',\n> > > >>>>> 'jpeg/exif.cpp',\n> > > >>>>> 'jpeg/post_processor_jpeg.cpp',\n> > > >>>>> + 'jpeg/thumbnailer.cpp',\n> > > >>>>> ])\n> > > >>>>> android_camera_metadata_sources = files([\n> > > >>>>>\n>\n> --\n> Regards,\n>\n> Laurent Pinchart","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 5EF4CBDB13\n\tfor ;\n\tFri, 23 Oct 2020 00:37:25 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id A7B1E615D6;\n\tFri, 23 Oct 2020 02:37:24 +0200 (CEST)","from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com\n\t[IPv6:2a00:1450:4864:20::52e])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C4A8E6034E\n\tfor ;\n\tFri, 23 Oct 2020 02:37:21 +0200 (CEST)","by mail-ed1-x52e.google.com with SMTP id w23so3598926edl.0\n\tfor ;\n\tThu, 22 Oct 2020 17:37:21 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=chromium.org header.i=@chromium.org\n\theader.b=\"LSz4b21U\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org;\n\ts=google; \n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=tqouv1kFvOzlbSIXB4i+M/DudYxfSwjzwkRQOlWFgKY=;\n\tb=LSz4b21UEwFEjwOS/sataMi0IEK0N6uKKauB8n+f9Fa18Z4SQM7vDpOiZ4p4oY9g0f\n\tCjJFQsZRToWO9/XOJ1mTXxR20bBWD8bKTe3NDsShmniXQ6CbycHVdUXGYwy2YTqppXSd\n\tEUi6D14eSe1uIFUZZS1R7r2BQG1b2uiBLbfrU=","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=tqouv1kFvOzlbSIXB4i+M/DudYxfSwjzwkRQOlWFgKY=;\n\tb=O+HdC1fVrk/UkKNUplpWjeZj+ubjslzqLgSRvJ9mwI5vbPLJgFZC6vaXEBODVnHjQB\n\tAvQ4RxMR/qWcfuIR5lG9EzL4rQTAuUzteFkLeI8QCR8djyfSVv8aV2mSEpZeDr+OD1pM\n\tbgvOOpTlIz04sONt5YOW4gdDez//bkPb4rLOUK5Xh12TvL1eJZ0S15a5C5rg6lgGVJHF\n\tmjR08iI6ZbKEFN/9l6YKfNp+bR2SSCY3+7C8MhuNiueTF1sw4hopawUOTqfJ/S8DtHlg\n\tNroEITNrKtEYuFFwkIJ8jFbxgRBr2ukWFJZR5IMSfm8XqTSyVKX9xBkyya5VjnTOvL6f\n\tBk0g==","X-Gm-Message-State":"AOAM530tgDSD142nOnwViNAJ5WdjugXEKYiujbNGNd6XoYlRoZuEgHsM\n\tExKOZFVKo/JG2l1MixQB9mEf83FDHOxfNX3vJ6U04YszgIOg+g==","X-Google-Smtp-Source":"ABdhPJymE9I4et+mzB/G0BaJq3I2md+IKdAyJRcSLrDPqHM8juxrNpSRiVH0y8VIRjoduJzEwWBLzynJotzHhrBEomE=","X-Received":"by 2002:a05:6402:602:: with SMTP id\n\tn2mr4677581edv.327.1603413441135; \n\tThu, 22 Oct 2020 17:37:21 -0700 (PDT)","MIME-Version":"1.0","References":"<20201021080806.46636-1-email@uajain.com>\n\t<20201021080806.46636-3-email@uajain.com>\n\t\n\t\n\t<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>\n\t<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>\n\t<9ca3b492-8468-e9cf-b103-b4350be472b9@ideasonboard.com>\n\t\n\t<20201022135914.GA3938@pendragon.ideasonboard.com>","In-Reply-To":"<20201022135914.GA3938@pendragon.ideasonboard.com>","From":"Hirokazu Honda ","Date":"Fri, 23 Oct 2020 09:37:09 +0900","Message-ID":"","To":"Laurent Pinchart ","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13422,"web_url":"https://patchwork.libcamera.org/comment/13422/","msgid":"<20201023035906.GA4113@pendragon.ideasonboard.com>","date":"2020-10-23T03:59:06","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Hiro-san,\n\nOn Fri, Oct 23, 2020 at 09:37:09AM +0900, Hirokazu Honda wrote:\n> On Thu, Oct 22, 2020 at 11:00 PM Laurent Pinchart wrote:\n> > On Thu, Oct 22, 2020 at 08:42:01PM +0900, Hirokazu Honda wrote:\n> > > Hi Umang,\n> > > Thanks for the work. I couldn't have time to review these today and\n> > > will review tomorrow.\n> > > In my understanding, the thumbnail class is basically to scale frames\n> > > to the given destination.\n> > > Can we generalize it as PostProcessor interface so that we will be\n> > > able to make use of it for down-scaling regardless of whether jpeg\n> > > encoding is required.\n> > > It will be necessary when Android HAL adaptation layer has to output\n> > > multiple different NV12 streams while the native camera can produce a\n> > > single NV12 stream simultaneously.\n> >\n> > One point to consider is that the thumbnailer is a down-scaler using a\n> > nearest-neighbour algorithm. It's good enough for thumbnails, but will\n> > lead to awful quality when scaling streams. For that, I think we need a\n> > better scaler implementation, and we could then replace the thumbnailer\n> > class with it.\n> \n> We can select nearest-neighbor and bilinear algorithms in\n> libyuv::NV12Scale with the filtering mode argument[1].\n> We may want to specify the scale algorithm on a class construction.\n> I would use libyuv here for better performance and less buggy (+ less\n> lines to be reviewed).\n> What do you'all think?\n\nWe have considered libyuv early on, and I think it's an interesting\noption moving forward. The reason we have decided to go for a minimal\nscaler implementation to create JPEG thumbnails is to avoid adding an\nexternal dependency that isn't packaged by distributions. This allows us\nto finalize JPEG thumbnail support without delay, and decide on the\nlonger term solution separately.\n\nIt would be very nice if Google could work with distributions to package\nlibyuv. I'd like to avoid including a copy of libyuv in libcamera.\n\nOne last comment to provide all the design background. We have also\nconsidered creating a format conversion library in libcamera, to gather\ncode from various projects that exist out there. There's format\nconversion in libv4l2, in raw2rgbpnm, in projects such as ffmpeg, and of\ncourse in libyuv. I would be great if that could be consolidated and\npackaged, but that's a big task in itself.\n\n> [1] https://chromium.googlesource.com/libyuv/libyuv/+/refs/heads/master/include/libyuv/scale.h#148\n>\n> > > On Wed, Oct 21, 2020 at 8:09 PM Kieran Bingham wrote:\n> > > > On 21/10/2020 12:03, Umang Jain wrote:\n> > > > > On 10/21/20 4:25 PM, Kieran Bingham wrote:\n> > > > >> On 21/10/2020 11:51, Umang Jain wrote:\n> > > > >>> On 10/21/20 2:49 PM, Kieran Bingham wrote:\n> > > > >>>> On 21/10/2020 09:08, Umang Jain wrote:\n> > > > >>>>> Add a basic image thumbnailer for NV12 frames being captured.\n> > > > >>>>> It shall generate a thumbnail image to be embedded as a part of\n> > > > >>>>> EXIF metadata of the frame. The output of the thumbnail will still\n> > > > >>>>> be NV12.\n> > > > >>>>>\n> > > > >>>>> Signed-off-by: Umang Jain \n> > > > >>>>> ---\n> > > > >>>>> src/android/jpeg/post_processor_jpeg.cpp | 39 +++++++++\n> > > > >>>>> src/android/jpeg/post_processor_jpeg.h | 3 +\n> > > > >>>>> src/android/jpeg/thumbnailer.cpp | 100 +++++++++++++++++++++++\n> > > > >>>>> src/android/jpeg/thumbnailer.h | 40 +++++++++\n> > > > >>>>> src/android/meson.build | 1 +\n> > > > >>>>> 5 files changed, 183 insertions(+)\n> > > > >>>>> create mode 100644 src/android/jpeg/thumbnailer.cpp\n> > > > >>>>> create mode 100644 src/android/jpeg/thumbnailer.h\n> > > > >>>>>\n> > > > >>>>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp\n> > > > >>>>> b/src/android/jpeg/post_processor_jpeg.cpp\n> > > > >>>>> index 9d452b7..f5f1f78 100644\n> > > > >>>>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n> > > > >>>>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n> > > > >>>>> @@ -11,6 +11,7 @@\n> > > > >>>>> #include \"../camera_metadata.h\"\n> > > > >>>>> #include \"encoder_libjpeg.h\"\n> > > > >>>>> #include \"exif.h\"\n> > > > >>>>> +#include \"thumbnailer.h\"\n> > > > >>>>> #include \n> > > > >>>>> @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const StreamConfiguration &inCfg,\n> > > > >>>>> return encoder_->configure(inCfg);\n> > > > >>>>> }\n> > > > >>>>> +void PostProcessorJpeg::generateThumbnail(const libcamera::Span &source,\n> > > > >>>>> + std::vector &thumbnail)\n> > > > >>>>> +{\n> > > > >>>>> + libcamera::Span destination;\n> > > > >>>>> + Thumbnailer thumbnailer;\n> > > > >>>>> +\n> > > > >>>>> + thumbnailer.configure(streamSize_, formats::NV12);\n> > > > >>>>> + libcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n> > > > >>>>> + thumbnailer.scaleBuffer(source, thumbnail);\n> > > > >>>>> +\n> > > > >>>>> + if (thumbnail.data()) {\n> > > > >>>>> + StreamConfiguration thumbnailCfg;\n> > > > >>>>> +\n> > > > >>>>> + std::unique_ptr encoder =\n> > > > >>>>> + std::make_unique();\n> > > > >>>>> +\n> > > > >>>>> + thumbnailCfg.pixelFormat = formats::NV12;\n> > > > >>>>> + thumbnailCfg.size = targetSize;\n> > > > >>>>> + encoder->configure(thumbnailCfg);\n> > > > >>>>\n> > > > >>>> thumbnail.capacity() might be quite low here.\n> > > > >>>> We need to make sure the vector is big enough at this point, you might\n> > > > >>>> need to do something like:\n> > > > >>>>\n> > > > >>>> thumbnail.resize(targetSize.width * targetSize.height * 4);\n> > > > >>>\n> > > > >>> I am not sure I follow. This is compressing the thumbnail part right? So\n> > > > >>> thumbnail is the \"source\" for the encoder here (Please refer to\n> > > > >>> ->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n> > > > >>\n> > > > >> Ugh, sorry - tired eyes indeed. Yes, please ignore those comments.\n> > > > >>\n> > > > >>>> Really we should obtain that size from the encoder. I thought we had a\n> > > > >>>> helper to do that already, but it seems we don't.\n> > > > >>>> We could/should add Encoder::maxOutput(); which would ask the encoder\n> > > > >>>> \"For your given configuration, what is the maximum number of bytes you\n> > > > >>>> might output\".\n> > > > >>>>\n> > > > >>>> Then of course we'd do:\n> > > > >>>> thumbnail.resize(encoder.maxOutput());\n> > > > >>>>\n> > > > >>>>\n> > > > >>>>> + int jpeg_size = encoder->encode({ thumbnail.data(), thumbnail.capacity() },\n> > > > >>>>> + destination, { });\n> > > > >>>\n> > > > >>> As said above, thumbnail is the source here. The compressed thumbnail\n> > > > >>> output is carried in destination. And, destination is a local span<>\n> > > > >>> here. Does it makes sense?\n> > > > >>\n> > > > >> Yes, I had mixed up source and destination I'm sorry.\n> > > > >>\n> > > > >> So who/where is allocating the destination?\n> > > > >\n> > > > > Ah, that's a good question and I think I just found a bug. I currently\n> > > > > pass on a locally allocated Span<> 'destination' to the encoder. But\n> > > > > there is no explicit allocation here. I think I need to point / create\n> > > > > the Span with the output size / bytes and then pass to the encode().\n> > > >\n> > > > :-) Ok - then please apply all of my earlier comments to that part\n> > > > instead. ;-)\n> > > >\n> > > > We certainly need somewhere to store the compressed image, to be able to\n> > > > pass it into the exif :-D\n> > > >\n> > > > >>>>> + LOG(JPEG, Info) << \"Thumbnail compress returned \"\n> > > > >>>>> + << jpeg_size << \" bytes\";\n> > > > >>>>\n> > > > >>>> And I presume we could then do an:\n> > > > >>>> thumbnail.resize(jpeg_size);\n> > > > >>>>\n> > > > >>>> here to update the thumbnail with the correct size. I'd be weary of the\n> > > > >>>> resize operations doing lots of re-allocations though, so perhaps we\n> > > > >>>> want to minimize that. But lets get to something that works first\n> > > > >>>> before worrying about optimising.\n> > > > >>>>\n> > > > >>>>> + }\n> > > > >>>>> +}\n> > > > >>>>> +\n> > > > >>>>> int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n> > > > >>>>> const libcamera::Span &destination,\n> > > > >>>>> CameraMetadata *metadata)\n> > > > >>>>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const\n> > > > >>>>> libcamera::FrameBuffer *source,\n> > > > >>>>> return jpeg_size;\n> > > > >>>>> }\n> > > > >>>>> + std::vector thumbnail;\n> > > > >>>>\n> > > > >>>> You need to resize this somewhere.\n> > > > >>>> Edit: Now seen a better place above ;-)\n> > > > >>>\n> > > > >>> The thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\n> > > > >>> We just pass in to the thumbnailer, while keeping its ownership in\n> > > > >>> PostProcessorJpeg.\n> > > > >>>\n> > > > >>>>> + generateThumbnail(destination, thumbnail);\n> > > > >>>>> + /*\n> > > > >>>>> + * \\todo: Write the compressed thumbnail to a file for inspection.\n> > > > >>>>> + * (I) Check if we can still write the thumbnail to EXIF here.\n> > > > >>>>> + * If not, we might need to move the thumbnailer logic to encoder.\n> > > > >>>>> + * And if we do that, first we need to make sure we get can\n> > > > >>>>> + * compressed data written to destination first before calling\n> > > > >>>>> + * jpeg_finish_compress operation somehow. Thumbnailing will\n> > > > >>>>> + * only occur if we have compressed data available first.\n> > > > >>>>> + */\n> > > > >>>>> +\n> > > > >>>>> /*\n> > > > >>>>> * Fill in the JPEG blob header.\n> > > > >>>>> *\n> > > > >>>>> diff --git a/src/android/jpeg/post_processor_jpeg.h\n> > > > >>>>> b/src/android/jpeg/post_processor_jpeg.h\n> > > > >>>>> index 62c8650..05601ee 100644\n> > > > >>>>> --- a/src/android/jpeg/post_processor_jpeg.h\n> > > > >>>>> +++ b/src/android/jpeg/post_processor_jpeg.h\n> > > > >>>>> @@ -28,6 +28,9 @@ public:\n> > > > >>>>> CameraMetadata *metadata) override;\n> > > > >>>>> private:\n> > > > >>>>> + void generateThumbnail(const libcamera::Span &source,\n> > > > >>>>> + std::vector &thumbnail);\n> > > > >>>>> +\n> > > > >>>>> CameraDevice *cameraDevice_;\n> > > > >>>>> std::unique_ptr encoder_;\n> > > > >>>>> libcamera::Size streamSize_;\n> > > > >>>>> diff --git a/src/android/jpeg/thumbnailer.cpp\n> > > > >>>>> b/src/android/jpeg/thumbnailer.cpp\n> > > > >>>>> new file mode 100644\n> > > > >>>>> index 0000000..3163576\n> > > > >>>>> --- /dev/null\n> > > > >>>>> +++ b/src/android/jpeg/thumbnailer.cpp\n> > > > >>>>> @@ -0,0 +1,100 @@\n> > > > >>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > > >>>>> +/*\n> > > > >>>>> + * Copyright (C) 2020, Google Inc.\n> > > > >>>>> + *\n> > > > >>>>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n> > > > >>>>> + */\n> > > > >>>>> +\n> > > > >>>>> +#include \"thumbnailer.h\"\n> > > > >>>>> +\n> > > > >>>>> +#include \n> > > > >>>>> +\n> > > > >>>>> +#include \"libcamera/internal/file.h\"\n> > > > >>>>> +#include \"libcamera/internal/log.h\"\n> > > > >>>>> +\n> > > > >>>>> +using namespace libcamera;\n> > > > >>>>> +\n> > > > >>>>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n> > > > >>>>> +\n> > > > >>>>> +Thumbnailer::Thumbnailer()\n> > > > >>>>> + : validConfiguration_(false)\n> > > > >>>>> +{\n> > > > >>>>> +}\n> > > > >>>>> +\n> > > > >>>>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat pixelFormat)\n> > > > >>>>> +{\n> > > > >>>>> + sourceSize_ = sourceSize;\n> > > > >>>>> + pixelFormat_ = pixelFormat;\n> > > > >>>>> +\n> > > > >>>>> + if (pixelFormat_ != formats::NV12) {\n> > > > >>>>> + LOG (Thumbnailer, Error) << \"Failed to configure: Pixel Format \"\n> > > > >>>>> + << pixelFormat_.toString() << \" unsupported.\";\n> > > > >>>>> + return;\n> > > > >>>>> + }\n> > > > >>>>> +\n> > > > >>>>> + validConfiguration_ = true;\n> > > > >>>>> +}\n> > > > >>>>> +\n> > > > >>>>> +/*\n> > > > >>>>> + * The Exif specification recommends the width of the thumbnail to be a\n> > > > >>>>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding height\n> > > > >>>>> + * keeping the aspect ratio same as of the source.\n> > > > >>>>> + */\n> > > > >>>>> +Size Thumbnailer::computeThumbnailSize()\n> > > > >>>>> +{\n> > > > >>>>> + unsigned int targetHeight;\n> > > > >>>>> + unsigned int targetWidth = 160;\n> > > > >>>>> +\n> > > > >>>>> + targetHeight = targetWidth * sourceSize_.height / sourceSize_.width;\n> > > > >>>>> +\n> > > > >>>>> + if (targetHeight & 1)\n> > > > >>>>> + targetHeight++;\n> > > > >>>>> +\n> > > > >>>>> + return Size(targetWidth, targetHeight);\n> > > > >>>>> +}\n> > > > >>>>> +\n> > > > >>>>> +void\n> > > > >>>>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n> > > > >>>>> + std::vector &destination)\n> > > > >>>>> +{\n> > > > >>>>> + if (!validConfiguration_) {\n> > > > >>>>> + LOG(Thumbnailer, Error) << \"config is unconfigured or invalid.\";\n> > > > >>>>> + return;\n> > > > >>>>> + }\n> > > > >>>>> +\n> > > > >>>>> + targetSize_ = computeThumbnailSize();\n> > > > >>>>> +\n> > > > >>>>> + const unsigned int sw = sourceSize_.width;\n> > > > >>>>> + const unsigned int sh = sourceSize_.height;\n> > > > >>>>> + const unsigned int tw = targetSize_.width;\n> > > > >>>>> + const unsigned int th = targetSize_.height;\n> > > > >>>>> +\n> > > > >>>>> + /* Image scaling block implementing nearest-neighbour algorithm. */\n> > > > >>>>> + unsigned char *src = static_cast(source.data());\n> > > > >>>>> + unsigned char *src_c = src + sh * sw;\n> > > > >>>>> + unsigned char *src_cb, *src_cr;\n> > > > >>>>> +\n> > > > >>>>> + size_t dstSize = (th * tw) + ((th/2) * tw);\n> > > > >>>>> + destination.reserve(dstSize);\n> > > > >>>>> + unsigned char *dst = destination.data();\n> > > > >>>>> + unsigned char *dst_c = dst + th * tw;\n> > > > >>>>> +\n> > > > >>>>> + for (unsigned int y = 0; y < th; y+=2) {\n> > > > >>>>> + unsigned int sourceY = (sh*y + th/2) / th;\n> > > > >>>>> +\n> > > > >>>>> + src_cb = src_c + (sourceY/2) * sw + 0;\n> > > > >>>>> + src_cr = src_c + (sourceY/2) * sw + 1;\n> > > > >>>>> +\n> > > > >>>>> + for (unsigned int x = 0; x < tw; x+=2) {\n> > > > >>>>> + unsigned int sourceX = (sw*x + tw/2) / tw;\n> > > > >>>>> +\n> > > > >>>>> + dst[y * tw + x] = src[sw * sourceY + sourceX];\n> > > > >>>>> + dst[(y+1) * tw + x] = src[sw * (sourceY+1) + sourceX];\n> > > > >>>>> + dst[y * tw + (x+1)] = src[sw * sourceY + (sourceX+1)];\n> > > > >>>>> + dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) + (sourceX+1)];\n> > > > >>>>> +\n> > > > >>>>> + dst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n> > > > >>>>> + dst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n> > > > >>>>> + }\n> > > > >>>>> + }\n> > > > >>>>> +}\n> > > > >>>>> diff --git a/src/android/jpeg/thumbnailer.h\n> > > > >>>>> b/src/android/jpeg/thumbnailer.h\n> > > > >>>>> new file mode 100644\n> > > > >>>>> index 0000000..bab9855\n> > > > >>>>> --- /dev/null\n> > > > >>>>> +++ b/src/android/jpeg/thumbnailer.h\n> > > > >>>>> @@ -0,0 +1,40 @@\n> > > > >>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > > >>>>> +/*\n> > > > >>>>> + * Copyright (C) 2020, Google Inc.\n> > > > >>>>> + *\n> > > > >>>>> + * thumbnailer.h - Basic image thumbnailer from NV12\n> > > > >>>>> + */\n> > > > >>>>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n> > > > >>>>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n> > > > >>>>> +\n> > > > >>>>> +#include \n> > > > >>>>> +\n> > > > >>>>> +#include \"libcamera/internal/buffer.h\"\n> > > > >>>>> +#include \"libcamera/internal/formats.h\"\n> > > > >>>>> +\n> > > > >>>>> +class Thumbnailer\n> > > > >>>>> +{\n> > > > >>>>> +public:\n> > > > >>>>> + Thumbnailer();\n> > > > >>>>> +\n> > > > >>>>> + void configure(const libcamera::Size &sourceSize,\n> > > > >>>>> + libcamera::PixelFormat pixelFormat);\n> > > > >>>>> +\n> > > > >>>>> + /*\n> > > > >>>>> + * \\todo: Discuss if we can return targetSize_ via configure() or\n> > > > >>>>> + * scaleBuffer(). We need targetSize_ to re-encode the scaled\n> > > > >>>>> buffer\n> > > > >>>>> + * via encoder in PostProcssorJpeg::writeThumbnail().\n> > > > >>>>> + */\n> > > > >>>>> + libcamera::Size computeThumbnailSize();\n> > > > >>>>> + void scaleBuffer(const libcamera::Span &source,\n> > > > >>>>> + std::vector &dest);\n> > > > >>>>> +\n> > > > >>>>> +private:\n> > > > >>>>> + libcamera::PixelFormat pixelFormat_;\n> > > > >>>>> + libcamera::Size sourceSize_;\n> > > > >>>>> + libcamera::Size targetSize_;\n> > > > >>>>> +\n> > > > >>>>> + bool validConfiguration_;\n> > > > >>>>> +};\n> > > > >>>>> +\n> > > > >>>>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n> > > > >>>>> diff --git a/src/android/meson.build b/src/android/meson.build\n> > > > >>>>> index 5a01bea..3905e2f 100644\n> > > > >>>>> --- a/src/android/meson.build\n> > > > >>>>> +++ b/src/android/meson.build\n> > > > >>>>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n> > > > >>>>> 'jpeg/encoder_libjpeg.cpp',\n> > > > >>>>> 'jpeg/exif.cpp',\n> > > > >>>>> 'jpeg/post_processor_jpeg.cpp',\n> > > > >>>>> + 'jpeg/thumbnailer.cpp',\n> > > > >>>>> ])\n> > > > >>>>> android_camera_metadata_sources = files([\n> > > > >>>>>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id CFE88C3D3C\n\tfor ;\n\tFri, 23 Oct 2020 03:59:54 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 43AD2615D4;\n\tFri, 23 Oct 2020 05:59:54 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id D9D4C6034E\n\tfor ;\n\tFri, 23 Oct 2020 05:59:52 +0200 (CEST)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 3C0EA53;\n\tFri, 23 Oct 2020 05:59:52 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"roj5rkDc\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1603425592;\n\tbh=pM/Cw+10SroQgoXU4iqOqnEIqTyXNFUhgGTnlfGQfec=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=roj5rkDcwghBj1kG0TzYoQxC3AjC0hIQaLryyWQe0w9z9N3YjWMcx+A3OrxKAoFwj\n\tbErsNGXmEsokrIxgrvpDZXvGbPLvf2bqQMqechf+MQkk7UDqnVmM1vxiQqTD3KfXhO\n\tgueA2aPKGKzyWBo/h5dB2cEEnhD1ELGoKeqClsak=","Date":"Fri, 23 Oct 2020 06:59:06 +0300","From":"Laurent Pinchart ","To":"Hirokazu Honda ","Message-ID":"<20201023035906.GA4113@pendragon.ideasonboard.com>","References":"<20201021080806.46636-1-email@uajain.com>\n\t<20201021080806.46636-3-email@uajain.com>\n\t\n\t\n\t<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>\n\t<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>\n\t<9ca3b492-8468-e9cf-b103-b4350be472b9@ideasonboard.com>\n\t\n\t<20201022135914.GA3938@pendragon.ideasonboard.com>\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13439,"web_url":"https://patchwork.libcamera.org/comment/13439/","msgid":"<734c408b-2e5b-6ee4-af07-f9c45416abce@ideasonboard.com>","date":"2020-10-23T08:39:35","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi Hiro-san, Laurent,\n\nOn 23/10/2020 04:59, Laurent Pinchart wrote:\n> Hi Hiro-san,\n> \n> On Fri, Oct 23, 2020 at 09:37:09AM +0900, Hirokazu Honda wrote:\n>> On Thu, Oct 22, 2020 at 11:00 PM Laurent Pinchart wrote:\n>>> On Thu, Oct 22, 2020 at 08:42:01PM +0900, Hirokazu Honda wrote:\n>>>> Hi Umang,\n>>>> Thanks for the work. I couldn't have time to review these today and\n>>>> will review tomorrow.\n>>>> In my understanding, the thumbnail class is basically to scale frames\n>>>> to the given destination.\n>>>> Can we generalize it as PostProcessor interface so that we will be\n>>>> able to make use of it for down-scaling regardless of whether jpeg\n>>>> encoding is required.\n>>>> It will be necessary when Android HAL adaptation layer has to output\n>>>> multiple different NV12 streams while the native camera can produce a\n>>>> single NV12 stream simultaneously.\n>>>\n>>> One point to consider is that the thumbnailer is a down-scaler using a\n>>> nearest-neighbour algorithm. It's good enough for thumbnails, but will\n>>> lead to awful quality when scaling streams. For that, I think we need a\n>>> better scaler implementation, and we could then replace the thumbnailer\n>>> class with it.\n>>\n>> We can select nearest-neighbor and bilinear algorithms in\n>> libyuv::NV12Scale with the filtering mode argument[1].\n>> We may want to specify the scale algorithm on a class construction.\n>> I would use libyuv here for better performance and less buggy (+ less\n>> lines to be reviewed).\n>> What do you'all think?\n> \n> We have considered libyuv early on, and I think it's an interesting\n> option moving forward. The reason we have decided to go for a minimal\n> scaler implementation to create JPEG thumbnails is to avoid adding an\n> external dependency that isn't packaged by distributions. This allows us\n> to finalize JPEG thumbnail support without delay, and decide on the\n> longer term solution separately.\n> \n> It would be very nice if Google could work with distributions to package\n> libyuv. I'd like to avoid including a copy of libyuv in libcamera.\n\nLack of packaging is a big factor here. If it's not available we can't\nuse it ;-)\n\nHowever I looked into this briefly yesterday, - we could use a meson\nwrap. But I don't know how that will affect us distributing packages, as\nthen we also have to distribute libyuv.\n\nOr perhaps the meson wrap would link statically.\n\n\n> One last comment to provide all the design background. We have also\n> considered creating a format conversion library in libcamera, to gather\n> code from various projects that exist out there. There's format\n> conversion in libv4l2, in raw2rgbpnm, in projects such as ffmpeg, and of\n> course in libyuv. I would be great if that could be consolidated and\n> packaged, but that's a big task in itself.\n\nAnd that's the bigger picture. This should be consolidated. And that\nconsolidated library should support more than yuv ;-)\n\nGiven it would be a software format convertor, and scaler, and ideally\nwith hardware acceleration support where possible, I think that quickly\nbecomes libisp - and the feature creep will ... creep in ;-)\n\n\n\n> \n>> [1] https://chromium.googlesource.com/libyuv/libyuv/+/refs/heads/master/include/libyuv/scale.h#148\n>>\n>>>> On Wed, Oct 21, 2020 at 8:09 PM Kieran Bingham wrote:\n>>>>> On 21/10/2020 12:03, Umang Jain wrote:\n>>>>>> On 10/21/20 4:25 PM, Kieran Bingham wrote:\n>>>>>>> On 21/10/2020 11:51, Umang Jain wrote:\n>>>>>>>> On 10/21/20 2:49 PM, Kieran Bingham wrote:\n>>>>>>>>> On 21/10/2020 09:08, Umang Jain wrote:\n>>>>>>>>>> Add a basic image thumbnailer for NV12 frames being captured.\n>>>>>>>>>> It shall generate a thumbnail image to be embedded as a part of\n>>>>>>>>>> EXIF metadata of the frame. The output of the thumbnail will still\n>>>>>>>>>> be NV12.\n>>>>>>>>>>\n>>>>>>>>>> Signed-off-by: Umang Jain \n>>>>>>>>>> ---\n>>>>>>>>>> src/android/jpeg/post_processor_jpeg.cpp | 39 +++++++++\n>>>>>>>>>> src/android/jpeg/post_processor_jpeg.h | 3 +\n>>>>>>>>>> src/android/jpeg/thumbnailer.cpp | 100 +++++++++++++++++++++++\n>>>>>>>>>> src/android/jpeg/thumbnailer.h | 40 +++++++++\n>>>>>>>>>> src/android/meson.build | 1 +\n>>>>>>>>>> 5 files changed, 183 insertions(+)\n>>>>>>>>>> create mode 100644 src/android/jpeg/thumbnailer.cpp\n>>>>>>>>>> create mode 100644 src/android/jpeg/thumbnailer.h\n>>>>>>>>>>\n>>>>>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp\n>>>>>>>>>> b/src/android/jpeg/post_processor_jpeg.cpp\n>>>>>>>>>> index 9d452b7..f5f1f78 100644\n>>>>>>>>>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n>>>>>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n>>>>>>>>>> @@ -11,6 +11,7 @@\n>>>>>>>>>> #include \"../camera_metadata.h\"\n>>>>>>>>>> #include \"encoder_libjpeg.h\"\n>>>>>>>>>> #include \"exif.h\"\n>>>>>>>>>> +#include \"thumbnailer.h\"\n>>>>>>>>>> #include \n>>>>>>>>>> @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const StreamConfiguration &inCfg,\n>>>>>>>>>> return encoder_->configure(inCfg);\n>>>>>>>>>> }\n>>>>>>>>>> +void PostProcessorJpeg::generateThumbnail(const libcamera::Span &source,\n>>>>>>>>>> + std::vector &thumbnail)\n>>>>>>>>>> +{\n>>>>>>>>>> + libcamera::Span destination;\n>>>>>>>>>> + Thumbnailer thumbnailer;\n>>>>>>>>>> +\n>>>>>>>>>> + thumbnailer.configure(streamSize_, formats::NV12);\n>>>>>>>>>> + libcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n>>>>>>>>>> + thumbnailer.scaleBuffer(source, thumbnail);\n>>>>>>>>>> +\n>>>>>>>>>> + if (thumbnail.data()) {\n>>>>>>>>>> + StreamConfiguration thumbnailCfg;\n>>>>>>>>>> +\n>>>>>>>>>> + std::unique_ptr encoder =\n>>>>>>>>>> + std::make_unique();\n>>>>>>>>>> +\n>>>>>>>>>> + thumbnailCfg.pixelFormat = formats::NV12;\n>>>>>>>>>> + thumbnailCfg.size = targetSize;\n>>>>>>>>>> + encoder->configure(thumbnailCfg);\n>>>>>>>>>\n>>>>>>>>> thumbnail.capacity() might be quite low here.\n>>>>>>>>> We need to make sure the vector is big enough at this point, you might\n>>>>>>>>> need to do something like:\n>>>>>>>>>\n>>>>>>>>> thumbnail.resize(targetSize.width * targetSize.height * 4);\n>>>>>>>>\n>>>>>>>> I am not sure I follow. This is compressing the thumbnail part right? So\n>>>>>>>> thumbnail is the \"source\" for the encoder here (Please refer to\n>>>>>>>> ->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n>>>>>>>\n>>>>>>> Ugh, sorry - tired eyes indeed. Yes, please ignore those comments.\n>>>>>>>\n>>>>>>>>> Really we should obtain that size from the encoder. I thought we had a\n>>>>>>>>> helper to do that already, but it seems we don't.\n>>>>>>>>> We could/should add Encoder::maxOutput(); which would ask the encoder\n>>>>>>>>> \"For your given configuration, what is the maximum number of bytes you\n>>>>>>>>> might output\".\n>>>>>>>>>\n>>>>>>>>> Then of course we'd do:\n>>>>>>>>> thumbnail.resize(encoder.maxOutput());\n>>>>>>>>>\n>>>>>>>>>\n>>>>>>>>>> + int jpeg_size = encoder->encode({ thumbnail.data(), thumbnail.capacity() },\n>>>>>>>>>> + destination, { });\n>>>>>>>>\n>>>>>>>> As said above, thumbnail is the source here. The compressed thumbnail\n>>>>>>>> output is carried in destination. And, destination is a local span<>\n>>>>>>>> here. Does it makes sense?\n>>>>>>>\n>>>>>>> Yes, I had mixed up source and destination I'm sorry.\n>>>>>>>\n>>>>>>> So who/where is allocating the destination?\n>>>>>>\n>>>>>> Ah, that's a good question and I think I just found a bug. I currently\n>>>>>> pass on a locally allocated Span<> 'destination' to the encoder. But\n>>>>>> there is no explicit allocation here. I think I need to point / create\n>>>>>> the Span with the output size / bytes and then pass to the encode().\n>>>>>\n>>>>> :-) Ok - then please apply all of my earlier comments to that part\n>>>>> instead. ;-)\n>>>>>\n>>>>> We certainly need somewhere to store the compressed image, to be able to\n>>>>> pass it into the exif :-D\n>>>>>\n>>>>>>>>>> + LOG(JPEG, Info) << \"Thumbnail compress returned \"\n>>>>>>>>>> + << jpeg_size << \" bytes\";\n>>>>>>>>>\n>>>>>>>>> And I presume we could then do an:\n>>>>>>>>> thumbnail.resize(jpeg_size);\n>>>>>>>>>\n>>>>>>>>> here to update the thumbnail with the correct size. I'd be weary of the\n>>>>>>>>> resize operations doing lots of re-allocations though, so perhaps we\n>>>>>>>>> want to minimize that. But lets get to something that works first\n>>>>>>>>> before worrying about optimising.\n>>>>>>>>>\n>>>>>>>>>> + }\n>>>>>>>>>> +}\n>>>>>>>>>> +\n>>>>>>>>>> int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n>>>>>>>>>> const libcamera::Span &destination,\n>>>>>>>>>> CameraMetadata *metadata)\n>>>>>>>>>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const\n>>>>>>>>>> libcamera::FrameBuffer *source,\n>>>>>>>>>> return jpeg_size;\n>>>>>>>>>> }\n>>>>>>>>>> + std::vector thumbnail;\n>>>>>>>>>\n>>>>>>>>> You need to resize this somewhere.\n>>>>>>>>> Edit: Now seen a better place above ;-)\n>>>>>>>>\n>>>>>>>> The thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\n>>>>>>>> We just pass in to the thumbnailer, while keeping its ownership in\n>>>>>>>> PostProcessorJpeg.\n>>>>>>>>\n>>>>>>>>>> + generateThumbnail(destination, thumbnail);\n>>>>>>>>>> + /*\n>>>>>>>>>> + * \\todo: Write the compressed thumbnail to a file for inspection.\n>>>>>>>>>> + * (I) Check if we can still write the thumbnail to EXIF here.\n>>>>>>>>>> + * If not, we might need to move the thumbnailer logic to encoder.\n>>>>>>>>>> + * And if we do that, first we need to make sure we get can\n>>>>>>>>>> + * compressed data written to destination first before calling\n>>>>>>>>>> + * jpeg_finish_compress operation somehow. Thumbnailing will\n>>>>>>>>>> + * only occur if we have compressed data available first.\n>>>>>>>>>> + */\n>>>>>>>>>> +\n>>>>>>>>>> /*\n>>>>>>>>>> * Fill in the JPEG blob header.\n>>>>>>>>>> *\n>>>>>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.h\n>>>>>>>>>> b/src/android/jpeg/post_processor_jpeg.h\n>>>>>>>>>> index 62c8650..05601ee 100644\n>>>>>>>>>> --- a/src/android/jpeg/post_processor_jpeg.h\n>>>>>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.h\n>>>>>>>>>> @@ -28,6 +28,9 @@ public:\n>>>>>>>>>> CameraMetadata *metadata) override;\n>>>>>>>>>> private:\n>>>>>>>>>> + void generateThumbnail(const libcamera::Span &source,\n>>>>>>>>>> + std::vector &thumbnail);\n>>>>>>>>>> +\n>>>>>>>>>> CameraDevice *cameraDevice_;\n>>>>>>>>>> std::unique_ptr encoder_;\n>>>>>>>>>> libcamera::Size streamSize_;\n>>>>>>>>>> diff --git a/src/android/jpeg/thumbnailer.cpp\n>>>>>>>>>> b/src/android/jpeg/thumbnailer.cpp\n>>>>>>>>>> new file mode 100644\n>>>>>>>>>> index 0000000..3163576\n>>>>>>>>>> --- /dev/null\n>>>>>>>>>> +++ b/src/android/jpeg/thumbnailer.cpp\n>>>>>>>>>> @@ -0,0 +1,100 @@\n>>>>>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n>>>>>>>>>> +/*\n>>>>>>>>>> + * Copyright (C) 2020, Google Inc.\n>>>>>>>>>> + *\n>>>>>>>>>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n>>>>>>>>>> + */\n>>>>>>>>>> +\n>>>>>>>>>> +#include \"thumbnailer.h\"\n>>>>>>>>>> +\n>>>>>>>>>> +#include \n>>>>>>>>>> +\n>>>>>>>>>> +#include \"libcamera/internal/file.h\"\n>>>>>>>>>> +#include \"libcamera/internal/log.h\"\n>>>>>>>>>> +\n>>>>>>>>>> +using namespace libcamera;\n>>>>>>>>>> +\n>>>>>>>>>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n>>>>>>>>>> +\n>>>>>>>>>> +Thumbnailer::Thumbnailer()\n>>>>>>>>>> + : validConfiguration_(false)\n>>>>>>>>>> +{\n>>>>>>>>>> +}\n>>>>>>>>>> +\n>>>>>>>>>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat pixelFormat)\n>>>>>>>>>> +{\n>>>>>>>>>> + sourceSize_ = sourceSize;\n>>>>>>>>>> + pixelFormat_ = pixelFormat;\n>>>>>>>>>> +\n>>>>>>>>>> + if (pixelFormat_ != formats::NV12) {\n>>>>>>>>>> + LOG (Thumbnailer, Error) << \"Failed to configure: Pixel Format \"\n>>>>>>>>>> + << pixelFormat_.toString() << \" unsupported.\";\n>>>>>>>>>> + return;\n>>>>>>>>>> + }\n>>>>>>>>>> +\n>>>>>>>>>> + validConfiguration_ = true;\n>>>>>>>>>> +}\n>>>>>>>>>> +\n>>>>>>>>>> +/*\n>>>>>>>>>> + * The Exif specification recommends the width of the thumbnail to be a\n>>>>>>>>>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding height\n>>>>>>>>>> + * keeping the aspect ratio same as of the source.\n>>>>>>>>>> + */\n>>>>>>>>>> +Size Thumbnailer::computeThumbnailSize()\n>>>>>>>>>> +{\n>>>>>>>>>> + unsigned int targetHeight;\n>>>>>>>>>> + unsigned int targetWidth = 160;\n>>>>>>>>>> +\n>>>>>>>>>> + targetHeight = targetWidth * sourceSize_.height / sourceSize_.width;\n>>>>>>>>>> +\n>>>>>>>>>> + if (targetHeight & 1)\n>>>>>>>>>> + targetHeight++;\n>>>>>>>>>> +\n>>>>>>>>>> + return Size(targetWidth, targetHeight);\n>>>>>>>>>> +}\n>>>>>>>>>> +\n>>>>>>>>>> +void\n>>>>>>>>>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n>>>>>>>>>> + std::vector &destination)\n>>>>>>>>>> +{\n>>>>>>>>>> + if (!validConfiguration_) {\n>>>>>>>>>> + LOG(Thumbnailer, Error) << \"config is unconfigured or invalid.\";\n>>>>>>>>>> + return;\n>>>>>>>>>> + }\n>>>>>>>>>> +\n>>>>>>>>>> + targetSize_ = computeThumbnailSize();\n>>>>>>>>>> +\n>>>>>>>>>> + const unsigned int sw = sourceSize_.width;\n>>>>>>>>>> + const unsigned int sh = sourceSize_.height;\n>>>>>>>>>> + const unsigned int tw = targetSize_.width;\n>>>>>>>>>> + const unsigned int th = targetSize_.height;\n>>>>>>>>>> +\n>>>>>>>>>> + /* Image scaling block implementing nearest-neighbour algorithm. */\n>>>>>>>>>> + unsigned char *src = static_cast(source.data());\n>>>>>>>>>> + unsigned char *src_c = src + sh * sw;\n>>>>>>>>>> + unsigned char *src_cb, *src_cr;\n>>>>>>>>>> +\n>>>>>>>>>> + size_t dstSize = (th * tw) + ((th/2) * tw);\n>>>>>>>>>> + destination.reserve(dstSize);\n>>>>>>>>>> + unsigned char *dst = destination.data();\n>>>>>>>>>> + unsigned char *dst_c = dst + th * tw;\n>>>>>>>>>> +\n>>>>>>>>>> + for (unsigned int y = 0; y < th; y+=2) {\n>>>>>>>>>> + unsigned int sourceY = (sh*y + th/2) / th;\n>>>>>>>>>> +\n>>>>>>>>>> + src_cb = src_c + (sourceY/2) * sw + 0;\n>>>>>>>>>> + src_cr = src_c + (sourceY/2) * sw + 1;\n>>>>>>>>>> +\n>>>>>>>>>> + for (unsigned int x = 0; x < tw; x+=2) {\n>>>>>>>>>> + unsigned int sourceX = (sw*x + tw/2) / tw;\n>>>>>>>>>> +\n>>>>>>>>>> + dst[y * tw + x] = src[sw * sourceY + sourceX];\n>>>>>>>>>> + dst[(y+1) * tw + x] = src[sw * (sourceY+1) + sourceX];\n>>>>>>>>>> + dst[y * tw + (x+1)] = src[sw * sourceY + (sourceX+1)];\n>>>>>>>>>> + dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) + (sourceX+1)];\n>>>>>>>>>> +\n>>>>>>>>>> + dst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n>>>>>>>>>> + dst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n>>>>>>>>>> + }\n>>>>>>>>>> + }\n>>>>>>>>>> +}\n>>>>>>>>>> diff --git a/src/android/jpeg/thumbnailer.h\n>>>>>>>>>> b/src/android/jpeg/thumbnailer.h\n>>>>>>>>>> new file mode 100644\n>>>>>>>>>> index 0000000..bab9855\n>>>>>>>>>> --- /dev/null\n>>>>>>>>>> +++ b/src/android/jpeg/thumbnailer.h\n>>>>>>>>>> @@ -0,0 +1,40 @@\n>>>>>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n>>>>>>>>>> +/*\n>>>>>>>>>> + * Copyright (C) 2020, Google Inc.\n>>>>>>>>>> + *\n>>>>>>>>>> + * thumbnailer.h - Basic image thumbnailer from NV12\n>>>>>>>>>> + */\n>>>>>>>>>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n>>>>>>>>>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n>>>>>>>>>> +\n>>>>>>>>>> +#include \n>>>>>>>>>> +\n>>>>>>>>>> +#include \"libcamera/internal/buffer.h\"\n>>>>>>>>>> +#include \"libcamera/internal/formats.h\"\n>>>>>>>>>> +\n>>>>>>>>>> +class Thumbnailer\n>>>>>>>>>> +{\n>>>>>>>>>> +public:\n>>>>>>>>>> + Thumbnailer();\n>>>>>>>>>> +\n>>>>>>>>>> + void configure(const libcamera::Size &sourceSize,\n>>>>>>>>>> + libcamera::PixelFormat pixelFormat);\n>>>>>>>>>> +\n>>>>>>>>>> + /*\n>>>>>>>>>> + * \\todo: Discuss if we can return targetSize_ via configure() or\n>>>>>>>>>> + * scaleBuffer(). We need targetSize_ to re-encode the scaled\n>>>>>>>>>> buffer\n>>>>>>>>>> + * via encoder in PostProcssorJpeg::writeThumbnail().\n>>>>>>>>>> + */\n>>>>>>>>>> + libcamera::Size computeThumbnailSize();\n>>>>>>>>>> + void scaleBuffer(const libcamera::Span &source,\n>>>>>>>>>> + std::vector &dest);\n>>>>>>>>>> +\n>>>>>>>>>> +private:\n>>>>>>>>>> + libcamera::PixelFormat pixelFormat_;\n>>>>>>>>>> + libcamera::Size sourceSize_;\n>>>>>>>>>> + libcamera::Size targetSize_;\n>>>>>>>>>> +\n>>>>>>>>>> + bool validConfiguration_;\n>>>>>>>>>> +};\n>>>>>>>>>> +\n>>>>>>>>>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n>>>>>>>>>> diff --git a/src/android/meson.build b/src/android/meson.build\n>>>>>>>>>> index 5a01bea..3905e2f 100644\n>>>>>>>>>> --- a/src/android/meson.build\n>>>>>>>>>> +++ b/src/android/meson.build\n>>>>>>>>>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n>>>>>>>>>> 'jpeg/encoder_libjpeg.cpp',\n>>>>>>>>>> 'jpeg/exif.cpp',\n>>>>>>>>>> 'jpeg/post_processor_jpeg.cpp',\n>>>>>>>>>> + 'jpeg/thumbnailer.cpp',\n>>>>>>>>>> ])\n>>>>>>>>>> android_camera_metadata_sources = files([\n>>>>>>>>>>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id E4A32BDB13\n\tfor ;\n\tFri, 23 Oct 2020 08:39:41 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 62C8D615D6;\n\tFri, 23 Oct 2020 10:39:41 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id AA57C60350\n\tfor ;\n\tFri, 23 Oct 2020 10:39:39 +0200 (CEST)","from [192.168.0.20]\n\t(cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 08654B26;\n\tFri, 23 Oct 2020 10:39:38 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"aRJm6B57\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1603442379;\n\tbh=uMKP28D99NpdEGeArdrJqDFY9QoKzL+Exl4yny3oYHI=;\n\th=Reply-To:Subject:To:Cc:References:From:Date:In-Reply-To:From;\n\tb=aRJm6B57nH+GmWEBnmmamPWa2kGRrun8YS3KRekmNp0TqFe2wq5jXKqGtDK25rvGM\n\tFUgxtov1iQ7Jrjd2fZenVegAaBGw3D0//nT9BtF5lPk6Ptx6koJZLDbEXnlE2845Ek\n\tpICz57fYEyMu3+I5jrbaIpB52tx7bkMR6ytSPKOw=","To":"Laurent Pinchart ,\n\tHirokazu Honda ","References":"<20201021080806.46636-1-email@uajain.com>\n\t<20201021080806.46636-3-email@uajain.com>\n\t\n\t\n\t<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>\n\t<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>\n\t<9ca3b492-8468-e9cf-b103-b4350be472b9@ideasonboard.com>\n\t\n\t<20201022135914.GA3938@pendragon.ideasonboard.com>\n\t\n\t<20201023035906.GA4113@pendragon.ideasonboard.com>","From":"Kieran Bingham ","Autocrypt":"addr=kieran.bingham@ideasonboard.com; keydata=\n\tmQINBFYE/WYBEACs1PwjMD9rgCu1hlIiUA1AXR4rv2v+BCLUq//vrX5S5bjzxKAryRf0uHat\n\tV/zwz6hiDrZuHUACDB7X8OaQcwhLaVlq6byfoBr25+hbZG7G3+5EUl9cQ7dQEdvNj6V6y/SC\n\trRanWfelwQThCHckbobWiQJfK9n7rYNcPMq9B8e9F020LFH7Kj6YmO95ewJGgLm+idg1Kb3C\n\tpotzWkXc1xmPzcQ1fvQMOfMwdS+4SNw4rY9f07Xb2K99rjMwZVDgESKIzhsDB5GY465sCsiQ\n\tcSAZRxqE49RTBq2+EQsbrQpIc8XiffAB8qexh5/QPzCmR4kJgCGeHIXBtgRj+nIkCJPZvZtf\n\tKr2EAbc6tgg6DkAEHJb+1okosV09+0+TXywYvtEop/WUOWQ+zo+Y/OBd+8Ptgt1pDRyOBzL8\n\tRXa8ZqRf0Mwg75D+dKntZeJHzPRJyrlfQokngAAs4PaFt6UfS+ypMAF37T6CeDArQC41V3ko\n\tlPn1yMsVD0p+6i3DPvA/GPIksDC4owjnzVX9kM8Zc5Cx+XoAN0w5Eqo4t6qEVbuettxx55gq\n\t8K8FieAjgjMSxngo/HST8TpFeqI5nVeq0/lqtBRQKumuIqDg+Bkr4L1V/PSB6XgQcOdhtd36\n\tOe9X9dXB8YSNt7VjOcO7BTmFn/Z8r92mSAfHXpb07YJWJosQOQARAQABtDBLaWVyYW4gQmlu\n\tZ2hhbSA8a2llcmFuLmJpbmdoYW1AaWRlYXNvbmJvYXJkLmNvbT6JAlcEEwEKAEECGwMFCwkI\n\tBwIGFQgJCgsCBBYCAwECHgECF4ACGQEWIQSQLdeYP70o/eNy1HqhHkZyEKRh/QUCXWTtygUJ\n\tCyJXZAAKCRChHkZyEKRh/f8dEACTDsbLN2nioNZMwyLuQRUAFcXNolDX48xcUXsWS2QjxaPm\n\tVsJx8Uy8aYkS85mdPBh0C83OovQR/OVbr8AxhGvYqBs3nQvbWuTl/+4od7DfK2VZOoKBAu5S\n\tQK2FYuUcikDqYcFWJ8DQnubxfE8dvzojHEkXw0sA4igINHDDFX3HJGZtLio+WpEFQtCbfTAG\n\tYZslasz1YZRbwEdSsmO3/kqy5eMnczlm8a21A3fKUo3g8oAZEFM+f4DUNzqIltg31OAB/kZS\n\tenKZQ/SWC8PmLg/ZXBrReYakxXtkP6w3FwMlzOlhGxqhIRNiAJfXJBaRhuUWzPOpEDE9q5YJ\n\tBmqQL2WJm1VSNNVxbXJHpaWMH1sA2R00vmvRrPXGwyIO0IPYeUYQa3gsy6k+En/aMQJd27dp\n\taScf9am9PFICPY5T4ppneeJLif2lyLojo0mcHOV+uyrds9XkLpp14GfTkeKPdPMrLLTsHRfH\n\tfA4I4OBpRrEPiGIZB/0im98MkGY/Mu6qxeZmYLCcgD6qz4idOvfgVOrNh+aA8HzIVR+RMW8H\n\tQGBN9f0E3kfwxuhl3omo6V7lDw8XOdmuWZNC9zPq1UfryVHANYbLGz9KJ4Aw6M+OgBC2JpkD\n\thXMdHUkC+d20dwXrwHTlrJi1YNp6rBc+xald3wsUPOZ5z8moTHUX/uPA/qhGsbkCDQRWBP1m\n\tARAAzijkb+Sau4hAncr1JjOY+KyFEdUNxRy+hqTJdJfaYihxyaj0Ee0P0zEi35CbE6lgU0Uz\n\ttih9fiUbSV3wfsWqg1Ut3/5rTKu7kLFp15kF7eqvV4uezXRD3Qu4yjv/rMmEJbbD4cTvGCYI\n\td6MDC417f7vK3hCbCVIZSp3GXxyC1LU+UQr3fFcOyCwmP9vDUR9JV0BSqHHxRDdpUXE26Dk6\n\tmhf0V1YkspE5St814ETXpEus2urZE5yJIUROlWPIL+hm3NEWfAP06vsQUyLvr/GtbOT79vXl\n\tEn1aulcYyu20dRRxhkQ6iILaURcxIAVJJKPi8dsoMnS8pB0QW12AHWuirPF0g6DiuUfPmrA5\n\tPKe56IGlpkjc8cO51lIxHkWTpCMWigRdPDexKX+Sb+W9QWK/0JjIc4t3KBaiG8O4yRX8ml2R\n\t+rxfAVKM6V769P/hWoRGdgUMgYHFpHGSgEt80OKK5HeUPy2cngDUXzwrqiM5Sz6Od0qw5pCk\n\tNlXqI0W/who0iSVM+8+RmyY0OEkxEcci7rRLsGnM15B5PjLJjh1f2ULYkv8s4SnDwMZ/kE04\n\t/UqCMK/KnX8pwXEMCjz0h6qWNpGwJ0/tYIgQJZh6bqkvBrDogAvuhf60Sogw+mH8b+PBlx1L\n\toeTK396wc+4c3BfiC6pNtUS5GpsPMMjYMk7kVvEAEQEAAYkCPAQYAQoAJgIbDBYhBJAt15g/\n\tvSj943LUeqEeRnIQpGH9BQJdizzIBQkLSKZiAAoJEKEeRnIQpGH9eYgQAJpjaWNgqNOnMTmD\n\tMJggbwjIotypzIXfhHNCeTkG7+qCDlSaBPclcPGYrTwCt0YWPU2TgGgJrVhYT20ierN8LUvj\n\t6qOPTd+Uk7NFzL65qkh80ZKNBFddx1AabQpSVQKbdcLb8OFs85kuSvFdgqZwgxA1vl4TFhNz\n\tPZ79NAmXLackAx3sOVFhk4WQaKRshCB7cSl+RIng5S/ThOBlwNlcKG7j7W2MC06BlTbdEkUp\n\tECzuuRBv8wX4OQl+hbWbB/VKIx5HKlLu1eypen/5lNVzSqMMIYkkZcjV2SWQyUGxSwq0O/sx\n\tS0A8/atCHUXOboUsn54qdxrVDaK+6jIAuo8JiRWctP16KjzUM7MO0/+4zllM8EY57rXrj48j\n\tsbEYX0YQnzaj+jO6kJtoZsIaYR7rMMq9aUAjyiaEZpmP1qF/2sYenDx0Fg2BSlLvLvXM0vU8\n\tpQk3kgDu7kb/7PRYrZvBsr21EIQoIjXbZxDz/o7z95frkP71EaICttZ6k9q5oxxA5WC6sTXc\n\tMW8zs8avFNuA9VpXt0YupJd2ijtZy2mpZNG02fFVXhIn4G807G7+9mhuC4XG5rKlBBUXTvPU\n\tAfYnB4JBDLmLzBFavQfvonSfbitgXwCG3vS+9HEwAjU30Bar1PEOmIbiAoMzuKeRm2LVpmq4\n\tWZw01QYHU/GUV/zHJSFk","Organization":"Ideas on Board","Message-ID":"<734c408b-2e5b-6ee4-af07-f9c45416abce@ideasonboard.com>","Date":"Fri, 23 Oct 2020 09:39:35 +0100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101\n\tThunderbird/68.10.0","MIME-Version":"1.0","In-Reply-To":"<20201023035906.GA4113@pendragon.ideasonboard.com>","Content-Language":"en-GB","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Reply-To":"kieran.bingham@ideasonboard.com","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13482,"web_url":"https://patchwork.libcamera.org/comment/13482/","msgid":"","date":"2020-10-26T07:29:20","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":63,"url":"https://patchwork.libcamera.org/api/people/63/","name":"Hirokazu Honda","email":"hiroh@chromium.org"},"content":"Hi Laurent and Kieran,\n\nOn Fri, Oct 23, 2020 at 5:39 PM Kieran Bingham\n wrote:\n>\n> Hi Hiro-san, Laurent,\n>\n> On 23/10/2020 04:59, Laurent Pinchart wrote:\n> > Hi Hiro-san,\n> >\n> > On Fri, Oct 23, 2020 at 09:37:09AM +0900, Hirokazu Honda wrote:\n> >> On Thu, Oct 22, 2020 at 11:00 PM Laurent Pinchart wrote:\n> >>> On Thu, Oct 22, 2020 at 08:42:01PM +0900, Hirokazu Honda wrote:\n> >>>> Hi Umang,\n> >>>> Thanks for the work. I couldn't have time to review these today and\n> >>>> will review tomorrow.\n> >>>> In my understanding, the thumbnail class is basically to scale frames\n> >>>> to the given destination.\n> >>>> Can we generalize it as PostProcessor interface so that we will be\n> >>>> able to make use of it for down-scaling regardless of whether jpeg\n> >>>> encoding is required.\n> >>>> It will be necessary when Android HAL adaptation layer has to output\n> >>>> multiple different NV12 streams while the native camera can produce a\n> >>>> single NV12 stream simultaneously.\n> >>>\n> >>> One point to consider is that the thumbnailer is a down-scaler using a\n> >>> nearest-neighbour algorithm. It's good enough for thumbnails, but will\n> >>> lead to awful quality when scaling streams. For that, I think we need a\n> >>> better scaler implementation, and we could then replace the thumbnailer\n> >>> class with it.\n> >>\n> >> We can select nearest-neighbor and bilinear algorithms in\n> >> libyuv::NV12Scale with the filtering mode argument[1].\n> >> We may want to specify the scale algorithm on a class construction.\n> >> I would use libyuv here for better performance and less buggy (+ less\n> >> lines to be reviewed).\n> >> What do you'all think?\n> >\n> > We have considered libyuv early on, and I think it's an interesting\n> > option moving forward. The reason we have decided to go for a minimal\n> > scaler implementation to create JPEG thumbnails is to avoid adding an\n> > external dependency that isn't packaged by distributions. This allows us\n> > to finalize JPEG thumbnail support without delay, and decide on the\n> > longer term solution separately.\n> >\n> > It would be very nice if Google could work with distributions to package\n> > libyuv. I'd like to avoid including a copy of libyuv in libcamera.\n>\n> Lack of packaging is a big factor here. If it's not available we can't\n> use it ;-)\n>\n\nThanks. That is fair enough.\n\n> However I looked into this briefly yesterday, - we could use a meson\n> wrap. But I don't know how that will affect us distributing packages, as\n> then we also have to distribute libyuv.\n>\n> Or perhaps the meson wrap would link statically.\n>\n\nCan we download libyuv somewhere in libcamera, e.g. //third_party,\nduring meson config if it is necessary?\nThe code under //third_party is ignored by .gitignore, i.e., is not a\npart of libcamera tree.\nWe may want to maintain a commit hash to be checked out in the libcamera tree.\n\nBest Regards,\n-Hiro\n\n>\n> > One last comment to provide all the design background. We have also\n> > considered creating a format conversion library in libcamera, to gather\n> > code from various projects that exist out there. There's format\n> > conversion in libv4l2, in raw2rgbpnm, in projects such as ffmpeg, and of\n> > course in libyuv. I would be great if that could be consolidated and\n> > packaged, but that's a big task in itself.\n>\n> And that's the bigger picture. This should be consolidated. And that\n> consolidated library should support more than yuv ;-)\n>\n> Given it would be a software format convertor, and scaler, and ideally\n> with hardware acceleration support where possible, I think that quickly\n> becomes libisp - and the feature creep will ... creep in ;-)\n>\n>\n>\n> >\n> >> [1] https://chromium.googlesource.com/libyuv/libyuv/+/refs/heads/master/include/libyuv/scale.h#148\n> >>\n> >>>> On Wed, Oct 21, 2020 at 8:09 PM Kieran Bingham wrote:\n> >>>>> On 21/10/2020 12:03, Umang Jain wrote:\n> >>>>>> On 10/21/20 4:25 PM, Kieran Bingham wrote:\n> >>>>>>> On 21/10/2020 11:51, Umang Jain wrote:\n> >>>>>>>> On 10/21/20 2:49 PM, Kieran Bingham wrote:\n> >>>>>>>>> On 21/10/2020 09:08, Umang Jain wrote:\n> >>>>>>>>>> Add a basic image thumbnailer for NV12 frames being captured.\n> >>>>>>>>>> It shall generate a thumbnail image to be embedded as a part of\n> >>>>>>>>>> EXIF metadata of the frame. The output of the thumbnail will still\n> >>>>>>>>>> be NV12.\n> >>>>>>>>>>\n> >>>>>>>>>> Signed-off-by: Umang Jain \n> >>>>>>>>>> ---\n> >>>>>>>>>> src/android/jpeg/post_processor_jpeg.cpp | 39 +++++++++\n> >>>>>>>>>> src/android/jpeg/post_processor_jpeg.h | 3 +\n> >>>>>>>>>> src/android/jpeg/thumbnailer.cpp | 100 +++++++++++++++++++++++\n> >>>>>>>>>> src/android/jpeg/thumbnailer.h | 40 +++++++++\n> >>>>>>>>>> src/android/meson.build | 1 +\n> >>>>>>>>>> 5 files changed, 183 insertions(+)\n> >>>>>>>>>> create mode 100644 src/android/jpeg/thumbnailer.cpp\n> >>>>>>>>>> create mode 100644 src/android/jpeg/thumbnailer.h\n> >>>>>>>>>>\n> >>>>>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp\n> >>>>>>>>>> b/src/android/jpeg/post_processor_jpeg.cpp\n> >>>>>>>>>> index 9d452b7..f5f1f78 100644\n> >>>>>>>>>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n> >>>>>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n> >>>>>>>>>> @@ -11,6 +11,7 @@\n> >>>>>>>>>> #include \"../camera_metadata.h\"\n> >>>>>>>>>> #include \"encoder_libjpeg.h\"\n> >>>>>>>>>> #include \"exif.h\"\n> >>>>>>>>>> +#include \"thumbnailer.h\"\n> >>>>>>>>>> #include \n> >>>>>>>>>> @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const StreamConfiguration &inCfg,\n> >>>>>>>>>> return encoder_->configure(inCfg);\n> >>>>>>>>>> }\n> >>>>>>>>>> +void PostProcessorJpeg::generateThumbnail(const libcamera::Span &source,\n> >>>>>>>>>> + std::vector &thumbnail)\n> >>>>>>>>>> +{\n> >>>>>>>>>> + libcamera::Span destination;\n> >>>>>>>>>> + Thumbnailer thumbnailer;\n> >>>>>>>>>> +\n> >>>>>>>>>> + thumbnailer.configure(streamSize_, formats::NV12);\n> >>>>>>>>>> + libcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n> >>>>>>>>>> + thumbnailer.scaleBuffer(source, thumbnail);\n> >>>>>>>>>> +\n> >>>>>>>>>> + if (thumbnail.data()) {\n> >>>>>>>>>> + StreamConfiguration thumbnailCfg;\n> >>>>>>>>>> +\n> >>>>>>>>>> + std::unique_ptr encoder =\n> >>>>>>>>>> + std::make_unique();\n> >>>>>>>>>> +\n> >>>>>>>>>> + thumbnailCfg.pixelFormat = formats::NV12;\n> >>>>>>>>>> + thumbnailCfg.size = targetSize;\n> >>>>>>>>>> + encoder->configure(thumbnailCfg);\n> >>>>>>>>>\n> >>>>>>>>> thumbnail.capacity() might be quite low here.\n> >>>>>>>>> We need to make sure the vector is big enough at this point, you might\n> >>>>>>>>> need to do something like:\n> >>>>>>>>>\n> >>>>>>>>> thumbnail.resize(targetSize.width * targetSize.height * 4);\n> >>>>>>>>\n> >>>>>>>> I am not sure I follow. This is compressing the thumbnail part right? So\n> >>>>>>>> thumbnail is the \"source\" for the encoder here (Please refer to\n> >>>>>>>> ->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n> >>>>>>>\n> >>>>>>> Ugh, sorry - tired eyes indeed. Yes, please ignore those comments.\n> >>>>>>>\n> >>>>>>>>> Really we should obtain that size from the encoder. I thought we had a\n> >>>>>>>>> helper to do that already, but it seems we don't.\n> >>>>>>>>> We could/should add Encoder::maxOutput(); which would ask the encoder\n> >>>>>>>>> \"For your given configuration, what is the maximum number of bytes you\n> >>>>>>>>> might output\".\n> >>>>>>>>>\n> >>>>>>>>> Then of course we'd do:\n> >>>>>>>>> thumbnail.resize(encoder.maxOutput());\n> >>>>>>>>>\n> >>>>>>>>>\n> >>>>>>>>>> + int jpeg_size = encoder->encode({ thumbnail.data(), thumbnail.capacity() },\n> >>>>>>>>>> + destination, { });\n> >>>>>>>>\n> >>>>>>>> As said above, thumbnail is the source here. The compressed thumbnail\n> >>>>>>>> output is carried in destination. And, destination is a local span<>\n> >>>>>>>> here. Does it makes sense?\n> >>>>>>>\n> >>>>>>> Yes, I had mixed up source and destination I'm sorry.\n> >>>>>>>\n> >>>>>>> So who/where is allocating the destination?\n> >>>>>>\n> >>>>>> Ah, that's a good question and I think I just found a bug. I currently\n> >>>>>> pass on a locally allocated Span<> 'destination' to the encoder. But\n> >>>>>> there is no explicit allocation here. I think I need to point / create\n> >>>>>> the Span with the output size / bytes and then pass to the encode().\n> >>>>>\n> >>>>> :-) Ok - then please apply all of my earlier comments to that part\n> >>>>> instead. ;-)\n> >>>>>\n> >>>>> We certainly need somewhere to store the compressed image, to be able to\n> >>>>> pass it into the exif :-D\n> >>>>>\n> >>>>>>>>>> + LOG(JPEG, Info) << \"Thumbnail compress returned \"\n> >>>>>>>>>> + << jpeg_size << \" bytes\";\n> >>>>>>>>>\n> >>>>>>>>> And I presume we could then do an:\n> >>>>>>>>> thumbnail.resize(jpeg_size);\n> >>>>>>>>>\n> >>>>>>>>> here to update the thumbnail with the correct size. I'd be weary of the\n> >>>>>>>>> resize operations doing lots of re-allocations though, so perhaps we\n> >>>>>>>>> want to minimize that. But lets get to something that works first\n> >>>>>>>>> before worrying about optimising.\n> >>>>>>>>>\n> >>>>>>>>>> + }\n> >>>>>>>>>> +}\n> >>>>>>>>>> +\n> >>>>>>>>>> int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n> >>>>>>>>>> const libcamera::Span &destination,\n> >>>>>>>>>> CameraMetadata *metadata)\n> >>>>>>>>>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const\n> >>>>>>>>>> libcamera::FrameBuffer *source,\n> >>>>>>>>>> return jpeg_size;\n> >>>>>>>>>> }\n> >>>>>>>>>> + std::vector thumbnail;\n> >>>>>>>>>\n> >>>>>>>>> You need to resize this somewhere.\n> >>>>>>>>> Edit: Now seen a better place above ;-)\n> >>>>>>>>\n> >>>>>>>> The thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\n> >>>>>>>> We just pass in to the thumbnailer, while keeping its ownership in\n> >>>>>>>> PostProcessorJpeg.\n> >>>>>>>>\n> >>>>>>>>>> + generateThumbnail(destination, thumbnail);\n> >>>>>>>>>> + /*\n> >>>>>>>>>> + * \\todo: Write the compressed thumbnail to a file for inspection.\n> >>>>>>>>>> + * (I) Check if we can still write the thumbnail to EXIF here.\n> >>>>>>>>>> + * If not, we might need to move the thumbnailer logic to encoder.\n> >>>>>>>>>> + * And if we do that, first we need to make sure we get can\n> >>>>>>>>>> + * compressed data written to destination first before calling\n> >>>>>>>>>> + * jpeg_finish_compress operation somehow. Thumbnailing will\n> >>>>>>>>>> + * only occur if we have compressed data available first.\n> >>>>>>>>>> + */\n> >>>>>>>>>> +\n> >>>>>>>>>> /*\n> >>>>>>>>>> * Fill in the JPEG blob header.\n> >>>>>>>>>> *\n> >>>>>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.h\n> >>>>>>>>>> b/src/android/jpeg/post_processor_jpeg.h\n> >>>>>>>>>> index 62c8650..05601ee 100644\n> >>>>>>>>>> --- a/src/android/jpeg/post_processor_jpeg.h\n> >>>>>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.h\n> >>>>>>>>>> @@ -28,6 +28,9 @@ public:\n> >>>>>>>>>> CameraMetadata *metadata) override;\n> >>>>>>>>>> private:\n> >>>>>>>>>> + void generateThumbnail(const libcamera::Span &source,\n> >>>>>>>>>> + std::vector &thumbnail);\n> >>>>>>>>>> +\n> >>>>>>>>>> CameraDevice *cameraDevice_;\n> >>>>>>>>>> std::unique_ptr encoder_;\n> >>>>>>>>>> libcamera::Size streamSize_;\n> >>>>>>>>>> diff --git a/src/android/jpeg/thumbnailer.cpp\n> >>>>>>>>>> b/src/android/jpeg/thumbnailer.cpp\n> >>>>>>>>>> new file mode 100644\n> >>>>>>>>>> index 0000000..3163576\n> >>>>>>>>>> --- /dev/null\n> >>>>>>>>>> +++ b/src/android/jpeg/thumbnailer.cpp\n> >>>>>>>>>> @@ -0,0 +1,100 @@\n> >>>>>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> >>>>>>>>>> +/*\n> >>>>>>>>>> + * Copyright (C) 2020, Google Inc.\n> >>>>>>>>>> + *\n> >>>>>>>>>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n> >>>>>>>>>> + */\n> >>>>>>>>>> +\n> >>>>>>>>>> +#include \"thumbnailer.h\"\n> >>>>>>>>>> +\n> >>>>>>>>>> +#include \n> >>>>>>>>>> +\n> >>>>>>>>>> +#include \"libcamera/internal/file.h\"\n> >>>>>>>>>> +#include \"libcamera/internal/log.h\"\n> >>>>>>>>>> +\n> >>>>>>>>>> +using namespace libcamera;\n> >>>>>>>>>> +\n> >>>>>>>>>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n> >>>>>>>>>> +\n> >>>>>>>>>> +Thumbnailer::Thumbnailer()\n> >>>>>>>>>> + : validConfiguration_(false)\n> >>>>>>>>>> +{\n> >>>>>>>>>> +}\n> >>>>>>>>>> +\n> >>>>>>>>>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat pixelFormat)\n> >>>>>>>>>> +{\n> >>>>>>>>>> + sourceSize_ = sourceSize;\n> >>>>>>>>>> + pixelFormat_ = pixelFormat;\n> >>>>>>>>>> +\n> >>>>>>>>>> + if (pixelFormat_ != formats::NV12) {\n> >>>>>>>>>> + LOG (Thumbnailer, Error) << \"Failed to configure: Pixel Format \"\n> >>>>>>>>>> + << pixelFormat_.toString() << \" unsupported.\";\n> >>>>>>>>>> + return;\n> >>>>>>>>>> + }\n> >>>>>>>>>> +\n> >>>>>>>>>> + validConfiguration_ = true;\n> >>>>>>>>>> +}\n> >>>>>>>>>> +\n> >>>>>>>>>> +/*\n> >>>>>>>>>> + * The Exif specification recommends the width of the thumbnail to be a\n> >>>>>>>>>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding height\n> >>>>>>>>>> + * keeping the aspect ratio same as of the source.\n> >>>>>>>>>> + */\n> >>>>>>>>>> +Size Thumbnailer::computeThumbnailSize()\n> >>>>>>>>>> +{\n> >>>>>>>>>> + unsigned int targetHeight;\n> >>>>>>>>>> + unsigned int targetWidth = 160;\n> >>>>>>>>>> +\n> >>>>>>>>>> + targetHeight = targetWidth * sourceSize_.height / sourceSize_.width;\n> >>>>>>>>>> +\n> >>>>>>>>>> + if (targetHeight & 1)\n> >>>>>>>>>> + targetHeight++;\n> >>>>>>>>>> +\n> >>>>>>>>>> + return Size(targetWidth, targetHeight);\n> >>>>>>>>>> +}\n> >>>>>>>>>> +\n> >>>>>>>>>> +void\n> >>>>>>>>>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n> >>>>>>>>>> + std::vector &destination)\n> >>>>>>>>>> +{\n> >>>>>>>>>> + if (!validConfiguration_) {\n> >>>>>>>>>> + LOG(Thumbnailer, Error) << \"config is unconfigured or invalid.\";\n> >>>>>>>>>> + return;\n> >>>>>>>>>> + }\n> >>>>>>>>>> +\n> >>>>>>>>>> + targetSize_ = computeThumbnailSize();\n> >>>>>>>>>> +\n> >>>>>>>>>> + const unsigned int sw = sourceSize_.width;\n> >>>>>>>>>> + const unsigned int sh = sourceSize_.height;\n> >>>>>>>>>> + const unsigned int tw = targetSize_.width;\n> >>>>>>>>>> + const unsigned int th = targetSize_.height;\n> >>>>>>>>>> +\n> >>>>>>>>>> + /* Image scaling block implementing nearest-neighbour algorithm. */\n> >>>>>>>>>> + unsigned char *src = static_cast(source.data());\n> >>>>>>>>>> + unsigned char *src_c = src + sh * sw;\n> >>>>>>>>>> + unsigned char *src_cb, *src_cr;\n> >>>>>>>>>> +\n> >>>>>>>>>> + size_t dstSize = (th * tw) + ((th/2) * tw);\n> >>>>>>>>>> + destination.reserve(dstSize);\n> >>>>>>>>>> + unsigned char *dst = destination.data();\n> >>>>>>>>>> + unsigned char *dst_c = dst + th * tw;\n> >>>>>>>>>> +\n> >>>>>>>>>> + for (unsigned int y = 0; y < th; y+=2) {\n> >>>>>>>>>> + unsigned int sourceY = (sh*y + th/2) / th;\n> >>>>>>>>>> +\n> >>>>>>>>>> + src_cb = src_c + (sourceY/2) * sw + 0;\n> >>>>>>>>>> + src_cr = src_c + (sourceY/2) * sw + 1;\n> >>>>>>>>>> +\n> >>>>>>>>>> + for (unsigned int x = 0; x < tw; x+=2) {\n> >>>>>>>>>> + unsigned int sourceX = (sw*x + tw/2) / tw;\n> >>>>>>>>>> +\n> >>>>>>>>>> + dst[y * tw + x] = src[sw * sourceY + sourceX];\n> >>>>>>>>>> + dst[(y+1) * tw + x] = src[sw * (sourceY+1) + sourceX];\n> >>>>>>>>>> + dst[y * tw + (x+1)] = src[sw * sourceY + (sourceX+1)];\n> >>>>>>>>>> + dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) + (sourceX+1)];\n> >>>>>>>>>> +\n> >>>>>>>>>> + dst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n> >>>>>>>>>> + dst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n> >>>>>>>>>> + }\n> >>>>>>>>>> + }\n> >>>>>>>>>> +}\n> >>>>>>>>>> diff --git a/src/android/jpeg/thumbnailer.h\n> >>>>>>>>>> b/src/android/jpeg/thumbnailer.h\n> >>>>>>>>>> new file mode 100644\n> >>>>>>>>>> index 0000000..bab9855\n> >>>>>>>>>> --- /dev/null\n> >>>>>>>>>> +++ b/src/android/jpeg/thumbnailer.h\n> >>>>>>>>>> @@ -0,0 +1,40 @@\n> >>>>>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> >>>>>>>>>> +/*\n> >>>>>>>>>> + * Copyright (C) 2020, Google Inc.\n> >>>>>>>>>> + *\n> >>>>>>>>>> + * thumbnailer.h - Basic image thumbnailer from NV12\n> >>>>>>>>>> + */\n> >>>>>>>>>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n> >>>>>>>>>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n> >>>>>>>>>> +\n> >>>>>>>>>> +#include \n> >>>>>>>>>> +\n> >>>>>>>>>> +#include \"libcamera/internal/buffer.h\"\n> >>>>>>>>>> +#include \"libcamera/internal/formats.h\"\n> >>>>>>>>>> +\n> >>>>>>>>>> +class Thumbnailer\n> >>>>>>>>>> +{\n> >>>>>>>>>> +public:\n> >>>>>>>>>> + Thumbnailer();\n> >>>>>>>>>> +\n> >>>>>>>>>> + void configure(const libcamera::Size &sourceSize,\n> >>>>>>>>>> + libcamera::PixelFormat pixelFormat);\n> >>>>>>>>>> +\n> >>>>>>>>>> + /*\n> >>>>>>>>>> + * \\todo: Discuss if we can return targetSize_ via configure() or\n> >>>>>>>>>> + * scaleBuffer(). We need targetSize_ to re-encode the scaled\n> >>>>>>>>>> buffer\n> >>>>>>>>>> + * via encoder in PostProcssorJpeg::writeThumbnail().\n> >>>>>>>>>> + */\n> >>>>>>>>>> + libcamera::Size computeThumbnailSize();\n> >>>>>>>>>> + void scaleBuffer(const libcamera::Span &source,\n> >>>>>>>>>> + std::vector &dest);\n> >>>>>>>>>> +\n> >>>>>>>>>> +private:\n> >>>>>>>>>> + libcamera::PixelFormat pixelFormat_;\n> >>>>>>>>>> + libcamera::Size sourceSize_;\n> >>>>>>>>>> + libcamera::Size targetSize_;\n> >>>>>>>>>> +\n> >>>>>>>>>> + bool validConfiguration_;\n> >>>>>>>>>> +};\n> >>>>>>>>>> +\n> >>>>>>>>>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n> >>>>>>>>>> diff --git a/src/android/meson.build b/src/android/meson.build\n> >>>>>>>>>> index 5a01bea..3905e2f 100644\n> >>>>>>>>>> --- a/src/android/meson.build\n> >>>>>>>>>> +++ b/src/android/meson.build\n> >>>>>>>>>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n> >>>>>>>>>> 'jpeg/encoder_libjpeg.cpp',\n> >>>>>>>>>> 'jpeg/exif.cpp',\n> >>>>>>>>>> 'jpeg/post_processor_jpeg.cpp',\n> >>>>>>>>>> + 'jpeg/thumbnailer.cpp',\n> >>>>>>>>>> ])\n> >>>>>>>>>> android_camera_metadata_sources = files([\n> >>>>>>>>>>\n> >\n>\n> --\n> Regards\n> --\n> Kieran","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C35CAC3B5C\n\tfor ;\n\tMon, 26 Oct 2020 07:29:33 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 4733A61E01;\n\tMon, 26 Oct 2020 08:29:33 +0100 (CET)","from mail-ed1-x536.google.com (mail-ed1-x536.google.com\n\t[IPv6:2a00:1450:4864:20::536])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 622196052D\n\tfor ;\n\tMon, 26 Oct 2020 08:29:32 +0100 (CET)","by mail-ed1-x536.google.com with SMTP id t20so8184009edr.11\n\tfor ;\n\tMon, 26 Oct 2020 00:29:32 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=chromium.org header.i=@chromium.org\n\theader.b=\"FzZcMkNk\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org;\n\ts=google; \n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=xuSHgx/jhYkvOMcwMlcPar4pR4s4X3WiMmvidjckgFE=;\n\tb=FzZcMkNkBEfssvh6Czr/SOC7Fwvelu7i4J5wXX9O1o5oSOxdQuvg9U9iS4Sh7Qu1Lp\n\tLcT8MmspMLJ0DyDaETFmooCdQFu4FXqDIzU5ocW0W1LuqEFLhE7MYaYIE13o3wa1/zas\n\t5YqE2dGfImpUzyNpwh9VAwKwwgJj9SN9HENh8=","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=xuSHgx/jhYkvOMcwMlcPar4pR4s4X3WiMmvidjckgFE=;\n\tb=UT4als8u20jzZBHGhVyAfECu0ufLHqpg1aI8w4lj0jyUCCnpGF9S9opJ2D3JTZ7ea7\n\t09Ukb8GAkr07lcR5PXhkadQpCY4RurVkcOL4wn+7Dgd32TBYSql4jdWJWY2vVZ8r4kP8\n\t/YPpfJBfzZUpnkN7o30ttJqrNPuC+OmgE8vu5nZbPnsEnTDF772ds4BTMmIMF/RRVECk\n\tHNeabHzXRgjxs9WycqXV/nvfDT89qj0B8C9xjS+UU9OUxnmhZI1K3EjgQG+ukKW1Juep\n\txp5FARcVv5RKWxOIgDsr7ti2CV5cwapGNnYy43jNUiunUpQITUulBqW9EMJiFO7CEzPM\n\tDTag==","X-Gm-Message-State":"AOAM530GkZUeRcX6KuFsNp0qNWhaeYi1KSCgIsFi6rT5HCTni1Cc70in\n\tYAgKO0gXxrwXsp6Z5oDzclpJYKaMiRsRfh/PIvQETypUqwg=","X-Google-Smtp-Source":"ABdhPJzzvqYlybfnzvEbkbwX8Hfv9ZNaKMp6B+Q63lq3JpLRu9z1XsO5yfS30i4EjLmykbyullaBctUCLziKl8JFNjU=","X-Received":"by 2002:aa7:ccc8:: with SMTP id\n\ty8mr14841160edt.325.1603697371525; \n\tMon, 26 Oct 2020 00:29:31 -0700 (PDT)","MIME-Version":"1.0","References":"<20201021080806.46636-1-email@uajain.com>\n\t<20201021080806.46636-3-email@uajain.com>\n\t\n\t\n\t<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>\n\t<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>\n\t<9ca3b492-8468-e9cf-b103-b4350be472b9@ideasonboard.com>\n\t\n\t<20201022135914.GA3938@pendragon.ideasonboard.com>\n\t\n\t<20201023035906.GA4113@pendragon.ideasonboard.com>\n\t<734c408b-2e5b-6ee4-af07-f9c45416abce@ideasonboard.com>","In-Reply-To":"<734c408b-2e5b-6ee4-af07-f9c45416abce@ideasonboard.com>","From":"Hirokazu Honda ","Date":"Mon, 26 Oct 2020 16:29:20 +0900","Message-ID":"","To":"Kieran Bingham ","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13499,"web_url":"https://patchwork.libcamera.org/comment/13499/","msgid":"<20201027010400.GO3756@pendragon.ideasonboard.com>","date":"2020-10-27T01:04:00","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Hiro-san,\n\nOn Mon, Oct 26, 2020 at 04:29:20PM +0900, Hirokazu Honda wrote:\n> On Fri, Oct 23, 2020 at 5:39 PM Kieran Bingham wrote:\n> > On 23/10/2020 04:59, Laurent Pinchart wrote:\n> > > On Fri, Oct 23, 2020 at 09:37:09AM +0900, Hirokazu Honda wrote:\n> > >> On Thu, Oct 22, 2020 at 11:00 PM Laurent Pinchart wrote:\n> > >>> On Thu, Oct 22, 2020 at 08:42:01PM +0900, Hirokazu Honda wrote:\n> > >>>> Hi Umang,\n> > >>>> Thanks for the work. I couldn't have time to review these today and\n> > >>>> will review tomorrow.\n> > >>>> In my understanding, the thumbnail class is basically to scale frames\n> > >>>> to the given destination.\n> > >>>> Can we generalize it as PostProcessor interface so that we will be\n> > >>>> able to make use of it for down-scaling regardless of whether jpeg\n> > >>>> encoding is required.\n> > >>>> It will be necessary when Android HAL adaptation layer has to output\n> > >>>> multiple different NV12 streams while the native camera can produce a\n> > >>>> single NV12 stream simultaneously.\n> > >>>\n> > >>> One point to consider is that the thumbnailer is a down-scaler using a\n> > >>> nearest-neighbour algorithm. It's good enough for thumbnails, but will\n> > >>> lead to awful quality when scaling streams. For that, I think we need a\n> > >>> better scaler implementation, and we could then replace the thumbnailer\n> > >>> class with it.\n> > >>\n> > >> We can select nearest-neighbor and bilinear algorithms in\n> > >> libyuv::NV12Scale with the filtering mode argument[1].\n> > >> We may want to specify the scale algorithm on a class construction.\n> > >> I would use libyuv here for better performance and less buggy (+ less\n> > >> lines to be reviewed).\n> > >> What do you'all think?\n> > >\n> > > We have considered libyuv early on, and I think it's an interesting\n> > > option moving forward. The reason we have decided to go for a minimal\n> > > scaler implementation to create JPEG thumbnails is to avoid adding an\n> > > external dependency that isn't packaged by distributions. This allows us\n> > > to finalize JPEG thumbnail support without delay, and decide on the\n> > > longer term solution separately.\n> > >\n> > > It would be very nice if Google could work with distributions to package\n> > > libyuv. I'd like to avoid including a copy of libyuv in libcamera.\n> >\n> > Lack of packaging is a big factor here. If it's not available we can't\n> > use it ;-)\n> \n> Thanks. That is fair enough.\n> \n> > However I looked into this briefly yesterday, - we could use a meson\n> > wrap. But I don't know how that will affect us distributing packages, as\n> > then we also have to distribute libyuv.\n> >\n> > Or perhaps the meson wrap would link statically.\n> \n> Can we download libyuv somewhere in libcamera, e.g. //third_party,\n> during meson config if it is necessary?\n> The code under //third_party is ignored by .gitignore, i.e., is not a\n> part of libcamera tree.\n> We may want to maintain a commit hash to be checked out in the libcamera tree.\n\nAs Kieran mentioned, we could use a meson wrap, and link statically to\nlibyuv. If we rely on a particular commit of libyuv then I think we\ncould as well copy the source code in the libcamera repository, to avoid\ndepending on an internet connection at meson config time. I'm however\nnot sure what issues we would encounter on the way. libyuv is integrated\nwith the Chromium code base, but looking, for instance, at the buildroot\npackage, it seems fairly easy to compile it standalone. We may need to\nadd support for the meson build system though.\n\nAs a conclusion, I think we'll need to figure out how we can use libyuv,\nbut not as a prerequisite for the JPEG thumbnail work. The best option\nwould be if Google maintained libyuv as a standalone library and worked\non getting it packaged in distributions :-)\n\n> > > One last comment to provide all the design background. We have also\n> > > considered creating a format conversion library in libcamera, to gather\n> > > code from various projects that exist out there. There's format\n> > > conversion in libv4l2, in raw2rgbpnm, in projects such as ffmpeg, and of\n> > > course in libyuv. I would be great if that could be consolidated and\n> > > packaged, but that's a big task in itself.\n> >\n> > And that's the bigger picture. This should be consolidated. And that\n> > consolidated library should support more than yuv ;-)\n> >\n> > Given it would be a software format convertor, and scaler, and ideally\n> > with hardware acceleration support where possible, I think that quickly\n> > becomes libisp - and the feature creep will ... creep in ;-)\n> >\n> > >> [1] https://chromium.googlesource.com/libyuv/libyuv/+/refs/heads/master/include/libyuv/scale.h#148\n> > >>\n> > >>>> On Wed, Oct 21, 2020 at 8:09 PM Kieran Bingham wrote:\n> > >>>>> On 21/10/2020 12:03, Umang Jain wrote:\n> > >>>>>> On 10/21/20 4:25 PM, Kieran Bingham wrote:\n> > >>>>>>> On 21/10/2020 11:51, Umang Jain wrote:\n> > >>>>>>>> On 10/21/20 2:49 PM, Kieran Bingham wrote:\n> > >>>>>>>>> On 21/10/2020 09:08, Umang Jain wrote:\n> > >>>>>>>>>> Add a basic image thumbnailer for NV12 frames being captured.\n> > >>>>>>>>>> It shall generate a thumbnail image to be embedded as a part of\n> > >>>>>>>>>> EXIF metadata of the frame. The output of the thumbnail will still\n> > >>>>>>>>>> be NV12.\n> > >>>>>>>>>>\n> > >>>>>>>>>> Signed-off-by: Umang Jain \n> > >>>>>>>>>> ---\n> > >>>>>>>>>> src/android/jpeg/post_processor_jpeg.cpp | 39 +++++++++\n> > >>>>>>>>>> src/android/jpeg/post_processor_jpeg.h | 3 +\n> > >>>>>>>>>> src/android/jpeg/thumbnailer.cpp | 100 +++++++++++++++++++++++\n> > >>>>>>>>>> src/android/jpeg/thumbnailer.h | 40 +++++++++\n> > >>>>>>>>>> src/android/meson.build | 1 +\n> > >>>>>>>>>> 5 files changed, 183 insertions(+)\n> > >>>>>>>>>> create mode 100644 src/android/jpeg/thumbnailer.cpp\n> > >>>>>>>>>> create mode 100644 src/android/jpeg/thumbnailer.h\n> > >>>>>>>>>>\n> > >>>>>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp\n> > >>>>>>>>>> b/src/android/jpeg/post_processor_jpeg.cpp\n> > >>>>>>>>>> index 9d452b7..f5f1f78 100644\n> > >>>>>>>>>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n> > >>>>>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n> > >>>>>>>>>> @@ -11,6 +11,7 @@\n> > >>>>>>>>>> #include \"../camera_metadata.h\"\n> > >>>>>>>>>> #include \"encoder_libjpeg.h\"\n> > >>>>>>>>>> #include \"exif.h\"\n> > >>>>>>>>>> +#include \"thumbnailer.h\"\n> > >>>>>>>>>> #include \n> > >>>>>>>>>> @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const StreamConfiguration &inCfg,\n> > >>>>>>>>>> return encoder_->configure(inCfg);\n> > >>>>>>>>>> }\n> > >>>>>>>>>> +void PostProcessorJpeg::generateThumbnail(const libcamera::Span &source,\n> > >>>>>>>>>> + std::vector &thumbnail)\n> > >>>>>>>>>> +{\n> > >>>>>>>>>> + libcamera::Span destination;\n> > >>>>>>>>>> + Thumbnailer thumbnailer;\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + thumbnailer.configure(streamSize_, formats::NV12);\n> > >>>>>>>>>> + libcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n> > >>>>>>>>>> + thumbnailer.scaleBuffer(source, thumbnail);\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + if (thumbnail.data()) {\n> > >>>>>>>>>> + StreamConfiguration thumbnailCfg;\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + std::unique_ptr encoder =\n> > >>>>>>>>>> + std::make_unique();\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + thumbnailCfg.pixelFormat = formats::NV12;\n> > >>>>>>>>>> + thumbnailCfg.size = targetSize;\n> > >>>>>>>>>> + encoder->configure(thumbnailCfg);\n> > >>>>>>>>>\n> > >>>>>>>>> thumbnail.capacity() might be quite low here.\n> > >>>>>>>>> We need to make sure the vector is big enough at this point, you might\n> > >>>>>>>>> need to do something like:\n> > >>>>>>>>>\n> > >>>>>>>>> thumbnail.resize(targetSize.width * targetSize.height * 4);\n> > >>>>>>>>\n> > >>>>>>>> I am not sure I follow. This is compressing the thumbnail part right? So\n> > >>>>>>>> thumbnail is the \"source\" for the encoder here (Please refer to\n> > >>>>>>>> ->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n> > >>>>>>>\n> > >>>>>>> Ugh, sorry - tired eyes indeed. Yes, please ignore those comments.\n> > >>>>>>>\n> > >>>>>>>>> Really we should obtain that size from the encoder. I thought we had a\n> > >>>>>>>>> helper to do that already, but it seems we don't.\n> > >>>>>>>>> We could/should add Encoder::maxOutput(); which would ask the encoder\n> > >>>>>>>>> \"For your given configuration, what is the maximum number of bytes you\n> > >>>>>>>>> might output\".\n> > >>>>>>>>>\n> > >>>>>>>>> Then of course we'd do:\n> > >>>>>>>>> thumbnail.resize(encoder.maxOutput());\n> > >>>>>>>>>\n> > >>>>>>>>>\n> > >>>>>>>>>> + int jpeg_size = encoder->encode({ thumbnail.data(), thumbnail.capacity() },\n> > >>>>>>>>>> + destination, { });\n> > >>>>>>>>\n> > >>>>>>>> As said above, thumbnail is the source here. The compressed thumbnail\n> > >>>>>>>> output is carried in destination. And, destination is a local span<>\n> > >>>>>>>> here. Does it makes sense?\n> > >>>>>>>\n> > >>>>>>> Yes, I had mixed up source and destination I'm sorry.\n> > >>>>>>>\n> > >>>>>>> So who/where is allocating the destination?\n> > >>>>>>\n> > >>>>>> Ah, that's a good question and I think I just found a bug. I currently\n> > >>>>>> pass on a locally allocated Span<> 'destination' to the encoder. But\n> > >>>>>> there is no explicit allocation here. I think I need to point / create\n> > >>>>>> the Span with the output size / bytes and then pass to the encode().\n> > >>>>>\n> > >>>>> :-) Ok - then please apply all of my earlier comments to that part\n> > >>>>> instead. ;-)\n> > >>>>>\n> > >>>>> We certainly need somewhere to store the compressed image, to be able to\n> > >>>>> pass it into the exif :-D\n> > >>>>>\n> > >>>>>>>>>> + LOG(JPEG, Info) << \"Thumbnail compress returned \"\n> > >>>>>>>>>> + << jpeg_size << \" bytes\";\n> > >>>>>>>>>\n> > >>>>>>>>> And I presume we could then do an:\n> > >>>>>>>>> thumbnail.resize(jpeg_size);\n> > >>>>>>>>>\n> > >>>>>>>>> here to update the thumbnail with the correct size. I'd be weary of the\n> > >>>>>>>>> resize operations doing lots of re-allocations though, so perhaps we\n> > >>>>>>>>> want to minimize that. But lets get to something that works first\n> > >>>>>>>>> before worrying about optimising.\n> > >>>>>>>>>\n> > >>>>>>>>>> + }\n> > >>>>>>>>>> +}\n> > >>>>>>>>>> +\n> > >>>>>>>>>> int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n> > >>>>>>>>>> const libcamera::Span &destination,\n> > >>>>>>>>>> CameraMetadata *metadata)\n> > >>>>>>>>>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const\n> > >>>>>>>>>> libcamera::FrameBuffer *source,\n> > >>>>>>>>>> return jpeg_size;\n> > >>>>>>>>>> }\n> > >>>>>>>>>> + std::vector thumbnail;\n> > >>>>>>>>>\n> > >>>>>>>>> You need to resize this somewhere.\n> > >>>>>>>>> Edit: Now seen a better place above ;-)\n> > >>>>>>>>\n> > >>>>>>>> The thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\n> > >>>>>>>> We just pass in to the thumbnailer, while keeping its ownership in\n> > >>>>>>>> PostProcessorJpeg.\n> > >>>>>>>>\n> > >>>>>>>>>> + generateThumbnail(destination, thumbnail);\n> > >>>>>>>>>> + /*\n> > >>>>>>>>>> + * \\todo: Write the compressed thumbnail to a file for inspection.\n> > >>>>>>>>>> + * (I) Check if we can still write the thumbnail to EXIF here.\n> > >>>>>>>>>> + * If not, we might need to move the thumbnailer logic to encoder.\n> > >>>>>>>>>> + * And if we do that, first we need to make sure we get can\n> > >>>>>>>>>> + * compressed data written to destination first before calling\n> > >>>>>>>>>> + * jpeg_finish_compress operation somehow. Thumbnailing will\n> > >>>>>>>>>> + * only occur if we have compressed data available first.\n> > >>>>>>>>>> + */\n> > >>>>>>>>>> +\n> > >>>>>>>>>> /*\n> > >>>>>>>>>> * Fill in the JPEG blob header.\n> > >>>>>>>>>> *\n> > >>>>>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.h\n> > >>>>>>>>>> b/src/android/jpeg/post_processor_jpeg.h\n> > >>>>>>>>>> index 62c8650..05601ee 100644\n> > >>>>>>>>>> --- a/src/android/jpeg/post_processor_jpeg.h\n> > >>>>>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.h\n> > >>>>>>>>>> @@ -28,6 +28,9 @@ public:\n> > >>>>>>>>>> CameraMetadata *metadata) override;\n> > >>>>>>>>>> private:\n> > >>>>>>>>>> + void generateThumbnail(const libcamera::Span &source,\n> > >>>>>>>>>> + std::vector &thumbnail);\n> > >>>>>>>>>> +\n> > >>>>>>>>>> CameraDevice *cameraDevice_;\n> > >>>>>>>>>> std::unique_ptr encoder_;\n> > >>>>>>>>>> libcamera::Size streamSize_;\n> > >>>>>>>>>> diff --git a/src/android/jpeg/thumbnailer.cpp\n> > >>>>>>>>>> b/src/android/jpeg/thumbnailer.cpp\n> > >>>>>>>>>> new file mode 100644\n> > >>>>>>>>>> index 0000000..3163576\n> > >>>>>>>>>> --- /dev/null\n> > >>>>>>>>>> +++ b/src/android/jpeg/thumbnailer.cpp\n> > >>>>>>>>>> @@ -0,0 +1,100 @@\n> > >>>>>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > >>>>>>>>>> +/*\n> > >>>>>>>>>> + * Copyright (C) 2020, Google Inc.\n> > >>>>>>>>>> + *\n> > >>>>>>>>>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n> > >>>>>>>>>> + */\n> > >>>>>>>>>> +\n> > >>>>>>>>>> +#include \"thumbnailer.h\"\n> > >>>>>>>>>> +\n> > >>>>>>>>>> +#include \n> > >>>>>>>>>> +\n> > >>>>>>>>>> +#include \"libcamera/internal/file.h\"\n> > >>>>>>>>>> +#include \"libcamera/internal/log.h\"\n> > >>>>>>>>>> +\n> > >>>>>>>>>> +using namespace libcamera;\n> > >>>>>>>>>> +\n> > >>>>>>>>>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n> > >>>>>>>>>> +\n> > >>>>>>>>>> +Thumbnailer::Thumbnailer()\n> > >>>>>>>>>> + : validConfiguration_(false)\n> > >>>>>>>>>> +{\n> > >>>>>>>>>> +}\n> > >>>>>>>>>> +\n> > >>>>>>>>>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat pixelFormat)\n> > >>>>>>>>>> +{\n> > >>>>>>>>>> + sourceSize_ = sourceSize;\n> > >>>>>>>>>> + pixelFormat_ = pixelFormat;\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + if (pixelFormat_ != formats::NV12) {\n> > >>>>>>>>>> + LOG (Thumbnailer, Error) << \"Failed to configure: Pixel Format \"\n> > >>>>>>>>>> + << pixelFormat_.toString() << \" unsupported.\";\n> > >>>>>>>>>> + return;\n> > >>>>>>>>>> + }\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + validConfiguration_ = true;\n> > >>>>>>>>>> +}\n> > >>>>>>>>>> +\n> > >>>>>>>>>> +/*\n> > >>>>>>>>>> + * The Exif specification recommends the width of the thumbnail to be a\n> > >>>>>>>>>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding height\n> > >>>>>>>>>> + * keeping the aspect ratio same as of the source.\n> > >>>>>>>>>> + */\n> > >>>>>>>>>> +Size Thumbnailer::computeThumbnailSize()\n> > >>>>>>>>>> +{\n> > >>>>>>>>>> + unsigned int targetHeight;\n> > >>>>>>>>>> + unsigned int targetWidth = 160;\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + targetHeight = targetWidth * sourceSize_.height / sourceSize_.width;\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + if (targetHeight & 1)\n> > >>>>>>>>>> + targetHeight++;\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + return Size(targetWidth, targetHeight);\n> > >>>>>>>>>> +}\n> > >>>>>>>>>> +\n> > >>>>>>>>>> +void\n> > >>>>>>>>>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n> > >>>>>>>>>> + std::vector &destination)\n> > >>>>>>>>>> +{\n> > >>>>>>>>>> + if (!validConfiguration_) {\n> > >>>>>>>>>> + LOG(Thumbnailer, Error) << \"config is unconfigured or invalid.\";\n> > >>>>>>>>>> + return;\n> > >>>>>>>>>> + }\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + targetSize_ = computeThumbnailSize();\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + const unsigned int sw = sourceSize_.width;\n> > >>>>>>>>>> + const unsigned int sh = sourceSize_.height;\n> > >>>>>>>>>> + const unsigned int tw = targetSize_.width;\n> > >>>>>>>>>> + const unsigned int th = targetSize_.height;\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + /* Image scaling block implementing nearest-neighbour algorithm. */\n> > >>>>>>>>>> + unsigned char *src = static_cast(source.data());\n> > >>>>>>>>>> + unsigned char *src_c = src + sh * sw;\n> > >>>>>>>>>> + unsigned char *src_cb, *src_cr;\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + size_t dstSize = (th * tw) + ((th/2) * tw);\n> > >>>>>>>>>> + destination.reserve(dstSize);\n> > >>>>>>>>>> + unsigned char *dst = destination.data();\n> > >>>>>>>>>> + unsigned char *dst_c = dst + th * tw;\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + for (unsigned int y = 0; y < th; y+=2) {\n> > >>>>>>>>>> + unsigned int sourceY = (sh*y + th/2) / th;\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + src_cb = src_c + (sourceY/2) * sw + 0;\n> > >>>>>>>>>> + src_cr = src_c + (sourceY/2) * sw + 1;\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + for (unsigned int x = 0; x < tw; x+=2) {\n> > >>>>>>>>>> + unsigned int sourceX = (sw*x + tw/2) / tw;\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + dst[y * tw + x] = src[sw * sourceY + sourceX];\n> > >>>>>>>>>> + dst[(y+1) * tw + x] = src[sw * (sourceY+1) + sourceX];\n> > >>>>>>>>>> + dst[y * tw + (x+1)] = src[sw * sourceY + (sourceX+1)];\n> > >>>>>>>>>> + dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) + (sourceX+1)];\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + dst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n> > >>>>>>>>>> + dst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n> > >>>>>>>>>> + }\n> > >>>>>>>>>> + }\n> > >>>>>>>>>> +}\n> > >>>>>>>>>> diff --git a/src/android/jpeg/thumbnailer.h\n> > >>>>>>>>>> b/src/android/jpeg/thumbnailer.h\n> > >>>>>>>>>> new file mode 100644\n> > >>>>>>>>>> index 0000000..bab9855\n> > >>>>>>>>>> --- /dev/null\n> > >>>>>>>>>> +++ b/src/android/jpeg/thumbnailer.h\n> > >>>>>>>>>> @@ -0,0 +1,40 @@\n> > >>>>>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > >>>>>>>>>> +/*\n> > >>>>>>>>>> + * Copyright (C) 2020, Google Inc.\n> > >>>>>>>>>> + *\n> > >>>>>>>>>> + * thumbnailer.h - Basic image thumbnailer from NV12\n> > >>>>>>>>>> + */\n> > >>>>>>>>>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n> > >>>>>>>>>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n> > >>>>>>>>>> +\n> > >>>>>>>>>> +#include \n> > >>>>>>>>>> +\n> > >>>>>>>>>> +#include \"libcamera/internal/buffer.h\"\n> > >>>>>>>>>> +#include \"libcamera/internal/formats.h\"\n> > >>>>>>>>>> +\n> > >>>>>>>>>> +class Thumbnailer\n> > >>>>>>>>>> +{\n> > >>>>>>>>>> +public:\n> > >>>>>>>>>> + Thumbnailer();\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + void configure(const libcamera::Size &sourceSize,\n> > >>>>>>>>>> + libcamera::PixelFormat pixelFormat);\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + /*\n> > >>>>>>>>>> + * \\todo: Discuss if we can return targetSize_ via configure() or\n> > >>>>>>>>>> + * scaleBuffer(). We need targetSize_ to re-encode the scaled\n> > >>>>>>>>>> buffer\n> > >>>>>>>>>> + * via encoder in PostProcssorJpeg::writeThumbnail().\n> > >>>>>>>>>> + */\n> > >>>>>>>>>> + libcamera::Size computeThumbnailSize();\n> > >>>>>>>>>> + void scaleBuffer(const libcamera::Span &source,\n> > >>>>>>>>>> + std::vector &dest);\n> > >>>>>>>>>> +\n> > >>>>>>>>>> +private:\n> > >>>>>>>>>> + libcamera::PixelFormat pixelFormat_;\n> > >>>>>>>>>> + libcamera::Size sourceSize_;\n> > >>>>>>>>>> + libcamera::Size targetSize_;\n> > >>>>>>>>>> +\n> > >>>>>>>>>> + bool validConfiguration_;\n> > >>>>>>>>>> +};\n> > >>>>>>>>>> +\n> > >>>>>>>>>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n> > >>>>>>>>>> diff --git a/src/android/meson.build b/src/android/meson.build\n> > >>>>>>>>>> index 5a01bea..3905e2f 100644\n> > >>>>>>>>>> --- a/src/android/meson.build\n> > >>>>>>>>>> +++ b/src/android/meson.build\n> > >>>>>>>>>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n> > >>>>>>>>>> 'jpeg/encoder_libjpeg.cpp',\n> > >>>>>>>>>> 'jpeg/exif.cpp',\n> > >>>>>>>>>> 'jpeg/post_processor_jpeg.cpp',\n> > >>>>>>>>>> + 'jpeg/thumbnailer.cpp',\n> > >>>>>>>>>> ])\n> > >>>>>>>>>> android_camera_metadata_sources = files([\n> > >>>>>>>>>>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 1D573C3B5C\n\tfor ;\n\tTue, 27 Oct 2020 01:04:50 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 91ADE6206C;\n\tTue, 27 Oct 2020 02:04:49 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id D501E6034C\n\tfor ;\n\tTue, 27 Oct 2020 02:04:47 +0100 (CET)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 4575F81;\n\tTue, 27 Oct 2020 02:04:47 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"MpwCLBs0\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1603760687;\n\tbh=KoP8ylGNQqc+/1cKHbhsu++15pwI0HFxcmB9DFE22Ns=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=MpwCLBs0+rzuvQ587DQrBhASbsj/GB+9zsB051K5H/FQU7EMJwKr5odqcxKgnPwSa\n\tqNapI5NbulQK4PGxsRxCm5SWNkMsFoBefr3ZjCsKo6FNEvbkxxv/3p1pjOdzmR5ccw\n\t/mQjVjrELdAtY4WslaussppUb+7tuNueGIsoChuk=","Date":"Tue, 27 Oct 2020 03:04:00 +0200","From":"Laurent Pinchart ","To":"Hirokazu Honda ","Message-ID":"<20201027010400.GO3756@pendragon.ideasonboard.com>","References":"\n\t<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>\n\t<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>\n\t<9ca3b492-8468-e9cf-b103-b4350be472b9@ideasonboard.com>\n\t\n\t<20201022135914.GA3938@pendragon.ideasonboard.com>\n\t\n\t<20201023035906.GA4113@pendragon.ideasonboard.com>\n\t<734c408b-2e5b-6ee4-af07-f9c45416abce@ideasonboard.com>\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13501,"web_url":"https://patchwork.libcamera.org/comment/13501/","msgid":"","date":"2020-10-27T02:58:46","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":63,"url":"https://patchwork.libcamera.org/api/people/63/","name":"Hirokazu Honda","email":"hiroh@chromium.org"},"content":"Hi, Laurent.\n\nOn Tue, Oct 27, 2020 at 10:04 AM Laurent Pinchart\n wrote:\n>\n> Hi Hiro-san,\n>\n> On Mon, Oct 26, 2020 at 04:29:20PM +0900, Hirokazu Honda wrote:\n> > On Fri, Oct 23, 2020 at 5:39 PM Kieran Bingham wrote:\n> > > On 23/10/2020 04:59, Laurent Pinchart wrote:\n> > > > On Fri, Oct 23, 2020 at 09:37:09AM +0900, Hirokazu Honda wrote:\n> > > >> On Thu, Oct 22, 2020 at 11:00 PM Laurent Pinchart wrote:\n> > > >>> On Thu, Oct 22, 2020 at 08:42:01PM +0900, Hirokazu Honda wrote:\n> > > >>>> Hi Umang,\n> > > >>>> Thanks for the work. I couldn't have time to review these today and\n> > > >>>> will review tomorrow.\n> > > >>>> In my understanding, the thumbnail class is basically to scale frames\n> > > >>>> to the given destination.\n> > > >>>> Can we generalize it as PostProcessor interface so that we will be\n> > > >>>> able to make use of it for down-scaling regardless of whether jpeg\n> > > >>>> encoding is required.\n> > > >>>> It will be necessary when Android HAL adaptation layer has to output\n> > > >>>> multiple different NV12 streams while the native camera can produce a\n> > > >>>> single NV12 stream simultaneously.\n> > > >>>\n> > > >>> One point to consider is that the thumbnailer is a down-scaler using a\n> > > >>> nearest-neighbour algorithm. It's good enough for thumbnails, but will\n> > > >>> lead to awful quality when scaling streams. For that, I think we need a\n> > > >>> better scaler implementation, and we could then replace the thumbnailer\n> > > >>> class with it.\n> > > >>\n> > > >> We can select nearest-neighbor and bilinear algorithms in\n> > > >> libyuv::NV12Scale with the filtering mode argument[1].\n> > > >> We may want to specify the scale algorithm on a class construction.\n> > > >> I would use libyuv here for better performance and less buggy (+ less\n> > > >> lines to be reviewed).\n> > > >> What do you'all think?\n> > > >\n> > > > We have considered libyuv early on, and I think it's an interesting\n> > > > option moving forward. The reason we have decided to go for a minimal\n> > > > scaler implementation to create JPEG thumbnails is to avoid adding an\n> > > > external dependency that isn't packaged by distributions. This allows us\n> > > > to finalize JPEG thumbnail support without delay, and decide on the\n> > > > longer term solution separately.\n> > > >\n> > > > It would be very nice if Google could work with distributions to package\n> > > > libyuv. I'd like to avoid including a copy of libyuv in libcamera.\n> > >\n> > > Lack of packaging is a big factor here. If it's not available we can't\n> > > use it ;-)\n> >\n> > Thanks. That is fair enough.\n> >\n> > > However I looked into this briefly yesterday, - we could use a meson\n> > > wrap. But I don't know how that will affect us distributing packages, as\n> > > then we also have to distribute libyuv.\n> > >\n> > > Or perhaps the meson wrap would link statically.\n> >\n> > Can we download libyuv somewhere in libcamera, e.g. //third_party,\n> > during meson config if it is necessary?\n> > The code under //third_party is ignored by .gitignore, i.e., is not a\n> > part of libcamera tree.\n> > We may want to maintain a commit hash to be checked out in the libcamera tree.\n>\n> As Kieran mentioned, we could use a meson wrap, and link statically to\n> libyuv. If we rely on a particular commit of libyuv then I think we\n> could as well copy the source code in the libcamera repository, to avoid\n> depending on an internet connection at meson config time. I'm however\n> not sure what issues we would encounter on the way. libyuv is integrated\n> with the Chromium code base, but looking, for instance, at the buildroot\n> package, it seems fairly easy to compile it standalone. We may need to\n> add support for the meson build system though.\n>\n> As a conclusion, I think we'll need to figure out how we can use libyuv,\n> but not as a prerequisite for the JPEG thumbnail work. The best option\n> would be if Google maintained libyuv as a standalone library and worked\n> on getting it packaged in distributions :-)\n>\n\nI understood. I am ok in this way now to not block this work.\nSorry Uman for flooding in the thread.\nI will take a look the code today.\n\nBest Regards,\n-Hiro\n> > > > One last comment to provide all the design background. We have also\n> > > > considered creating a format conversion library in libcamera, to gather\n> > > > code from various projects that exist out there. There's format\n> > > > conversion in libv4l2, in raw2rgbpnm, in projects such as ffmpeg, and of\n> > > > course in libyuv. I would be great if that could be consolidated and\n> > > > packaged, but that's a big task in itself.\n> > >\n> > > And that's the bigger picture. This should be consolidated. And that\n> > > consolidated library should support more than yuv ;-)\n> > >\n> > > Given it would be a software format convertor, and scaler, and ideally\n> > > with hardware acceleration support where possible, I think that quickly\n> > > becomes libisp - and the feature creep will ... creep in ;-)\n> > >\n> > > >> [1] https://chromium.googlesource.com/libyuv/libyuv/+/refs/heads/master/include/libyuv/scale.h#148\n> > > >>\n> > > >>>> On Wed, Oct 21, 2020 at 8:09 PM Kieran Bingham wrote:\n> > > >>>>> On 21/10/2020 12:03, Umang Jain wrote:\n> > > >>>>>> On 10/21/20 4:25 PM, Kieran Bingham wrote:\n> > > >>>>>>> On 21/10/2020 11:51, Umang Jain wrote:\n> > > >>>>>>>> On 10/21/20 2:49 PM, Kieran Bingham wrote:\n> > > >>>>>>>>> On 21/10/2020 09:08, Umang Jain wrote:\n> > > >>>>>>>>>> Add a basic image thumbnailer for NV12 frames being captured.\n> > > >>>>>>>>>> It shall generate a thumbnail image to be embedded as a part of\n> > > >>>>>>>>>> EXIF metadata of the frame. The output of the thumbnail will still\n> > > >>>>>>>>>> be NV12.\n> > > >>>>>>>>>>\n> > > >>>>>>>>>> Signed-off-by: Umang Jain \n> > > >>>>>>>>>> ---\n> > > >>>>>>>>>> src/android/jpeg/post_processor_jpeg.cpp | 39 +++++++++\n> > > >>>>>>>>>> src/android/jpeg/post_processor_jpeg.h | 3 +\n> > > >>>>>>>>>> src/android/jpeg/thumbnailer.cpp | 100 +++++++++++++++++++++++\n> > > >>>>>>>>>> src/android/jpeg/thumbnailer.h | 40 +++++++++\n> > > >>>>>>>>>> src/android/meson.build | 1 +\n> > > >>>>>>>>>> 5 files changed, 183 insertions(+)\n> > > >>>>>>>>>> create mode 100644 src/android/jpeg/thumbnailer.cpp\n> > > >>>>>>>>>> create mode 100644 src/android/jpeg/thumbnailer.h\n> > > >>>>>>>>>>\n> > > >>>>>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp\n> > > >>>>>>>>>> b/src/android/jpeg/post_processor_jpeg.cpp\n> > > >>>>>>>>>> index 9d452b7..f5f1f78 100644\n> > > >>>>>>>>>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n> > > >>>>>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n> > > >>>>>>>>>> @@ -11,6 +11,7 @@\n> > > >>>>>>>>>> #include \"../camera_metadata.h\"\n> > > >>>>>>>>>> #include \"encoder_libjpeg.h\"\n> > > >>>>>>>>>> #include \"exif.h\"\n> > > >>>>>>>>>> +#include \"thumbnailer.h\"\n> > > >>>>>>>>>> #include \n> > > >>>>>>>>>> @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const StreamConfiguration &inCfg,\n> > > >>>>>>>>>> return encoder_->configure(inCfg);\n> > > >>>>>>>>>> }\n> > > >>>>>>>>>> +void PostProcessorJpeg::generateThumbnail(const libcamera::Span &source,\n> > > >>>>>>>>>> + std::vector &thumbnail)\n> > > >>>>>>>>>> +{\n> > > >>>>>>>>>> + libcamera::Span destination;\n> > > >>>>>>>>>> + Thumbnailer thumbnailer;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + thumbnailer.configure(streamSize_, formats::NV12);\n> > > >>>>>>>>>> + libcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n> > > >>>>>>>>>> + thumbnailer.scaleBuffer(source, thumbnail);\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + if (thumbnail.data()) {\n> > > >>>>>>>>>> + StreamConfiguration thumbnailCfg;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + std::unique_ptr encoder =\n> > > >>>>>>>>>> + std::make_unique();\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + thumbnailCfg.pixelFormat = formats::NV12;\n> > > >>>>>>>>>> + thumbnailCfg.size = targetSize;\n> > > >>>>>>>>>> + encoder->configure(thumbnailCfg);\n> > > >>>>>>>>>\n> > > >>>>>>>>> thumbnail.capacity() might be quite low here.\n> > > >>>>>>>>> We need to make sure the vector is big enough at this point, you might\n> > > >>>>>>>>> need to do something like:\n> > > >>>>>>>>>\n> > > >>>>>>>>> thumbnail.resize(targetSize.width * targetSize.height * 4);\n> > > >>>>>>>>\n> > > >>>>>>>> I am not sure I follow. This is compressing the thumbnail part right? So\n> > > >>>>>>>> thumbnail is the \"source\" for the encoder here (Please refer to\n> > > >>>>>>>> ->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n> > > >>>>>>>\n> > > >>>>>>> Ugh, sorry - tired eyes indeed. Yes, please ignore those comments.\n> > > >>>>>>>\n> > > >>>>>>>>> Really we should obtain that size from the encoder. I thought we had a\n> > > >>>>>>>>> helper to do that already, but it seems we don't.\n> > > >>>>>>>>> We could/should add Encoder::maxOutput(); which would ask the encoder\n> > > >>>>>>>>> \"For your given configuration, what is the maximum number of bytes you\n> > > >>>>>>>>> might output\".\n> > > >>>>>>>>>\n> > > >>>>>>>>> Then of course we'd do:\n> > > >>>>>>>>> thumbnail.resize(encoder.maxOutput());\n> > > >>>>>>>>>\n> > > >>>>>>>>>\n> > > >>>>>>>>>> + int jpeg_size = encoder->encode({ thumbnail.data(), thumbnail.capacity() },\n> > > >>>>>>>>>> + destination, { });\n> > > >>>>>>>>\n> > > >>>>>>>> As said above, thumbnail is the source here. The compressed thumbnail\n> > > >>>>>>>> output is carried in destination. And, destination is a local span<>\n> > > >>>>>>>> here. Does it makes sense?\n> > > >>>>>>>\n> > > >>>>>>> Yes, I had mixed up source and destination I'm sorry.\n> > > >>>>>>>\n> > > >>>>>>> So who/where is allocating the destination?\n> > > >>>>>>\n> > > >>>>>> Ah, that's a good question and I think I just found a bug. I currently\n> > > >>>>>> pass on a locally allocated Span<> 'destination' to the encoder. But\n> > > >>>>>> there is no explicit allocation here. I think I need to point / create\n> > > >>>>>> the Span with the output size / bytes and then pass to the encode().\n> > > >>>>>\n> > > >>>>> :-) Ok - then please apply all of my earlier comments to that part\n> > > >>>>> instead. ;-)\n> > > >>>>>\n> > > >>>>> We certainly need somewhere to store the compressed image, to be able to\n> > > >>>>> pass it into the exif :-D\n> > > >>>>>\n> > > >>>>>>>>>> + LOG(JPEG, Info) << \"Thumbnail compress returned \"\n> > > >>>>>>>>>> + << jpeg_size << \" bytes\";\n> > > >>>>>>>>>\n> > > >>>>>>>>> And I presume we could then do an:\n> > > >>>>>>>>> thumbnail.resize(jpeg_size);\n> > > >>>>>>>>>\n> > > >>>>>>>>> here to update the thumbnail with the correct size. I'd be weary of the\n> > > >>>>>>>>> resize operations doing lots of re-allocations though, so perhaps we\n> > > >>>>>>>>> want to minimize that. But lets get to something that works first\n> > > >>>>>>>>> before worrying about optimising.\n> > > >>>>>>>>>\n> > > >>>>>>>>>> + }\n> > > >>>>>>>>>> +}\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n> > > >>>>>>>>>> const libcamera::Span &destination,\n> > > >>>>>>>>>> CameraMetadata *metadata)\n> > > >>>>>>>>>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const\n> > > >>>>>>>>>> libcamera::FrameBuffer *source,\n> > > >>>>>>>>>> return jpeg_size;\n> > > >>>>>>>>>> }\n> > > >>>>>>>>>> + std::vector thumbnail;\n> > > >>>>>>>>>\n> > > >>>>>>>>> You need to resize this somewhere.\n> > > >>>>>>>>> Edit: Now seen a better place above ;-)\n> > > >>>>>>>>\n> > > >>>>>>>> The thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\n> > > >>>>>>>> We just pass in to the thumbnailer, while keeping its ownership in\n> > > >>>>>>>> PostProcessorJpeg.\n> > > >>>>>>>>\n> > > >>>>>>>>>> + generateThumbnail(destination, thumbnail);\n> > > >>>>>>>>>> + /*\n> > > >>>>>>>>>> + * \\todo: Write the compressed thumbnail to a file for inspection.\n> > > >>>>>>>>>> + * (I) Check if we can still write the thumbnail to EXIF here.\n> > > >>>>>>>>>> + * If not, we might need to move the thumbnailer logic to encoder.\n> > > >>>>>>>>>> + * And if we do that, first we need to make sure we get can\n> > > >>>>>>>>>> + * compressed data written to destination first before calling\n> > > >>>>>>>>>> + * jpeg_finish_compress operation somehow. Thumbnailing will\n> > > >>>>>>>>>> + * only occur if we have compressed data available first.\n> > > >>>>>>>>>> + */\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> /*\n> > > >>>>>>>>>> * Fill in the JPEG blob header.\n> > > >>>>>>>>>> *\n> > > >>>>>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.h\n> > > >>>>>>>>>> b/src/android/jpeg/post_processor_jpeg.h\n> > > >>>>>>>>>> index 62c8650..05601ee 100644\n> > > >>>>>>>>>> --- a/src/android/jpeg/post_processor_jpeg.h\n> > > >>>>>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.h\n> > > >>>>>>>>>> @@ -28,6 +28,9 @@ public:\n> > > >>>>>>>>>> CameraMetadata *metadata) override;\n> > > >>>>>>>>>> private:\n> > > >>>>>>>>>> + void generateThumbnail(const libcamera::Span &source,\n> > > >>>>>>>>>> + std::vector &thumbnail);\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> CameraDevice *cameraDevice_;\n> > > >>>>>>>>>> std::unique_ptr encoder_;\n> > > >>>>>>>>>> libcamera::Size streamSize_;\n> > > >>>>>>>>>> diff --git a/src/android/jpeg/thumbnailer.cpp\n> > > >>>>>>>>>> b/src/android/jpeg/thumbnailer.cpp\n> > > >>>>>>>>>> new file mode 100644\n> > > >>>>>>>>>> index 0000000..3163576\n> > > >>>>>>>>>> --- /dev/null\n> > > >>>>>>>>>> +++ b/src/android/jpeg/thumbnailer.cpp\n> > > >>>>>>>>>> @@ -0,0 +1,100 @@\n> > > >>>>>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > >>>>>>>>>> +/*\n> > > >>>>>>>>>> + * Copyright (C) 2020, Google Inc.\n> > > >>>>>>>>>> + *\n> > > >>>>>>>>>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n> > > >>>>>>>>>> + */\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +#include \"thumbnailer.h\"\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +#include \n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +#include \"libcamera/internal/file.h\"\n> > > >>>>>>>>>> +#include \"libcamera/internal/log.h\"\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +using namespace libcamera;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +Thumbnailer::Thumbnailer()\n> > > >>>>>>>>>> + : validConfiguration_(false)\n> > > >>>>>>>>>> +{\n> > > >>>>>>>>>> +}\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat pixelFormat)\n> > > >>>>>>>>>> +{\n> > > >>>>>>>>>> + sourceSize_ = sourceSize;\n> > > >>>>>>>>>> + pixelFormat_ = pixelFormat;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + if (pixelFormat_ != formats::NV12) {\n> > > >>>>>>>>>> + LOG (Thumbnailer, Error) << \"Failed to configure: Pixel Format \"\n> > > >>>>>>>>>> + << pixelFormat_.toString() << \" unsupported.\";\n> > > >>>>>>>>>> + return;\n> > > >>>>>>>>>> + }\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + validConfiguration_ = true;\n> > > >>>>>>>>>> +}\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +/*\n> > > >>>>>>>>>> + * The Exif specification recommends the width of the thumbnail to be a\n> > > >>>>>>>>>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding height\n> > > >>>>>>>>>> + * keeping the aspect ratio same as of the source.\n> > > >>>>>>>>>> + */\n> > > >>>>>>>>>> +Size Thumbnailer::computeThumbnailSize()\n> > > >>>>>>>>>> +{\n> > > >>>>>>>>>> + unsigned int targetHeight;\n> > > >>>>>>>>>> + unsigned int targetWidth = 160;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + targetHeight = targetWidth * sourceSize_.height / sourceSize_.width;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + if (targetHeight & 1)\n> > > >>>>>>>>>> + targetHeight++;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + return Size(targetWidth, targetHeight);\n> > > >>>>>>>>>> +}\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +void\n> > > >>>>>>>>>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n> > > >>>>>>>>>> + std::vector &destination)\n> > > >>>>>>>>>> +{\n> > > >>>>>>>>>> + if (!validConfiguration_) {\n> > > >>>>>>>>>> + LOG(Thumbnailer, Error) << \"config is unconfigured or invalid.\";\n> > > >>>>>>>>>> + return;\n> > > >>>>>>>>>> + }\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + targetSize_ = computeThumbnailSize();\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + const unsigned int sw = sourceSize_.width;\n> > > >>>>>>>>>> + const unsigned int sh = sourceSize_.height;\n> > > >>>>>>>>>> + const unsigned int tw = targetSize_.width;\n> > > >>>>>>>>>> + const unsigned int th = targetSize_.height;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + /* Image scaling block implementing nearest-neighbour algorithm. */\n> > > >>>>>>>>>> + unsigned char *src = static_cast(source.data());\n> > > >>>>>>>>>> + unsigned char *src_c = src + sh * sw;\n> > > >>>>>>>>>> + unsigned char *src_cb, *src_cr;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + size_t dstSize = (th * tw) + ((th/2) * tw);\n> > > >>>>>>>>>> + destination.reserve(dstSize);\n> > > >>>>>>>>>> + unsigned char *dst = destination.data();\n> > > >>>>>>>>>> + unsigned char *dst_c = dst + th * tw;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + for (unsigned int y = 0; y < th; y+=2) {\n> > > >>>>>>>>>> + unsigned int sourceY = (sh*y + th/2) / th;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + src_cb = src_c + (sourceY/2) * sw + 0;\n> > > >>>>>>>>>> + src_cr = src_c + (sourceY/2) * sw + 1;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + for (unsigned int x = 0; x < tw; x+=2) {\n> > > >>>>>>>>>> + unsigned int sourceX = (sw*x + tw/2) / tw;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + dst[y * tw + x] = src[sw * sourceY + sourceX];\n> > > >>>>>>>>>> + dst[(y+1) * tw + x] = src[sw * (sourceY+1) + sourceX];\n> > > >>>>>>>>>> + dst[y * tw + (x+1)] = src[sw * sourceY + (sourceX+1)];\n> > > >>>>>>>>>> + dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) + (sourceX+1)];\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + dst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n> > > >>>>>>>>>> + dst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n> > > >>>>>>>>>> + }\n> > > >>>>>>>>>> + }\n> > > >>>>>>>>>> +}\n> > > >>>>>>>>>> diff --git a/src/android/jpeg/thumbnailer.h\n> > > >>>>>>>>>> b/src/android/jpeg/thumbnailer.h\n> > > >>>>>>>>>> new file mode 100644\n> > > >>>>>>>>>> index 0000000..bab9855\n> > > >>>>>>>>>> --- /dev/null\n> > > >>>>>>>>>> +++ b/src/android/jpeg/thumbnailer.h\n> > > >>>>>>>>>> @@ -0,0 +1,40 @@\n> > > >>>>>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > >>>>>>>>>> +/*\n> > > >>>>>>>>>> + * Copyright (C) 2020, Google Inc.\n> > > >>>>>>>>>> + *\n> > > >>>>>>>>>> + * thumbnailer.h - Basic image thumbnailer from NV12\n> > > >>>>>>>>>> + */\n> > > >>>>>>>>>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n> > > >>>>>>>>>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +#include \n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +#include \"libcamera/internal/buffer.h\"\n> > > >>>>>>>>>> +#include \"libcamera/internal/formats.h\"\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +class Thumbnailer\n> > > >>>>>>>>>> +{\n> > > >>>>>>>>>> +public:\n> > > >>>>>>>>>> + Thumbnailer();\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + void configure(const libcamera::Size &sourceSize,\n> > > >>>>>>>>>> + libcamera::PixelFormat pixelFormat);\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + /*\n> > > >>>>>>>>>> + * \\todo: Discuss if we can return targetSize_ via configure() or\n> > > >>>>>>>>>> + * scaleBuffer(). We need targetSize_ to re-encode the scaled\n> > > >>>>>>>>>> buffer\n> > > >>>>>>>>>> + * via encoder in PostProcssorJpeg::writeThumbnail().\n> > > >>>>>>>>>> + */\n> > > >>>>>>>>>> + libcamera::Size computeThumbnailSize();\n> > > >>>>>>>>>> + void scaleBuffer(const libcamera::Span &source,\n> > > >>>>>>>>>> + std::vector &dest);\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +private:\n> > > >>>>>>>>>> + libcamera::PixelFormat pixelFormat_;\n> > > >>>>>>>>>> + libcamera::Size sourceSize_;\n> > > >>>>>>>>>> + libcamera::Size targetSize_;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + bool validConfiguration_;\n> > > >>>>>>>>>> +};\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n> > > >>>>>>>>>> diff --git a/src/android/meson.build b/src/android/meson.build\n> > > >>>>>>>>>> index 5a01bea..3905e2f 100644\n> > > >>>>>>>>>> --- a/src/android/meson.build\n> > > >>>>>>>>>> +++ b/src/android/meson.build\n> > > >>>>>>>>>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n> > > >>>>>>>>>> 'jpeg/encoder_libjpeg.cpp',\n> > > >>>>>>>>>> 'jpeg/exif.cpp',\n> > > >>>>>>>>>> 'jpeg/post_processor_jpeg.cpp',\n> > > >>>>>>>>>> + 'jpeg/thumbnailer.cpp',\n> > > >>>>>>>>>> ])\n> > > >>>>>>>>>> android_camera_metadata_sources = files([\n> > > >>>>>>>>>>\n>\n> --\n> Regards,\n>\n> Laurent Pinchart","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 0398DC3B5C\n\tfor ;\n\tTue, 27 Oct 2020 02:58:59 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 7BC306206C;\n\tTue, 27 Oct 2020 03:58:58 +0100 (CET)","from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com\n\t[IPv6:2a00:1450:4864:20::62b])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id C02C36034C\n\tfor ;\n\tTue, 27 Oct 2020 03:58:57 +0100 (CET)","by mail-ej1-x62b.google.com with SMTP id d6so39192ejb.11\n\tfor ;\n\tMon, 26 Oct 2020 19:58:57 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=chromium.org header.i=@chromium.org\n\theader.b=\"lMnMqNxI\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org;\n\ts=google; \n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=KqYpXJOXVZizPva5/JOF4wl663cGmkS85GsUSaVCh3g=;\n\tb=lMnMqNxIj8PVZpIsu+bfdRf9cq4SUiwov8kfvJksNI/EgdqhDFmRHLCMFoYHWRpVci\n\tOaVHBvLqpA1O4kxF0D34UPCxJ4FzKpegCTTnJZUwhUKe5bRILGWczwE9UE5kzck46vDb\n\tFE/Oi4YNvL17GiwC/0CZgG7+s1/k0/ZSg/SaQ=","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=KqYpXJOXVZizPva5/JOF4wl663cGmkS85GsUSaVCh3g=;\n\tb=qg6W8AmOFyOVmPgEfyBdjFqYg0rRhNYKUKf5qosI1VGk67emqTF26EtFm68B0Yq5MB\n\t0TJJkTzyq6NrVTr9IClSOKRWR0zfjuHc6tJ/N/kTLaj/ysZ948CCnhkZI7aRlnKflniW\n\tGWsY4QvVrGT7MKcMVf0lvyHJm1uwANYHErrbmd1Zyp0JS5Z1x7GsKMJbRXZUiL4ohjpT\n\tUjOm0/q3UpX3OCKBxTyMPmqZmMKz4FDTgz8POP8uJ8oKsu+ojDtJybstT8bWqEJroWwm\n\trysiYbhwrQH808wluHI/vcgV5wGrSAFaRqQPgsz9IkkdW+6VUgDsJAds+ucZ9yEX4wsn\n\tYVTw==","X-Gm-Message-State":"AOAM533XVn4Oz15BJK4LoB5KYrqCypfeHqHGZk0R1rP4r5doTwptEHC7\n\twKWODC5C7R8Wd7jwsNkIJD6HgctvHlzd/AHyLEML7g==","X-Google-Smtp-Source":"ABdhPJz1B+ZFVFt6UlZVA1UDl2KDP18ZfMDLfjpOjuoM257rfVfO6+EoQ7ZiGyScYOmzQqMzjCyjnbOVGIEW/bZoWfk=","X-Received":"by 2002:a17:906:2894:: with SMTP id\n\to20mr296417ejd.221.1603767537079; \n\tMon, 26 Oct 2020 19:58:57 -0700 (PDT)","MIME-Version":"1.0","References":"\n\t<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>\n\t<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>\n\t<9ca3b492-8468-e9cf-b103-b4350be472b9@ideasonboard.com>\n\t\n\t<20201022135914.GA3938@pendragon.ideasonboard.com>\n\t\n\t<20201023035906.GA4113@pendragon.ideasonboard.com>\n\t<734c408b-2e5b-6ee4-af07-f9c45416abce@ideasonboard.com>\n\t\n\t<20201027010400.GO3756@pendragon.ideasonboard.com>","In-Reply-To":"<20201027010400.GO3756@pendragon.ideasonboard.com>","From":"Hirokazu Honda ","Date":"Tue, 27 Oct 2020 11:58:46 +0900","Message-ID":"","To":"Laurent Pinchart ","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13502,"web_url":"https://patchwork.libcamera.org/comment/13502/","msgid":"<1ceb246c-223e-a8b1-1389-2b90a12cde3d@uajain.com>","date":"2020-10-27T03:43:32","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":1,"url":"https://patchwork.libcamera.org/api/people/1/","name":"Umang Jain","email":"email@uajain.com"},"content":"Hi Hiro,\n\nOn 10/27/20 8:28 AM, Hirokazu Honda wrote:\n> Hi, Laurent.\n>\n> On Tue, Oct 27, 2020 at 10:04 AM Laurent Pinchart\n> wrote:\n>> Hi Hiro-san,\n>>\n>> On Mon, Oct 26, 2020 at 04:29:20PM +0900, Hirokazu Honda wrote:\n>>> On Fri, Oct 23, 2020 at 5:39 PM Kieran Bingham wrote:\n>>>> On 23/10/2020 04:59, Laurent Pinchart wrote:\n>>>>> On Fri, Oct 23, 2020 at 09:37:09AM +0900, Hirokazu Honda wrote:\n>>>>>> On Thu, Oct 22, 2020 at 11:00 PM Laurent Pinchart wrote:\n>>>>>>> On Thu, Oct 22, 2020 at 08:42:01PM +0900, Hirokazu Honda wrote:\n>>>>>>>> Hi Umang,\n>>>>>>>> Thanks for the work. I couldn't have time to review these today and\n>>>>>>>> will review tomorrow.\n>>>>>>>> In my understanding, the thumbnail class is basically to scale frames\n>>>>>>>> to the given destination.\n>>>>>>>> Can we generalize it as PostProcessor interface so that we will be\n>>>>>>>> able to make use of it for down-scaling regardless of whether jpeg\n>>>>>>>> encoding is required.\n>>>>>>>> It will be necessary when Android HAL adaptation layer has to output\n>>>>>>>> multiple different NV12 streams while the native camera can produce a\n>>>>>>>> single NV12 stream simultaneously.\n>>>>>>> One point to consider is that the thumbnailer is a down-scaler using a\n>>>>>>> nearest-neighbour algorithm. It's good enough for thumbnails, but will\n>>>>>>> lead to awful quality when scaling streams. For that, I think we need a\n>>>>>>> better scaler implementation, and we could then replace the thumbnailer\n>>>>>>> class with it.\n>>>>>> We can select nearest-neighbor and bilinear algorithms in\n>>>>>> libyuv::NV12Scale with the filtering mode argument[1].\n>>>>>> We may want to specify the scale algorithm on a class construction.\n>>>>>> I would use libyuv here for better performance and less buggy (+ less\n>>>>>> lines to be reviewed).\n>>>>>> What do you'all think?\n>>>>> We have considered libyuv early on, and I think it's an interesting\n>>>>> option moving forward. The reason we have decided to go for a minimal\n>>>>> scaler implementation to create JPEG thumbnails is to avoid adding an\n>>>>> external dependency that isn't packaged by distributions. This allows us\n>>>>> to finalize JPEG thumbnail support without delay, and decide on the\n>>>>> longer term solution separately.\n>>>>>\n>>>>> It would be very nice if Google could work with distributions to package\n>>>>> libyuv. I'd like to avoid including a copy of libyuv in libcamera.\n>>>> Lack of packaging is a big factor here. If it's not available we can't\n>>>> use it ;-)\n>>> Thanks. That is fair enough.\n>>>\n>>>> However I looked into this briefly yesterday, - we could use a meson\n>>>> wrap. But I don't know how that will affect us distributing packages, as\n>>>> then we also have to distribute libyuv.\n>>>>\n>>>> Or perhaps the meson wrap would link statically.\n>>> Can we download libyuv somewhere in libcamera, e.g. //third_party,\n>>> during meson config if it is necessary?\n>>> The code under //third_party is ignored by .gitignore, i.e., is not a\n>>> part of libcamera tree.\n>>> We may want to maintain a commit hash to be checked out in the libcamera tree.\n>> As Kieran mentioned, we could use a meson wrap, and link statically to\n>> libyuv. If we rely on a particular commit of libyuv then I think we\n>> could as well copy the source code in the libcamera repository, to avoid\n>> depending on an internet connection at meson config time. I'm however\n>> not sure what issues we would encounter on the way. libyuv is integrated\n>> with the Chromium code base, but looking, for instance, at the buildroot\n>> package, it seems fairly easy to compile it standalone. We may need to\n>> add support for the meson build system though.\n>>\n>> As a conclusion, I think we'll need to figure out how we can use libyuv,\n>> but not as a prerequisite for the JPEG thumbnail work. The best option\n>> would be if Google maintained libyuv as a standalone library and worked\n>> on getting it packaged in distributions :-)\n>>\n> I understood. I am ok in this way now to not block this work.\n> Sorry Uman for flooding in the thread.\n> I will take a look the code today.\nNo issues. Please make sure if you looking at the code, that I have \nsubmitted the final implementation under the subject \"android: jpeg: \nexif: Embed a JPEG-encoded thumbnail\" on the list. I forgot to CC you on \nthe previous  but is it OK if I CC you on futher iterations of the work?\n>\n> Best Regards,\n> -Hiro\n>>>>> One last comment to provide all the design background. We have also\n>>>>> considered creating a format conversion library in libcamera, to gather\n>>>>> code from various projects that exist out there. There's format\n>>>>> conversion in libv4l2, in raw2rgbpnm, in projects such as ffmpeg, and of\n>>>>> course in libyuv. I would be great if that could be consolidated and\n>>>>> packaged, but that's a big task in itself.\n>>>> And that's the bigger picture. This should be consolidated. And that\n>>>> consolidated library should support more than yuv ;-)\n>>>>\n>>>> Given it would be a software format convertor, and scaler, and ideally\n>>>> with hardware acceleration support where possible, I think that quickly\n>>>> becomes libisp - and the feature creep will ... creep in ;-)\n>>>>\n>>>>>> [1] https://chromium.googlesource.com/libyuv/libyuv/+/refs/heads/master/include/libyuv/scale.h#148\n>>>>>>\n>>>>>>>> On Wed, Oct 21, 2020 at 8:09 PM Kieran Bingham wrote:\n>>>>>>>>> On 21/10/2020 12:03, Umang Jain wrote:\n>>>>>>>>>> On 10/21/20 4:25 PM, Kieran Bingham wrote:\n>>>>>>>>>>> On 21/10/2020 11:51, Umang Jain wrote:\n>>>>>>>>>>>> On 10/21/20 2:49 PM, Kieran Bingham wrote:\n>>>>>>>>>>>>> On 21/10/2020 09:08, Umang Jain wrote:\n>>>>>>>>>>>>>> Add a basic image thumbnailer for NV12 frames being captured.\n>>>>>>>>>>>>>> It shall generate a thumbnail image to be embedded as a part of\n>>>>>>>>>>>>>> EXIF metadata of the frame. The output of the thumbnail will still\n>>>>>>>>>>>>>> be NV12.\n>>>>>>>>>>>>>>\n>>>>>>>>>>>>>> Signed-off-by: Umang Jain \n>>>>>>>>>>>>>> ---\n>>>>>>>>>>>>>> src/android/jpeg/post_processor_jpeg.cpp | 39 +++++++++\n>>>>>>>>>>>>>> src/android/jpeg/post_processor_jpeg.h | 3 +\n>>>>>>>>>>>>>> src/android/jpeg/thumbnailer.cpp | 100 +++++++++++++++++++++++\n>>>>>>>>>>>>>> src/android/jpeg/thumbnailer.h | 40 +++++++++\n>>>>>>>>>>>>>> src/android/meson.build | 1 +\n>>>>>>>>>>>>>> 5 files changed, 183 insertions(+)\n>>>>>>>>>>>>>> create mode 100644 src/android/jpeg/thumbnailer.cpp\n>>>>>>>>>>>>>> create mode 100644 src/android/jpeg/thumbnailer.h\n>>>>>>>>>>>>>>\n>>>>>>>>>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp\n>>>>>>>>>>>>>> b/src/android/jpeg/post_processor_jpeg.cpp\n>>>>>>>>>>>>>> index 9d452b7..f5f1f78 100644\n>>>>>>>>>>>>>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n>>>>>>>>>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n>>>>>>>>>>>>>> @@ -11,6 +11,7 @@\n>>>>>>>>>>>>>> #include \"../camera_metadata.h\"\n>>>>>>>>>>>>>> #include \"encoder_libjpeg.h\"\n>>>>>>>>>>>>>> #include \"exif.h\"\n>>>>>>>>>>>>>> +#include \"thumbnailer.h\"\n>>>>>>>>>>>>>> #include \n>>>>>>>>>>>>>> @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const StreamConfiguration &inCfg,\n>>>>>>>>>>>>>> return encoder_->configure(inCfg);\n>>>>>>>>>>>>>> }\n>>>>>>>>>>>>>> +void PostProcessorJpeg::generateThumbnail(const libcamera::Span &source,\n>>>>>>>>>>>>>> + std::vector &thumbnail)\n>>>>>>>>>>>>>> +{\n>>>>>>>>>>>>>> + libcamera::Span destination;\n>>>>>>>>>>>>>> + Thumbnailer thumbnailer;\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + thumbnailer.configure(streamSize_, formats::NV12);\n>>>>>>>>>>>>>> + libcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n>>>>>>>>>>>>>> + thumbnailer.scaleBuffer(source, thumbnail);\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + if (thumbnail.data()) {\n>>>>>>>>>>>>>> + StreamConfiguration thumbnailCfg;\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + std::unique_ptr encoder =\n>>>>>>>>>>>>>> + std::make_unique();\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + thumbnailCfg.pixelFormat = formats::NV12;\n>>>>>>>>>>>>>> + thumbnailCfg.size = targetSize;\n>>>>>>>>>>>>>> + encoder->configure(thumbnailCfg);\n>>>>>>>>>>>>> thumbnail.capacity() might be quite low here.\n>>>>>>>>>>>>> We need to make sure the vector is big enough at this point, you might\n>>>>>>>>>>>>> need to do something like:\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> thumbnail.resize(targetSize.width * targetSize.height * 4);\n>>>>>>>>>>>> I am not sure I follow. This is compressing the thumbnail part right? So\n>>>>>>>>>>>> thumbnail is the \"source\" for the encoder here (Please refer to\n>>>>>>>>>>>> ->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n>>>>>>>>>>> Ugh, sorry - tired eyes indeed. Yes, please ignore those comments.\n>>>>>>>>>>>\n>>>>>>>>>>>>> Really we should obtain that size from the encoder. I thought we had a\n>>>>>>>>>>>>> helper to do that already, but it seems we don't.\n>>>>>>>>>>>>> We could/should add Encoder::maxOutput(); which would ask the encoder\n>>>>>>>>>>>>> \"For your given configuration, what is the maximum number of bytes you\n>>>>>>>>>>>>> might output\".\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Then of course we'd do:\n>>>>>>>>>>>>> thumbnail.resize(encoder.maxOutput());\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>> + int jpeg_size = encoder->encode({ thumbnail.data(), thumbnail.capacity() },\n>>>>>>>>>>>>>> + destination, { });\n>>>>>>>>>>>> As said above, thumbnail is the source here. The compressed thumbnail\n>>>>>>>>>>>> output is carried in destination. And, destination is a local span<>\n>>>>>>>>>>>> here. Does it makes sense?\n>>>>>>>>>>> Yes, I had mixed up source and destination I'm sorry.\n>>>>>>>>>>>\n>>>>>>>>>>> So who/where is allocating the destination?\n>>>>>>>>>> Ah, that's a good question and I think I just found a bug. I currently\n>>>>>>>>>> pass on a locally allocated Span<> 'destination' to the encoder. But\n>>>>>>>>>> there is no explicit allocation here. I think I need to point / create\n>>>>>>>>>> the Span with the output size / bytes and then pass to the encode().\n>>>>>>>>> :-) Ok - then please apply all of my earlier comments to that part\n>>>>>>>>> instead. ;-)\n>>>>>>>>>\n>>>>>>>>> We certainly need somewhere to store the compressed image, to be able to\n>>>>>>>>> pass it into the exif :-D\n>>>>>>>>>\n>>>>>>>>>>>>>> + LOG(JPEG, Info) << \"Thumbnail compress returned \"\n>>>>>>>>>>>>>> + << jpeg_size << \" bytes\";\n>>>>>>>>>>>>> And I presume we could then do an:\n>>>>>>>>>>>>> thumbnail.resize(jpeg_size);\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> here to update the thumbnail with the correct size. I'd be weary of the\n>>>>>>>>>>>>> resize operations doing lots of re-allocations though, so perhaps we\n>>>>>>>>>>>>> want to minimize that. But lets get to something that works first\n>>>>>>>>>>>>> before worrying about optimising.\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>> + }\n>>>>>>>>>>>>>> +}\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n>>>>>>>>>>>>>> const libcamera::Span &destination,\n>>>>>>>>>>>>>> CameraMetadata *metadata)\n>>>>>>>>>>>>>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const\n>>>>>>>>>>>>>> libcamera::FrameBuffer *source,\n>>>>>>>>>>>>>> return jpeg_size;\n>>>>>>>>>>>>>> }\n>>>>>>>>>>>>>> + std::vector thumbnail;\n>>>>>>>>>>>>> You need to resize this somewhere.\n>>>>>>>>>>>>> Edit: Now seen a better place above ;-)\n>>>>>>>>>>>> The thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\n>>>>>>>>>>>> We just pass in to the thumbnailer, while keeping its ownership in\n>>>>>>>>>>>> PostProcessorJpeg.\n>>>>>>>>>>>>\n>>>>>>>>>>>>>> + generateThumbnail(destination, thumbnail);\n>>>>>>>>>>>>>> + /*\n>>>>>>>>>>>>>> + * \\todo: Write the compressed thumbnail to a file for inspection.\n>>>>>>>>>>>>>> + * (I) Check if we can still write the thumbnail to EXIF here.\n>>>>>>>>>>>>>> + * If not, we might need to move the thumbnailer logic to encoder.\n>>>>>>>>>>>>>> + * And if we do that, first we need to make sure we get can\n>>>>>>>>>>>>>> + * compressed data written to destination first before calling\n>>>>>>>>>>>>>> + * jpeg_finish_compress operation somehow. Thumbnailing will\n>>>>>>>>>>>>>> + * only occur if we have compressed data available first.\n>>>>>>>>>>>>>> + */\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> /*\n>>>>>>>>>>>>>> * Fill in the JPEG blob header.\n>>>>>>>>>>>>>> *\n>>>>>>>>>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.h\n>>>>>>>>>>>>>> b/src/android/jpeg/post_processor_jpeg.h\n>>>>>>>>>>>>>> index 62c8650..05601ee 100644\n>>>>>>>>>>>>>> --- a/src/android/jpeg/post_processor_jpeg.h\n>>>>>>>>>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.h\n>>>>>>>>>>>>>> @@ -28,6 +28,9 @@ public:\n>>>>>>>>>>>>>> CameraMetadata *metadata) override;\n>>>>>>>>>>>>>> private:\n>>>>>>>>>>>>>> + void generateThumbnail(const libcamera::Span &source,\n>>>>>>>>>>>>>> + std::vector &thumbnail);\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> CameraDevice *cameraDevice_;\n>>>>>>>>>>>>>> std::unique_ptr encoder_;\n>>>>>>>>>>>>>> libcamera::Size streamSize_;\n>>>>>>>>>>>>>> diff --git a/src/android/jpeg/thumbnailer.cpp\n>>>>>>>>>>>>>> b/src/android/jpeg/thumbnailer.cpp\n>>>>>>>>>>>>>> new file mode 100644\n>>>>>>>>>>>>>> index 0000000..3163576\n>>>>>>>>>>>>>> --- /dev/null\n>>>>>>>>>>>>>> +++ b/src/android/jpeg/thumbnailer.cpp\n>>>>>>>>>>>>>> @@ -0,0 +1,100 @@\n>>>>>>>>>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n>>>>>>>>>>>>>> +/*\n>>>>>>>>>>>>>> + * Copyright (C) 2020, Google Inc.\n>>>>>>>>>>>>>> + *\n>>>>>>>>>>>>>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n>>>>>>>>>>>>>> + */\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> +#include \"thumbnailer.h\"\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> +#include \n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> +#include \"libcamera/internal/file.h\"\n>>>>>>>>>>>>>> +#include \"libcamera/internal/log.h\"\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> +using namespace libcamera;\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> +Thumbnailer::Thumbnailer()\n>>>>>>>>>>>>>> + : validConfiguration_(false)\n>>>>>>>>>>>>>> +{\n>>>>>>>>>>>>>> +}\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat pixelFormat)\n>>>>>>>>>>>>>> +{\n>>>>>>>>>>>>>> + sourceSize_ = sourceSize;\n>>>>>>>>>>>>>> + pixelFormat_ = pixelFormat;\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + if (pixelFormat_ != formats::NV12) {\n>>>>>>>>>>>>>> + LOG (Thumbnailer, Error) << \"Failed to configure: Pixel Format \"\n>>>>>>>>>>>>>> + << pixelFormat_.toString() << \" unsupported.\";\n>>>>>>>>>>>>>> + return;\n>>>>>>>>>>>>>> + }\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + validConfiguration_ = true;\n>>>>>>>>>>>>>> +}\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> +/*\n>>>>>>>>>>>>>> + * The Exif specification recommends the width of the thumbnail to be a\n>>>>>>>>>>>>>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding height\n>>>>>>>>>>>>>> + * keeping the aspect ratio same as of the source.\n>>>>>>>>>>>>>> + */\n>>>>>>>>>>>>>> +Size Thumbnailer::computeThumbnailSize()\n>>>>>>>>>>>>>> +{\n>>>>>>>>>>>>>> + unsigned int targetHeight;\n>>>>>>>>>>>>>> + unsigned int targetWidth = 160;\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + targetHeight = targetWidth * sourceSize_.height / sourceSize_.width;\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + if (targetHeight & 1)\n>>>>>>>>>>>>>> + targetHeight++;\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + return Size(targetWidth, targetHeight);\n>>>>>>>>>>>>>> +}\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> +void\n>>>>>>>>>>>>>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n>>>>>>>>>>>>>> + std::vector &destination)\n>>>>>>>>>>>>>> +{\n>>>>>>>>>>>>>> + if (!validConfiguration_) {\n>>>>>>>>>>>>>> + LOG(Thumbnailer, Error) << \"config is unconfigured or invalid.\";\n>>>>>>>>>>>>>> + return;\n>>>>>>>>>>>>>> + }\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + targetSize_ = computeThumbnailSize();\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + const unsigned int sw = sourceSize_.width;\n>>>>>>>>>>>>>> + const unsigned int sh = sourceSize_.height;\n>>>>>>>>>>>>>> + const unsigned int tw = targetSize_.width;\n>>>>>>>>>>>>>> + const unsigned int th = targetSize_.height;\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + /* Image scaling block implementing nearest-neighbour algorithm. */\n>>>>>>>>>>>>>> + unsigned char *src = static_cast(source.data());\n>>>>>>>>>>>>>> + unsigned char *src_c = src + sh * sw;\n>>>>>>>>>>>>>> + unsigned char *src_cb, *src_cr;\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + size_t dstSize = (th * tw) + ((th/2) * tw);\n>>>>>>>>>>>>>> + destination.reserve(dstSize);\n>>>>>>>>>>>>>> + unsigned char *dst = destination.data();\n>>>>>>>>>>>>>> + unsigned char *dst_c = dst + th * tw;\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + for (unsigned int y = 0; y < th; y+=2) {\n>>>>>>>>>>>>>> + unsigned int sourceY = (sh*y + th/2) / th;\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + src_cb = src_c + (sourceY/2) * sw + 0;\n>>>>>>>>>>>>>> + src_cr = src_c + (sourceY/2) * sw + 1;\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + for (unsigned int x = 0; x < tw; x+=2) {\n>>>>>>>>>>>>>> + unsigned int sourceX = (sw*x + tw/2) / tw;\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + dst[y * tw + x] = src[sw * sourceY + sourceX];\n>>>>>>>>>>>>>> + dst[(y+1) * tw + x] = src[sw * (sourceY+1) + sourceX];\n>>>>>>>>>>>>>> + dst[y * tw + (x+1)] = src[sw * sourceY + (sourceX+1)];\n>>>>>>>>>>>>>> + dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) + (sourceX+1)];\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + dst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n>>>>>>>>>>>>>> + dst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n>>>>>>>>>>>>>> + }\n>>>>>>>>>>>>>> + }\n>>>>>>>>>>>>>> +}\n>>>>>>>>>>>>>> diff --git a/src/android/jpeg/thumbnailer.h\n>>>>>>>>>>>>>> b/src/android/jpeg/thumbnailer.h\n>>>>>>>>>>>>>> new file mode 100644\n>>>>>>>>>>>>>> index 0000000..bab9855\n>>>>>>>>>>>>>> --- /dev/null\n>>>>>>>>>>>>>> +++ b/src/android/jpeg/thumbnailer.h\n>>>>>>>>>>>>>> @@ -0,0 +1,40 @@\n>>>>>>>>>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n>>>>>>>>>>>>>> +/*\n>>>>>>>>>>>>>> + * Copyright (C) 2020, Google Inc.\n>>>>>>>>>>>>>> + *\n>>>>>>>>>>>>>> + * thumbnailer.h - Basic image thumbnailer from NV12\n>>>>>>>>>>>>>> + */\n>>>>>>>>>>>>>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n>>>>>>>>>>>>>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> +#include \n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> +#include \"libcamera/internal/buffer.h\"\n>>>>>>>>>>>>>> +#include \"libcamera/internal/formats.h\"\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> +class Thumbnailer\n>>>>>>>>>>>>>> +{\n>>>>>>>>>>>>>> +public:\n>>>>>>>>>>>>>> + Thumbnailer();\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + void configure(const libcamera::Size &sourceSize,\n>>>>>>>>>>>>>> + libcamera::PixelFormat pixelFormat);\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + /*\n>>>>>>>>>>>>>> + * \\todo: Discuss if we can return targetSize_ via configure() or\n>>>>>>>>>>>>>> + * scaleBuffer(). We need targetSize_ to re-encode the scaled\n>>>>>>>>>>>>>> buffer\n>>>>>>>>>>>>>> + * via encoder in PostProcssorJpeg::writeThumbnail().\n>>>>>>>>>>>>>> + */\n>>>>>>>>>>>>>> + libcamera::Size computeThumbnailSize();\n>>>>>>>>>>>>>> + void scaleBuffer(const libcamera::Span &source,\n>>>>>>>>>>>>>> + std::vector &dest);\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> +private:\n>>>>>>>>>>>>>> + libcamera::PixelFormat pixelFormat_;\n>>>>>>>>>>>>>> + libcamera::Size sourceSize_;\n>>>>>>>>>>>>>> + libcamera::Size targetSize_;\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> + bool validConfiguration_;\n>>>>>>>>>>>>>> +};\n>>>>>>>>>>>>>> +\n>>>>>>>>>>>>>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n>>>>>>>>>>>>>> diff --git a/src/android/meson.build b/src/android/meson.build\n>>>>>>>>>>>>>> index 5a01bea..3905e2f 100644\n>>>>>>>>>>>>>> --- a/src/android/meson.build\n>>>>>>>>>>>>>> +++ b/src/android/meson.build\n>>>>>>>>>>>>>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n>>>>>>>>>>>>>> 'jpeg/encoder_libjpeg.cpp',\n>>>>>>>>>>>>>> 'jpeg/exif.cpp',\n>>>>>>>>>>>>>> 'jpeg/post_processor_jpeg.cpp',\n>>>>>>>>>>>>>> + 'jpeg/thumbnailer.cpp',\n>>>>>>>>>>>>>> ])\n>>>>>>>>>>>>>> android_camera_metadata_sources = files([\n>>>>>>>>>>>>>>\n>> --\n>> Regards,\n>>\n>> Laurent Pinchart\n> _______________________________________________\n> libcamera-devel mailing list\n> libcamera-devel@lists.libcamera.org\n> https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id D8D04BDB1E\n\tfor ;\n\tTue, 27 Oct 2020 03:43:39 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 5524C62069;\n\tTue, 27 Oct 2020 04:43:39 +0100 (CET)","from mail.uajain.com (static.126.159.217.95.clients.your-server.de\n\t[95.217.159.126])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 0012F6034C\n\tfor ;\n\tTue, 27 Oct 2020 04:43:36 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=uajain.com header.i=@uajain.com\n\theader.b=\"ZyVIK+//\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=uajain.com; s=mail;\n\tt=1603770216; bh=Dj/LtRIn1FMSyCn/GM0RpcTZqOxhL8WdTVl6sYB8NnU=;\n\th=Subject:To:References:From:In-Reply-To;\n\tb=ZyVIK+//lhPRXtYrgp1jpwj3fRNotZLjKRVPj3AzE2LzVxWo+HP7EI+5+ZEZMmrHR\n\tPmR5SVgWtHmlKAl1rpAuQtjEAuUMMv0nQIxMNQB2o1QQXyK/AuFMOPXrXFCUuBwXGm\n\tagMsgjFC+/kflnp/vpxZm9fMRS6ZLMrfns6Lc2VWeLG86hIzmZfdz4pMUZUbY5g/sd\n\t4HSXtIMi0utgGFOokgYUGMPxTQNXLoe0GKIc4d8QRY6cJGaO0WwvmNVi+qudzysBZ1\n\t4kFuXoesxwtrsLduqKQrEJQEoN9+NAkXe1zBevi8p7LYtvL31vzYaRdnTlnr1R/wkl\n\t9esCbjWotfuhQ==","To":"libcamera-devel@lists.libcamera.org","References":"\n\t<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>\n\t<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>\n\t<9ca3b492-8468-e9cf-b103-b4350be472b9@ideasonboard.com>\n\t\n\t<20201022135914.GA3938@pendragon.ideasonboard.com>\n\t\n\t<20201023035906.GA4113@pendragon.ideasonboard.com>\n\t<734c408b-2e5b-6ee4-af07-f9c45416abce@ideasonboard.com>\n\t\n\t<20201027010400.GO3756@pendragon.ideasonboard.com>\n\t","From":"Umang Jain ","Message-ID":"<1ceb246c-223e-a8b1-1389-2b90a12cde3d@uajain.com>","Date":"Tue, 27 Oct 2020 09:13:32 +0530","Mime-Version":"1.0","In-Reply-To":"","Content-Language":"en-US","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Transfer-Encoding":"base64","Content-Type":"text/plain; charset=\"utf-8\"; Format=\"flowed\"","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":14179,"web_url":"https://patchwork.libcamera.org/comment/14179/","msgid":"","date":"2020-12-10T05:32:39","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":9,"url":"https://patchwork.libcamera.org/api/people/9/","name":"Tomasz Figa","email":"tfiga@chromium.org"},"content":"On Tue, Oct 27, 2020 at 10:04 AM Laurent Pinchart\n wrote:\n>\n> Hi Hiro-san,\n>\n> On Mon, Oct 26, 2020 at 04:29:20PM +0900, Hirokazu Honda wrote:\n> > On Fri, Oct 23, 2020 at 5:39 PM Kieran Bingham wrote:\n> > > On 23/10/2020 04:59, Laurent Pinchart wrote:\n> > > > On Fri, Oct 23, 2020 at 09:37:09AM +0900, Hirokazu Honda wrote:\n> > > >> On Thu, Oct 22, 2020 at 11:00 PM Laurent Pinchart wrote:\n> > > >>> On Thu, Oct 22, 2020 at 08:42:01PM +0900, Hirokazu Honda wrote:\n> > > >>>> Hi Umang,\n> > > >>>> Thanks for the work. I couldn't have time to review these today and\n> > > >>>> will review tomorrow.\n> > > >>>> In my understanding, the thumbnail class is basically to scale frames\n> > > >>>> to the given destination.\n> > > >>>> Can we generalize it as PostProcessor interface so that we will be\n> > > >>>> able to make use of it for down-scaling regardless of whether jpeg\n> > > >>>> encoding is required.\n> > > >>>> It will be necessary when Android HAL adaptation layer has to output\n> > > >>>> multiple different NV12 streams while the native camera can produce a\n> > > >>>> single NV12 stream simultaneously.\n> > > >>>\n> > > >>> One point to consider is that the thumbnailer is a down-scaler using a\n> > > >>> nearest-neighbour algorithm. It's good enough for thumbnails, but will\n> > > >>> lead to awful quality when scaling streams. For that, I think we need a\n> > > >>> better scaler implementation, and we could then replace the thumbnailer\n> > > >>> class with it.\n> > > >>\n> > > >> We can select nearest-neighbor and bilinear algorithms in\n> > > >> libyuv::NV12Scale with the filtering mode argument[1].\n> > > >> We may want to specify the scale algorithm on a class construction.\n> > > >> I would use libyuv here for better performance and less buggy (+ less\n> > > >> lines to be reviewed).\n> > > >> What do you'all think?\n> > > >\n> > > > We have considered libyuv early on, and I think it's an interesting\n> > > > option moving forward. The reason we have decided to go for a minimal\n> > > > scaler implementation to create JPEG thumbnails is to avoid adding an\n> > > > external dependency that isn't packaged by distributions. This allows us\n> > > > to finalize JPEG thumbnail support without delay, and decide on the\n> > > > longer term solution separately.\n> > > >\n> > > > It would be very nice if Google could work with distributions to package\n> > > > libyuv. I'd like to avoid including a copy of libyuv in libcamera.\n> > >\n> > > Lack of packaging is a big factor here. If it's not available we can't\n> > > use it ;-)\n> >\n> > Thanks. That is fair enough.\n> >\n> > > However I looked into this briefly yesterday, - we could use a meson\n> > > wrap. But I don't know how that will affect us distributing packages, as\n> > > then we also have to distribute libyuv.\n> > >\n> > > Or perhaps the meson wrap would link statically.\n> >\n> > Can we download libyuv somewhere in libcamera, e.g. //third_party,\n> > during meson config if it is necessary?\n> > The code under //third_party is ignored by .gitignore, i.e., is not a\n> > part of libcamera tree.\n> > We may want to maintain a commit hash to be checked out in the libcamera tree.\n>\n> As Kieran mentioned, we could use a meson wrap, and link statically to\n> libyuv. If we rely on a particular commit of libyuv then I think we\n> could as well copy the source code in the libcamera repository, to avoid\n> depending on an internet connection at meson config time. I'm however\n> not sure what issues we would encounter on the way. libyuv is integrated\n> with the Chromium code base, but looking, for instance, at the buildroot\n> package, it seems fairly easy to compile it standalone. We may need to\n> add support for the meson build system though.\n\nIf the alternative is to implement the conversion routines ourselves,\nI guess it's not any better than just forking libyuv permanently and\npossibly submitting any relevant changes back there to avoid diverging\ntoo much, especially given that libyuv has a proven and very well\noptimized implementation.\n\nRegardless of that, I'll check internally if we can do anything to\nmake it easier to use outside of Chromium.\n\nBest regards,\nTomasz\n\n>\n> As a conclusion, I think we'll need to figure out how we can use libyuv,\n> but not as a prerequisite for the JPEG thumbnail work. The best option\n> would be if Google maintained libyuv as a standalone library and worked\n> on getting it packaged in distributions :-)\n>\n> > > > One last comment to provide all the design background. We have also\n> > > > considered creating a format conversion library in libcamera, to gather\n> > > > code from various projects that exist out there. There's format\n> > > > conversion in libv4l2, in raw2rgbpnm, in projects such as ffmpeg, and of\n> > > > course in libyuv. I would be great if that could be consolidated and\n> > > > packaged, but that's a big task in itself.\n> > >\n> > > And that's the bigger picture. This should be consolidated. And that\n> > > consolidated library should support more than yuv ;-)\n> > >\n> > > Given it would be a software format convertor, and scaler, and ideally\n> > > with hardware acceleration support where possible, I think that quickly\n> > > becomes libisp - and the feature creep will ... creep in ;-)\n> > >\n> > > >> [1] https://chromium.googlesource.com/libyuv/libyuv/+/refs/heads/master/include/libyuv/scale.h#148\n> > > >>\n> > > >>>> On Wed, Oct 21, 2020 at 8:09 PM Kieran Bingham wrote:\n> > > >>>>> On 21/10/2020 12:03, Umang Jain wrote:\n> > > >>>>>> On 10/21/20 4:25 PM, Kieran Bingham wrote:\n> > > >>>>>>> On 21/10/2020 11:51, Umang Jain wrote:\n> > > >>>>>>>> On 10/21/20 2:49 PM, Kieran Bingham wrote:\n> > > >>>>>>>>> On 21/10/2020 09:08, Umang Jain wrote:\n> > > >>>>>>>>>> Add a basic image thumbnailer for NV12 frames being captured.\n> > > >>>>>>>>>> It shall generate a thumbnail image to be embedded as a part of\n> > > >>>>>>>>>> EXIF metadata of the frame. The output of the thumbnail will still\n> > > >>>>>>>>>> be NV12.\n> > > >>>>>>>>>>\n> > > >>>>>>>>>> Signed-off-by: Umang Jain \n> > > >>>>>>>>>> ---\n> > > >>>>>>>>>> src/android/jpeg/post_processor_jpeg.cpp | 39 +++++++++\n> > > >>>>>>>>>> src/android/jpeg/post_processor_jpeg.h | 3 +\n> > > >>>>>>>>>> src/android/jpeg/thumbnailer.cpp | 100 +++++++++++++++++++++++\n> > > >>>>>>>>>> src/android/jpeg/thumbnailer.h | 40 +++++++++\n> > > >>>>>>>>>> src/android/meson.build | 1 +\n> > > >>>>>>>>>> 5 files changed, 183 insertions(+)\n> > > >>>>>>>>>> create mode 100644 src/android/jpeg/thumbnailer.cpp\n> > > >>>>>>>>>> create mode 100644 src/android/jpeg/thumbnailer.h\n> > > >>>>>>>>>>\n> > > >>>>>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp\n> > > >>>>>>>>>> b/src/android/jpeg/post_processor_jpeg.cpp\n> > > >>>>>>>>>> index 9d452b7..f5f1f78 100644\n> > > >>>>>>>>>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n> > > >>>>>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n> > > >>>>>>>>>> @@ -11,6 +11,7 @@\n> > > >>>>>>>>>> #include \"../camera_metadata.h\"\n> > > >>>>>>>>>> #include \"encoder_libjpeg.h\"\n> > > >>>>>>>>>> #include \"exif.h\"\n> > > >>>>>>>>>> +#include \"thumbnailer.h\"\n> > > >>>>>>>>>> #include \n> > > >>>>>>>>>> @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const StreamConfiguration &inCfg,\n> > > >>>>>>>>>> return encoder_->configure(inCfg);\n> > > >>>>>>>>>> }\n> > > >>>>>>>>>> +void PostProcessorJpeg::generateThumbnail(const libcamera::Span &source,\n> > > >>>>>>>>>> + std::vector &thumbnail)\n> > > >>>>>>>>>> +{\n> > > >>>>>>>>>> + libcamera::Span destination;\n> > > >>>>>>>>>> + Thumbnailer thumbnailer;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + thumbnailer.configure(streamSize_, formats::NV12);\n> > > >>>>>>>>>> + libcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n> > > >>>>>>>>>> + thumbnailer.scaleBuffer(source, thumbnail);\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + if (thumbnail.data()) {\n> > > >>>>>>>>>> + StreamConfiguration thumbnailCfg;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + std::unique_ptr encoder =\n> > > >>>>>>>>>> + std::make_unique();\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + thumbnailCfg.pixelFormat = formats::NV12;\n> > > >>>>>>>>>> + thumbnailCfg.size = targetSize;\n> > > >>>>>>>>>> + encoder->configure(thumbnailCfg);\n> > > >>>>>>>>>\n> > > >>>>>>>>> thumbnail.capacity() might be quite low here.\n> > > >>>>>>>>> We need to make sure the vector is big enough at this point, you might\n> > > >>>>>>>>> need to do something like:\n> > > >>>>>>>>>\n> > > >>>>>>>>> thumbnail.resize(targetSize.width * targetSize.height * 4);\n> > > >>>>>>>>\n> > > >>>>>>>> I am not sure I follow. This is compressing the thumbnail part right? So\n> > > >>>>>>>> thumbnail is the \"source\" for the encoder here (Please refer to\n> > > >>>>>>>> ->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n> > > >>>>>>>\n> > > >>>>>>> Ugh, sorry - tired eyes indeed. Yes, please ignore those comments.\n> > > >>>>>>>\n> > > >>>>>>>>> Really we should obtain that size from the encoder. I thought we had a\n> > > >>>>>>>>> helper to do that already, but it seems we don't.\n> > > >>>>>>>>> We could/should add Encoder::maxOutput(); which would ask the encoder\n> > > >>>>>>>>> \"For your given configuration, what is the maximum number of bytes you\n> > > >>>>>>>>> might output\".\n> > > >>>>>>>>>\n> > > >>>>>>>>> Then of course we'd do:\n> > > >>>>>>>>> thumbnail.resize(encoder.maxOutput());\n> > > >>>>>>>>>\n> > > >>>>>>>>>\n> > > >>>>>>>>>> + int jpeg_size = encoder->encode({ thumbnail.data(), thumbnail.capacity() },\n> > > >>>>>>>>>> + destination, { });\n> > > >>>>>>>>\n> > > >>>>>>>> As said above, thumbnail is the source here. The compressed thumbnail\n> > > >>>>>>>> output is carried in destination. And, destination is a local span<>\n> > > >>>>>>>> here. Does it makes sense?\n> > > >>>>>>>\n> > > >>>>>>> Yes, I had mixed up source and destination I'm sorry.\n> > > >>>>>>>\n> > > >>>>>>> So who/where is allocating the destination?\n> > > >>>>>>\n> > > >>>>>> Ah, that's a good question and I think I just found a bug. I currently\n> > > >>>>>> pass on a locally allocated Span<> 'destination' to the encoder. But\n> > > >>>>>> there is no explicit allocation here. I think I need to point / create\n> > > >>>>>> the Span with the output size / bytes and then pass to the encode().\n> > > >>>>>\n> > > >>>>> :-) Ok - then please apply all of my earlier comments to that part\n> > > >>>>> instead. ;-)\n> > > >>>>>\n> > > >>>>> We certainly need somewhere to store the compressed image, to be able to\n> > > >>>>> pass it into the exif :-D\n> > > >>>>>\n> > > >>>>>>>>>> + LOG(JPEG, Info) << \"Thumbnail compress returned \"\n> > > >>>>>>>>>> + << jpeg_size << \" bytes\";\n> > > >>>>>>>>>\n> > > >>>>>>>>> And I presume we could then do an:\n> > > >>>>>>>>> thumbnail.resize(jpeg_size);\n> > > >>>>>>>>>\n> > > >>>>>>>>> here to update the thumbnail with the correct size. I'd be weary of the\n> > > >>>>>>>>> resize operations doing lots of re-allocations though, so perhaps we\n> > > >>>>>>>>> want to minimize that. But lets get to something that works first\n> > > >>>>>>>>> before worrying about optimising.\n> > > >>>>>>>>>\n> > > >>>>>>>>>> + }\n> > > >>>>>>>>>> +}\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n> > > >>>>>>>>>> const libcamera::Span &destination,\n> > > >>>>>>>>>> CameraMetadata *metadata)\n> > > >>>>>>>>>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const\n> > > >>>>>>>>>> libcamera::FrameBuffer *source,\n> > > >>>>>>>>>> return jpeg_size;\n> > > >>>>>>>>>> }\n> > > >>>>>>>>>> + std::vector thumbnail;\n> > > >>>>>>>>>\n> > > >>>>>>>>> You need to resize this somewhere.\n> > > >>>>>>>>> Edit: Now seen a better place above ;-)\n> > > >>>>>>>>\n> > > >>>>>>>> The thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\n> > > >>>>>>>> We just pass in to the thumbnailer, while keeping its ownership in\n> > > >>>>>>>> PostProcessorJpeg.\n> > > >>>>>>>>\n> > > >>>>>>>>>> + generateThumbnail(destination, thumbnail);\n> > > >>>>>>>>>> + /*\n> > > >>>>>>>>>> + * \\todo: Write the compressed thumbnail to a file for inspection.\n> > > >>>>>>>>>> + * (I) Check if we can still write the thumbnail to EXIF here.\n> > > >>>>>>>>>> + * If not, we might need to move the thumbnailer logic to encoder.\n> > > >>>>>>>>>> + * And if we do that, first we need to make sure we get can\n> > > >>>>>>>>>> + * compressed data written to destination first before calling\n> > > >>>>>>>>>> + * jpeg_finish_compress operation somehow. Thumbnailing will\n> > > >>>>>>>>>> + * only occur if we have compressed data available first.\n> > > >>>>>>>>>> + */\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> /*\n> > > >>>>>>>>>> * Fill in the JPEG blob header.\n> > > >>>>>>>>>> *\n> > > >>>>>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.h\n> > > >>>>>>>>>> b/src/android/jpeg/post_processor_jpeg.h\n> > > >>>>>>>>>> index 62c8650..05601ee 100644\n> > > >>>>>>>>>> --- a/src/android/jpeg/post_processor_jpeg.h\n> > > >>>>>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.h\n> > > >>>>>>>>>> @@ -28,6 +28,9 @@ public:\n> > > >>>>>>>>>> CameraMetadata *metadata) override;\n> > > >>>>>>>>>> private:\n> > > >>>>>>>>>> + void generateThumbnail(const libcamera::Span &source,\n> > > >>>>>>>>>> + std::vector &thumbnail);\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> CameraDevice *cameraDevice_;\n> > > >>>>>>>>>> std::unique_ptr encoder_;\n> > > >>>>>>>>>> libcamera::Size streamSize_;\n> > > >>>>>>>>>> diff --git a/src/android/jpeg/thumbnailer.cpp\n> > > >>>>>>>>>> b/src/android/jpeg/thumbnailer.cpp\n> > > >>>>>>>>>> new file mode 100644\n> > > >>>>>>>>>> index 0000000..3163576\n> > > >>>>>>>>>> --- /dev/null\n> > > >>>>>>>>>> +++ b/src/android/jpeg/thumbnailer.cpp\n> > > >>>>>>>>>> @@ -0,0 +1,100 @@\n> > > >>>>>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > >>>>>>>>>> +/*\n> > > >>>>>>>>>> + * Copyright (C) 2020, Google Inc.\n> > > >>>>>>>>>> + *\n> > > >>>>>>>>>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n> > > >>>>>>>>>> + */\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +#include \"thumbnailer.h\"\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +#include \n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +#include \"libcamera/internal/file.h\"\n> > > >>>>>>>>>> +#include \"libcamera/internal/log.h\"\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +using namespace libcamera;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +Thumbnailer::Thumbnailer()\n> > > >>>>>>>>>> + : validConfiguration_(false)\n> > > >>>>>>>>>> +{\n> > > >>>>>>>>>> +}\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat pixelFormat)\n> > > >>>>>>>>>> +{\n> > > >>>>>>>>>> + sourceSize_ = sourceSize;\n> > > >>>>>>>>>> + pixelFormat_ = pixelFormat;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + if (pixelFormat_ != formats::NV12) {\n> > > >>>>>>>>>> + LOG (Thumbnailer, Error) << \"Failed to configure: Pixel Format \"\n> > > >>>>>>>>>> + << pixelFormat_.toString() << \" unsupported.\";\n> > > >>>>>>>>>> + return;\n> > > >>>>>>>>>> + }\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + validConfiguration_ = true;\n> > > >>>>>>>>>> +}\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +/*\n> > > >>>>>>>>>> + * The Exif specification recommends the width of the thumbnail to be a\n> > > >>>>>>>>>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding height\n> > > >>>>>>>>>> + * keeping the aspect ratio same as of the source.\n> > > >>>>>>>>>> + */\n> > > >>>>>>>>>> +Size Thumbnailer::computeThumbnailSize()\n> > > >>>>>>>>>> +{\n> > > >>>>>>>>>> + unsigned int targetHeight;\n> > > >>>>>>>>>> + unsigned int targetWidth = 160;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + targetHeight = targetWidth * sourceSize_.height / sourceSize_.width;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + if (targetHeight & 1)\n> > > >>>>>>>>>> + targetHeight++;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + return Size(targetWidth, targetHeight);\n> > > >>>>>>>>>> +}\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +void\n> > > >>>>>>>>>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n> > > >>>>>>>>>> + std::vector &destination)\n> > > >>>>>>>>>> +{\n> > > >>>>>>>>>> + if (!validConfiguration_) {\n> > > >>>>>>>>>> + LOG(Thumbnailer, Error) << \"config is unconfigured or invalid.\";\n> > > >>>>>>>>>> + return;\n> > > >>>>>>>>>> + }\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + targetSize_ = computeThumbnailSize();\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + const unsigned int sw = sourceSize_.width;\n> > > >>>>>>>>>> + const unsigned int sh = sourceSize_.height;\n> > > >>>>>>>>>> + const unsigned int tw = targetSize_.width;\n> > > >>>>>>>>>> + const unsigned int th = targetSize_.height;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + /* Image scaling block implementing nearest-neighbour algorithm. */\n> > > >>>>>>>>>> + unsigned char *src = static_cast(source.data());\n> > > >>>>>>>>>> + unsigned char *src_c = src + sh * sw;\n> > > >>>>>>>>>> + unsigned char *src_cb, *src_cr;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + size_t dstSize = (th * tw) + ((th/2) * tw);\n> > > >>>>>>>>>> + destination.reserve(dstSize);\n> > > >>>>>>>>>> + unsigned char *dst = destination.data();\n> > > >>>>>>>>>> + unsigned char *dst_c = dst + th * tw;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + for (unsigned int y = 0; y < th; y+=2) {\n> > > >>>>>>>>>> + unsigned int sourceY = (sh*y + th/2) / th;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + src_cb = src_c + (sourceY/2) * sw + 0;\n> > > >>>>>>>>>> + src_cr = src_c + (sourceY/2) * sw + 1;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + for (unsigned int x = 0; x < tw; x+=2) {\n> > > >>>>>>>>>> + unsigned int sourceX = (sw*x + tw/2) / tw;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + dst[y * tw + x] = src[sw * sourceY + sourceX];\n> > > >>>>>>>>>> + dst[(y+1) * tw + x] = src[sw * (sourceY+1) + sourceX];\n> > > >>>>>>>>>> + dst[y * tw + (x+1)] = src[sw * sourceY + (sourceX+1)];\n> > > >>>>>>>>>> + dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) + (sourceX+1)];\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + dst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n> > > >>>>>>>>>> + dst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n> > > >>>>>>>>>> + }\n> > > >>>>>>>>>> + }\n> > > >>>>>>>>>> +}\n> > > >>>>>>>>>> diff --git a/src/android/jpeg/thumbnailer.h\n> > > >>>>>>>>>> b/src/android/jpeg/thumbnailer.h\n> > > >>>>>>>>>> new file mode 100644\n> > > >>>>>>>>>> index 0000000..bab9855\n> > > >>>>>>>>>> --- /dev/null\n> > > >>>>>>>>>> +++ b/src/android/jpeg/thumbnailer.h\n> > > >>>>>>>>>> @@ -0,0 +1,40 @@\n> > > >>>>>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > > >>>>>>>>>> +/*\n> > > >>>>>>>>>> + * Copyright (C) 2020, Google Inc.\n> > > >>>>>>>>>> + *\n> > > >>>>>>>>>> + * thumbnailer.h - Basic image thumbnailer from NV12\n> > > >>>>>>>>>> + */\n> > > >>>>>>>>>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n> > > >>>>>>>>>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +#include \n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +#include \"libcamera/internal/buffer.h\"\n> > > >>>>>>>>>> +#include \"libcamera/internal/formats.h\"\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +class Thumbnailer\n> > > >>>>>>>>>> +{\n> > > >>>>>>>>>> +public:\n> > > >>>>>>>>>> + Thumbnailer();\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + void configure(const libcamera::Size &sourceSize,\n> > > >>>>>>>>>> + libcamera::PixelFormat pixelFormat);\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + /*\n> > > >>>>>>>>>> + * \\todo: Discuss if we can return targetSize_ via configure() or\n> > > >>>>>>>>>> + * scaleBuffer(). We need targetSize_ to re-encode the scaled\n> > > >>>>>>>>>> buffer\n> > > >>>>>>>>>> + * via encoder in PostProcssorJpeg::writeThumbnail().\n> > > >>>>>>>>>> + */\n> > > >>>>>>>>>> + libcamera::Size computeThumbnailSize();\n> > > >>>>>>>>>> + void scaleBuffer(const libcamera::Span &source,\n> > > >>>>>>>>>> + std::vector &dest);\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +private:\n> > > >>>>>>>>>> + libcamera::PixelFormat pixelFormat_;\n> > > >>>>>>>>>> + libcamera::Size sourceSize_;\n> > > >>>>>>>>>> + libcamera::Size targetSize_;\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> + bool validConfiguration_;\n> > > >>>>>>>>>> +};\n> > > >>>>>>>>>> +\n> > > >>>>>>>>>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n> > > >>>>>>>>>> diff --git a/src/android/meson.build b/src/android/meson.build\n> > > >>>>>>>>>> index 5a01bea..3905e2f 100644\n> > > >>>>>>>>>> --- a/src/android/meson.build\n> > > >>>>>>>>>> +++ b/src/android/meson.build\n> > > >>>>>>>>>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n> > > >>>>>>>>>> 'jpeg/encoder_libjpeg.cpp',\n> > > >>>>>>>>>> 'jpeg/exif.cpp',\n> > > >>>>>>>>>> 'jpeg/post_processor_jpeg.cpp',\n> > > >>>>>>>>>> + 'jpeg/thumbnailer.cpp',\n> > > >>>>>>>>>> ])\n> > > >>>>>>>>>> android_camera_metadata_sources = files([\n> > > >>>>>>>>>>\n>\n> --\n> Regards,\n>\n> Laurent Pinchart\n> _______________________________________________\n> libcamera-devel mailing list\n> libcamera-devel@lists.libcamera.org\n> https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id C483CBD808\n\tfor ;\n\tThu, 10 Dec 2020 05:32:55 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 29ED767F70;\n\tThu, 10 Dec 2020 06:32:55 +0100 (CET)","from mail-ed1-x536.google.com (mail-ed1-x536.google.com\n\t[IPv6:2a00:1450:4864:20::536])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 51BD860322\n\tfor ;\n\tThu, 10 Dec 2020 06:32:53 +0100 (CET)","by mail-ed1-x536.google.com with SMTP id b73so4138449edf.13\n\tfor ;\n\tWed, 09 Dec 2020 21:32:53 -0800 (PST)","from mail-wr1-f48.google.com (mail-wr1-f48.google.com.\n\t[209.85.221.48]) by smtp.gmail.com with ESMTPSA id\n\tu23sm3436666ejy.87.2020.12.09.21.32.51\n\tfor \n\t(version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);\n\tWed, 09 Dec 2020 21:32:51 -0800 (PST)","by mail-wr1-f48.google.com with SMTP id t4so4100286wrr.12\n\tfor ;\n\tWed, 09 Dec 2020 21:32:51 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=chromium.org header.i=@chromium.org\n\theader.b=\"TkOEulP4\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org;\n\ts=google; \n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=qjBXvbpD8YvOz2D6u83mmLkm9EfU4sBkdSyoZfIwiUo=;\n\tb=TkOEulP43CZfZiYmj2S2jVY0DtvODpTUKWItTr/azoILcTtdIYAdmBsG7mBRIEFD69\n\tcIjfZyUVlfuD43qmto/xhtOPTNbbNeJFzKANJ2aFxvSOB+HKc/1xxGUuWXz+d7AJ3OPI\n\te5YY7jFSfNIJYt79JHviivgfYGcfmlWWAu11Q=","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=qjBXvbpD8YvOz2D6u83mmLkm9EfU4sBkdSyoZfIwiUo=;\n\tb=cqppbKK02Dwb/v/CUlgzZ+WxT1w8nJN8LljG1qDXAckA1wA0f1nyLzjTuqpSkbld5Q\n\tica0aBWyLUiX+d1bPpSPwwhPxcgMI8Bp+ld25zdmfLhXUIJ7/oVcUpG5vxYy8VZbPRuG\n\tc2BCc9CJfsJzjg1v9rYk8hJuiNlagQPVECp4KpfW4uF5DH/15CMz8nTUbjE/79vfIVrI\n\tHXeSZJ1HHKRTAv8upYy7RxmaCrMrD+V0pgPqykDducoeVLrcXhxag0tjlMmGIzr9Chn5\n\tcsWZmnYdTXr8iLH1don0Jl1wF5a8PZAUVsq4upX+VGLa8uijplYOCxM41JqyN2EB7yyt\n\tj4cA==","X-Gm-Message-State":"AOAM532OBPk2WvLxvn+baRokZFFMNSSoGMDqQlU9JP38hlIXNi/O/tVy\n\tEmL8793K3KTcS73k+OoYU0Gq9BL7VYPSv4rM","X-Google-Smtp-Source":"ABdhPJy5pt7hd7I2Q0ydGiwWX6wkDcbRI7Li3ReBh9FxIgRbnjgdGIuXKrDdmbqlGHGwZVovejhpSA==","X-Received":["by 2002:a05:6402:3192:: with SMTP id\n\tdi18mr5030750edb.332.1607578372363; \n\tWed, 09 Dec 2020 21:32:52 -0800 (PST)","by 2002:adf:bb0e:: with SMTP id\n\tr14mr6282719wrg.159.1607578370634; \n\tWed, 09 Dec 2020 21:32:50 -0800 (PST)"],"MIME-Version":"1.0","References":"\n\t<9ae3994c-e566-5410-e362-50a0dc7aefbd@ideasonboard.com>\n\t<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>\n\t<9ca3b492-8468-e9cf-b103-b4350be472b9@ideasonboard.com>\n\t\n\t<20201022135914.GA3938@pendragon.ideasonboard.com>\n\t\n\t<20201023035906.GA4113@pendragon.ideasonboard.com>\n\t<734c408b-2e5b-6ee4-af07-f9c45416abce@ideasonboard.com>\n\t\n\t<20201027010400.GO3756@pendragon.ideasonboard.com>","In-Reply-To":"<20201027010400.GO3756@pendragon.ideasonboard.com>","From":"Tomasz Figa ","Date":"Thu, 10 Dec 2020 14:32:39 +0900","X-Gmail-Original-Message-ID":"","Message-ID":"","To":"Laurent Pinchart ","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":14180,"web_url":"https://patchwork.libcamera.org/comment/14180/","msgid":"","date":"2020-12-10T07:45:32","subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Tomasz,\n\nOn Thu, Dec 10, 2020 at 02:32:39PM +0900, Tomasz Figa wrote:\n> On Tue, Oct 27, 2020 at 10:04 AM Laurent Pinchart wrote:\n> > On Mon, Oct 26, 2020 at 04:29:20PM +0900, Hirokazu Honda wrote:\n> >> On Fri, Oct 23, 2020 at 5:39 PM Kieran Bingham wrote:\n> >>> On 23/10/2020 04:59, Laurent Pinchart wrote:\n> >>>> On Fri, Oct 23, 2020 at 09:37:09AM +0900, Hirokazu Honda wrote:\n> >>>>> On Thu, Oct 22, 2020 at 11:00 PM Laurent Pinchart wrote:\n> >>>>>> On Thu, Oct 22, 2020 at 08:42:01PM +0900, Hirokazu Honda wrote:\n> >>>>>>> Hi Umang,\n> >>>>>>> Thanks for the work. I couldn't have time to review these today and\n> >>>>>>> will review tomorrow.\n> >>>>>>> In my understanding, the thumbnail class is basically to scale frames\n> >>>>>>> to the given destination.\n> >>>>>>> Can we generalize it as PostProcessor interface so that we will be\n> >>>>>>> able to make use of it for down-scaling regardless of whether jpeg\n> >>>>>>> encoding is required.\n> >>>>>>> It will be necessary when Android HAL adaptation layer has to output\n> >>>>>>> multiple different NV12 streams while the native camera can produce a\n> >>>>>>> single NV12 stream simultaneously.\n> >>>>>>\n> >>>>>> One point to consider is that the thumbnailer is a down-scaler using a\n> >>>>>> nearest-neighbour algorithm. It's good enough for thumbnails, but will\n> >>>>>> lead to awful quality when scaling streams. For that, I think we need a\n> >>>>>> better scaler implementation, and we could then replace the thumbnailer\n> >>>>>> class with it.\n> >>>>>\n> >>>>> We can select nearest-neighbor and bilinear algorithms in\n> >>>>> libyuv::NV12Scale with the filtering mode argument[1].\n> >>>>> We may want to specify the scale algorithm on a class construction.\n> >>>>> I would use libyuv here for better performance and less buggy (+ less\n> >>>>> lines to be reviewed).\n> >>>>> What do you'all think?\n> >>>>\n> >>>> We have considered libyuv early on, and I think it's an interesting\n> >>>> option moving forward. The reason we have decided to go for a minimal\n> >>>> scaler implementation to create JPEG thumbnails is to avoid adding an\n> >>>> external dependency that isn't packaged by distributions. This allows us\n> >>>> to finalize JPEG thumbnail support without delay, and decide on the\n> >>>> longer term solution separately.\n> >>>>\n> >>>> It would be very nice if Google could work with distributions to package\n> >>>> libyuv. I'd like to avoid including a copy of libyuv in libcamera.\n> >>>\n> >>> Lack of packaging is a big factor here. If it's not available we can't\n> >>> use it ;-)\n> >>\n> >> Thanks. That is fair enough.\n> >>\n> >>> However I looked into this briefly yesterday, - we could use a meson\n> >>> wrap. But I don't know how that will affect us distributing packages, as\n> >>> then we also have to distribute libyuv.\n> >>>\n> >>> Or perhaps the meson wrap would link statically.\n> >>\n> >> Can we download libyuv somewhere in libcamera, e.g. //third_party,\n> >> during meson config if it is necessary?\n> >> The code under //third_party is ignored by .gitignore, i.e., is not a\n> >> part of libcamera tree.\n> >> We may want to maintain a commit hash to be checked out in the libcamera tree.\n> >\n> > As Kieran mentioned, we could use a meson wrap, and link statically to\n> > libyuv. If we rely on a particular commit of libyuv then I think we\n> > could as well copy the source code in the libcamera repository, to avoid\n> > depending on an internet connection at meson config time. I'm however\n> > not sure what issues we would encounter on the way. libyuv is integrated\n> > with the Chromium code base, but looking, for instance, at the buildroot\n> > package, it seems fairly easy to compile it standalone. We may need to\n> > add support for the meson build system though.\n> \n> If the alternative is to implement the conversion routines ourselves,\n> I guess it's not any better than just forking libyuv permanently and\n> possibly submitting any relevant changes back there to avoid diverging\n> too much, especially given that libyuv has a proven and very well\n> optimized implementation.\n\nI agree with you.\n\n> Regardless of that, I'll check internally if we can do anything to\n> make it easier to use outside of Chromium.\n\nThat would be great, I think it would benefit way more projects than\njust libcamera.\n\n> > As a conclusion, I think we'll need to figure out how we can use libyuv,\n> > but not as a prerequisite for the JPEG thumbnail work. The best option\n> > would be if Google maintained libyuv as a standalone library and worked\n> > on getting it packaged in distributions :-)\n> >\n> >>>> One last comment to provide all the design background. We have also\n> >>>> considered creating a format conversion library in libcamera, to gather\n> >>>> code from various projects that exist out there. There's format\n> >>>> conversion in libv4l2, in raw2rgbpnm, in projects such as ffmpeg, and of\n> >>>> course in libyuv. I would be great if that could be consolidated and\n> >>>> packaged, but that's a big task in itself.\n> >>>\n> >>> And that's the bigger picture. This should be consolidated. And that\n> >>> consolidated library should support more than yuv ;-)\n> >>>\n> >>> Given it would be a software format convertor, and scaler, and ideally\n> >>> with hardware acceleration support where possible, I think that quickly\n> >>> becomes libisp - and the feature creep will ... creep in ;-)\n> >>>\n> >>>>> [1] https://chromium.googlesource.com/libyuv/libyuv/+/refs/heads/master/include/libyuv/scale.h#148\n> >>>>>\n> >>>>>>> On Wed, Oct 21, 2020 at 8:09 PM Kieran Bingham wrote:\n> >>>>>>>> On 21/10/2020 12:03, Umang Jain wrote:\n> >>>>>>>>> On 10/21/20 4:25 PM, Kieran Bingham wrote:\n> >>>>>>>>>> On 21/10/2020 11:51, Umang Jain wrote:\n> >>>>>>>>>>> On 10/21/20 2:49 PM, Kieran Bingham wrote:\n> >>>>>>>>>>>> On 21/10/2020 09:08, Umang Jain wrote:\n> >>>>>>>>>>>>> Add a basic image thumbnailer for NV12 frames being captured.\n> >>>>>>>>>>>>> It shall generate a thumbnail image to be embedded as a part of\n> >>>>>>>>>>>>> EXIF metadata of the frame. The output of the thumbnail will still\n> >>>>>>>>>>>>> be NV12.\n> >>>>>>>>>>>>>\n> >>>>>>>>>>>>> Signed-off-by: Umang Jain \n> >>>>>>>>>>>>> ---\n> >>>>>>>>>>>>> src/android/jpeg/post_processor_jpeg.cpp | 39 +++++++++\n> >>>>>>>>>>>>> src/android/jpeg/post_processor_jpeg.h | 3 +\n> >>>>>>>>>>>>> src/android/jpeg/thumbnailer.cpp | 100 +++++++++++++++++++++++\n> >>>>>>>>>>>>> src/android/jpeg/thumbnailer.h | 40 +++++++++\n> >>>>>>>>>>>>> src/android/meson.build | 1 +\n> >>>>>>>>>>>>> 5 files changed, 183 insertions(+)\n> >>>>>>>>>>>>> create mode 100644 src/android/jpeg/thumbnailer.cpp\n> >>>>>>>>>>>>> create mode 100644 src/android/jpeg/thumbnailer.h\n> >>>>>>>>>>>>>\n> >>>>>>>>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.cpp\n> >>>>>>>>>>>>> b/src/android/jpeg/post_processor_jpeg.cpp\n> >>>>>>>>>>>>> index 9d452b7..f5f1f78 100644\n> >>>>>>>>>>>>> --- a/src/android/jpeg/post_processor_jpeg.cpp\n> >>>>>>>>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.cpp\n> >>>>>>>>>>>>> @@ -11,6 +11,7 @@\n> >>>>>>>>>>>>> #include \"../camera_metadata.h\"\n> >>>>>>>>>>>>> #include \"encoder_libjpeg.h\"\n> >>>>>>>>>>>>> #include \"exif.h\"\n> >>>>>>>>>>>>> +#include \"thumbnailer.h\"\n> >>>>>>>>>>>>> #include \n> >>>>>>>>>>>>> @@ -44,6 +45,32 @@ int PostProcessorJpeg::configure(const StreamConfiguration &inCfg,\n> >>>>>>>>>>>>> return encoder_->configure(inCfg);\n> >>>>>>>>>>>>> }\n> >>>>>>>>>>>>> +void PostProcessorJpeg::generateThumbnail(const libcamera::Span &source,\n> >>>>>>>>>>>>> + std::vector &thumbnail)\n> >>>>>>>>>>>>> +{\n> >>>>>>>>>>>>> + libcamera::Span destination;\n> >>>>>>>>>>>>> + Thumbnailer thumbnailer;\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + thumbnailer.configure(streamSize_, formats::NV12);\n> >>>>>>>>>>>>> + libcamera::Size targetSize = thumbnailer.computeThumbnailSize();\n> >>>>>>>>>>>>> + thumbnailer.scaleBuffer(source, thumbnail);\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + if (thumbnail.data()) {\n> >>>>>>>>>>>>> + StreamConfiguration thumbnailCfg;\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + std::unique_ptr encoder =\n> >>>>>>>>>>>>> + std::make_unique();\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + thumbnailCfg.pixelFormat = formats::NV12;\n> >>>>>>>>>>>>> + thumbnailCfg.size = targetSize;\n> >>>>>>>>>>>>> + encoder->configure(thumbnailCfg);\n> >>>>>>>>>>>>\n> >>>>>>>>>>>> thumbnail.capacity() might be quite low here.\n> >>>>>>>>>>>> We need to make sure the vector is big enough at this point, you might\n> >>>>>>>>>>>> need to do something like:\n> >>>>>>>>>>>>\n> >>>>>>>>>>>> thumbnail.resize(targetSize.width * targetSize.height * 4);\n> >>>>>>>>>>>\n> >>>>>>>>>>> I am not sure I follow. This is compressing the thumbnail part right? So\n> >>>>>>>>>>> thumbnail is the \"source\" for the encoder here (Please refer to\n> >>>>>>>>>>> ->encode() below) and why would we resize the source(i.e. 'thumbnail')?\n> >>>>>>>>>>\n> >>>>>>>>>> Ugh, sorry - tired eyes indeed. Yes, please ignore those comments.\n> >>>>>>>>>>\n> >>>>>>>>>>>> Really we should obtain that size from the encoder. I thought we had a\n> >>>>>>>>>>>> helper to do that already, but it seems we don't.\n> >>>>>>>>>>>> We could/should add Encoder::maxOutput(); which would ask the encoder\n> >>>>>>>>>>>> \"For your given configuration, what is the maximum number of bytes you\n> >>>>>>>>>>>> might output\".\n> >>>>>>>>>>>>\n> >>>>>>>>>>>> Then of course we'd do:\n> >>>>>>>>>>>> thumbnail.resize(encoder.maxOutput());\n> >>>>>>>>>>>>\n> >>>>>>>>>>>>\n> >>>>>>>>>>>>> + int jpeg_size = encoder->encode({ thumbnail.data(), thumbnail.capacity() },\n> >>>>>>>>>>>>> + destination, { });\n> >>>>>>>>>>>\n> >>>>>>>>>>> As said above, thumbnail is the source here. The compressed thumbnail\n> >>>>>>>>>>> output is carried in destination. And, destination is a local span<>\n> >>>>>>>>>>> here. Does it makes sense?\n> >>>>>>>>>>\n> >>>>>>>>>> Yes, I had mixed up source and destination I'm sorry.\n> >>>>>>>>>>\n> >>>>>>>>>> So who/where is allocating the destination?\n> >>>>>>>>>\n> >>>>>>>>> Ah, that's a good question and I think I just found a bug. I currently\n> >>>>>>>>> pass on a locally allocated Span<> 'destination' to the encoder. But\n> >>>>>>>>> there is no explicit allocation here. I think I need to point / create\n> >>>>>>>>> the Span with the output size / bytes and then pass to the encode().\n> >>>>>>>>\n> >>>>>>>> :-) Ok - then please apply all of my earlier comments to that part\n> >>>>>>>> instead. ;-)\n> >>>>>>>>\n> >>>>>>>> We certainly need somewhere to store the compressed image, to be able to\n> >>>>>>>> pass it into the exif :-D\n> >>>>>>>>\n> >>>>>>>>>>>>> + LOG(JPEG, Info) << \"Thumbnail compress returned \"\n> >>>>>>>>>>>>> + << jpeg_size << \" bytes\";\n> >>>>>>>>>>>>\n> >>>>>>>>>>>> And I presume we could then do an:\n> >>>>>>>>>>>> thumbnail.resize(jpeg_size);\n> >>>>>>>>>>>>\n> >>>>>>>>>>>> here to update the thumbnail with the correct size. I'd be weary of the\n> >>>>>>>>>>>> resize operations doing lots of re-allocations though, so perhaps we\n> >>>>>>>>>>>> want to minimize that. But lets get to something that works first\n> >>>>>>>>>>>> before worrying about optimising.\n> >>>>>>>>>>>>\n> >>>>>>>>>>>>> + }\n> >>>>>>>>>>>>> +}\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> int PostProcessorJpeg::process(const libcamera::FrameBuffer *source,\n> >>>>>>>>>>>>> const libcamera::Span &destination,\n> >>>>>>>>>>>>> CameraMetadata *metadata)\n> >>>>>>>>>>>>> @@ -73,6 +100,18 @@ int PostProcessorJpeg::process(const\n> >>>>>>>>>>>>> libcamera::FrameBuffer *source,\n> >>>>>>>>>>>>> return jpeg_size;\n> >>>>>>>>>>>>> }\n> >>>>>>>>>>>>> + std::vector thumbnail;\n> >>>>>>>>>>>>\n> >>>>>>>>>>>> You need to resize this somewhere.\n> >>>>>>>>>>>> Edit: Now seen a better place above ;-)\n> >>>>>>>>>>>\n> >>>>>>>>>>> The thumbnail vector just gets resized in Thumbnailer::scaleBuffer(). :)\n> >>>>>>>>>>> We just pass in to the thumbnailer, while keeping its ownership in\n> >>>>>>>>>>> PostProcessorJpeg.\n> >>>>>>>>>>>\n> >>>>>>>>>>>>> + generateThumbnail(destination, thumbnail);\n> >>>>>>>>>>>>> + /*\n> >>>>>>>>>>>>> + * \\todo: Write the compressed thumbnail to a file for inspection.\n> >>>>>>>>>>>>> + * (I) Check if we can still write the thumbnail to EXIF here.\n> >>>>>>>>>>>>> + * If not, we might need to move the thumbnailer logic to encoder.\n> >>>>>>>>>>>>> + * And if we do that, first we need to make sure we get can\n> >>>>>>>>>>>>> + * compressed data written to destination first before calling\n> >>>>>>>>>>>>> + * jpeg_finish_compress operation somehow. Thumbnailing will\n> >>>>>>>>>>>>> + * only occur if we have compressed data available first.\n> >>>>>>>>>>>>> + */\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> /*\n> >>>>>>>>>>>>> * Fill in the JPEG blob header.\n> >>>>>>>>>>>>> *\n> >>>>>>>>>>>>> diff --git a/src/android/jpeg/post_processor_jpeg.h\n> >>>>>>>>>>>>> b/src/android/jpeg/post_processor_jpeg.h\n> >>>>>>>>>>>>> index 62c8650..05601ee 100644\n> >>>>>>>>>>>>> --- a/src/android/jpeg/post_processor_jpeg.h\n> >>>>>>>>>>>>> +++ b/src/android/jpeg/post_processor_jpeg.h\n> >>>>>>>>>>>>> @@ -28,6 +28,9 @@ public:\n> >>>>>>>>>>>>> CameraMetadata *metadata) override;\n> >>>>>>>>>>>>> private:\n> >>>>>>>>>>>>> + void generateThumbnail(const libcamera::Span &source,\n> >>>>>>>>>>>>> + std::vector &thumbnail);\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> CameraDevice *cameraDevice_;\n> >>>>>>>>>>>>> std::unique_ptr encoder_;\n> >>>>>>>>>>>>> libcamera::Size streamSize_;\n> >>>>>>>>>>>>> diff --git a/src/android/jpeg/thumbnailer.cpp\n> >>>>>>>>>>>>> b/src/android/jpeg/thumbnailer.cpp\n> >>>>>>>>>>>>> new file mode 100644\n> >>>>>>>>>>>>> index 0000000..3163576\n> >>>>>>>>>>>>> --- /dev/null\n> >>>>>>>>>>>>> +++ b/src/android/jpeg/thumbnailer.cpp\n> >>>>>>>>>>>>> @@ -0,0 +1,100 @@\n> >>>>>>>>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> >>>>>>>>>>>>> +/*\n> >>>>>>>>>>>>> + * Copyright (C) 2020, Google Inc.\n> >>>>>>>>>>>>> + *\n> >>>>>>>>>>>>> + * thumbnailer.cpp - Basic image thumbnailer from NV12\n> >>>>>>>>>>>>> + */\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> +#include \"thumbnailer.h\"\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> +#include \n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> +#include \"libcamera/internal/file.h\"\n> >>>>>>>>>>>>> +#include \"libcamera/internal/log.h\"\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> +using namespace libcamera;\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> +LOG_DEFINE_CATEGORY(Thumbnailer)\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> +Thumbnailer::Thumbnailer()\n> >>>>>>>>>>>>> + : validConfiguration_(false)\n> >>>>>>>>>>>>> +{\n> >>>>>>>>>>>>> +}\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> +void Thumbnailer::configure(const Size &sourceSize, PixelFormat pixelFormat)\n> >>>>>>>>>>>>> +{\n> >>>>>>>>>>>>> + sourceSize_ = sourceSize;\n> >>>>>>>>>>>>> + pixelFormat_ = pixelFormat;\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + if (pixelFormat_ != formats::NV12) {\n> >>>>>>>>>>>>> + LOG (Thumbnailer, Error) << \"Failed to configure: Pixel Format \"\n> >>>>>>>>>>>>> + << pixelFormat_.toString() << \" unsupported.\";\n> >>>>>>>>>>>>> + return;\n> >>>>>>>>>>>>> + }\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + validConfiguration_ = true;\n> >>>>>>>>>>>>> +}\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> +/*\n> >>>>>>>>>>>>> + * The Exif specification recommends the width of the thumbnail to be a\n> >>>>>>>>>>>>> + * mutiple of 16 (section 4.8.1). Hence, compute the corresponding height\n> >>>>>>>>>>>>> + * keeping the aspect ratio same as of the source.\n> >>>>>>>>>>>>> + */\n> >>>>>>>>>>>>> +Size Thumbnailer::computeThumbnailSize()\n> >>>>>>>>>>>>> +{\n> >>>>>>>>>>>>> + unsigned int targetHeight;\n> >>>>>>>>>>>>> + unsigned int targetWidth = 160;\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + targetHeight = targetWidth * sourceSize_.height / sourceSize_.width;\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + if (targetHeight & 1)\n> >>>>>>>>>>>>> + targetHeight++;\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + return Size(targetWidth, targetHeight);\n> >>>>>>>>>>>>> +}\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> +void\n> >>>>>>>>>>>>> +Thumbnailer::scaleBuffer(const libcamera::Span &source,\n> >>>>>>>>>>>>> + std::vector &destination)\n> >>>>>>>>>>>>> +{\n> >>>>>>>>>>>>> + if (!validConfiguration_) {\n> >>>>>>>>>>>>> + LOG(Thumbnailer, Error) << \"config is unconfigured or invalid.\";\n> >>>>>>>>>>>>> + return;\n> >>>>>>>>>>>>> + }\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + targetSize_ = computeThumbnailSize();\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + const unsigned int sw = sourceSize_.width;\n> >>>>>>>>>>>>> + const unsigned int sh = sourceSize_.height;\n> >>>>>>>>>>>>> + const unsigned int tw = targetSize_.width;\n> >>>>>>>>>>>>> + const unsigned int th = targetSize_.height;\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + /* Image scaling block implementing nearest-neighbour algorithm. */\n> >>>>>>>>>>>>> + unsigned char *src = static_cast(source.data());\n> >>>>>>>>>>>>> + unsigned char *src_c = src + sh * sw;\n> >>>>>>>>>>>>> + unsigned char *src_cb, *src_cr;\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + size_t dstSize = (th * tw) + ((th/2) * tw);\n> >>>>>>>>>>>>> + destination.reserve(dstSize);\n> >>>>>>>>>>>>> + unsigned char *dst = destination.data();\n> >>>>>>>>>>>>> + unsigned char *dst_c = dst + th * tw;\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + for (unsigned int y = 0; y < th; y+=2) {\n> >>>>>>>>>>>>> + unsigned int sourceY = (sh*y + th/2) / th;\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + src_cb = src_c + (sourceY/2) * sw + 0;\n> >>>>>>>>>>>>> + src_cr = src_c + (sourceY/2) * sw + 1;\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + for (unsigned int x = 0; x < tw; x+=2) {\n> >>>>>>>>>>>>> + unsigned int sourceX = (sw*x + tw/2) / tw;\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + dst[y * tw + x] = src[sw * sourceY + sourceX];\n> >>>>>>>>>>>>> + dst[(y+1) * tw + x] = src[sw * (sourceY+1) + sourceX];\n> >>>>>>>>>>>>> + dst[y * tw + (x+1)] = src[sw * sourceY + (sourceX+1)];\n> >>>>>>>>>>>>> + dst[(y+1) * tw + (x+1)] = src[sw * (sourceY+1) + (sourceX+1)];\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + dst_c[(y/2) * tw + x + 0] = src_cb[(sourceX/2) * 2];\n> >>>>>>>>>>>>> + dst_c[(y/2) * tw + x + 1] = src_cr[(sourceX/2) * 2];\n> >>>>>>>>>>>>> + }\n> >>>>>>>>>>>>> + }\n> >>>>>>>>>>>>> +}\n> >>>>>>>>>>>>> diff --git a/src/android/jpeg/thumbnailer.h\n> >>>>>>>>>>>>> b/src/android/jpeg/thumbnailer.h\n> >>>>>>>>>>>>> new file mode 100644\n> >>>>>>>>>>>>> index 0000000..bab9855\n> >>>>>>>>>>>>> --- /dev/null\n> >>>>>>>>>>>>> +++ b/src/android/jpeg/thumbnailer.h\n> >>>>>>>>>>>>> @@ -0,0 +1,40 @@\n> >>>>>>>>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> >>>>>>>>>>>>> +/*\n> >>>>>>>>>>>>> + * Copyright (C) 2020, Google Inc.\n> >>>>>>>>>>>>> + *\n> >>>>>>>>>>>>> + * thumbnailer.h - Basic image thumbnailer from NV12\n> >>>>>>>>>>>>> + */\n> >>>>>>>>>>>>> +#ifndef __ANDROID_JPEG_THUMBNAILER_H__\n> >>>>>>>>>>>>> +#define __ANDROID_JPEG_THUMBNAILER_H__\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> +#include \n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> +#include \"libcamera/internal/buffer.h\"\n> >>>>>>>>>>>>> +#include \"libcamera/internal/formats.h\"\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> +class Thumbnailer\n> >>>>>>>>>>>>> +{\n> >>>>>>>>>>>>> +public:\n> >>>>>>>>>>>>> + Thumbnailer();\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + void configure(const libcamera::Size &sourceSize,\n> >>>>>>>>>>>>> + libcamera::PixelFormat pixelFormat);\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + /*\n> >>>>>>>>>>>>> + * \\todo: Discuss if we can return targetSize_ via configure() or\n> >>>>>>>>>>>>> + * scaleBuffer(). We need targetSize_ to re-encode the scaled\n> >>>>>>>>>>>>> buffer\n> >>>>>>>>>>>>> + * via encoder in PostProcssorJpeg::writeThumbnail().\n> >>>>>>>>>>>>> + */\n> >>>>>>>>>>>>> + libcamera::Size computeThumbnailSize();\n> >>>>>>>>>>>>> + void scaleBuffer(const libcamera::Span &source,\n> >>>>>>>>>>>>> + std::vector &dest);\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> +private:\n> >>>>>>>>>>>>> + libcamera::PixelFormat pixelFormat_;\n> >>>>>>>>>>>>> + libcamera::Size sourceSize_;\n> >>>>>>>>>>>>> + libcamera::Size targetSize_;\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> + bool validConfiguration_;\n> >>>>>>>>>>>>> +};\n> >>>>>>>>>>>>> +\n> >>>>>>>>>>>>> +#endif /* __ANDROID_JPEG_THUMBNAILER_H__ */\n> >>>>>>>>>>>>> diff --git a/src/android/meson.build b/src/android/meson.build\n> >>>>>>>>>>>>> index 5a01bea..3905e2f 100644\n> >>>>>>>>>>>>> --- a/src/android/meson.build\n> >>>>>>>>>>>>> +++ b/src/android/meson.build\n> >>>>>>>>>>>>> @@ -25,6 +25,7 @@ android_hal_sources = files([\n> >>>>>>>>>>>>> 'jpeg/encoder_libjpeg.cpp',\n> >>>>>>>>>>>>> 'jpeg/exif.cpp',\n> >>>>>>>>>>>>> 'jpeg/post_processor_jpeg.cpp',\n> >>>>>>>>>>>>> + 'jpeg/thumbnailer.cpp',\n> >>>>>>>>>>>>> ])\n> >>>>>>>>>>>>> android_camera_metadata_sources = files([\n> >>>>>>>>>>>>>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 3B16CBD80A\n\tfor ;\n\tThu, 10 Dec 2020 07:45:40 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 7726E60322;\n\tThu, 10 Dec 2020 08:45:39 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 1FD7260322\n\tfor ;\n\tThu, 10 Dec 2020 08:45:38 +0100 (CET)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 7D965DD;\n\tThu, 10 Dec 2020 08:45:37 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"BRONBzE7\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1607586337;\n\tbh=YN2mtcgWUvER3DZ2Qe4Qf7s7Z6Tl6DjpFxOpG5lY5h8=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=BRONBzE7iP2zGU++S0BIMsm9xeHXTM9rd6Ds1JU2/+6M0Ah3ybe3FQq0IqkBpuMBK\n\t2EVLPKmpthVjDFD0jiRs5XXZ8PZ1Q/UOnZN8oqWNHfJ8CfxH3P8zKKzbJZBjvQBQav\n\th6sVQg48+OIsA14W1VoBwVvRrbUo6Hnv2Hjtf79w=","Date":"Thu, 10 Dec 2020 09:45:32 +0200","From":"Laurent Pinchart ","To":"Tomasz Figa ","Message-ID":"","References":"<0d9afed4-49e1-43f8-9b39-84b7fedfbe5d@uajain.com>\n\t<9ca3b492-8468-e9cf-b103-b4350be472b9@ideasonboard.com>\n\t\n\t<20201022135914.GA3938@pendragon.ideasonboard.com>\n\t\n\t<20201023035906.GA4113@pendragon.ideasonboard.com>\n\t<734c408b-2e5b-6ee4-af07-f9c45416abce@ideasonboard.com>\n\t\n\t<20201027010400.GO3756@pendragon.ideasonboard.com>\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [RFC PATCH 2/2] android: jpeg: Add a basic\n\tNV12 image thumbnailer","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera devel ","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]