From patchwork Thu Dec 1 09:27:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Harvey Yang X-Patchwork-Id: 17926 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id CE83EC3284 for ; Thu, 1 Dec 2022 09:27:50 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8ACCC63345; Thu, 1 Dec 2022 10:27:50 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1669886870; bh=Z5msnFGTDVHbIG7OKAF2z9RNv9X6UVpxLCMk5dyA1kE=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=eG0G9jJBzuA45YSpr3eQI2IBN/PcxttjNt2q4jN99d7MiebxqNAtjX9XBQ7NAXJHG uZu+c5ikX9SH4Gjsx76fh3YT+AIXGHP6IhNs7yUBhatW+h5p2qLYYsms5Qw3CmbxsO eZJlSxkosbCpLm2/DQ20D2/5+JBKXjPUKMtsxpr8rtJwhESpxiKfThddFYrJJucr+9 kxEaY67syKtDmRyU00K4kM+qLVzhK1718BWpahwmpxgfdSONQiQtyand83fTHZ57jH +zuxdQlzEayX50FLBEee8fNyoJem92ZOvXNhHfeIdBMxOKo9hQ/bCSUM9Nad08MUGr 5bB1NJkEiNXog== Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 8C1E363336 for ; Thu, 1 Dec 2022 10:27:48 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="OmTjnt8P"; dkim-atps=neutral Received: by mail-pf1-x435.google.com with SMTP id z17so1344222pff.1 for ; Thu, 01 Dec 2022 01:27:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GppenPRfYE9AngDAbvr9EW4YaLp/osn673u8/Q+g9zI=; b=OmTjnt8PoPHnUybMEwHPsc4if0S4Q9Ker5ri3dR/3QeSx40U4446fCnygfWUkH9eZ8 2UXNWb549AWxsO5GZliW0Y0zejAJwmTNw2jvKZ0s5AJHruUkz5otlma5KS9qbIwvvvsc 4EY1nJZtefNKEVRuZspLZPToIhifLh6O16+gQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GppenPRfYE9AngDAbvr9EW4YaLp/osn673u8/Q+g9zI=; b=xXYZVCs/wWb/1KHpef9A/a5tiGHncTkOAkmKWBSyaguF7U46SlsJoA3GAd4AmNSefY z5slT4HNz2ElaVzzgeWj7+N8hk3jwD4sqp7rMCEjRSWlZsVbXw6vvicQSRl1m1sGoEAe rSwXAY2FEvwZlpmgwPJQAOpxJtn2gVVVjQLl/JeRX21yrQbfZQem6tFrIZ3XFgOjMWKd AiwaRhxgkFMQm2dA++0eqT9swRqnjgnSCkiooof1eSfnQ/Bv9WaNBC5DiCYHtKiITL4V 4WMK6q32suQSjE4RJ+vBluGIqn3S0IMd3v7YDG1FxPvqlkNh03gBzSNGNk122y6OIM6m exqA== X-Gm-Message-State: ANoB5pm6Fl21IrOjAwiHRbZGBTJq4kIJE2Ik/OhleLSPV2PHSIn256lk qqYwzFFUs1OdI0KKtNJvyDYLrIeDqkMBB594 X-Google-Smtp-Source: AA0mqf6SsrDN2eOU+sRt8bPL8mFcLriaLrJaBEmS4BT56WPRfOhh1WHQ/zP7KRNFAR23g3ghz/aJ1A== X-Received: by 2002:a63:d117:0:b0:476:c781:d3ae with SMTP id k23-20020a63d117000000b00476c781d3aemr40493070pgg.183.1669886866648; Thu, 01 Dec 2022 01:27:46 -0800 (PST) Received: from chenghaoyang-low.c.googlers.com.com (46.165.189.35.bc.googleusercontent.com. [35.189.165.46]) by smtp.gmail.com with ESMTPSA id z9-20020a1709027e8900b00186b8752a78sm3108050pla.80.2022.12.01.01.27.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Dec 2022 01:27:46 -0800 (PST) X-Google-Original-From: Harvey Yang To: libcamera-devel@lists.libcamera.org Date: Thu, 1 Dec 2022 09:27:31 +0000 Message-Id: <20221201092733.2042078-5-chenghaoyang@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog In-Reply-To: <20221201092733.2042078-1-chenghaoyang@google.com> References: <20221201092733.2042078-1-chenghaoyang@google.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v7 4/6] Move generateThumbnail from PostProcessorJpeg to Encoder X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Harvey Yang via libcamera-devel From: Harvey Yang Reply-To: Harvey Yang Cc: Harvey Yang Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Harvey Yang In the following patch, generateThumbnail will have a different implementation in the jea encoder. Therefore, this patch moves the generateThumbnail function from PostProcessorJpeg to Encoder. Signed-off-by: Harvey Yang --- src/android/jpeg/encoder.h | 5 + src/android/jpeg/encoder_libjpeg.cpp | 122 ++++++++++++++++++----- src/android/jpeg/encoder_libjpeg.h | 28 +++++- src/android/jpeg/post_processor_jpeg.cpp | 54 +--------- src/android/jpeg/post_processor_jpeg.h | 11 +- 5 files changed, 130 insertions(+), 90 deletions(-) diff --git a/src/android/jpeg/encoder.h b/src/android/jpeg/encoder.h index b974d367..7dc1ee27 100644 --- a/src/android/jpeg/encoder.h +++ b/src/android/jpeg/encoder.h @@ -22,4 +22,9 @@ public: libcamera::Span destination, libcamera::Span exifData, unsigned int quality) = 0; + virtual int generateThumbnail( + const libcamera::FrameBuffer &source, + const libcamera::Size &targetSize, + unsigned int quality, + std::vector *thumbnail) = 0; }; diff --git a/src/android/jpeg/encoder_libjpeg.cpp b/src/android/jpeg/encoder_libjpeg.cpp index fd62bd9c..cc37fde3 100644 --- a/src/android/jpeg/encoder_libjpeg.cpp +++ b/src/android/jpeg/encoder_libjpeg.cpp @@ -71,29 +71,43 @@ const struct JPEGPixelFormatInfo &findPixelInfo(const PixelFormat &format) EncoderLibJpeg::EncoderLibJpeg() { /* \todo Expand error handling coverage with a custom handler. */ - compress_.err = jpeg_std_error(&jerr_); + image_data_.compress.err = jpeg_std_error(&image_data_.jerr); + thumbnail_data_.compress.err = jpeg_std_error(&thumbnail_data_.jerr); - jpeg_create_compress(&compress_); + jpeg_create_compress(&image_data_.compress); + jpeg_create_compress(&thumbnail_data_.compress); } EncoderLibJpeg::~EncoderLibJpeg() { - jpeg_destroy_compress(&compress_); + jpeg_destroy_compress(&image_data_.compress); + jpeg_destroy_compress(&thumbnail_data_.compress); } int EncoderLibJpeg::configure(const StreamConfiguration &cfg) { - const struct JPEGPixelFormatInfo info = findPixelInfo(cfg.pixelFormat); + thumbnailer_.configure(cfg.size, cfg.pixelFormat); + + return configureEncoder(&image_data_.compress, cfg.pixelFormat, + cfg.size); +} + +int EncoderLibJpeg::configureEncoder(struct jpeg_compress_struct *compress, + libcamera::PixelFormat pixelFormat, + libcamera::Size size) +{ + const struct JPEGPixelFormatInfo info = findPixelInfo(pixelFormat); + if (info.colorSpace == JCS_UNKNOWN) return -ENOTSUP; - compress_.image_width = cfg.size.width; - compress_.image_height = cfg.size.height; - compress_.in_color_space = info.colorSpace; + compress->image_width = size.width; + compress->image_height = size.height; + compress->in_color_space = info.colorSpace; - compress_.input_components = info.colorSpace == JCS_GRAYSCALE ? 1 : 3; + compress->input_components = info.colorSpace == JCS_GRAYSCALE ? 1 : 3; - jpeg_set_defaults(&compress_); + jpeg_set_defaults(compress); pixelFormatInfo_ = &info.pixelFormatInfo; @@ -107,13 +121,13 @@ void EncoderLibJpeg::compressRGB(const std::vector> &planes) { unsigned char *src = const_cast(planes[0].data()); /* \todo Stride information should come from buffer configuration. */ - unsigned int stride = pixelFormatInfo_->stride(compress_.image_width, 0); + unsigned int stride = pixelFormatInfo_->stride(compress_->image_width, 0); JSAMPROW row_pointer[1]; - while (compress_.next_scanline < compress_.image_height) { - row_pointer[0] = &src[compress_.next_scanline * stride]; - jpeg_write_scanlines(&compress_, row_pointer, 1); + while (compress_->next_scanline < compress_->image_height) { + row_pointer[0] = &src[compress_->next_scanline * stride]; + jpeg_write_scanlines(compress_, row_pointer, 1); } } @@ -123,7 +137,7 @@ void EncoderLibJpeg::compressRGB(const std::vector> &planes) */ void EncoderLibJpeg::compressNV(const std::vector> &planes) { - uint8_t tmprowbuf[compress_.image_width * 3]; + uint8_t tmprowbuf[compress_->image_width * 3]; /* * \todo Use the raw api, and only unpack the cb/cr samples to new line @@ -133,10 +147,10 @@ void EncoderLibJpeg::compressNV(const std::vector> &planes) * Possible hints at: * https://sourceforge.net/p/libjpeg/mailman/message/30815123/ */ - unsigned int y_stride = pixelFormatInfo_->stride(compress_.image_width, 0); - unsigned int c_stride = pixelFormatInfo_->stride(compress_.image_width, 1); + unsigned int y_stride = pixelFormatInfo_->stride(compress_->image_width, 0); + unsigned int c_stride = pixelFormatInfo_->stride(compress_->image_width, 1); - unsigned int horzSubSample = 2 * compress_.image_width / c_stride; + unsigned int horzSubSample = 2 * compress_->image_width / c_stride; unsigned int vertSubSample = pixelFormatInfo_->planes[1].verticalSubSampling; unsigned int c_inc = horzSubSample == 1 ? 2 : 0; @@ -149,14 +163,14 @@ void EncoderLibJpeg::compressNV(const std::vector> &planes) JSAMPROW row_pointer[1]; row_pointer[0] = &tmprowbuf[0]; - for (unsigned int y = 0; y < compress_.image_height; y++) { + for (unsigned int y = 0; y < compress_->image_height; y++) { unsigned char *dst = &tmprowbuf[0]; const unsigned char *src_y = src + y * y_stride; const unsigned char *src_cb = src_c + (y / vertSubSample) * c_stride + cb_pos; const unsigned char *src_cr = src_c + (y / vertSubSample) * c_stride + cr_pos; - for (unsigned int x = 0; x < compress_.image_width; x += 2) { + for (unsigned int x = 0; x < compress_->image_width; x += 2) { dst[0] = *src_y; dst[1] = *src_cb; dst[2] = *src_cr; @@ -174,13 +188,67 @@ void EncoderLibJpeg::compressNV(const std::vector> &planes) dst += 3; } - jpeg_write_scanlines(&compress_, row_pointer, 1); + jpeg_write_scanlines(compress_, row_pointer, 1); } } +int EncoderLibJpeg::generateThumbnail(const libcamera::FrameBuffer &source, + const libcamera::Size &targetSize, + unsigned int quality, + std::vector *thumbnail) +{ + /* Stores the raw scaled-down thumbnail bytes. */ + std::vector rawThumbnail; + + thumbnailer_.createThumbnail(source, targetSize, &rawThumbnail); + + int ret = configureEncoder(&thumbnail_data_.compress, + thumbnailer_.pixelFormat(), targetSize); + compress_ = &thumbnail_data_.compress; + + if (!rawThumbnail.empty() && !ret) { + /* + * \todo Avoid value-initialization of all elements of the + * vector. + */ + thumbnail->resize(rawThumbnail.size()); + + /* + * Split planes manually as the encoder expects a vector of + * planes. + * + * \todo Pass a vector of planes directly to + * Thumbnailer::createThumbnailer above and remove the manual + * planes split from here. + */ + std::vector> thumbnailPlanes; + const PixelFormatInfo &formatNV12 = + PixelFormatInfo::info(formats::NV12); + size_t yPlaneSize = formatNV12.planeSize(targetSize, 0); + size_t uvPlaneSize = formatNV12.planeSize(targetSize, 1); + thumbnailPlanes.push_back({ rawThumbnail.data(), yPlaneSize }); + thumbnailPlanes.push_back({ rawThumbnail.data() + yPlaneSize, + uvPlaneSize }); + + int jpeg_size = encode(thumbnailPlanes, *thumbnail, {}, + quality); + thumbnail->resize(jpeg_size); + + LOG(JPEG, Debug) + << "Thumbnail compress returned " + << jpeg_size << " bytes"; + + return jpeg_size; + } + + return -1; +} + int EncoderLibJpeg::encode(const FrameBuffer &source, Span dest, Span exifData, unsigned int quality) { + compress_ = &image_data_.compress; + MappedFrameBuffer frame(&source, MappedFrameBuffer::MapFlag::Read); if (!frame.isValid()) { LOG(JPEG, Error) << "Failed to map FrameBuffer : " @@ -198,7 +266,7 @@ int EncoderLibJpeg::encode(const std::vector> &src, unsigned char *destination = dest.data(); unsigned long size = dest.size(); - jpeg_set_quality(&compress_, quality, TRUE); + jpeg_set_quality(compress_, quality, TRUE); /* * The jpeg_mem_dest will reallocate if the required size is not @@ -208,18 +276,18 @@ int EncoderLibJpeg::encode(const std::vector> &src, * \todo Implement our own custom memory destination to prevent * reallocation and prefer failure with correct reporting. */ - jpeg_mem_dest(&compress_, &destination, &size); + jpeg_mem_dest(compress_, &destination, &size); - jpeg_start_compress(&compress_, TRUE); + jpeg_start_compress(compress_, TRUE); if (exifData.size()) /* Store Exif data in the JPEG_APP1 data block. */ - jpeg_write_marker(&compress_, JPEG_APP0 + 1, + jpeg_write_marker(compress_, JPEG_APP0 + 1, static_cast(exifData.data()), exifData.size()); - LOG(JPEG, Debug) << "JPEG Encode Starting:" << compress_.image_width - << "x" << compress_.image_height; + LOG(JPEG, Debug) << "JPEG Encode Starting:" << compress_->image_width + << "x" << compress_->image_height; ASSERT(src.size() == pixelFormatInfo_->numPlanes()); @@ -228,7 +296,7 @@ int EncoderLibJpeg::encode(const std::vector> &src, else compressRGB(src); - jpeg_finish_compress(&compress_); + jpeg_finish_compress(compress_); return size; } diff --git a/src/android/jpeg/encoder_libjpeg.h b/src/android/jpeg/encoder_libjpeg.h index 1b3ac067..1ec2ba13 100644 --- a/src/android/jpeg/encoder_libjpeg.h +++ b/src/android/jpeg/encoder_libjpeg.h @@ -15,6 +15,8 @@ #include +#include "thumbnailer.h" + class EncoderLibJpeg : public Encoder { public: @@ -26,20 +28,40 @@ public: libcamera::Span destination, libcamera::Span exifData, unsigned int quality) override; + int generateThumbnail( + const libcamera::FrameBuffer &source, + const libcamera::Size &targetSize, + unsigned int quality, + std::vector *thumbnail) override; + +private: + struct JpegData { + struct jpeg_compress_struct compress; + struct jpeg_error_mgr jerr; + }; + + int configureEncoder(struct jpeg_compress_struct *compress, + libcamera::PixelFormat pixelFormat, + libcamera::Size size); + int encode(const std::vector> &planes, libcamera::Span destination, libcamera::Span exifData, unsigned int quality); -private: void compressRGB(const std::vector> &planes); void compressNV(const std::vector> &planes); - struct jpeg_compress_struct compress_; - struct jpeg_error_mgr jerr_; + JpegData image_data_; + JpegData thumbnail_data_; + + // |&image_data_.compress| or |&thumbnail_data_.compress|. + struct jpeg_compress_struct *compress_; const libcamera::PixelFormatInfo *pixelFormatInfo_; bool nv_; bool nvSwap_; + + Thumbnailer thumbnailer_; }; diff --git a/src/android/jpeg/post_processor_jpeg.cpp b/src/android/jpeg/post_processor_jpeg.cpp index 0cf56716..60eebb11 100644 --- a/src/android/jpeg/post_processor_jpeg.cpp +++ b/src/android/jpeg/post_processor_jpeg.cpp @@ -44,60 +44,11 @@ int PostProcessorJpeg::configure(const StreamConfiguration &inCfg, streamSize_ = outCfg.size; - thumbnailer_.configure(inCfg.size, inCfg.pixelFormat); - encoder_ = std::make_unique(); return encoder_->configure(inCfg); } -void PostProcessorJpeg::generateThumbnail(const FrameBuffer &source, - const Size &targetSize, - unsigned int quality, - std::vector *thumbnail) -{ - /* Stores the raw scaled-down thumbnail bytes. */ - std::vector rawThumbnail; - - thumbnailer_.createThumbnail(source, targetSize, &rawThumbnail); - - StreamConfiguration thCfg; - thCfg.size = targetSize; - thCfg.pixelFormat = thumbnailer_.pixelFormat(); - int ret = thumbnailEncoder_.configure(thCfg); - - if (!rawThumbnail.empty() && !ret) { - /* - * \todo Avoid value-initialization of all elements of the - * vector. - */ - thumbnail->resize(rawThumbnail.size()); - - /* - * Split planes manually as the encoder expects a vector of - * planes. - * - * \todo Pass a vector of planes directly to - * Thumbnailer::createThumbnailer above and remove the manual - * planes split from here. - */ - std::vector> thumbnailPlanes; - const PixelFormatInfo &formatNV12 = PixelFormatInfo::info(formats::NV12); - size_t yPlaneSize = formatNV12.planeSize(targetSize, 0); - size_t uvPlaneSize = formatNV12.planeSize(targetSize, 1); - thumbnailPlanes.push_back({ rawThumbnail.data(), yPlaneSize }); - thumbnailPlanes.push_back({ rawThumbnail.data() + yPlaneSize, uvPlaneSize }); - - int jpeg_size = thumbnailEncoder_.encode(thumbnailPlanes, - *thumbnail, {}, quality); - thumbnail->resize(jpeg_size); - - LOG(JPEG, Debug) - << "Thumbnail compress returned " - << jpeg_size << " bytes"; - } -} - void PostProcessorJpeg::process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) { ASSERT(encoder_); @@ -164,8 +115,9 @@ void PostProcessorJpeg::process(Camera3RequestDescriptor::StreamBuffer *streamBu if (thumbnailSize != Size(0, 0)) { std::vector thumbnail; - generateThumbnail(source, thumbnailSize, quality, &thumbnail); - if (!thumbnail.empty()) + ret = encoder_->generateThumbnail(source, thumbnailSize, + quality, &thumbnail); + if (ret > 0 && !thumbnail.empty()) exif.setThumbnail(std::move(thumbnail), Exif::Compression::JPEG); } diff --git a/src/android/jpeg/post_processor_jpeg.h b/src/android/jpeg/post_processor_jpeg.h index 98309b01..55b23d7d 100644 --- a/src/android/jpeg/post_processor_jpeg.h +++ b/src/android/jpeg/post_processor_jpeg.h @@ -8,11 +8,11 @@ #pragma once #include "../post_processor.h" -#include "encoder_libjpeg.h" -#include "thumbnailer.h" #include +#include "encoder.h" + class CameraDevice; class PostProcessorJpeg : public PostProcessor @@ -25,14 +25,7 @@ public: void process(Camera3RequestDescriptor::StreamBuffer *streamBuffer) override; private: - void generateThumbnail(const libcamera::FrameBuffer &source, - const libcamera::Size &targetSize, - unsigned int quality, - std::vector *thumbnail); - CameraDevice *const cameraDevice_; std::unique_ptr encoder_; libcamera::Size streamSize_; - EncoderLibJpeg thumbnailEncoder_; - Thumbnailer thumbnailer_; };