From patchwork Tue Oct 22 07:43:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Harvey Yang X-Patchwork-Id: 21729 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 24014C3274 for ; Tue, 22 Oct 2024 07:46:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 9E679653AA; Tue, 22 Oct 2024 09:46:08 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="PqJWYSRp"; dkim-atps=neutral Received: from mail-pg1-x531.google.com (mail-pg1-x531.google.com [IPv6:2607:f8b0:4864:20::531]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 50F416539D for ; Tue, 22 Oct 2024 09:46:01 +0200 (CEST) Received: by mail-pg1-x531.google.com with SMTP id 41be03b00d2f7-7ea80863f12so2303174a12.1 for ; Tue, 22 Oct 2024 00:46:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1729583159; x=1730187959; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CSWjmxQnTZaXkoxNFGFOs0Q/kFfJAZ8PWYLlCD+UfXQ=; b=PqJWYSRprbzWUVf6RLA54/PNovrdZacVP0aQfekq+l/IXA+fqg3qXsUTB02YOj27kS OQPQi0gwMZUMlZEf+YlRYJA6f2ZvP8IffnIq55/nlGej4DtTwP9kL/FKVjRm2BtPGqKC Wgk+TQjigDtsPuDIGNgwX35ziWIZcW7SfrBxg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729583159; x=1730187959; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CSWjmxQnTZaXkoxNFGFOs0Q/kFfJAZ8PWYLlCD+UfXQ=; b=SWNU8ktUKtK9K1HG7QnuldD8DwaCdJCwLnxgjBMCLcJ8AaKXOStbu56KcrvpPKeZhG rodsXp2VrcOjmGB3Rj0BwpHpzAuqNpLcmVhycAl7cB4ruyrQw8awrmlcBNPlcBtzxmId x/7LH6oYBZ5FRBoOxdX3VL4tuMYldzj6p03kUNQc/UP6E8rI+NFFZ83OSMyGRdHj0kiq 4QImQdz7p3B8e/AthgmYT5J7sviGkjWWFynLprSlAcjuSSab6MuHB0L1cTzVjb/O7neo A8+3KWeY0UjPpNypEHwx1eLy8q9UsJqE7N1zVqdBi73M76GuKJ3/tTKpo/aNk30Gxy9l OcZw== X-Gm-Message-State: AOJu0YxwqJhARg5SPRmUKcU3oCfryKusgDR0Yc7OJrZ1J843QYcFNZL7 WFbWMa4rWZwdueysgOQv44mLWxuFvoCbG442lfktFvrP5D/eP1Lt4rJuLiOpo7eRwEY93ATLU3s = X-Google-Smtp-Source: AGHT+IHudDE1Nn9fTMsk+PriPh/gQjq32SRiFscB9vtOE73eJYCMsfz6GGAA+WLjFpqOWfvRQxid/Q== X-Received: by 2002:a05:6a20:b908:b0:1d9:21cb:ecdb with SMTP id adf61e73a8af0-1d92c5839f4mr14880519637.41.1729583159077; Tue, 22 Oct 2024 00:45:59 -0700 (PDT) Received: from chenghaoyang-low.c.googlers.com.com (27.247.221.35.bc.googleusercontent.com. [35.221.247.27]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71ec13d75aesm4091024b3a.124.2024.10.22.00.45.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Oct 2024 00:45:58 -0700 (PDT) From: Harvey Yang To: libcamera-devel@lists.libcamera.org Cc: Harvey Yang , Konami Shu , Yunke Cao , Tomasz Figa , Jacopo Mondi Subject: [PATCH v16 5/7] libcamera: virtual: Add ImageFrameGenerator Date: Tue, 22 Oct 2024 07:43:41 +0000 Message-ID: <20241022074544.3790451-6-chenghaoyang@chromium.org> X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog In-Reply-To: <20241022074544.3790451-1-chenghaoyang@chromium.org> References: <20241022074544.3790451-1-chenghaoyang@chromium.org> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Besides TestPatternGenerator, this patch adds ImageFrameGenerator that loads real images (jpg / jpeg for now) as the source and generates scaled frames. Signed-off-by: Konami Shu Co-developed-by: Harvey Yang Signed-off-by: Harvey Yang Co-developed-by: Yunke Cao Signed-off-by: Yunke Cao Co-developed-by: Tomasz Figa Signed-off-by: Tomasz Figa Reviewed-by: Jacopo Mondi Reviewed-by: Kieran Bingham --- .../virtual/image_frame_generator.cpp | 178 ++++++++++++++++++ .../pipeline/virtual/image_frame_generator.h | 50 +++++ src/libcamera/pipeline/virtual/meson.build | 4 + 3 files changed, 232 insertions(+) create mode 100644 src/libcamera/pipeline/virtual/image_frame_generator.cpp create mode 100644 src/libcamera/pipeline/virtual/image_frame_generator.h diff --git a/src/libcamera/pipeline/virtual/image_frame_generator.cpp b/src/libcamera/pipeline/virtual/image_frame_generator.cpp new file mode 100644 index 000000000..e140969c8 --- /dev/null +++ b/src/libcamera/pipeline/virtual/image_frame_generator.cpp @@ -0,0 +1,178 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Google Inc. + * + * Derived class of FrameGenerator for generating frames from images + */ + +#include "image_frame_generator.h" + +#include + +#include +#include + +#include + +#include "libcamera/internal/mapped_framebuffer.h" + +#include "libyuv/convert.h" +#include "libyuv/scale.h" + +namespace libcamera { + +LOG_DECLARE_CATEGORY(Virtual) + +/* + * Factory function to create an ImageFrameGenerator object. + * Read the images and convert them to buffers in NV12 format. + * Store the pointers to the buffers to a list (imageFrameDatas) + */ +std::unique_ptr +ImageFrameGenerator::create(ImageFrames &imageFrames) +{ + std::unique_ptr imageFrameGenerator = + std::make_unique(); + imageFrameGenerator->imageFrames_ = &imageFrames; + + /* + * For each file in the directory, load the image, + * convert it to NV12, and store the pointer. + */ + for (unsigned int i = 0; i < imageFrames.number.value_or(1); i++) { + std::filesystem::path path; + if (!imageFrames.number) + /* If the path is to an image */ + path = imageFrames.path; + else + /* If the path is to a directory */ + path = imageFrames.path / (std::to_string(i) + ".jpg"); + + File file(path); + if (!file.open(File::OpenModeFlag::ReadOnly)) { + LOG(Virtual, Error) << "Failed to open image file " << file.fileName() + << ": " << strerror(file.error()); + return nullptr; + } + + /* Read the image file to data */ + auto fileSize = file.size(); + auto buffer = std::make_unique(fileSize); + if (file.read({ buffer.get(), static_cast(fileSize) }) != fileSize) { + LOG(Virtual, Error) << "Failed to read file " << file.fileName() + << ": " << strerror(file.error()); + return nullptr; + } + + /* Get the width and height of the image */ + int width, height; + if (libyuv::MJPGSize(buffer.get(), fileSize, &width, &height)) { + LOG(Virtual, Error) << "Failed to get the size of the image file: " + << file.fileName(); + return nullptr; + } + + std::unique_ptr dstY = + std::make_unique(width * height); + std::unique_ptr dstUV = + std::make_unique(width * height / 2); + int ret = libyuv::MJPGToNV12(buffer.get(), fileSize, + dstY.get(), width, dstUV.get(), + width, width, height, width, height); + if (ret != 0) + LOG(Virtual, Error) << "MJPGToNV12() failed with " << ret; + + imageFrameGenerator->imageFrameDatas_.emplace_back( + ImageFrameData{ std::move(dstY), std::move(dstUV), + Size(width, height) }); + } + + return imageFrameGenerator; +} + +/* + * \var ImageFrameGenerator::frameRepeat + * \brief Number of frames to repeat before proceeding to the next frame + */ + +/* Scale the buffers for image frames. */ +void ImageFrameGenerator::configure(const Size &size) +{ + /* Reset the source images to prevent multiple configuration calls */ + scaledFrameDatas_.clear(); + frameIndex_ = 0; + parameter_ = 0; + + for (unsigned int i = 0; i < imageFrames_->number.value_or(1); i++) { + /* Scale the imageFrameDatas_ to scaledY and scaledUV */ + unsigned int halfSizeWidth = (size.width + 1) / 2; + unsigned int halfSizeHeight = (size.height + 1) / 2; + std::unique_ptr scaledY = + std::make_unique(size.width * size.height); + std::unique_ptr scaledUV = + std::make_unique(halfSizeWidth * halfSizeHeight * 2); + auto &src = imageFrameDatas_[i]; + + /* + * \todo Some platforms might enforce stride due to GPU. + * The width needs to be a multiple of the stride to work + * properly for now. + */ + libyuv::NV12Scale(src.Y.get(), src.size.width, + src.UV.get(), src.size.width, + src.size.width, src.size.height, + scaledY.get(), size.width, scaledUV.get(), size.width, + size.width, size.height, libyuv::FilterMode::kFilterBilinear); + + scaledFrameDatas_.emplace_back( + ImageFrameData{ std::move(scaledY), std::move(scaledUV), size }); + } +} + +int ImageFrameGenerator::generateFrame(const Size &size, const FrameBuffer *buffer) +{ + ASSERT(!scaledFrameDatas_.empty()); + + MappedFrameBuffer mappedFrameBuffer(buffer, MappedFrameBuffer::MapFlag::Write); + + auto planes = mappedFrameBuffer.planes(); + + /* Loop only around the number of images available */ + frameIndex_ %= imageFrames_->number.value_or(1); + + /* Write the scaledY and scaledUV to the mapped frame buffer */ + libyuv::NV12Copy(scaledFrameDatas_[frameIndex_].Y.get(), size.width, + scaledFrameDatas_[frameIndex_].UV.get(), size.width, planes[0].begin(), + size.width, planes[1].begin(), size.width, + size.width, size.height); + + /* Proceed to the next image every 4 frames */ + /* \todo Consider setting the frameRepeat in the config file */ + parameter_++; + if (parameter_ % frameRepeat == 0) + frameIndex_++; + + return 0; +} + +/* + * \var ImageFrameGenerator::imageFrameDatas_ + * \brief List of pointers to the not scaled image buffers + */ + +/* + * \var ImageFrameGenerator::scaledFrameDatas_ + * \brief List of pointers to the scaled image buffers + */ + +/* + * \var ImageFrameGenerator::imageFrames_ + * \brief Pointer to the imageFrames_ in VirtualCameraData + */ + +/* + * \var ImageFrameGenerator::parameter_ + * \brief Speed parameter. Change to the next image every parameter_ frames + */ + +} /* namespace libcamera */ diff --git a/src/libcamera/pipeline/virtual/image_frame_generator.h b/src/libcamera/pipeline/virtual/image_frame_generator.h new file mode 100644 index 000000000..e072a47b8 --- /dev/null +++ b/src/libcamera/pipeline/virtual/image_frame_generator.h @@ -0,0 +1,50 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Google Inc. + * + * Derived class of FrameGenerator for generating frames from images + */ + +#pragma once + +#include +#include +#include +#include +#include + +#include "frame_generator.h" + +namespace libcamera { + +/* Frame configuration provided by the config file */ +struct ImageFrames { + std::filesystem::path path; + std::optional number; +}; + +class ImageFrameGenerator : public FrameGenerator +{ +public: + static std::unique_ptr create(ImageFrames &imageFrames); + +private: + static constexpr unsigned int frameRepeat = 4; + + struct ImageFrameData { + std::unique_ptr Y; + std::unique_ptr UV; + Size size; + }; + + void configure(const Size &size) override; + int generateFrame(const Size &size, const FrameBuffer *buffer) override; + + std::vector imageFrameDatas_; + std::vector scaledFrameDatas_; + ImageFrames *imageFrames_; + unsigned int frameIndex_; + unsigned int parameter_; +}; + +} /* namespace libcamera */ diff --git a/src/libcamera/pipeline/virtual/meson.build b/src/libcamera/pipeline/virtual/meson.build index 0c537777f..bb38c193c 100644 --- a/src/libcamera/pipeline/virtual/meson.build +++ b/src/libcamera/pipeline/virtual/meson.build @@ -1,8 +1,12 @@ # SPDX-License-Identifier: CC0-1.0 libcamera_internal_sources += files([ + 'image_frame_generator.cpp', 'test_pattern_generator.cpp', 'virtual.cpp', ]) +libjpeg = dependency('libjpeg', required : true) + libcamera_deps += [libyuv_dep] +libcamera_deps += [libjpeg]