{"id":21507,"url":"https://patchwork.libcamera.org/api/patches/21507/?format=json","web_url":"https://patchwork.libcamera.org/patch/21507/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20241004095946.264537-6-chenghaoyang@google.com>","date":"2024-10-04T09:59:18","name":"[15,5/7] libcamera: virtual: Add ImageFrameGenerator","commit_ref":null,"pull_url":null,"state":"superseded","archived":false,"hash":"d23620927d9f1b3cee39eac395d87e9b68a68a30","submitter":{"id":117,"url":"https://patchwork.libcamera.org/api/people/117/?format=json","name":"Cheng-Hao Yang","email":"chenghaoyang@chromium.org"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/21507/mbox/","series":[{"id":4656,"url":"https://patchwork.libcamera.org/api/series/4656/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=4656","date":"2024-10-04T09:59:14","name":"[15,1/7] libcamera: add DmaBufAllocator::exportBuffers()","version":1,"mbox":"https://patchwork.libcamera.org/series/4656/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/21507/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/21507/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 74F35C3257\n\tfor ;\n\tFri, 4 Oct 2024 10:00:12 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id ED89663537;\n\tFri, 4 Oct 2024 12:00:11 +0200 (CEST)","from mail-pf1-x42b.google.com (mail-pf1-x42b.google.com\n\t[IPv6:2607:f8b0:4864:20::42b])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 5573063533\n\tfor ;\n\tFri, 4 Oct 2024 12:00:02 +0200 (CEST)","by mail-pf1-x42b.google.com with SMTP id\n\td2e1a72fcca58-71dba8b05cbso1930626b3a.3\n\tfor ;\n\tFri, 04 Oct 2024 03:00:02 -0700 (PDT)","from chenghaoyang-low.c.googlers.com.com\n\t(27.247.221.35.bc.googleusercontent.com. [35.221.247.27])\n\tby smtp.gmail.com with ESMTPSA id\n\td2e1a72fcca58-71ded2ad931sm316911b3a.41.2024.10.04.02.59.58\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tFri, 04 Oct 2024 02:59:59 -0700 (PDT)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=chromium.org header.i=@chromium.org\n\theader.b=\"PnKCrs1T\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=chromium.org; s=google; t=1728036000; x=1728640800;\n\tdarn=lists.libcamera.org; \n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:cc:to:from:from:to:cc:subject:date\n\t:message-id:reply-to;\n\tbh=3JyVolO8XjOiqTX/jZ0em4R/BrPOODb1SlyVieKwbNY=;\n\tb=PnKCrs1TLAWO8AtLYHrqX4qAkPPpvFrtp5GRoNPJjUoGvmhHlK1HGbdywv+LyVDXyd\n\txzL2Wqb1ana3lnmxZ+oMcWwmte5At/DZglQhMGizMBDh7eF/aNYKpbZaNGMO5tEU7OuV\n\tC5URvJje+NGkZk38TdCnAgc0wgV0kBgugZMj8=","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20230601; t=1728036000; x=1728640800;\n\th=content-transfer-encoding:mime-version:references:in-reply-to\n\t:message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc\n\t:subject:date:message-id:reply-to;\n\tbh=3JyVolO8XjOiqTX/jZ0em4R/BrPOODb1SlyVieKwbNY=;\n\tb=rrXRQMd9iJveMgVtocOX2AMr1mr37WVWMpoCbqgl8KIhGEFlgbrX27aGYrYe3UY8+A\n\tCzujG+yCP1aSJJH1Ov8Ro4jMxUbQbB6YeRFZVVTwvFVoCcNXN0eXdrka5bvwHcF74ysU\n\t8BV35myuVDQIZGdhXXnN1LjKOYwgeHPMSfntNi7epcNglCfUp6DDup8XA1qUNTndBKiG\n\tASr8WvsBIPv2cvDmq0ioW9bQtuzJMqHM8/1DkTZS4dHYsDQytl5X2hkAQ0e+I/wABsUQ\n\tlkcoiIvRKEWHlNyVAaCJfdiAutNr5Svi0mO9U14QEp5+6EWcU74hVbYZOveShZyiRupH\n\tS3eg==","X-Gm-Message-State":"AOJu0Yx4gfqLAHkMkbrY5P4bVwr9g4LETE4FkPQsJWpuqLlZWOqy8HWi\n\tjqTepP7+0B68Yi/nBE0hNnW2KbWCJpYsnypjLD5FXnLruRufTX0GOEX7R8rQxAxH4TxBbBdMCSi\n\tFTw==","X-Google-Smtp-Source":"AGHT+IHtzlSSeVn24RMNWsuuP4JpmyDUU8MP+jJ7KjzwEt4jw/5xCVuSKKx0ORA4br/UyEZtmy8eIg==","X-Received":"by 2002:a05:6a00:14ca:b0:70d:22b5:5420 with SMTP id\n\td2e1a72fcca58-71de23e7fe9mr3582048b3a.15.1728035999886; \n\tFri, 04 Oct 2024 02:59:59 -0700 (PDT)","From":"Harvey Yang ","X-Google-Original-From":"Harvey Yang ","To":"libcamera-devel@lists.libcamera.org","Cc":"Harvey Yang , Konami Shu , \n\tHarvey Yang , Yunke Cao ,\n\tTomasz Figa ,\n\tJacopo Mondi ","Subject":"[PATCH 15 5/7] libcamera: virtual: Add ImageFrameGenerator","Date":"Fri, 4 Oct 2024 09:59:18 +0000","Message-ID":"<20241004095946.264537-6-chenghaoyang@google.com>","X-Mailer":"git-send-email 2.47.0.rc0.187.ge670bccf7e-goog","In-Reply-To":"<20241004095946.264537-1-chenghaoyang@google.com>","References":"<20241004095946.264537-1-chenghaoyang@google.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"Besides TestPatternGenerator, this patch adds ImageFrameGenerator that\nloads real images (jpg / jpeg for now) as the source and generates\nscaled frames.\n\nSigned-off-by: Konami Shu \nCo-developed-by: Harvey Yang \nCo-developed-by: Yunke Cao \nCo-developed-by: Tomasz Figa \nReviewed-by: Jacopo Mondi \n---\n .../virtual/image_frame_generator.cpp | 179 ++++++++++++++++++\n .../pipeline/virtual/image_frame_generator.h | 51 +++++\n src/libcamera/pipeline/virtual/meson.build | 4 +\n 3 files changed, 234 insertions(+)\n create mode 100644 src/libcamera/pipeline/virtual/image_frame_generator.cpp\n create mode 100644 src/libcamera/pipeline/virtual/image_frame_generator.h","diff":"diff --git a/src/libcamera/pipeline/virtual/image_frame_generator.cpp b/src/libcamera/pipeline/virtual/image_frame_generator.cpp\nnew file mode 100644\nindex 000000000..04382beb7\n--- /dev/null\n+++ b/src/libcamera/pipeline/virtual/image_frame_generator.cpp\n@@ -0,0 +1,179 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2024, Google Inc.\n+ *\n+ * image_frame_generator.cpp - Derived class of FrameGenerator for\n+ * generating frames from images\n+ */\n+\n+#include \"image_frame_generator.h\"\n+\n+#include \n+\n+#include \n+#include \n+\n+#include \n+\n+#include \"libcamera/internal/mapped_framebuffer.h\"\n+\n+#include \"libyuv/convert.h\"\n+#include \"libyuv/scale.h\"\n+\n+namespace libcamera {\n+\n+LOG_DECLARE_CATEGORY(Virtual)\n+\n+/*\n+ * Factory function to create an ImageFrameGenerator object.\n+ * Read the images and convert them to buffers in NV12 format.\n+ * Store the pointers to the buffers to a list (imageFrameDatas)\n+ */\n+std::unique_ptr\n+ImageFrameGenerator::create(ImageFrames &imageFrames)\n+{\n+\tstd::unique_ptr imageFrameGenerator =\n+\t\tstd::make_unique();\n+\timageFrameGenerator->imageFrames_ = &imageFrames;\n+\n+\t/*\n+\t * For each file in the directory, load the image,\n+\t * convert it to NV12, and store the pointer.\n+\t */\n+\tfor (unsigned int i = 0; i < imageFrames.number.value_or(1); i++) {\n+\t\tstd::filesystem::path path;\n+\t\tif (!imageFrames.number)\n+\t\t\t/* If the path is to an image */\n+\t\t\tpath = imageFrames.path;\n+\t\telse\n+\t\t\t/* If the path is to a directory */\n+\t\t\tpath = imageFrames.path / (std::to_string(i) + \".jpg\");\n+\n+\t\tFile file(path);\n+\t\tif (!file.open(File::OpenModeFlag::ReadOnly)) {\n+\t\t\tLOG(Virtual, Error) << \"Failed to open image file \" << file.fileName()\n+\t\t\t\t\t << \": \" << strerror(file.error());\n+\t\t\treturn nullptr;\n+\t\t}\n+\n+\t\t/* Read the image file to data */\n+\t\tauto fileSize = file.size();\n+\t\tauto buffer = std::make_unique(fileSize);\n+\t\tif (file.read({ buffer.get(), static_cast(fileSize) }) != fileSize) {\n+\t\t\tLOG(Virtual, Error) << \"Failed to read file \" << file.fileName()\n+\t\t\t\t\t << \": \" << strerror(file.error());\n+\t\t\treturn nullptr;\n+\t\t}\n+\n+\t\t/* Get the width and height of the image */\n+\t\tint width, height;\n+\t\tif (libyuv::MJPGSize(buffer.get(), fileSize, &width, &height)) {\n+\t\t\tLOG(Virtual, Error) << \"Failed to get the size of the image file: \"\n+\t\t\t\t\t << file.fileName();\n+\t\t\treturn nullptr;\n+\t\t}\n+\n+\t\tstd::unique_ptr dstY =\n+\t\t\tstd::make_unique(width * height);\n+\t\tstd::unique_ptr dstUV =\n+\t\t\tstd::make_unique(width * height / 2);\n+\t\tint ret = libyuv::MJPGToNV12(buffer.get(), fileSize,\n+\t\t\t\t\t dstY.get(), width, dstUV.get(),\n+\t\t\t\t\t width, width, height, width, height);\n+\t\tif (ret != 0)\n+\t\t\tLOG(Virtual, Error) << \"MJPGToNV12() failed with \" << ret;\n+\n+\t\timageFrameGenerator->imageFrameDatas_.emplace_back(\n+\t\t\tImageFrameData{ std::move(dstY), std::move(dstUV),\n+\t\t\t\t\tSize(width, height) });\n+\t}\n+\n+\treturn imageFrameGenerator;\n+}\n+\n+/*\n+ * \\var ImageFrameGenerator::frameRepeat\n+ * \\brief Number of frames to repeat before proceeding to the next frame\n+ */\n+\n+/* Scale the buffers for image frames. */\n+void ImageFrameGenerator::configure(const Size &size)\n+{\n+\t/* Reset the source images to prevent multiple configuration calls */\n+\tscaledFrameDatas_.clear();\n+\tframeIndex_ = 0;\n+\tparameter_ = 0;\n+\n+\tfor (unsigned int i = 0; i < imageFrames_->number.value_or(1); i++) {\n+\t\t/* Scale the imageFrameDatas_ to scaledY and scaledUV */\n+\t\tunsigned int halfSizeWidth = (size.width + 1) / 2;\n+\t\tunsigned int halfSizeHeight = (size.height + 1) / 2;\n+\t\tstd::unique_ptr scaledY =\n+\t\t\tstd::make_unique(size.width * size.height);\n+\t\tstd::unique_ptr scaledUV =\n+\t\t\tstd::make_unique(halfSizeWidth * halfSizeHeight * 2);\n+\t\tauto &src = imageFrameDatas_[i];\n+\n+\t\t/*\n+\t\t * \\todo Some platforms might enforce stride due to GPU.\n+ * The width needs to be a multiple of the stride to work\n+ * properly for now.\n+\t\t */\n+\t\tlibyuv::NV12Scale(src.Y.get(), src.size.width,\n+\t\t\t\t src.UV.get(), src.size.width,\n+\t\t\t\t src.size.width, src.size.height,\n+\t\t\t\t scaledY.get(), size.width, scaledUV.get(), size.width,\n+\t\t\t\t size.width, size.height, libyuv::FilterMode::kFilterBilinear);\n+\n+\t\tscaledFrameDatas_.emplace_back(\n+\t\t\tImageFrameData{ std::move(scaledY), std::move(scaledUV), size });\n+\t}\n+}\n+\n+int ImageFrameGenerator::generateFrame(const Size &size, const FrameBuffer *buffer)\n+{\n+\tASSERT(!scaledFrameDatas_.empty());\n+\n+\tMappedFrameBuffer mappedFrameBuffer(buffer, MappedFrameBuffer::MapFlag::Write);\n+\n+\tauto planes = mappedFrameBuffer.planes();\n+\n+\t/* Loop only around the number of images available */\n+\tframeIndex_ %= imageFrames_->number.value_or(1);\n+\n+\t/* Write the scaledY and scaledUV to the mapped frame buffer */\n+\tlibyuv::NV12Copy(scaledFrameDatas_[frameIndex_].Y.get(), size.width,\n+\t\t\t scaledFrameDatas_[frameIndex_].UV.get(), size.width, planes[0].begin(),\n+\t\t\t size.width, planes[1].begin(), size.width,\n+\t\t\t size.width, size.height);\n+\n+\t/* Proceed to the next image every 4 frames */\n+\t/* \\todo Consider setting the frameRepeat in the config file */\n+\tparameter_++;\n+\tif (parameter_ % frameRepeat == 0)\n+\t\tframeIndex_++;\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * \\var ImageFrameGenerator::imageFrameDatas_\n+ * \\brief List of pointers to the not scaled image buffers\n+ */\n+\n+/*\n+ * \\var ImageFrameGenerator::scaledFrameDatas_\n+ * \\brief List of pointers to the scaled image buffers\n+ */\n+\n+/*\n+ * \\var ImageFrameGenerator::imageFrames_\n+ * \\brief Pointer to the imageFrames_ in VirtualCameraData\n+ */\n+\n+/*\n+ * \\var ImageFrameGenerator::parameter_\n+ * \\brief Speed parameter. Change to the next image every parameter_ frames\n+ */\n+\n+} /* namespace libcamera */\ndiff --git a/src/libcamera/pipeline/virtual/image_frame_generator.h b/src/libcamera/pipeline/virtual/image_frame_generator.h\nnew file mode 100644\nindex 000000000..a68094a66\n--- /dev/null\n+++ b/src/libcamera/pipeline/virtual/image_frame_generator.h\n@@ -0,0 +1,51 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2024, Google Inc.\n+ *\n+ * image_frame_generator.h - Derived class of FrameGenerator for\n+ * generating frames from images\n+ */\n+\n+#pragma once\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+#include \"frame_generator.h\"\n+\n+namespace libcamera {\n+\n+/* Frame configuration provided by the config file */\n+struct ImageFrames {\n+\tstd::filesystem::path path;\n+\tstd::optional number;\n+};\n+\n+class ImageFrameGenerator : public FrameGenerator\n+{\n+public:\n+\tstatic std::unique_ptr create(ImageFrames &imageFrames);\n+\n+private:\n+\tstatic constexpr unsigned int frameRepeat = 4;\n+\n+\tstruct ImageFrameData {\n+\t\tstd::unique_ptr Y;\n+\t\tstd::unique_ptr UV;\n+\t\tSize size;\n+\t};\n+\n+\tvoid configure(const Size &size) override;\n+\tint generateFrame(const Size &size, const FrameBuffer *buffer) override;\n+\n+\tstd::vector imageFrameDatas_;\n+\tstd::vector scaledFrameDatas_;\n+\tImageFrames *imageFrames_;\n+\tunsigned int frameIndex_;\n+\tunsigned int parameter_;\n+};\n+\n+} /* namespace libcamera */\ndiff --git a/src/libcamera/pipeline/virtual/meson.build b/src/libcamera/pipeline/virtual/meson.build\nindex 0c537777f..bb38c193c 100644\n--- a/src/libcamera/pipeline/virtual/meson.build\n+++ b/src/libcamera/pipeline/virtual/meson.build\n@@ -1,8 +1,12 @@\n # SPDX-License-Identifier: CC0-1.0\n \n libcamera_internal_sources += files([\n+ 'image_frame_generator.cpp',\n 'test_pattern_generator.cpp',\n 'virtual.cpp',\n ])\n \n+libjpeg = dependency('libjpeg', required : true)\n+\n libcamera_deps += [libyuv_dep]\n+libcamera_deps += [libjpeg]\n","prefixes":["15","5/7"]}