From patchwork Fri May 20 19:01:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Curtin X-Patchwork-Id: 15998 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 576C0C0F2A for ; Fri, 20 May 2022 19:02:12 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 09A776566C; Fri, 20 May 2022 21:02:12 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1653073332; bh=kuOMBE990UyKDL26DhGS/lF5uPAVQU7mbYEs+z1+2Io=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=ltsx9wb34Byc37FbGWVj5ZAbMc3kkpwg4daICah3iQbI80o2OuCUk2RmwRNvS4kCH WWmrdQd40zEYV6pOcC5/no6NYb1aRIvVEYsJ2rmC8goAdB5KZyAQJbO23q42hwFfvj qqx0Zsg7zQcP3nhyYx8/91AtgvQqmaxsxCiKsjxDyen+IZHmPRJs5aWxBWT9eepey8 TqL0FZ0wY+uRz1gDXB73KuUmUgwS25pSCdLelRkYruzBoDDcd1lRVRH0y/mreMedZe n8v4RaeszUOHzT7A6JfWZQ2UD9hv5T8ZVZRcN7ibh+ALoKCpjP2R7apYg9KXT9FthO 5o7yBmbSbMqjw== Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9F8A16566C for ; Fri, 20 May 2022 21:02:09 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="QVGFpqmR"; dkim-atps=neutral DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1653073328; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ipc2oFQ8lFHTrRTk3ZxYK5iWN39zwTKL23ESMd260vI=; b=QVGFpqmRREH1k7AI45KaS54AdlcxvXyB4a3+tLmqim0EljAOsAnyKQAVecrCsiQMVJ36jp e3/dqtIK4D96EUxbaxUohdznBd8wTM52J5WTPduyC2RMM8o1MMZBlnSG1/5zEfi0uv+e1O LovxIOJRSlsMvIRzurCLYJInMO1qOxw= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-433-bMkMGZtXO06QLN4hmsaPkA-1; Fri, 20 May 2022 15:02:07 -0400 X-MC-Unique: bMkMGZtXO06QLN4hmsaPkA-1 Received: by mail-wr1-f72.google.com with SMTP id l6-20020adfe586000000b0020d0ff79cecso2854802wrm.12 for ; Fri, 20 May 2022 12:02:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ipc2oFQ8lFHTrRTk3ZxYK5iWN39zwTKL23ESMd260vI=; b=rqhvM1DKtYJk8Nl3uf+GsGaByQfeDRAfc4qqXt7Xt/I3/J9l65+DpgQ4Ggg2HrfxPx hAmNJkComLr5lb9bb5jM0a1VvyAsZQvQHocuwiAINLn9gOCn+7oZV3nWVUNd20rr2kD3 6ybY6RHlE6GTr5oAi5lOW4WBY1xNdzFJd00UdWLMwbtwuj+X0tPIvHnLGp3DxtbXitdP juHj92NMPcijHsNyjBW1558OqjGiOiE6Ynx4t0h3CUQ2C25+kyOYtwNwSy6pb03zXHg+ LOSfn+jqp5nnkw1HHfPwJS6rAZ31MakRxV50Q8GwYYBj4NrYYiJdX8+HQ90xfQVvp2tO fRLA== X-Gm-Message-State: AOAM532zlcezHmGkqaFyYpL9QE4CTWnqLg3mV3sMEdsnFQO12j+ast5g UOq5iJ9V8KV4S6x0qCNtTQ+HVvH+bInhv7fC0isCrvrJv7WWnJbOjFsgngcdRjJR+/6mWb5m13J jD804+3EdSyRLBoO7CXjAqOO6aWVM1us/KrXmKXinWGwuvCGM+xDb6dvgAGuLh6v3rcanGM4Ej0 VABMYeqDyi X-Received: by 2002:a05:600c:3caa:b0:394:8fb8:716 with SMTP id bg42-20020a05600c3caa00b003948fb80716mr9800614wmb.105.1653073325556; Fri, 20 May 2022 12:02:05 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy4+uwfgR5k1+1jo/NaHdvdkrr0Ii8GThSo07Kzzd9tzddwEzkjTmxpZhWbO/VGwNmxwfbpMA== X-Received: by 2002:a05:600c:3caa:b0:394:8fb8:716 with SMTP id bg42-20020a05600c3caa00b003948fb80716mr9800587wmb.105.1653073325112; Fri, 20 May 2022 12:02:05 -0700 (PDT) Received: from p1.Home ([2001:8a0:6724:4500:a69c:e66f:828e:b340]) by smtp.gmail.com with ESMTPSA id u30-20020adfa19e000000b0020d10a249eesm3337016wru.13.2022.05.20.12.02.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 May 2022 12:02:04 -0700 (PDT) To: libcamera-devel@lists.libcamera.org, laurent.pinchart@ideasonboard.com, kieran.bingham@ideasonboard.com, jacopo@jmondi.org, javierm@redhat.com Date: Fri, 20 May 2022 20:01:05 +0100 Message-Id: <20220520190106.425386-4-ecurtin@redhat.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20220520190106.425386-1-ecurtin@redhat.com> References: <20220520190106.425386-1-ecurtin@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=ecurtin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [libcamera-devel] [PATCH v9 3/4] cam: sdl_sink: Add SDL sink with initial YUYV support X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Eric Curtin via libcamera-devel From: Eric Curtin Reply-To: Eric Curtin Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This adds more portability to existing cam sinks. You can pass a YUYV camera buffer and SDL will handle the pixel buffer conversion to the display. This allows cam reference implementation to display images on VMs, Mac M1, Raspberry Pi, etc. This also enables cam reference implementation, to run as a desktop application in Wayland or X11. SDL also has support for Android and ChromeOS which has not been tested. Also tested on simpledrm Raspberry Pi 4 framebuffer successfully where existing kms sink did not work. Can also be used as kmsdrm sink. Only supports one camera stream at present. Signed-off-by: Eric Curtin Reviewed-by: Laurent Pinchart Reviewed-by: Kieran Bingham Tested-by: Jacopo Mondi --- src/cam/camera_session.cpp | 8 ++ src/cam/main.cpp | 4 + src/cam/main.h | 1 + src/cam/meson.build | 12 +++ src/cam/sdl_sink.cpp | 194 +++++++++++++++++++++++++++++++++++ src/cam/sdl_sink.h | 49 +++++++++ src/cam/sdl_texture.cpp | 37 +++++++ src/cam/sdl_texture.h | 29 ++++++ src/cam/sdl_texture_yuyv.cpp | 20 ++++ src/cam/sdl_texture_yuyv.h | 17 +++ 10 files changed, 371 insertions(+) create mode 100644 src/cam/sdl_sink.cpp create mode 100644 src/cam/sdl_sink.h create mode 100644 src/cam/sdl_texture.cpp create mode 100644 src/cam/sdl_texture.h create mode 100644 src/cam/sdl_texture_yuyv.cpp create mode 100644 src/cam/sdl_texture_yuyv.h diff --git a/src/cam/camera_session.cpp b/src/cam/camera_session.cpp index efffafbf..336ae471 100644 --- a/src/cam/camera_session.cpp +++ b/src/cam/camera_session.cpp @@ -20,6 +20,9 @@ #include "kms_sink.h" #endif #include "main.h" +#ifdef HAVE_SDL +#include "sdl_sink.h" +#endif #include "stream_options.h" using namespace libcamera; @@ -186,6 +189,11 @@ int CameraSession::start() sink_ = std::make_unique(options_[OptDisplay].toString()); #endif +#ifdef HAVE_SDL + if (options_.isSet(OptSDL)) + sink_ = std::make_unique(); +#endif + if (options_.isSet(OptFile)) { if (!options_[OptFile].toString().empty()) sink_ = std::make_unique(streamNames_, diff --git a/src/cam/main.cpp b/src/cam/main.cpp index c7f664b9..962262a8 100644 --- a/src/cam/main.cpp +++ b/src/cam/main.cpp @@ -147,6 +147,10 @@ int CamApp::parseOptions(int argc, char *argv[]) "The default file name is 'frame-#.bin'.", "file", ArgumentOptional, "filename", false, OptCamera); +#ifdef HAVE_SDL + parser.addOption(OptSDL, OptionNone, "Display viewfinder through SDL", + "sdl", ArgumentNone, "", false, OptCamera); +#endif parser.addOption(OptStream, &streamKeyValue, "Set configuration of a camera stream", "stream", true, OptCamera); diff --git a/src/cam/main.h b/src/cam/main.h index 62f7bbc9..507a184f 100644 --- a/src/cam/main.h +++ b/src/cam/main.h @@ -17,6 +17,7 @@ enum { OptList = 'l', OptListProperties = 'p', OptMonitor = 'm', + OptSDL = 'S', OptStream = 's', OptListControls = 256, OptStrictFormats = 257, diff --git a/src/cam/meson.build b/src/cam/meson.build index 5bab8c9e..7d714152 100644 --- a/src/cam/meson.build +++ b/src/cam/meson.build @@ -32,12 +32,24 @@ if libdrm.found() ]) endif +libsdl2 = dependency('SDL2', required : false) + +if libsdl2.found() + cam_cpp_args += ['-DHAVE_SDL'] + cam_sources += files([ + 'sdl_sink.cpp', + 'sdl_texture.cpp', + 'sdl_texture_yuyv.cpp' + ]) +endif + cam = executable('cam', cam_sources, dependencies : [ libatomic, libcamera_public, libdrm, libevent, + libsdl2, ], cpp_args : cam_cpp_args, install : true) diff --git a/src/cam/sdl_sink.cpp b/src/cam/sdl_sink.cpp new file mode 100644 index 00000000..65430efb --- /dev/null +++ b/src/cam/sdl_sink.cpp @@ -0,0 +1,194 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2022, Ideas on Board Oy + * + * sdl_sink.h - SDL Sink + */ + +#include "sdl_sink.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "event_loop.h" +#include "image.h" +#include "sdl_texture_yuyv.h" + +using namespace libcamera; + +using namespace std::chrono_literals; + +SDLSink::SDLSink() + : window_(nullptr), renderer_(nullptr), rect_({}), + init_(false) +{ +} + +SDLSink::~SDLSink() +{ + stop(); +} + +int SDLSink::configure(const libcamera::CameraConfiguration &config) +{ + const int ret = FrameSink::configure(config); + if (ret < 0) + return ret; + + if (config.size() > 1) { + std::cerr + << "SDL sink only supports one camera stream at present, streaming first camera stream" + << std::endl; + } else if (config.empty()) { + std::cerr << "Require at least one camera stream to process" + << std::endl; + return -EINVAL; + } + + const libcamera::StreamConfiguration &cfg = config.at(0); + rect_.w = cfg.size.width; + rect_.h = cfg.size.height; + + switch (cfg.pixelFormat) { + case libcamera::formats::YUYV: + texture_ = std::make_unique(rect_); + break; + default: + std::cerr << "Unsupported pixel format " + << cfg.pixelFormat.toString() << std::endl; + return -EINVAL; + }; + + return 0; +} + +int SDLSink::start() +{ + int ret = SDL_Init(SDL_INIT_VIDEO); + if (ret) { + std::cerr << "Failed to initialize SDL: " << SDL_GetError() + << std::endl; + return ret; + } + + init_ = true; + window_ = SDL_CreateWindow("", SDL_WINDOWPOS_UNDEFINED, + SDL_WINDOWPOS_UNDEFINED, rect_.w, + rect_.h, + SDL_WINDOW_SHOWN | SDL_WINDOW_RESIZABLE); + if (!window_) { + std::cerr << "Failed to create SDL window: " << SDL_GetError() + << std::endl; + return -EINVAL; + } + + renderer_ = SDL_CreateRenderer(window_, -1, 0); + if (!renderer_) { + std::cerr << "Failed to create SDL renderer: " << SDL_GetError() + << std::endl; + return -EINVAL; + } + + ret = SDL_RenderSetLogicalSize(renderer_, rect_.w, rect_.h); + if (ret) { /* Not critical to set, don't return in this case, set for scaling purposes */ + std::cerr << "Failed to set SDL render logical size: " + << SDL_GetError() << std::endl; + } + + ret = texture_->create(renderer_); + if (ret) { + return ret; + } + + /* \todo Make the event cancellable to support stop/start cycles. */ + EventLoop::instance()->addTimerEvent( + 10ms, std::bind(&SDLSink::processSDLEvents, this)); + + return 0; +} + +int SDLSink::stop() +{ + if (texture_) { + texture_.reset(); + } + + if (renderer_) { + SDL_DestroyRenderer(renderer_); + renderer_ = nullptr; + } + + if (window_) { + SDL_DestroyWindow(window_); + window_ = nullptr; + } + + if (init_) { + SDL_Quit(); + init_ = false; + } + + return FrameSink::stop(); +} + +void SDLSink::mapBuffer(FrameBuffer *buffer) +{ + std::unique_ptr image = + Image::fromFrameBuffer(buffer, Image::MapMode::ReadOnly); + assert(image != nullptr); + + mappedBuffers_[buffer] = std::move(image); +} + +bool SDLSink::processRequest(Request *request) +{ + for (auto [stream, buffer] : request->buffers()) { + renderBuffer(buffer); + break; /* to be expanded to launch SDL window per buffer */ + } + + return true; +} + +/* + * Process SDL events, required for things like window resize and quit button + */ +void SDLSink::processSDLEvents() +{ + for (SDL_Event e; SDL_PollEvent(&e);) { + if (e.type == SDL_QUIT) { + /* Click close icon then quit */ + EventLoop::instance()->exit(0); + } + } +} + +void SDLSink::renderBuffer(FrameBuffer *buffer) +{ + Image *image = mappedBuffers_[buffer].get(); + + /* \todo Implement support for multi-planar formats. */ + const FrameMetadata::Plane &meta = + buffer->metadata().planes()[0]; + + Span data = image->data(0); + if (meta.bytesused > data.size()) + std::cerr << "payload size " << meta.bytesused + << " larger than plane size " << data.size() + << std::endl; + + texture_->update(data); + + SDL_RenderClear(renderer_); + SDL_RenderCopy(renderer_, texture_->get(), nullptr, nullptr); + SDL_RenderPresent(renderer_); +} diff --git a/src/cam/sdl_sink.h b/src/cam/sdl_sink.h new file mode 100644 index 00000000..83171cca --- /dev/null +++ b/src/cam/sdl_sink.h @@ -0,0 +1,49 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2022, Ideas on Board Oy + * + * sdl_sink.h - SDL Sink + */ + +#pragma once + +#include +#include + +#include + +#include + +#include "frame_sink.h" + +class Image; +class SDLTexture; + +class SDLSink : public FrameSink +{ +public: + SDLSink(); + ~SDLSink(); + + int configure(const libcamera::CameraConfiguration &config) override; + int start() override; + int stop() override; + void mapBuffer(libcamera::FrameBuffer *buffer) override; + + bool processRequest(libcamera::Request *request) override; + +private: + void renderBuffer(libcamera::FrameBuffer *buffer); + void processSDLEvents(); + + std::map> + mappedBuffers_; + + std::unique_ptr texture_; + + SDL_Window *window_; + SDL_Renderer *renderer_; + SDL_Rect rect_; + SDL_PixelFormatEnum pixelFormat_; + bool init_; +}; diff --git a/src/cam/sdl_texture.cpp b/src/cam/sdl_texture.cpp new file mode 100644 index 00000000..ac355a97 --- /dev/null +++ b/src/cam/sdl_texture.cpp @@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2022, Ideas on Board Oy + * + * sdl_texture.cpp - SDL Texture + */ + +#include "sdl_texture.h" + +#include + +SDLTexture::SDLTexture(const SDL_Rect &rect, SDL_PixelFormatEnum pixelFormat, + const int pitch) + : ptr_(nullptr), rect_(rect), pixelFormat_(pixelFormat), pitch_(pitch) +{ +} + +SDLTexture::~SDLTexture() +{ + if (ptr_) { + SDL_DestroyTexture(ptr_); + } +} + +int SDLTexture::create(SDL_Renderer *renderer_) +{ + ptr_ = SDL_CreateTexture(renderer_, pixelFormat_, + SDL_TEXTUREACCESS_STREAMING, rect_.w, + rect_.h); + if (!ptr_) { + std::cerr << "Failed to create SDL texture: " << SDL_GetError() + << std::endl; + return -ENOMEM; + } + + return 0; +} diff --git a/src/cam/sdl_texture.h b/src/cam/sdl_texture.h new file mode 100644 index 00000000..b04eece0 --- /dev/null +++ b/src/cam/sdl_texture.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2022, Ideas on Board Oy + * + * sdl_texture.h - SDL Texture + */ + +#pragma once + +#include + +#include "image.h" + +class SDLTexture +{ +public: + SDLTexture(const SDL_Rect &rect, SDL_PixelFormatEnum pixelFormat, + const int pitch); + virtual ~SDLTexture(); + int create(SDL_Renderer *renderer_); + virtual void update(const libcamera::Span &data) = 0; + SDL_Texture *get() const { return ptr_; } + +protected: + SDL_Texture *ptr_; + const SDL_Rect &rect_; + SDL_PixelFormatEnum pixelFormat_; + const int pitch_; +}; diff --git a/src/cam/sdl_texture_yuyv.cpp b/src/cam/sdl_texture_yuyv.cpp new file mode 100644 index 00000000..a219097b --- /dev/null +++ b/src/cam/sdl_texture_yuyv.cpp @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2022, Ideas on Board Oy + * + * sdl_texture_yuyv.cpp - SDL Texture YUYV + */ + +#include "sdl_texture_yuyv.h" + +using namespace libcamera; + +SDLTextureYUYV::SDLTextureYUYV(const SDL_Rect &sdlRect) + : SDLTexture(sdlRect, SDL_PIXELFORMAT_YUY2, 4 * ((sdlRect.w + 1) / 2)) +{ +} + +void SDLTextureYUYV::update(const Span &data) +{ + SDL_UpdateTexture(ptr_, &rect_, data.data(), pitch_); +} diff --git a/src/cam/sdl_texture_yuyv.h b/src/cam/sdl_texture_yuyv.h new file mode 100644 index 00000000..38c51c74 --- /dev/null +++ b/src/cam/sdl_texture_yuyv.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2022, Ideas on Board Oy + * + * sdl_texture_yuyv.h - SDL Texture YUYV + */ + +#pragma once + +#include "sdl_texture.h" + +class SDLTextureYUYV : public SDLTexture +{ +public: + SDLTextureYUYV(const SDL_Rect &sdlRect); + void update(const libcamera::Span &data) override; +};