From patchwork Thu May 5 15:18:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Curtin X-Patchwork-Id: 15801 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id E76CBC3256 for ; Thu, 5 May 2022 15:19:20 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 943956564C; Thu, 5 May 2022 17:19:20 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1651763960; bh=XDNlXpuJP0v9ny+9VUZFDkUVY2ZMrZkRKjCRG+cakmw=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=eUpzUo/5/LWOy/a9YljxOSMGTpETqWT9WN1b5ysdRi0o9AiIPaEPPI5EKbtQNsO/8 mSwpTy1yyvLsJxmTmmFNQvDj+fY+V8nhr+1W0QKDpenIxBkPklTQmRR27Zi1Dg1wsM x8sGfBZREAJBEaxLm3TWEWOF+8wDQ0o3fwHIHppxCgMwUhAFmG9swP3VPrFzF6Blyl lkoZMdDF3MqMXP4RVPrqhHo6cF8NDJSQejIzYCqJUo5/1TY/u1JkAPCUC9KCnweyYI AFpDx5D2WES++rvlAdQOgzIvNQNvzlsSwXwstzbpxPkL7WXmDaCIDFRdMpOVq0gdiP +qDU+1BthSY5w== Received: from us-smtp-delivery-74.mimecast.com (us-smtp-delivery-74.mimecast.com [170.10.133.74]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id ADD7860424 for ; Thu, 5 May 2022 17:19:18 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.b="bjCT/aCb"; dkim-atps=neutral DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1651763957; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=B9Ps2ABkfKYA90JWqdEc93XHx2W/OPs84CLEMF2jatc=; b=bjCT/aCb1F+FDj650Kw0+tqHG+Crl2Gq0qFYL2eqdcJrazexTY40/ZfdzUsKEd3EQ8Zzqr ZcLZdhOf7VjduyaIeQlNOix7LPmNyFqavYwlEdzNMXAOfZ6ZiAWBO8gDcGF5J/if/j1hYH Lac/gDzScnVqJkiLT9og/GmsZS+VQw8= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-656-Y_1DYaG_Mtmai2enAtdn3A-1; Thu, 05 May 2022 11:19:16 -0400 X-MC-Unique: Y_1DYaG_Mtmai2enAtdn3A-1 Received: by mail-wr1-f72.google.com with SMTP id o13-20020adfa10d000000b0020c6fa5a77cso1577313wro.23 for ; Thu, 05 May 2022 08:19:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=B9Ps2ABkfKYA90JWqdEc93XHx2W/OPs84CLEMF2jatc=; b=KwUGE4mLfpi0NCIlx+O83zuK4Dqx5UmUr5d6Jzk6t/xzyZ1+wHtOGzpMTtqQrz1fum hV3zK16TTdxujzbadSrHPY/Bk2FVvAWLFV0P0hLvXo14YfUEtc5tZdnFDsLU/+qQjjR2 ol/JzmIP87+0aD51GJWdTXuEc+z/0z6S01zBsUSuIiT5UZ/L7BB4ecQ9KN2LXuFyCpLs 40YHFIkTEZ5qnL5z/AI2kbNnI/mjhvxBBu9FEIxQ0F4VWTN3ZhyeTGBnLbHvFcvNZ2GG mk5iHojinwGMgt44o9zn78cgfGtNNVVIlXKK9hpjw3iPATfD0DfeTk4GGCFcR1VB3gys DWPQ== X-Gm-Message-State: AOAM533VqlMS3MsVXhi/H8IbQ1bgtDavCrGxosDn278tbtXqocZ1L0wS nd9egSzst9XiuaHhQA8Ww6pWZTqNEpIO8YzQOqoSs9X7H6JwXACthBxyFpuY8VvQGGLMVaiwTuS pYa/92ZRh843NkNvq0hM1WIwFF+qXI9jgxycxeYLNENvPxI9oK/CReEnc91Sugn1WCaoCGoHRI6 alqSaxNVRd X-Received: by 2002:a7b:c74f:0:b0:394:1ce3:cc42 with SMTP id w15-20020a7bc74f000000b003941ce3cc42mr5374624wmk.153.1651763954819; Thu, 05 May 2022 08:19:14 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyMPHDLg7erjUZAPisQSYmQBbwM08Wf3rDrCHXN5enrDN+5atmF9sYQWK6tYSEcQbEDtgRdYQ== X-Received: by 2002:a7b:c74f:0:b0:394:1ce3:cc42 with SMTP id w15-20020a7bc74f000000b003941ce3cc42mr5374579wmk.153.1651763954381; Thu, 05 May 2022 08:19:14 -0700 (PDT) Received: from localhost.localdomain ([2001:bb6:9634:7100:33b0:c1b1:8875:4adb]) by smtp.gmail.com with ESMTPSA id z6-20020a056000110600b0020ad05214d5sm1434819wrw.2.2022.05.05.08.19.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 May 2022 08:19:14 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Thu, 5 May 2022 16:18:50 +0100 Message-Id: <20220505151851.2421-3-ecurtin@redhat.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220505151851.2421-1-ecurtin@redhat.com> References: <20220505151851.2421-1-ecurtin@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=ecurtin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [libcamera-devel] [PATCH v8 3/4] cam: sdl_sink: Add SDL sink with initial YUYV support X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Eric Curtin via libcamera-devel From: Eric Curtin Reply-To: Eric Curtin Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" This adds more portability to existing cam sinks. You can pass a YUYV camera buffer and SDL will handle the pixel buffer conversion to the display. This allows cam reference implementation to display images on VMs, Mac M1, Raspberry Pi, etc. This also enables cam reference implementation, to run as a desktop application in wayland or x11. SDL also has support for Android and ChromeOS which has not been tested. Also tested on simpledrm raspberry pi 4 framebuffer successfully where existing kms sink did not work. Can also be used as kmsdrm sink. Only supports one camera stream at present. Signed-off-by: Eric Curtin Reviewed-by: Kieran Bingham --- src/cam/camera_session.cpp | 8 ++ src/cam/main.cpp | 4 + src/cam/main.h | 1 + src/cam/meson.build | 12 +++ src/cam/sdl_sink.cpp | 192 +++++++++++++++++++++++++++++++++++ src/cam/sdl_sink.h | 48 +++++++++ src/cam/sdl_texture.cpp | 32 ++++++ src/cam/sdl_texture.h | 23 +++++ src/cam/sdl_texture_yuyv.cpp | 13 +++ src/cam/sdl_texture_yuyv.h | 10 ++ 10 files changed, 343 insertions(+) create mode 100644 src/cam/sdl_sink.cpp create mode 100644 src/cam/sdl_sink.h create mode 100644 src/cam/sdl_texture.cpp create mode 100644 src/cam/sdl_texture.h create mode 100644 src/cam/sdl_texture_yuyv.cpp create mode 100644 src/cam/sdl_texture_yuyv.h diff --git a/src/cam/camera_session.cpp b/src/cam/camera_session.cpp index efffafbf..71e6bd60 100644 --- a/src/cam/camera_session.cpp +++ b/src/cam/camera_session.cpp @@ -19,6 +19,9 @@ #ifdef HAVE_KMS #include "kms_sink.h" #endif +#ifdef HAVE_SDL +#include "sdl_sink.h" +#endif #include "main.h" #include "stream_options.h" @@ -186,6 +189,11 @@ int CameraSession::start() sink_ = std::make_unique(options_[OptDisplay].toString()); #endif +#ifdef HAVE_SDL + if (options_.isSet(OptSDL)) + sink_ = std::make_unique(); +#endif + if (options_.isSet(OptFile)) { if (!options_[OptFile].toString().empty()) sink_ = std::make_unique(streamNames_, diff --git a/src/cam/main.cpp b/src/cam/main.cpp index c7f664b9..fd3108b0 100644 --- a/src/cam/main.cpp +++ b/src/cam/main.cpp @@ -137,6 +137,10 @@ int CamApp::parseOptions(int argc, char *argv[]) "Display viewfinder through DRM/KMS on specified connector", "display", ArgumentOptional, "connector", false, OptCamera); +#endif +#ifdef HAVE_SDL + parser.addOption(OptSDL, OptionNone, "Display viewfinder through SDL", + "sdl", ArgumentNone, "", false, OptCamera); #endif parser.addOption(OptFile, OptionString, "Write captured frames to disk\n" diff --git a/src/cam/main.h b/src/cam/main.h index 62f7bbc9..2b285808 100644 --- a/src/cam/main.h +++ b/src/cam/main.h @@ -18,6 +18,7 @@ enum { OptListProperties = 'p', OptMonitor = 'm', OptStream = 's', + OptSDL = 'S', OptListControls = 256, OptStrictFormats = 257, OptMetadata = 258, diff --git a/src/cam/meson.build b/src/cam/meson.build index 5bab8c9e..3032730b 100644 --- a/src/cam/meson.build +++ b/src/cam/meson.build @@ -32,11 +32,23 @@ if libdrm.found() ]) endif +libsdl2 = dependency('SDL2', required : false) + +if libsdl2.found() + cam_cpp_args += ['-DHAVE_SDL'] + cam_sources += files([ + 'sdl_sink.cpp', + 'sdl_texture.cpp', + 'sdl_texture_yuyv.cpp' + ]) +endif + cam = executable('cam', cam_sources, dependencies : [ libatomic, libcamera_public, libdrm, + libsdl2, libevent, ], cpp_args : cam_cpp_args, diff --git a/src/cam/sdl_sink.cpp b/src/cam/sdl_sink.cpp new file mode 100644 index 00000000..6fbeaf56 --- /dev/null +++ b/src/cam/sdl_sink.cpp @@ -0,0 +1,192 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * sdl_sink.cpp - SDL Sink + */ + +#include "sdl_sink.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "sdl_texture_yuyv.h" + +#include "event_loop.h" +#include "image.h" + +using namespace libcamera; + +SDLSink::SDLSink() + : sdlWindow_(nullptr), sdlRenderer_(nullptr), sdlRect_({}), + sdlInit_(false) +{ +} + +SDLSink::~SDLSink() +{ + stop(); +} + +int SDLSink::configure(const libcamera::CameraConfiguration &config) +{ + int ret = FrameSink::configure(config); + if (ret < 0) + return ret; + + if (config.size() > 1) { + std::cerr + << "SDL sink only supports one camera stream at present, streaming first camera stream" + << std::endl; + } else if (config.empty()) { + std::cerr << "Require at least one camera stream to process" + << std::endl; + return -EINVAL; + } + + const libcamera::StreamConfiguration &cfg = config.at(0); + sdlRect_.w = cfg.size.width; + sdlRect_.h = cfg.size.height; + + switch (cfg.pixelFormat) { + case libcamera::formats::YUYV: + sdlTexture_ = std::make_unique(sdlRect_); + break; + default: + std::cerr << "Unsupported pixel format " + << cfg.pixelFormat.toString() << std::endl; + return -EINVAL; + }; + + return 0; +} + +int SDLSink::start() +{ + int ret = SDL_Init(SDL_INIT_VIDEO); + if (ret) { + std::cerr << "Failed to initialize SDL: " << SDL_GetError() + << std::endl; + return ret; + } + + sdlInit_ = true; + sdlWindow_ = SDL_CreateWindow("", SDL_WINDOWPOS_UNDEFINED, + SDL_WINDOWPOS_UNDEFINED, sdlRect_.w, + sdlRect_.h, + SDL_WINDOW_SHOWN | SDL_WINDOW_RESIZABLE); + if (!sdlWindow_) { + std::cerr << "Failed to create SDL window: " << SDL_GetError() + << std::endl; + return -EINVAL; + } + + sdlRenderer_ = SDL_CreateRenderer(sdlWindow_, -1, 0); + if (!sdlRenderer_) { + std::cerr << "Failed to create SDL renderer: " << SDL_GetError() + << std::endl; + return -EINVAL; + } + + ret = SDL_RenderSetLogicalSize(sdlRenderer_, sdlRect_.w, sdlRect_.h); + if (ret) { /* Not critical to set, don't return in this case, set for scaling purposes */ + std::cerr << "Failed to set SDL render logical size: " + << SDL_GetError() << std::endl; + } + + ret = sdlTexture_->create(sdlRenderer_); + if (ret) { + return ret; + } + + EventLoop::instance()->addTimerEvent( + 10ms, std::bind(&SDLSink::processSDLEvents, this)); + + return 0; +} + +int SDLSink::stop() +{ + if (sdlTexture_) { + sdlTexture_->destroy(); + sdlTexture_ = nullptr; + } + + if (sdlRenderer_) { + SDL_DestroyRenderer(sdlRenderer_); + sdlRenderer_ = nullptr; + } + + if (sdlWindow_) { + SDL_DestroyWindow(sdlWindow_); + sdlWindow_ = nullptr; + } + + if (sdlInit_) { + SDL_Quit(); + sdlInit_ = false; + } + + return FrameSink::stop(); +} + +void SDLSink::mapBuffer(FrameBuffer *buffer) +{ + std::unique_ptr image = + Image::fromFrameBuffer(buffer, Image::MapMode::ReadOnly); + assert(image != nullptr); + + mappedBuffers_[buffer] = std::move(image); +} + +bool SDLSink::processRequest(Request *request) +{ + for (auto [stream, buffer] : request->buffers()) { + renderBuffer(buffer); + break; /* to be expanded to launch SDL window per buffer */ + } + + return true; +} + +/* + * Process SDL events, required for things like window resize and quit button + */ +void SDLSink::processSDLEvents() +{ + for (SDL_Event e; SDL_PollEvent(&e);) { + if (e.type == SDL_QUIT) { + /* Click close icon then quit */ + EventLoop::instance()->exit(0); + } + } +} + +void SDLSink::renderBuffer(FrameBuffer *buffer) +{ + Image *image = mappedBuffers_[buffer].get(); + + for (unsigned int i = 0; i < buffer->planes().size(); ++i) { + const FrameMetadata::Plane &meta = + buffer->metadata().planes()[i]; + + Span data = image->data(i); + if (meta.bytesused > data.size()) + std::cerr << "payload size " << meta.bytesused + << " larger than plane size " << data.size() + << std::endl; + + sdlTexture_->update(data); + } + + SDL_RenderClear(sdlRenderer_); + SDL_RenderCopy(sdlRenderer_, sdlTexture_->ptr_, nullptr, nullptr); + SDL_RenderPresent(sdlRenderer_); +} diff --git a/src/cam/sdl_sink.h b/src/cam/sdl_sink.h new file mode 100644 index 00000000..37dd9f7e --- /dev/null +++ b/src/cam/sdl_sink.h @@ -0,0 +1,48 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * sdl_sink.h - SDL Sink + */ + +#pragma once + +#include +#include +#include + +#include + +#include + +#include "frame_sink.h" +#include "sdl_texture.h" + +class Image; + +class SDLSink : public FrameSink +{ +public: + SDLSink(); + ~SDLSink(); + + int configure(const libcamera::CameraConfiguration &config) override; + int start() override; + int stop() override; + void mapBuffer(libcamera::FrameBuffer *buffer) override; + + bool processRequest(libcamera::Request *request) override; + +private: + void renderBuffer(libcamera::FrameBuffer *buffer); + void processSDLEvents(); + + std::map> + mappedBuffers_; + + std::unique_ptr sdlTexture_; + + SDL_Window *sdlWindow_; + SDL_Renderer *sdlRenderer_; + SDL_Rect sdlRect_; + SDL_PixelFormatEnum pixelFormat_; + bool sdlInit_; +}; diff --git a/src/cam/sdl_texture.cpp b/src/cam/sdl_texture.cpp new file mode 100644 index 00000000..b39080c9 --- /dev/null +++ b/src/cam/sdl_texture.cpp @@ -0,0 +1,32 @@ +#include "sdl_texture.h" + +#include + +SDLTexture::SDLTexture(const SDL_Rect &sdlRect, SDL_PixelFormatEnum pixelFormat, + const int pitch) + : sdlRect_(sdlRect), pixelFormat_(pixelFormat), pitch_(pitch) +{ +} + +SDLTexture::~SDLTexture() +{ +} + +int SDLTexture::create(SDL_Renderer *sdlRenderer_) +{ + ptr_ = SDL_CreateTexture(sdlRenderer_, pixelFormat_, + SDL_TEXTUREACCESS_STREAMING, sdlRect_.w, + sdlRect_.h); + if (!ptr_) { + std::cerr << "Failed to create SDL texture: " << SDL_GetError() + << std::endl; + return -ENOMEM; + } + + return 0; +} + +void SDLTexture::destroy() +{ + SDL_DestroyTexture(ptr_); +} diff --git a/src/cam/sdl_texture.h b/src/cam/sdl_texture.h new file mode 100644 index 00000000..f9525f7f --- /dev/null +++ b/src/cam/sdl_texture.h @@ -0,0 +1,23 @@ +#pragma once + +#include + +#include "image.h" + +class SDLTexture +{ +public: + SDLTexture(const SDL_Rect &sdlRect, SDL_PixelFormatEnum pixelFormat, + const int pitch); + virtual ~SDLTexture(); + int create(SDL_Renderer *sdlRenderer_); + void destroy(); + virtual void update(const libcamera::Span &data) = 0; + + SDL_Texture *ptr_; + +protected: + const SDL_Rect &sdlRect_; + SDL_PixelFormatEnum pixelFormat_; + const int pitch_; +}; diff --git a/src/cam/sdl_texture_yuyv.cpp b/src/cam/sdl_texture_yuyv.cpp new file mode 100644 index 00000000..cf519a5d --- /dev/null +++ b/src/cam/sdl_texture_yuyv.cpp @@ -0,0 +1,13 @@ +#include "sdl_texture_yuyv.h" + +using namespace libcamera; + +SDLTextureYUYV::SDLTextureYUYV(const SDL_Rect &sdlRect) + : SDLTexture(sdlRect, SDL_PIXELFORMAT_YUY2, 4 * ((sdlRect.w + 1) / 2)) +{ +} + +void SDLTextureYUYV::update(const Span &data) +{ + SDL_UpdateTexture(ptr_, &sdlRect_, data.data(), pitch_); +} diff --git a/src/cam/sdl_texture_yuyv.h b/src/cam/sdl_texture_yuyv.h new file mode 100644 index 00000000..ad70f14f --- /dev/null +++ b/src/cam/sdl_texture_yuyv.h @@ -0,0 +1,10 @@ +#pragma once + +#include "sdl_texture.h" + +class SDLTextureYUYV : public SDLTexture +{ +public: + SDLTextureYUYV(const SDL_Rect &sdlRect); + void update(const libcamera::Span &data) override; +};