From patchwork Sun Aug 14 16:07:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kunal Agarwal X-Patchwork-Id: 17119 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8401FC3272 for ; Sun, 14 Aug 2022 16:09:00 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2140661FC6; Sun, 14 Aug 2022 18:09:00 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=libcamera.org; s=mail; t=1660493340; bh=xDQy7Yy0hcbmpWo62Aixp8tF1ZzVL9XntObilG+0xww=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=fSE1ybFlUWdVuqegh14DNjX6/tB7dFer3hKsomeNwYpoZJziLsee2FjZOzvlYOTB8 0emJLoW3T2E/vmFgx2VYiYGTwEulbSFJeHpJ0zOr44cJffv27OtAVsvEDK65VRmIL0 ljkV9ZP86cOh+7lZ9m7IVyDeAR0jJOM16tuACHjaIOl1MfWwUPYxDRDgm6qPhVpbt+ rgpJdC4T5Tua5tugdzHZA4LUXsMC/XdNmVOGFLY55CyyEgk6T5K6wt8k6U1POg2Jag 3X49EhEl8uLwZTEAgV5DAhpCs17Re0tpYnw0vK0tmVyNFAgV7iQGI34dTPzN6zUcmY iY+hKlmc+3hNQ== Received: from mail-pf1-x430.google.com (mail-pf1-x430.google.com [IPv6:2607:f8b0:4864:20::430]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 064A661FBC for ; Sun, 14 Aug 2022 18:08:59 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="HKcLO1xd"; dkim-atps=neutral Received: by mail-pf1-x430.google.com with SMTP id k14so4978993pfh.0 for ; Sun, 14 Aug 2022 09:08:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=eQ+c35Jw1mdaAvRe1cjqx4UoVwJ02sxS2HxP9a+AEoo=; b=HKcLO1xdS+QANOigJsggytzfAnYBCDbIs2WLv2ygaDS1gJFguDolA1ybKri8VUOyFx /p2Z+1gRGj+TRaNO4YRXaxwBK4P7FflG4LUE2IgfTTKnk3e/2vv6+874lan5wCjRq2ON kNRjaBLtatB63QeEn9V5wUNBhE3Hm+hUJitT2qdVwDswBwAuf0NK+AUVRcV7ZZppUmq5 qsHHQvOlb1nowtIxBMg7Lj8zx/Uc1pqPtcEQ4U3VkAUv/d6v8XqCsZHwOBKuhDn6Vk94 tOI3o8yaGcUuw6J+SkpH5e80AzBCQU0gWkY3O+M/jkPMN8nJIikGYA7/MK46Bs02UnV1 +TLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=eQ+c35Jw1mdaAvRe1cjqx4UoVwJ02sxS2HxP9a+AEoo=; b=TrXCYtNYoo9LmJrrEtIxkHNbUy3dVQB6qgqScwYEUn3hMXXvfKhqyQcgMyg0fGRV+j RJx3TPvHGC0PFydXiegFFgJ5C6ihHNKeVu2cIWchCQCQHZ0iID4slvAxdKWafFa2B9Vx NTcte/2tZSXeEnXhIg+AL6oY+0ZeUq+G2x3VUsAymzXd5TKC/TH3EAtM9jI/h1IV6lys 6oSFg5nfTIkGDZhfsq2TjiAROle8NBrzUOieUD41QAsmGM3rXwfFWk80gvODUh09ptnt VW+UWbaE9/j5QvrlZZo9qWyjh57pNYRv9tah9JsReBVn3S8Jr/vM+ifPOa1qMC796x29 kweQ== X-Gm-Message-State: ACgBeo2kz5C8sIhmEhNDwIQFW2tGnfkebUCoWmFhxCVmw1ZHFvCDiIyn 1VZ6B6IPwCSqQoUFfHQFqqS+ajn5v8g= X-Google-Smtp-Source: AA6agR5aIGL3dY+ggWYLc0g3w0bIYMt2P2gl4N3XSsWCC0vjO1V8U0O7xlG5qtz/k4i73XR/daEBEA== X-Received: by 2002:aa7:9813:0:b0:52d:395d:c98d with SMTP id e19-20020aa79813000000b0052d395dc98dmr12611450pfl.55.1660493336987; Sun, 14 Aug 2022 09:08:56 -0700 (PDT) Received: from pop-os.localdomain ([115.96.217.20]) by smtp.googlemail.com with ESMTPSA id j6-20020a170902da8600b001709aea1516sm5576653plx.276.2022.08.14.09.08.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 14 Aug 2022 09:08:56 -0700 (PDT) To: libcamera-devel@lists.libcamera.org Date: Sun, 14 Aug 2022 21:37:45 +0530 Message-Id: <20220814160747.52093-4-kunalagarwal1072002@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220814160747.52093-1-kunalagarwal1072002@gmail.com> References: <20220814160747.52093-1-kunalagarwal1072002@gmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v3 4/6] pipeline: simple: GL Converter implementation X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Kunal Agarwal via libcamera-devel From: Kunal Agarwal Reply-To: Kunal Agarwal Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Drop-in replacement for already existing converter which performs debayering on Raw input data and outputs data in RGB format The GL converter uses GPU for debayering via OpenGL compute Shaders Signed-off-by: Kunal Agarwal --- .../pipeline/simple/converter_gl.cpp | 297 ++++++++++++++++++ src/libcamera/pipeline/simple/converter_gl.h | 116 +++++++ 2 files changed, 413 insertions(+) create mode 100644 src/libcamera/pipeline/simple/converter_gl.cpp create mode 100644 src/libcamera/pipeline/simple/converter_gl.h diff --git a/src/libcamera/pipeline/simple/converter_gl.cpp b/src/libcamera/pipeline/simple/converter_gl.cpp new file mode 100644 index 00000000..2d123da7 --- /dev/null +++ b/src/libcamera/pipeline/simple/converter_gl.cpp @@ -0,0 +1,297 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2022, Kunal Agarwal + * + * converter_gl.cpp - GL converter for debayering + */ + +#include "converter_gl.h" + +#include +#include + +#include + +#include +#include + +#include "libcamera/internal/formats.h" + +#include +#include + +#include "texture.h" + +namespace libcamera { + +LOG_DECLARE_CATEGORY(SimplePipeline) + +float rectangleVertices[] = { + /* Coords */ /* texCoords */ + 1.0f, -1.0f, 1.0f, 0.0f, + -1.0f, -1.0f, 0.0f, 0.0f, + -1.0f, 1.0f, 0.0f, 1.0f, + + 1.0f, 1.0f, 1.0f, 1.0f, + 1.0f, -1.0f, 1.0f, 0.0f, + -1.0f, 1.0f, 0.0f, 1.0f +}; + +int SimpleConverter::configure(const StreamConfiguration &inputCfg, + const std::vector> &outputCfgs) +{ + int ret = configureGL(inputCfg, outputCfgs.front()); + return ret; +} + +int SimpleConverter::configureGL(const StreamConfiguration &inputCfg, + const StreamConfiguration &outputCfg) +{ + informat_.size = inputCfg.size; + informat_.planes[0].bpl_ = inputCfg.stride; + outformat_.size = outputCfg.size; + outformat_.planes[0].bpl_ = outputCfg.stride; + return 0; +} + +std::vector SimpleConverter::formats([[maybe_unused]] PixelFormat input) +{ + return { + PixelFormat::fromString("RGB888"), + PixelFormat::fromString("ARGB8888"), + }; +} + +SizeRange SimpleConverter::sizes(const Size &input) +{ + SizeRange sizes({ 1, 1 }, input); + return sizes; +} + +std::tuple +SimpleConverter::strideAndFrameSize(const PixelFormat &pixelFormat, const Size &sz) +{ + const PixelFormatInfo &info = PixelFormatInfo::info(pixelFormat); + return std::make_tuple(info.stride(sz.width, 0, 1), info.frameSize(sz, 1)); +} + +int SimpleConverter::exportBuffers(unsigned int output, unsigned int count, + std::vector> *buffers) +{ + if (output != 0) + return -EINVAL; + + if (outputBuffers.size() > 0) + return -EINVAL; + + std::vector> out; + for (unsigned i = 0; i < count; ++i) { + auto tex = createBuffer(); + outputBuffers.emplace_back(tex.second); + buffers->push_back(std::move(tex.first)); + } + return count; +} + +std::pair, GlRenderTarget> SimpleConverter::createBuffer() +{ + bo = gbm_bo_create(gbm, outformat_.size.width, outformat_.size.height, + GBM_BO_FORMAT_ARGB8888, GBM_BO_USE_RENDERING); + if (!bo) + LOG(SimplePipeline, Error) << "GBM buffer not created "; + + unsigned int filedesc = gbm_bo_get_fd(bo); + + LOG(SimplePipeline, Debug) << "File Descriptor value: " << filedesc; + + DmabufImage dimg = importDmabuf(filedesc, outformat_.size, libcamera::formats::ARGB8888); + + std::vector planes; + UniqueFD fd(filedesc); + FrameBuffer::Plane plane; + plane.fd = SharedFD(std::move(fd)); + plane.offset = gbm_bo_get_offset(bo, 0); + plane.length = gbm_bo_get_stride_for_plane(bo, 0) * outformat_.size.height; + + planes.push_back(std::move(plane)); + + auto fb = std::make_unique(planes); + return std::make_pair(std::move(fb), GlRenderTarget(fb.get(), dimg)); +} + +SimpleConverter::DmabufImage SimpleConverter::importDmabuf(int fdesc, Size pixelSize, PixelFormat format) +{ + int bytes_per_pixel = 4; + EGLint const attrs[] = { + EGL_WIDTH, + (int)pixelSize.width, + EGL_HEIGHT, + (int)pixelSize.height, + EGL_LINUX_DRM_FOURCC_EXT, + (int)format.fourcc(), + EGL_DMA_BUF_PLANE0_FD_EXT, + fdesc, + EGL_DMA_BUF_PLANE0_OFFSET_EXT, + 0, + EGL_DMA_BUF_PLANE0_PITCH_EXT, + (int)pixelSize.width * bytes_per_pixel, + EGL_NONE, + }; + + EGLImageKHR image = eglCreateImageKHR( + display_, + EGL_NO_CONTEXT, + EGL_LINUX_DMA_BUF_EXT, + NULL, + attrs); + + int e = glGetError(); + + if (e != GL_NO_ERROR) + LOG(SimplePipeline, Error) << "GL_ERROR: " << e; + + GLuint texture; + glGenTextures(1, &texture); + struct DmabufImage img = { + .texture = texture, + .image = image, + }; + + glBindTexture(GL_TEXTURE_2D, texture); + auto glEGLImageTargetTexture2DOES = (PFNGLEGLIMAGETARGETTEXTURE2DOESPROC)eglGetProcAddress("glEGLImageTargetTexture2DOES"); + glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, image); + + return img; +} + +int SimpleConverter::start() +{ + eglBindAPI(EGL_OPENGL_API); + device_ = open("/dev/dri/card0", O_RDWR); + + if (!device_) + LOG(SimplePipeline, Error) << "GBM Device not opened "; + + gbm = gbm_create_device(device_); + + if (!gbm) + LOG(SimplePipeline, Error) << " GBM Device not created "; + + auto eglGetPlatformDisplayEXT = (PFNEGLGETPLATFORMDISPLAYEXTPROC)eglGetProcAddress("eglGetPlatformDisplayEXT"); + + /* get an EGL display connection */ + display_ = eglGetPlatformDisplayEXT(EGL_PLATFORM_GBM_MESA, gbm, NULL); + + /* initialize the EGL display connection */ + eglInitialize(display_, NULL, NULL); + EGLConfig config; + EGLint n_of_configs; + + eglGetConfigs(display_, &config, 1, &n_of_configs); + + context_ = eglCreateContext(display_, config, EGL_NO_CONTEXT, NULL); + + if (context_ == EGL_NO_CONTEXT) { + EGLint err = eglGetError(); + LOG(SimplePipeline, Error) << " Context creation failed: " << err; + return -1; + } + + /* connect the context to the surface */ + eglMakeCurrent(display_, EGL_NO_SURFACE, EGL_NO_SURFACE, context_); + + shaderProgram_.callShader("default.vert", "default.frag"); + framebufferProgram_.callShader("bayer_8.vert", "bayer_8.frag"); + framebufferProgram_.activate(); + glBindAttribLocation(framebufferProgram_.id(), 0, "vertexIn"); + glBindAttribLocation(framebufferProgram_.id(), 2, "textureIn"); + glUniform1i(glGetUniformLocation(framebufferProgram_.id(), "tex_y"), 0); + glUniform2f(glGetUniformLocation(framebufferProgram_.id(), "tex_step"), 1.0f / (informat_.planes[0].bpl_ - 1), + 1.0f / (informat_.size.height - 1)); + glUniform2i(glGetUniformLocation(framebufferProgram_.id(), "tex_size"), informat_.size.width, + informat_.size.height); + glUniform2f(glGetUniformLocation(framebufferProgram_.id(), "tex_bayer_first_red"), 0.0, 1.0); + + /* Prepare framebuffer rectangle VBO and VAO */ + + glGenVertexArrays(1, &rectVAO); + glGenBuffers(1, &rectVBO); + glBindVertexArray(rectVAO); + glBindBuffer(GL_ARRAY_BUFFER, rectVBO); + glBufferData(GL_ARRAY_BUFFER, sizeof(rectangleVertices), &rectangleVertices, GL_STATIC_DRAW); + glEnableVertexAttribArray(0); + glVertexAttribPointer(0, 2, GL_FLOAT, GL_FALSE, 4 * sizeof(float), (void *)0); + glEnableVertexAttribArray(1); + glVertexAttribPointer(1, 2, GL_FLOAT, GL_FALSE, 4 * sizeof(float), (void *)(2 * sizeof(float))); + + /* create FrameBuffer object */ + + glGenFramebuffers(1, &fbo_); + glBindFramebuffer(GL_FRAMEBUFFER, fbo_); + + return 0; +} + +int SimpleConverter::queueBuffers(FrameBuffer *input, + const std::map &outputs) +{ + int ret; + if (outputs.empty()) + return -EINVAL; + + for (auto &ib : outputs) { + ret = queueBufferGL(input, ib.second); + if (ret < 0) + return ret; + } + + return 0; +} + +int SimpleConverter::queueBufferGL(FrameBuffer *input, FrameBuffer *output) +{ + DmabufImage rend_tex = importDmabuf(output->planes()[0].fd.get(), outformat_.size, libcamera::formats::ARGB8888); + + Texture bayer(GL_TEXTURE_2D, rend_tex.texture); + bayer.initTexture(GL_TEXTURE0); + bayer.startTexture(mappedBuffers_[input].get(), GL_LUMINANCE, GL_UNSIGNED_BYTE, informat_.size); + bayer.unbind(); + + /* Error checking framebuffer*/ + GLenum fboStatus = glCheckFramebufferStatus(GL_FRAMEBUFFER); + + if (fboStatus != GL_FRAMEBUFFER_COMPLETE) + LOG(SimplePipeline, Debug) << "Framebuffer error: " << fboStatus; + + /* Main */ + /* Bind the custom framebuffer */ + glBindFramebuffer(GL_FRAMEBUFFER, fbo_); + /* Specify the color of the background */ + glClearColor(0.0f, 0.0f, 0.0f, 1.0f); + /* Clean the back buffer and assign the new color to it */ + glClear(GL_COLOR_BUFFER_BIT); + /* Bind the default framebuffer */ + glBindFramebuffer(GL_FRAMEBUFFER, 0); + /* Draw the framebuffer rectangle */ + framebufferProgram_.activate(); + glBindVertexArray(rectVAO); + bayer.bind(); + glDrawArrays(GL_TRIANGLES, 0, 6); + + return 0; +} + +void SimpleConverter::stop() +{ + /* Delete all the objects we've created */ + shaderProgram_.deleteProgram(); + glDeleteFramebuffers(1, &fbo_); + eglDestroyContext(display_, context_); + eglTerminate(display_); + + gbm_bo_destroy(bo); + gbm_device_destroy(gbm); + close(device_); +} + +} /* namespace libcamera */ diff --git a/src/libcamera/pipeline/simple/converter_gl.h b/src/libcamera/pipeline/simple/converter_gl.h new file mode 100644 index 00000000..a61cead0 --- /dev/null +++ b/src/libcamera/pipeline/simple/converter_gl.h @@ -0,0 +1,116 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2022, Kunal Agarwal + * + * converter_gl.cpp - GL converter for debayering + */ + +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include +#include + +#include "libcamera/internal/mapped_framebuffer.h" + +#include +#include + +#include "shader.h" + +namespace libcamera { + +class FrameBuffer; +class GlRenderTarget; + +class SimpleConverter +{ +public: + int configure(const StreamConfiguration &inputCfg, + const std::vector> &outputCfgs); + std::vector formats(PixelFormat input); + SizeRange sizes(const Size &input); + + std::tuple + strideAndFrameSize(const PixelFormat &pixelFormat, const Size &size); + + int queueBuffers(FrameBuffer *input, + const std::map &outputs); + + int start(); + void stop(); + + int exportBuffers(unsigned int output, unsigned int count, + std::vector> *buffers); + std::pair, GlRenderTarget> createBuffer(); + bool isValid() const { return true; } + + Signal inputBufferReady; + Signal outputBufferReady; + struct DmabufImage { + GLuint texture; + EGLImageKHR image; + }; + +private: + int configureGL(const StreamConfiguration &inputCfg, + const StreamConfiguration &outputCfg); + DmabufImage importDmabuf(int fdesc, Size pixelSize, PixelFormat format); + int queueBufferGL(FrameBuffer *input, FrameBuffer *output); + + std::map> + mappedBuffers_; + + struct ConverterFormat { + struct Plane { + uint32_t size_ = 0; + uint32_t bpl_ = 0; + }; + PixelFormat fourcc; + Size size; + std::array planes; + unsigned int planesCount = 0; + }; + + int device_; + unsigned int rectVAO, rectVBO; + EGLDisplay display_; + EGLContext context_; + + struct gbm_device *gbm; + struct gbm_bo *bo; + unsigned int fbo_; + + ConverterFormat informat_; + ConverterFormat outformat_; + ShaderProgram shaderProgram_; + ShaderProgram framebufferProgram_; + std::vector outputBuffers; + PFNEGLCREATEIMAGEKHRPROC eglCreateImageKHR = (PFNEGLCREATEIMAGEKHRPROC)eglGetProcAddress("eglCreateImageKHR"); +}; + +class GlRenderTarget +{ +public: + struct SimpleConverter::DmabufImage texture_; + + /* This is never to be dereferenced. Only serves for comparison */ + const FrameBuffer *buffer_; + + GlRenderTarget(FrameBuffer *buffer, struct SimpleConverter::DmabufImage texture) + : texture_(texture), buffer_(buffer) + { + } +}; + +} /* namespace libcamera */