From patchwork Thu Nov 20 23:33:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bryan O'Donoghue X-Patchwork-Id: 25135 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C6900BD80A for ; Thu, 20 Nov 2025 23:34:16 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 8B12B60AAB; Fri, 21 Nov 2025 00:34:16 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="MsmWUBz8"; dkim-atps=neutral Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 28B5B60AA2 for ; Fri, 21 Nov 2025 00:34:14 +0100 (CET) Received: by mail-wm1-x331.google.com with SMTP id 5b1f17b1804b1-47778b23f64so8188475e9.0 for ; Thu, 20 Nov 2025 15:34:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1763681653; x=1764286453; darn=lists.libcamera.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FL3Kok5kdPBn6nF4Kp6wV9C5qL/tIIY/ODlwHYZlWRQ=; b=MsmWUBz8VD8Z3VVQJqZdAJV/B7HkXKNoM4MmuqdzqAOktsnYusaxiz+gW4nG8omJ3i Qcr4uGIDUUxR3A8GIqwbj4dTf3no2IDWPs37ZN69xaBUQd2+yxBFb60VXQ0CQt7CzGea tI7URK2JN/okL/9PMZSdSMrdt88QyfqMBLf+jJok3HUWLxl/+EYRtedjr3iTaaifS2Bl ezcqvStmPT14k6IFZ9EH3eqTI4MjIyXCBV4TzkrCH/d6o8dxiDrXtEVh/kQFvcb85S+h NAGIF3ljyeym6G03wCgMxbWOokFC/sDfmXZX9oYn4vwxVNzDF8DdGeXcy1kpOMYnb38V ZSng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763681653; x=1764286453; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=FL3Kok5kdPBn6nF4Kp6wV9C5qL/tIIY/ODlwHYZlWRQ=; b=o3eJRZeQQDGA7DazOyAj/mZZHUbeq9qcDuHYanf7B0ZCxlacI3R8tHbxY2CtM16gaE VKaADWtsTfVr/5Iq/YEw0KWUWWGswvOIVLgjI29z2XxTpwfG2DUZow2L3KxFtA9uV08x cOT5q1nTATJ7S5qMUhSchCx3rdXN/x36yuTitbslPGZJQwfJrTuROk+FAu9j7KEx/t+q xKB4u2agWEMfB46x+DjUb9uSXN3GMt9L/NTq9Iu6I2s5yF6MpEE2i0k3am0h/DSrKJUA woqPxlMh6kzptldrGYfAZ3RmLxojJnSMa7/V+Tmf6wCwUVRmJxdhyaFGrobezQ3azg9A stZw== X-Gm-Message-State: AOJu0YwkR8/J4USeuuIcjsL6suswMteixMDHUgEQqMNlPkxn3dDp+tL9 uyQ5toW8kQ8x1uJ+iBYIRFaGgggkBGLsbJ+c4Cmy52JHW2O7jnGqg+LkwaTZB5aI186I9YbVMsQ VvhM7oCo= X-Gm-Gg: ASbGncs2Ck2/muGr7Nwi1TtkTu8LONYXUkoWf7trkaTUkYVkfv/E9vqsor/gYzSM8f5 LxBiNAAajIIXmX3zqEGVw1lh3dzhmE5LSxKNSQsufHFaRoNKxAWOKAZAgDxjAGRjdR7DvmjhWFx WEG6MWo1nQ4rXCBfrUjCgxsMkJ7VT6UPp5suNwLbFiYX0Jg3D8QFF8HV1mUQmTZcP/sRPAyXZlS Xdme4p/tHVsxKfsoyg7dYjtqVUMTRtdXhXfSk+2fZsVeR+sEWS6+NollrItcRg297Nwcc31eTVf Krji3S68H+Kly4wziS7RFtxua6v65jeF8Il38+RYScsIA6ZZYTXnRxg+/BL+TrSQBWp+cxROotf Q2Ik8TDszi/bYybtCQm1RejBZsm+zeijxCcQjeccqUiOnv9d2pT9g2PX+e4nt8n9MS3P6cDGCf9 EbcXXZyPKwBCtInA+XdAC2FqDQcsKuscMcTAVWthoZZfjPe76J3Tj/7i8VZTyvI90wV18= X-Google-Smtp-Source: AGHT+IEC61KkESHvBiVrUJPhFkJjaS8t3UhL7wL6DQVbcsFTIiydJOkKTCzw3OCR2+3/DsE3xxt/cw== X-Received: by 2002:a05:600c:a01:b0:477:55ce:f3c2 with SMTP id 5b1f17b1804b1-477c111d3camr607845e9.14.1763681653321; Thu, 20 Nov 2025 15:34:13 -0800 (PST) Received: from inspiron14p-linux.ht.home (188-141-3-146.dynamic.upc.ie. [188.141.3.146]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-42cb7fa3a81sm7984139f8f.26.2025.11.20.15.34.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Nov 2025 15:34:12 -0800 (PST) From: Bryan O'Donoghue To: libcamera-devel@lists.libcamera.org Cc: pavel@ucw.cz, Bryan O'Donoghue , Robert Mader Subject: [PATCH v4 11/23] libcamera: software_isp: debayer_egl: Add an eGL debayer class Date: Thu, 20 Nov 2025 23:33:35 +0000 Message-ID: <20251120233347.5046-12-bryan.odonoghue@linaro.org> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251120233347.5046-1-bryan.odonoghue@linaro.org> References: <20251120233347.5046-1-bryan.odonoghue@linaro.org> MIME-Version: 1.0 X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a class to run the existing glsl debayer shaders on a GBM surface. Signed-off-by: Robert Mader Co-developed-by: Robert Mader [bod took scaling and buffer size fixes from Robert] Signed-off-by: Bryan O'Donoghue --- src/libcamera/software_isp/debayer_egl.cpp | 668 +++++++++++++++++++++ src/libcamera/software_isp/debayer_egl.h | 177 ++++++ src/libcamera/software_isp/meson.build | 8 + 3 files changed, 853 insertions(+) create mode 100644 src/libcamera/software_isp/debayer_egl.cpp create mode 100644 src/libcamera/software_isp/debayer_egl.h diff --git a/src/libcamera/software_isp/debayer_egl.cpp b/src/libcamera/software_isp/debayer_egl.cpp new file mode 100644 index 000000000..cf40478fa --- /dev/null +++ b/src/libcamera/software_isp/debayer_egl.cpp @@ -0,0 +1,668 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2024, Linaro Ltd. + * + * Authors: + * Bryan O'Donoghue + * + * debayer_cpu.cpp - EGL based debayering class + */ + +#include +#include +#include + +#include + +#include "libcamera/base/utils.h" +#include "libcamera/internal/glsl_shaders.h" +#include "debayer_egl.h" + +namespace libcamera { + +DebayerEGL::DebayerEGL(std::unique_ptr stats, const GlobalConfiguration &configuration) + : Debayer(configuration), stats_(std::move(stats)) +{ + eglImageBayerIn_ = eglImageBayerOut_= eglImageRedLookup_ = eglImageBlueLookup_ = eglImageGreenLookup_ = NULL; +} + +DebayerEGL::~DebayerEGL() +{ + if (eglImageBlueLookup_) + delete eglImageBlueLookup_; + + if (eglImageGreenLookup_) + delete eglImageGreenLookup_; + + if (eglImageRedLookup_) + delete eglImageRedLookup_; + + if (eglImageBayerOut_) + delete eglImageBayerOut_; + + if (eglImageBayerIn_) + delete eglImageBayerIn_; +} + +int DebayerEGL::getInputConfig(PixelFormat inputFormat, DebayerInputConfig &config) +{ + BayerFormat bayerFormat = + BayerFormat::fromPixelFormat(inputFormat); + + if ((bayerFormat.bitDepth == 8 || bayerFormat.bitDepth == 10) && + bayerFormat.packing == BayerFormat::Packing::None && + isStandardBayerOrder(bayerFormat.order)) { + config.bpp = (bayerFormat.bitDepth + 7) & ~7; + config.patternSize.width = 2; + config.patternSize.height = 2; + config.outputFormats = std::vector({ formats::XRGB8888, + formats::ARGB8888, + formats::XBGR8888, + formats::ABGR8888 }); + return 0; + } + + if (bayerFormat.bitDepth == 10 && + bayerFormat.packing == BayerFormat::Packing::CSI2 && + isStandardBayerOrder(bayerFormat.order)) { + config.bpp = 10; + config.patternSize.width = 4; /* 5 bytes per *4* pixels */ + config.patternSize.height = 2; + config.outputFormats = std::vector({ formats::XRGB8888, + formats::ARGB8888, + formats::XBGR8888, + formats::ABGR8888 }); + return 0; + } + + LOG(Debayer, Error) + << "Unsupported input format " << inputFormat.toString(); + + return -EINVAL; +} + +int DebayerEGL::getOutputConfig(PixelFormat outputFormat, DebayerOutputConfig &config) +{ + if (outputFormat == formats::XRGB8888 || outputFormat == formats::ARGB8888 || + outputFormat == formats::XBGR8888 || outputFormat == formats::ABGR8888) { + config.bpp = 32; + return 0; + } + + LOG(Debayer, Error) + << "Unsupported output format " << outputFormat.toString(); + + return -EINVAL; +} + +int DebayerEGL::getShaderVariableLocations(void) +{ + attributeVertex_ = glGetAttribLocation(programId_, "vertexIn"); + attributeTexture_ = glGetAttribLocation(programId_, "textureIn"); + + textureUniformBayerDataIn_ = glGetUniformLocation(programId_, "tex_y"); + textureUniformRedLookupDataIn_ = glGetUniformLocation(programId_, "red_param"); + textureUniformGreenLookupDataIn_ = glGetUniformLocation(programId_, "green_param"); + textureUniformBlueLookupDataIn_ = glGetUniformLocation(programId_, "blue_param"); + ccmUniformDataIn_ = glGetUniformLocation(programId_, "ccm"); + blackLevelUniformDataIn_ = glGetUniformLocation(programId_, "blacklevel"); + gammaUniformDataIn_ = glGetUniformLocation(programId_, "gamma"); + contrastUniformDataIn_ = glGetUniformLocation(programId_, "contrast"); + + textureUniformStep_ = glGetUniformLocation(programId_, "tex_step"); + textureUniformSize_ = glGetUniformLocation(programId_, "tex_size"); + textureUniformStrideFactor_ = glGetUniformLocation(programId_, "stride_factor"); + textureUniformBayerFirstRed_ = glGetUniformLocation(programId_, "tex_bayer_first_red"); + textureUniformProjMatrix_ = glGetUniformLocation(programId_, "proj_matrix"); + + LOG(Debayer, Debug) << "vertexIn " << attributeVertex_ << " textureIn " << attributeTexture_ + << " tex_y " << textureUniformBayerDataIn_ + << " red_param " << textureUniformRedLookupDataIn_ + << " green_param " << textureUniformGreenLookupDataIn_ + << " blue_param " << textureUniformBlueLookupDataIn_ + << " ccm " << ccmUniformDataIn_ + << " blacklevel " << blackLevelUniformDataIn_ + << " gamma " << gammaUniformDataIn_ + << " contrast " << contrastUniformDataIn_ + << " tex_step " << textureUniformStep_ + << " tex_size " << textureUniformSize_ + << " stride_factor " << textureUniformStrideFactor_ + << " tex_bayer_first_red " << textureUniformBayerFirstRed_ + << " proj_matrix " << textureUniformProjMatrix_; + return 0; +} + +int DebayerEGL::initBayerShaders(PixelFormat inputFormat, PixelFormat outputFormat) +{ + std::vector shaderEnv; + unsigned int fragmentShaderDataLen; + unsigned char *fragmentShaderData; + unsigned int vertexShaderDataLen; + unsigned char *vertexShaderData; + GLenum err; + + // Target gles 100 glsl requires "#version x" as first directive in shader + egl_.pushEnv(shaderEnv, "#version 100"); + + // Specify GL_OES_EGL_image_external + egl_.pushEnv(shaderEnv, "#extension GL_OES_EGL_image_external: enable"); + + // Tell shaders how to re-order output taking account of how the + // pixels are actually stored by GBM + switch (outputFormat) { + case formats::ARGB8888: + case formats::XRGB8888: + break; + case formats::ABGR8888: + case formats::XBGR8888: + egl_.pushEnv(shaderEnv, "#define SWAP_BLUE"); + break; + default: + goto invalid_fmt; + } + + // Pixel location parameters + glFormat_ = GL_LUMINANCE; + bytesPerPixel_ = 1; + switch (inputFormat) { + case libcamera::formats::SBGGR8: + case libcamera::formats::SBGGR10_CSI2P: + case libcamera::formats::SBGGR12_CSI2P: + firstRed_x_ = 1.0; + firstRed_y_ = 1.0; + break; + case libcamera::formats::SGBRG8: + case libcamera::formats::SGBRG10_CSI2P: + case libcamera::formats::SGBRG12_CSI2P: + firstRed_x_ = 0.0; + firstRed_y_ = 1.0; + break; + case libcamera::formats::SGRBG8: + case libcamera::formats::SGRBG10_CSI2P: + case libcamera::formats::SGRBG12_CSI2P: + firstRed_x_ = 1.0; + firstRed_y_ = 0.0; + break; + case libcamera::formats::SRGGB8: + case libcamera::formats::SRGGB10_CSI2P: + case libcamera::formats::SRGGB12_CSI2P: + firstRed_x_ = 0.0; + firstRed_y_ = 0.0; + break; + default: + goto invalid_fmt; + break; + }; + + // Shader selection + switch (inputFormat) { + case libcamera::formats::SBGGR8: + case libcamera::formats::SGBRG8: + case libcamera::formats::SGRBG8: + case libcamera::formats::SRGGB8: + fragmentShaderData = bayer_unpacked_frag; + fragmentShaderDataLen = bayer_unpacked_frag_len; + vertexShaderData = bayer_unpacked_vert; + vertexShaderDataLen = bayer_unpacked_vert_len; + break; + case libcamera::formats::SBGGR10_CSI2P: + case libcamera::formats::SGBRG10_CSI2P: + case libcamera::formats::SGRBG10_CSI2P: + case libcamera::formats::SRGGB10_CSI2P: + egl_.pushEnv(shaderEnv, "#define RAW10P"); + if (BayerFormat::fromPixelFormat(inputFormat).packing == BayerFormat::Packing::None) { + fragmentShaderData = bayer_unpacked_frag; + fragmentShaderDataLen = bayer_unpacked_frag_len; + vertexShaderData = bayer_unpacked_vert; + vertexShaderDataLen = bayer_unpacked_vert_len; + glFormat_ = GL_RG; + bytesPerPixel_ = 2; + } else { + fragmentShaderData = bayer_1x_packed_frag; + fragmentShaderDataLen = bayer_1x_packed_frag_len; + vertexShaderData = identity_vert; + vertexShaderDataLen = identity_vert_len; + } + break; + case libcamera::formats::SBGGR12_CSI2P: + case libcamera::formats::SGBRG12_CSI2P: + case libcamera::formats::SGRBG12_CSI2P: + case libcamera::formats::SRGGB12_CSI2P: + egl_.pushEnv(shaderEnv, "#define RAW12P"); + if (BayerFormat::fromPixelFormat(inputFormat).packing == BayerFormat::Packing::None) { + fragmentShaderData = bayer_unpacked_frag; + fragmentShaderDataLen = bayer_unpacked_frag_len; + vertexShaderData = bayer_unpacked_vert; + vertexShaderDataLen = bayer_unpacked_vert_len; + glFormat_ = GL_RG; + bytesPerPixel_ = 2; + } else { + fragmentShaderData = bayer_1x_packed_frag; + fragmentShaderDataLen = bayer_1x_packed_frag_len; + vertexShaderData = identity_vert; + vertexShaderDataLen = identity_vert_len; + } + break; + default: + goto invalid_fmt; + break; + }; + + if (ccmEnabled_) { + // Run the CCM if available + egl_.pushEnv(shaderEnv, "#define APPLY_CCM_PARAMETERS"); + } else { + // Flag to shaders that we have parameter gain tables + egl_.pushEnv(shaderEnv, "#define APPLY_RGB_PARAMETERS"); + } + + if (egl_.compileVertexShader(vertexShaderId_, vertexShaderData, vertexShaderDataLen, shaderEnv)) + goto compile_fail; + + if (egl_.compileFragmentShader(fragmentShaderId_, fragmentShaderData, fragmentShaderDataLen, shaderEnv)) + goto compile_fail; + + if (egl_.linkProgram(programId_, vertexShaderId_, fragmentShaderId_)) + goto link_fail; + + egl_.dumpShaderSource(vertexShaderId_); + egl_.dumpShaderSource(fragmentShaderId_); + + /* Ensure we set the programId_ */ + egl_.useProgram(programId_); + err = glGetError(); + if (err != GL_NO_ERROR) + goto program_fail; + + if (getShaderVariableLocations()) + goto parameters_fail; + + return 0; + +parameters_fail: + LOG(Debayer, Error) << "Program parameters fail"; + return -ENODEV; + +program_fail: + LOG(Debayer, Error) << "Use program error " << err; + return -ENODEV; + +link_fail: + LOG(Debayer, Error) << "Linking program fail"; + return -ENODEV; + +compile_fail: + LOG(Debayer, Error) << "Compile debayer shaders fail"; + return -ENODEV; + +invalid_fmt: + LOG(Debayer, Error) << "Unsupported input output format combination"; + return -EINVAL; +} + +int DebayerEGL::configure(const StreamConfiguration &inputCfg, + const std::vector> &outputCfgs, + bool ccmEnabled) +{ + GLint maxTextureImageUnits; + + if (getInputConfig(inputCfg.pixelFormat, inputConfig_) != 0) + return -EINVAL; + + if (stats_->configure(inputCfg) != 0) + return -EINVAL; + + const Size &stats_pattern_size = stats_->patternSize(); + if (inputConfig_.patternSize.width != stats_pattern_size.width || + inputConfig_.patternSize.height != stats_pattern_size.height) { + LOG(Debayer, Error) + << "mismatching stats and debayer pattern sizes for " + << inputCfg.pixelFormat.toString(); + return -EINVAL; + } + + inputConfig_.stride = inputCfg.stride; + width_ = inputCfg.size.width; + height_ = inputCfg.size.height; + ccmEnabled_ = ccmEnabled; + + if (outputCfgs.size() != 1) { + LOG(Debayer, Error) + << "Unsupported number of output streams: " + << outputCfgs.size(); + return -EINVAL; + } + + if (gbmSurface_.createDevice()) + return -ENODEV; + + if (egl_.initEGLContext(&gbmSurface_)) + return -ENODEV; + + glGetIntegerv(GL_MAX_TEXTURE_IMAGE_UNITS, &maxTextureImageUnits); + + LOG(Debayer, Debug) << "Available fragment shader texture units " << maxTextureImageUnits; + + if (!ccmEnabled && maxTextureImageUnits < DEBAYER_EGL_MIN_SIMPLE_RGB_GAIN_TEXTURE_UNITS) { + LOG(Debayer, Error) << "Fragment shader texture unit count " << maxTextureImageUnits + << " required minimum for RGB gain table lookup " << DEBAYER_EGL_MIN_SIMPLE_RGB_GAIN_TEXTURE_UNITS + << " try using an identity CCM "; + return -ENODEV; + } + + StreamConfiguration &outputCfg = outputCfgs[0]; + SizeRange outSizeRange = sizes(inputCfg.pixelFormat, inputCfg.size); + std::tie(outputConfig_.stride, outputConfig_.frameSize) = + strideAndFrameSize(outputCfg.pixelFormat, outputCfg.size); + + if (!outSizeRange.contains(outputCfg.size) || outputConfig_.stride != outputCfg.stride) { + LOG(Debayer, Error) + << "Invalid output size/stride: " + << "\n " << outputCfg.size << " (" << outSizeRange << ")" + << "\n " << outputCfg.stride << " (" << outputConfig_.stride << ")"; + return -EINVAL; + } + + window_.x = ((inputCfg.size.width - outputCfg.size.width) / 2) & + ~(inputConfig_.patternSize.width - 1); + window_.y = ((inputCfg.size.height - outputCfg.size.height) / 2) & + ~(inputConfig_.patternSize.height - 1); + window_.width = outputCfg.size.width; + window_.height = outputCfg.size.height; + + /* + * Don't pass x,y from window_ since process() already adjusts for it. + * But crop the window to 2/3 of its width and height for speedup. + */ + stats_->setWindow(Rectangle(window_.size())); + + // Raw bayer input as texture + eglImageBayerIn_ = new eGLImage(width_, height_, 32, inputCfg.stride, GL_TEXTURE0, 0); + if (!eglImageBayerIn_) + return -ENOMEM; + + // Only do the RGB lookup table textures if CCM is disabled + if (!ccmEnabled_) { + + /// RGB correction tables as 2d textures + // eGL doesn't support glTexImage1D so we do a little hack with 2D to compensate + eglImageRedLookup_ = new eGLImage(DebayerParams::kRGBLookupSize, 1, 32, GL_TEXTURE1, 1); + if (!eglImageRedLookup_) + return -ENOMEM; + + eglImageGreenLookup_ = new eGLImage(DebayerParams::kRGBLookupSize, 1, 32, GL_TEXTURE2, 2); + if (!eglImageGreenLookup_) + return -ENOMEM; + + eglImageBlueLookup_ = new eGLImage(DebayerParams::kRGBLookupSize, 1, 32, GL_TEXTURE3, 3); + if (!eglImageBlueLookup_) + return -ENOMEM; + } + + eglImageBayerOut_ = new eGLImage(outputCfg.size.width, outputCfg.size.height, 32, outputCfg.stride, GL_TEXTURE4, 4); + if (!eglImageBayerOut_) + return -ENOMEM; + + if (initBayerShaders(inputCfg.pixelFormat, outputCfg.pixelFormat)) + return -EINVAL; + + return 0; +} + +Size DebayerEGL::patternSize(PixelFormat inputFormat) +{ + DebayerEGL::DebayerInputConfig config; + + if (getInputConfig(inputFormat, config) != 0) + return {}; + + return config.patternSize; +} + +std::vector DebayerEGL::formats(PixelFormat inputFormat) +{ + DebayerEGL::DebayerInputConfig config; + + if (getInputConfig(inputFormat, config) != 0) + return std::vector(); + + return config.outputFormats; +} + +std::tuple +DebayerEGL::strideAndFrameSize(const PixelFormat &outputFormat, const Size &size) +{ + DebayerEGL::DebayerOutputConfig config; + + if (getOutputConfig(outputFormat, config) != 0) + return std::make_tuple(0, 0); + + /* Align stride to 256 bytes as a generic GPU memory access alignment */ + unsigned int stride = libcamera::utils::alignUp(size.width * config.bpp / 8, 256); + + return std::make_tuple(stride, stride * size.height); +} + +void DebayerEGL::setShaderVariableValues(DebayerParams ¶ms) +{ + /* + * Raw Bayer 8-bit, and packed raw Bayer 10-bit/12-bit formats + * are stored in a GL_LUMINANCE texture. The texture width is + * equal to the stride. + */ + GLfloat firstRed[] = { firstRed_x_, firstRed_y_ }; + GLfloat imgSize[] = { (GLfloat)width_, + (GLfloat)height_ }; + GLfloat Step[] = { static_cast(bytesPerPixel_) / (inputConfig_.stride - 1), + 1.0f / (height_ - 1) }; + GLfloat Stride = 1.0f; + GLfloat scaleX = (GLfloat)window_.width / width_; + GLfloat scaleY = (GLfloat)window_.height / height_; + GLfloat transX = -(1.0f - scaleX); + GLfloat transY = -(1.0f - scaleY); + GLfloat scale = std::max(scaleX, scaleY); + GLfloat projMatrix[] = { + scale, 0, 0, 0, + 0, scale, 0, 0, + 0, 0, 1, 0, + transX, transY, 0, 1 + }; + + // vertexIn - bayer_8.vert + glEnableVertexAttribArray(attributeVertex_); + glVertexAttribPointer(attributeVertex_, 2, GL_FLOAT, GL_TRUE, + 2 * sizeof(GLfloat), vcoordinates); + + // textureIn - bayer_8.vert + glEnableVertexAttribArray(attributeTexture_); + glVertexAttribPointer(attributeTexture_, 2, GL_FLOAT, GL_TRUE, + 2 * sizeof(GLfloat), tcoordinates); + + // Set the sampler2D to the respective texture unit for each texutre + // To simultaneously sample multiple textures we need to use multiple + // texture units + glUniform1i(textureUniformBayerDataIn_, eglImageBayerIn_->texture_unit_uniform_id_); + + // These values are: + // firstRed = tex_bayer_first_red - bayer_8.vert + // imgSize = tex_size - bayer_8.vert + // step = tex_step - bayer_8.vert + // Stride = stride_factor identity.vert + // textureUniformProjMatri = No scaling + glUniform2fv(textureUniformBayerFirstRed_, 1, firstRed); + glUniform2fv(textureUniformSize_, 1, imgSize); + glUniform2fv(textureUniformStep_, 1, Step); + glUniform1f(textureUniformStrideFactor_, Stride); + glUniformMatrix4fv(textureUniformProjMatrix_, 1, GL_FALSE, projMatrix); + + LOG(Debayer, Debug) << "vertexIn " << attributeVertex_ << " textureIn " << attributeTexture_ + << " tex_y " << textureUniformBayerDataIn_ + << " red_param " << textureUniformRedLookupDataIn_ + << " green_param " << textureUniformGreenLookupDataIn_ + << " blue_param " << textureUniformBlueLookupDataIn_ + << " tex_step " << textureUniformStep_ + << " tex_size " << textureUniformSize_ + << " stride_factor " << textureUniformStrideFactor_ + << " tex_bayer_first_red " << textureUniformBayerFirstRed_; + + LOG (Debayer, Debug) << "textureUniformY_ = 0 " + << " firstRed.x " << firstRed[0] + << " firstRed.y " << firstRed[1] + << " textureUniformSize_.width " << imgSize[0] + << " textureUniformSize_.height " << imgSize[1] + << " textureUniformStep_.x " << Step[0] + << " textureUniformStep_.y " << Step[1] + << " textureUniformStrideFactor_ " << Stride + << " textureUniformProjMatrix_ " << textureUniformProjMatrix_; + + if (!ccmEnabled_) { + glUniform1i(textureUniformRedLookupDataIn_, eglImageRedLookup_->texture_unit_uniform_id_); + glUniform1i(textureUniformGreenLookupDataIn_, eglImageGreenLookup_->texture_unit_uniform_id_); + glUniform1i(textureUniformBlueLookupDataIn_, eglImageBlueLookup_->texture_unit_uniform_id_); + + egl_.createTexture2D(eglImageRedLookup_, GL_LUMINANCE, DebayerParams::kRGBLookupSize, 1, ¶ms.red); + egl_.createTexture2D(eglImageGreenLookup_, GL_LUMINANCE, DebayerParams::kRGBLookupSize, 1, ¶ms.green); + egl_.createTexture2D(eglImageBlueLookup_, GL_LUMINANCE, DebayerParams::kRGBLookupSize, 1, ¶ms.blue); + + LOG (Debayer, Debug) << "textureUniformRedLookupDataIn_ " << textureUniformRedLookupDataIn_ + << " textureUniformGreenLookupDataIn_ " << textureUniformGreenLookupDataIn_ + << " textureUniformBlueLookupDataIn_ " << textureUniformBlueLookupDataIn_; + } else { + GLfloat ccm[9] = { + params.ccm[0][0], params.ccm[0][1], params.ccm[0][2], + params.ccm[1][0], params.ccm[1][1], params.ccm[1][2], + params.ccm[2][0], params.ccm[2][1], params.ccm[2][2], + }; + glUniformMatrix3fv(ccmUniformDataIn_, 1, GL_FALSE, ccm); + LOG (Debayer, Debug) << " ccmUniformDataIn_ " << ccmUniformDataIn_ << " data " << params.ccm; + } + + /* + * 0 = Red, 1 = Green, 2 = Blue + */ + glUniform3f(blackLevelUniformDataIn_, params.blackLevel[0], params.blackLevel[1], params.blackLevel[2]); + LOG (Debayer, Debug) << " blackLevelUniformDataIn_ " << blackLevelUniformDataIn_ << " data " << params.blackLevel; + + /* + * Gamma + */ + glUniform1f(gammaUniformDataIn_, params.gamma); + LOG (Debayer, Debug) << " gammaUniformDataIn_ " << gammaUniformDataIn_ << " data " << params.gamma; + + /* + * Contrast + */ + glUniform1f(contrastUniformDataIn_, params.contrast); + LOG (Debayer, Debug) << " contrastUniformDataIn_ " << contrastUniformDataIn_ << " data " << params.contrast; + + return; +} + +int DebayerEGL::debayerGPU(MappedFrameBuffer &in, int out_fd, DebayerParams ¶ms) +{ + /* eGL context switch */ + egl_.makeCurrent(); + + /* Greate a standard texture input */ + egl_.createTexture2D(eglImageBayerIn_, glFormat_, inputConfig_.stride / bytesPerPixel_, height_, in.planes()[0].data()); + + /* Generate the output render framebuffer as render to texture */ + egl_.createOutputDMABufTexture2D(eglImageBayerOut_, out_fd); + + setShaderVariableValues(params); + glViewport(0, 0, width_, height_); + glClear(GL_COLOR_BUFFER_BIT); + glDrawArrays(GL_TRIANGLE_FAN, 0, DEBAYER_OPENGL_COORDS); + + GLenum err = glGetError(); + if (err != GL_NO_ERROR) { + LOG(eGL, Error) << "Drawing scene fail " << err; + return -ENODEV; + } else { + egl_.syncOutput(); + } + + return 0; +} + +void DebayerEGL::process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params) +{ + bench_.startFrame(); + + std::vector dmaSyncers; + + dmaSyncBegin(dmaSyncers, input, nullptr); + + setParams(params); + + /* Copy metadata from the input buffer */ + FrameMetadata &metadata = output->_d()->metadata(); + metadata.status = input->metadata().status; + metadata.sequence = input->metadata().sequence; + metadata.timestamp = input->metadata().timestamp; + + MappedFrameBuffer in(input, MappedFrameBuffer::MapFlag::Read); + if (!in.isValid()) { + LOG(Debayer, Error) << "mmap-ing buffer(s) failed"; + goto error; + } + + if (debayerGPU(in, output->planes()[0].fd.get(), params)) { + LOG(Debayer, Error) << "debayerGPU failed"; + goto error; + } + + bench_.finishFrame(); + + metadata.planes()[0].bytesused = output->planes()[0].length; + + /* Calculate stats for the whole frame */ + stats_->processFrame(frame, 0, input); + dmaSyncers.clear(); + + outputBufferReady.emit(output); + inputBufferReady.emit(input); + + return; + +error: + bench_.finishFrame(); + metadata.status = FrameMetadata::FrameError; + return; +} + +void DebayerEGL::stop() +{ + egl_.cleanUp(); +} + +SizeRange DebayerEGL::sizes(PixelFormat inputFormat, const Size &inputSize) +{ + Size patternSize = this->patternSize(inputFormat); + unsigned int borderHeight = patternSize.height; + + if (patternSize.isNull()) + return {}; + + /* No need for top/bottom border with a pattern height of 2 */ + if (patternSize.height == 2) + borderHeight = 0; + + /* + * For debayer interpolation a border is kept around the entire image + * and the minimum output size is pattern-height x pattern-width. + */ + if (inputSize.width < (3 * patternSize.width) || + inputSize.height < (2 * borderHeight + patternSize.height)) { + LOG(Debayer, Warning) + << "Input format size too small: " << inputSize.toString(); + return {}; + } + + return SizeRange(Size(patternSize.width, patternSize.height), + Size((inputSize.width - 2 * patternSize.width) & ~(patternSize.width - 1), + (inputSize.height - 2 * borderHeight) & ~(patternSize.height - 1)), + patternSize.width, patternSize.height); +} + +} /* namespace libcamera */ diff --git a/src/libcamera/software_isp/debayer_egl.h b/src/libcamera/software_isp/debayer_egl.h new file mode 100644 index 000000000..33067c463 --- /dev/null +++ b/src/libcamera/software_isp/debayer_egl.h @@ -0,0 +1,177 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2025, Bryan O'Donoghue. + * + * Authors: + * Bryan O'Donoghue + * + * debayer_opengl.h - EGL debayer header + */ + +#pragma once + +#include +#include +#include + +#define GL_GLEXT_PROTOTYPES +#define EGL_EGLEXT_PROTOTYPES +#include +#include +#include + +#include + +#include "debayer.h" + +#include "libcamera/internal/bayer_format.h" +#include "libcamera/internal/egl.h" +#include "libcamera/internal/framebuffer.h" +#include "libcamera/internal/mapped_framebuffer.h" +#include "libcamera/internal/software_isp/benchmark.h" +#include "libcamera/internal/software_isp/swstats_cpu.h" + +namespace libcamera { + +#define DEBAYER_EGL_MIN_SIMPLE_RGB_GAIN_TEXTURE_UNITS 4 +#define DEBAYER_OPENGL_COORDS 4 + +/** + * \class DebayerEGL + * \brief Class for debayering using an EGL Shader + * + * Implements an EGL shader based debayering solution. + */ +class DebayerEGL : public Debayer +{ +public: + /** + * \fn DebayerEGL::DebayerEGL(std::unique_ptr stats, const GlobalConfiguration &configuration) + * \brief Construct a DebayerEGL object + * \param[in] stats Statistics processing object + * \param[in] configuration Global configuration reference + */ + DebayerEGL(std::unique_ptr stats, const GlobalConfiguration &configuration); + ~DebayerEGL(); + + /* + * Setup the Debayer object according to the passed in parameters. + * Return 0 on success, a negative errno value on failure + * (unsupported parameters). + */ + int configure(const StreamConfiguration &inputCfg, + const std::vector> &outputCfgs, + bool ccmEnabled); + + /* + * Get width and height at which the bayer-pattern repeats. + * Return pattern-size or an empty Size for an unsupported inputFormat. + */ + Size patternSize(PixelFormat inputFormat); + + std::vector formats(PixelFormat input); + std::tuple strideAndFrameSize(const PixelFormat &outputFormat, const Size &size); + + void process(uint32_t frame, FrameBuffer *input, FrameBuffer *output, DebayerParams params); + void stop(); + + const SharedFD &getStatsFD() { return stats_->getStatsFD(); } + + /** + * \brief Get the output frame size. + * + * \return The output frame size. + */ + unsigned int frameSize() { return outputConfig_.frameSize; } + + SizeRange sizes(PixelFormat inputFormat, const Size &inputSize); + +private: + static int getInputConfig(PixelFormat inputFormat, DebayerInputConfig &config); + static int getOutputConfig(PixelFormat outputFormat, DebayerOutputConfig &config); + int setupStandardBayerOrder(BayerFormat::Order order); + void pushEnv(std::vector &shaderEnv, const char *str); + int initBayerShaders(PixelFormat inputFormat, PixelFormat outputFormat); + int initEGLContext(); + int generateTextures(); + int compileShaderProgram(GLuint &shaderId, GLenum shaderType, + unsigned char *shaderData, int shaderDataLen, + std::vector shaderEnv); + int linkShaderProgram(void); + int getShaderVariableLocations(); + void setShaderVariableValues(DebayerParams ¶ms); + void configureTexture(GLuint &texture); + int debayerGPU(MappedFrameBuffer &in, int out_fd, DebayerParams ¶ms); + + // Shader program identifiers + GLuint vertexShaderId_; + GLuint fragmentShaderId_; + GLuint programId_; + enum { + BAYER_INPUT_INDEX = 0, + BAYER_OUTPUT_INDEX, + BAYER_BUF_NUM, + }; + + // Pointer to object representing input texture + eGLImage *eglImageBayerIn_; + eGLImage *eglImageBayerOut_; + + eGLImage *eglImageRedLookup_; + eGLImage *eglImageGreenLookup_; + eGLImage *eglImageBlueLookup_; + + // Shader parameters + float firstRed_x_; + float firstRed_y_; + GLint attributeVertex_; + GLint attributeTexture_; + GLint textureUniformStep_; + GLint textureUniformSize_; + GLint textureUniformStrideFactor_; + GLint textureUniformBayerFirstRed_; + GLint textureUniformProjMatrix_; + + GLint textureUniformBayerDataIn_; + + // These textures will either point to simple RGB gains or to CCM lookup tables + GLint textureUniformRedLookupDataIn_; + GLint textureUniformGreenLookupDataIn_; + GLint textureUniformBlueLookupDataIn_; + + // Represent per-frame CCM as a uniform vector of floats 3 x 3 + GLint ccmUniformDataIn_; + bool ccmEnabled_; + + // Black Level compensation + GLint blackLevelUniformDataIn_; + + // Gamma + GLint gammaUniformDataIn_; + + // Contrast + GLint contrastUniformDataIn_; + + Rectangle window_; + std::unique_ptr stats_; + eGL egl_; + GBM gbmSurface_; + uint32_t width_; + uint32_t height_; + GLint glFormat_; + unsigned int bytesPerPixel_; + GLfloat vcoordinates[DEBAYER_OPENGL_COORDS][2] = { + { -1.0f, -1.0f }, + { -1.0f, +1.0f }, + { +1.0f, +1.0f }, + { +1.0f, -1.0f }, + }; + GLfloat tcoordinates[DEBAYER_OPENGL_COORDS][2] = { + { 0.0f, 0.0f }, + { 0.0f, 1.0f }, + { 1.0f, 1.0f }, + { 1.0f, 0.0f }, + }; +}; + +} /* namespace libcamera */ diff --git a/src/libcamera/software_isp/meson.build b/src/libcamera/software_isp/meson.build index 59fa5f02a..c61ac7d59 100644 --- a/src/libcamera/software_isp/meson.build +++ b/src/libcamera/software_isp/meson.build @@ -2,6 +2,7 @@ softisp_enabled = pipelines.contains('simple') summary({'SoftISP support' : softisp_enabled}, section : 'Configuration') +summary({'SoftISP GPU acceleration' : gles_headless_enabled}, section : 'Configuration') if not softisp_enabled subdir_done() @@ -14,3 +15,10 @@ libcamera_internal_sources += files([ 'software_isp.cpp', 'swstats_cpu.cpp', ]) + +if softisp_enabled and gles_headless_enabled + config_h.set('HAVE_DEBAYER_EGL', 1) + libcamera_internal_sources += files([ + 'debayer_egl.cpp', + ]) +endif