From patchwork Wed Jan 13 13:08:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 10853 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1B179BD80C for ; Wed, 13 Jan 2021 13:08:58 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DD4D8680E1; Wed, 13 Jan 2021 14:08:57 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="FPpPfBNc"; dkim-atps=neutral Received: from mail-lf1-x129.google.com (mail-lf1-x129.google.com [IPv6:2a00:1450:4864:20::129]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4E8616054E for ; Wed, 13 Jan 2021 14:08:56 +0100 (CET) Received: by mail-lf1-x129.google.com with SMTP id m12so2645359lfo.7 for ; Wed, 13 Jan 2021 05:08:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BpmDhIQhpX7hcjrbtD7qJnZrzb7EHJzsq72n+mHUsG4=; b=FPpPfBNcMp640yavsQquihNj95Fn2654plROBHq9a/Y5rlIyKX++hAIsxNY0FHNRGY okPw0KFfllZIc1QaFsY+gUA8XI43pLfHx0NF/fUaXoD0EYshLDNpVs+VUnmEeosDcnIl 5Bnl9A80PyQT4KDYqn2xWdpJXP104Y/2KUfTMukcc4yaaGQ2b/FQa/ys3YkML2WB4mYO gTpHaP575Ge/vjlAHTFJ5is3VuiQ2CTCEBfxuNC2I8cZA9PemL4E57/KVPnonilsnczc Oeu1muA9XwczmxVld6I3MH2RhGcA8srhXYmlmeVavqJYz6Dgb5g+iMU9WQE50uxRombF h3yA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BpmDhIQhpX7hcjrbtD7qJnZrzb7EHJzsq72n+mHUsG4=; b=UlEEY1ZXJ/yUS/6qonGG7m4i+ATVd9MIjWV+tXRbG5Qr5QwM/s0IleJfw4tffTSbV2 jgsGh3bptTqFoosB4db2yWyVvhoD9sh9RHOGHR1qzKsCj6CMjrs9VzMS9/guVxPzJqnv FEUcF0nIz4xmd8f8GhuO5ECUtorlQ534d8LMt9EgD8HeL1FOz2HbITO7I3fEIDLSvQQ6 lMnz1TxU/ueJQay8sDulllJrNWExIRYHIdiyRwJWGPgwI6kM+k/lAJtNrpWNjDtYR3iL dZnRw+rqh0KLf6OBhToLns6o1tb6laiDTGL/BrelffbFzH56bcaMsJP3G/u1tikU8COy puOw== X-Gm-Message-State: AOAM532j/8NMYW4XVDv6CIEvQ9LZ9E9LFD0XpgKHcn/fbEr98/Dm6lGP YV4MJg6n/MminCxpmKe4vTCFADQMLawFjQ== X-Google-Smtp-Source: ABdhPJw4mI5WTwQJDhNuknvCzos7lGYgAr9r8NTBCVGVWkBZS/7afBiaKf9l6n2dFk3lr9KhrBVQEw== X-Received: by 2002:a19:4f51:: with SMTP id a17mr915350lfk.522.1610543335067; Wed, 13 Jan 2021 05:08:55 -0800 (PST) Received: from localhost.localdomain ([85.249.43.69]) by smtp.googlemail.com with ESMTPSA id d2sm179477ljg.39.2021.01.13.05.08.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Jan 2021 05:08:54 -0800 (PST) From: Andrey Konovalov To: libcamera-devel@lists.libcamera.org, laurent.pinchart@ideasonboard.com Date: Wed, 13 Jan 2021 16:08:30 +0300 Message-Id: <20210113130832.15549-3-andrey.konovalov@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210113130832.15549-1-andrey.konovalov@linaro.org> References: <20210113130832.15549-1-andrey.konovalov@linaro.org> Subject: [libcamera-devel] [PATCH 2/4] qcam: viewfinder_gl: Add shader to render packed RAW10 formats X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: morgan@casual-effects.com MIME-Version: 1.0 Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The shader supports all 4 packed RAW10 variants. Simple bi-linear filtering is implemented. The 2 LS bits of the 10-bit colour values are dropped as the RGBA format we convert into has only 8 bits per colour. The texture coordinates passed to the fragment shader are ajusted to point to the nearest pixel in the image. This prevents artifacts when the image is scaled from the frame resolution to the window size. Signed-off-by: Andrey Konovalov --- src/qcam/assets/shader/bayer_1x_packed.frag | 174 ++++++++++++++++++++ src/qcam/assets/shader/shaders.qrc | 1 + src/qcam/viewfinder_gl.cpp | 77 ++++++++- src/qcam/viewfinder_gl.h | 6 + 4 files changed, 256 insertions(+), 2 deletions(-) create mode 100644 src/qcam/assets/shader/bayer_1x_packed.frag diff --git a/src/qcam/assets/shader/bayer_1x_packed.frag b/src/qcam/assets/shader/bayer_1x_packed.frag new file mode 100644 index 00000000..0a87c6db --- /dev/null +++ b/src/qcam/assets/shader/bayer_1x_packed.frag @@ -0,0 +1,174 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Based on the code from http://jgt.akpeters.com/papers/McGuire08/ + * + * Efficient, High-Quality Bayer Demosaic Filtering on GPUs + * + * Morgan McGuire + * + * This paper appears in issue Volume 13, Number 4. + * --------------------------------------------------------- + * Copyright (c) 2008, Morgan McGuire. All rights reserved. + * + * + * Modified by Linaro Ltd for 10/12-bit packed vs 8-bit raw Bayer format, + * and for simpler demosaic algorithm. + * Copyright (C) 2020, Linaro + * + * bayer_1x_packed.frag - Fragment shader code for raw Bayer 10-bit and 12-bit + * packed formats + */ + +#ifdef GL_ES +precision mediump float; +#endif + +varying vec2 textureOut; + +/* the texture size: tex_size.xy is in bytes, tex_size.zw is in pixels */ +uniform vec4 tex_size; +uniform vec2 tex_step; +uniform vec2 tex_bayer_first_red; + +uniform sampler2D tex_raw; + +void main(void) +{ + vec3 rgb; + + /* + * center.xy holds the coordinates of the pixel being sampled + * on the [0, 1] range. + * center.zw holds the coordinates of the pixel being sampled + * on the [0, width/height-1] range. + */ + vec4 center; + /* + * x-positions of the adjacent pixels on the [0, 1] range. + */ + vec2 xcoords; + /* + * y-positions of the adjacent pixels on the [0, 1] range. + */ + vec2 ycoords; + + /* + * The coordinates passed to the shader in textureOut may point + * to a place in between the pixels if the viewfinder window is scaled + * from the original captured frame size. Align them to the nearest + * pixel. + */ + center.zw = floor(textureOut * tex_size.zw); + center.y = center.w; + /* + * Add a small number (a few mantissa's LSBs) to avoid float + * representation issues. Maybe paranoic. + */ + center.x = BPP_X * center.z + 0.02; + + const float threshold_l = 0.127 /* fract(BPP_X) * 0.5 + 0.02 */; + const float threshold_h = 0.625 /* 1.0 - fract(BPP_X) * 1.5 */; + + float fract_x = fract(center.x); + /* + * The below floor() call ensures that center.x points + * at one of the bytes representing the 8 higher bits of + * the pixel value, not at the byte containing the LS bits + * of the group of the pixels. + */ + center.x = floor(center.x); + center.xy *= tex_step; + + xcoords = center.x + vec2(-tex_step.x, tex_step.x); + ycoords = center.y + vec2(-tex_step.y, tex_step.y); + /* + * If xcoords[0] points at the byte containing the LS bits + * of the previous group of the pixels, move xcoords[0] one + * byte back. + */ + xcoords[0] += (fract_x < threshold_l) ? -tex_step.x : 0.0; + /* + * If xcoords[1] points at the byte containing the LS bits + * of the current group of the pixels, move xcoords[1] one + * byte forward. + */ + xcoords[1] += (fract_x > threshold_h) ? tex_step.x : 0.0; + + vec2 alternate = mod(center.zw + tex_bayer_first_red, 2.0); + bool even_col = alternate.x < 1.0; + bool even_raw = alternate.y < 1.0; + + /* + * We need to sample the central pixel and the ones with offset + * of -1 to +1 pixel in both X and Y directions. Let's name these + * pixels as below, where C is the central pixel: + * +----+----+----+----+ + * | \ x| | | | + * |y \ | -1 | 0 | +1 | + * +----+----+----+----+ + * | +1 | D2 | A1 | D3 | + * +----+----+----+----+ + * | 0 | B0 | C | B1 | + * +----+----+----+----+ + * | -1 | D0 | A0 | D1 | + * +----+----+----+----+ + * In the below equations (0,-1).r means "r component of the texel + * shifted by -tex_step.y from the center.xy one" etc. + * In the even raw / even column (EE) case the colour values are: + * R = C = (0,0).r, + * G = (A0 + A1 + B0 + B1) / 4.0 = + * ( (0,-1).r + (0,1).r + (-1,0).r + (1,0).r ) / 4.0, + * B = (D0 + D1 + D2 + D3) / 4.0 = + * ( (-1,-1).r + (1,-1).r + (-1,1).r + (1,1).r ) / 4.0 + * For even raw / odd column (EO): + * R = (B0 + B1) / 2.0 = ( (-1,0).r + (1,0).r ) / 2.0, + * G = C = (0,0).r, + * B = (A0 + A1) / 2.0 = ( (0,-1).r + (0,1).r ) / 2.0 + * For odd raw / even column (OE): + * R = (A0 + A1) / 2.0 = ( (0,-1).r + (0,1).r ) / 2.0, + * G = C = (0,0).r, + * B = (B0 + B1) / 2.0 = ( (-1,0).r + (1,0).r ) / 2.0 + * For odd raw / odd column (OO): + * R = (D0 + D1 + D2 + D3) / 4.0 = + * ( (-1,-1).r + (1,-1).r + (-1,1).r + (1,1).r ) / 4.0, + * G = (A0 + A1 + B0 + B1) / 4.0 = + * ( (0,-1).r + (0,1).r + (-1,0).r + (1,0).r ) / 4.0, + * B = C = (0,0).r + */ + + /* + * Fetch the values and precalculate the terms: + * patterns.x = (A0 + A1) / 2.0 + * patterns.y = (B0 + B1) / 2.0 + * patterns.z = (A0 + A1 + B0 + B1) / 4.0 + * patterns.w = (D0 + D1 + D2 + D3) / 4.0 + */ + #define fetch(x, y) texture2D(tex_raw, vec2(x, y)).r + + float C = texture2D(tex_raw, center.xy).r; + vec4 patterns = vec4( + fetch(center.x, ycoords[0]), /* A0: (0,-1) */ + fetch(xcoords[0], center.y), /* B0: (-1,0) */ + fetch(xcoords[0], ycoords[0]), /* D0: (-1,-1) */ + fetch(xcoords[1], ycoords[0])); /* D1: (1,-1) */ + vec4 temp = vec4( + fetch(center.x, ycoords[1]), /* A1: (0,1) */ + fetch(xcoords[1], center.y), /* B1: (1,0) */ + fetch(xcoords[1], ycoords[1]), /* D3: (1,1) */ + fetch(xcoords[0], ycoords[1])); /* D2: (-1,1) */ + patterns = (patterns + temp) * 0.5; + /* .x = (A0 + A1) / 2.0, .y = (B0 + B1) / 2.0 */ + /* .z = (D0 + D3) / 2.0, .w = (D1 + D2) / 2.0 */ + patterns.w = (patterns.z + patterns.w) * 0.5; + patterns.z = (patterns.x + patterns.y) * 0.5; + + rgb = (even_col) ? + ((even_raw) ? + vec3(C, patterns.zw) : + vec3(patterns.x, C, patterns.y)) : + ((even_raw) ? + vec3(patterns.y, C, patterns.x) : + vec3(patterns.wz, C)); + + gl_FragColor = vec4(rgb, 1.0); +} diff --git a/src/qcam/assets/shader/shaders.qrc b/src/qcam/assets/shader/shaders.qrc index 8a8f9de1..d76d65c5 100644 --- a/src/qcam/assets/shader/shaders.qrc +++ b/src/qcam/assets/shader/shaders.qrc @@ -5,6 +5,7 @@ YUV_2_planes.frag YUV_3_planes.frag YUV_packed.frag + bayer_1x_packed.frag identity.vert diff --git a/src/qcam/viewfinder_gl.cpp b/src/qcam/viewfinder_gl.cpp index ff719418..b324d77f 100644 --- a/src/qcam/viewfinder_gl.cpp +++ b/src/qcam/viewfinder_gl.cpp @@ -36,6 +36,11 @@ static const QList supportedFormats{ libcamera::formats::RGBA8888, libcamera::formats::BGR888, libcamera::formats::RGB888, + /* Raw Bayer 10-bit packed */ + libcamera::formats::SBGGR10_CSI2P, + libcamera::formats::SGBRG10_CSI2P, + libcamera::formats::SGRBG10_CSI2P, + libcamera::formats::SRGGB10_CSI2P, }; ViewFinderGL::ViewFinderGL(QWidget *parent) @@ -114,6 +119,9 @@ bool ViewFinderGL::selectFormat(const libcamera::PixelFormat &format) { bool ret = true; + /* Set min/mag filters to GL_LINEAR by default. */ + textureMinMagFilters_ = GL_LINEAR; + fragmentShaderDefines_.clear(); switch (format) { @@ -203,6 +211,34 @@ bool ViewFinderGL::selectFormat(const libcamera::PixelFormat &format) fragmentShaderDefines_.append("#define RGB_PATTERN bgr"); fragmentShaderFile_ = ":RGB.frag"; break; + case libcamera::formats::SBGGR10_CSI2P: + firstRed_.setX(1.0); + firstRed_.setY(1.0); + fragmentShaderDefines_.append("#define BPP_X 1.25"); + fragmentShaderFile_ = ":bayer_1x_packed.frag"; + textureMinMagFilters_ = GL_NEAREST; + break; + case libcamera::formats::SGBRG10_CSI2P: + firstRed_.setX(0.0); + firstRed_.setY(1.0); + fragmentShaderDefines_.append("#define BPP_X 1.25"); + fragmentShaderFile_ = ":bayer_1x_packed.frag"; + textureMinMagFilters_ = GL_NEAREST; + break; + case libcamera::formats::SGRBG10_CSI2P: + firstRed_.setX(1.0); + firstRed_.setY(0.0); + fragmentShaderDefines_.append("#define BPP_X 1.25"); + fragmentShaderFile_ = ":bayer_1x_packed.frag"; + textureMinMagFilters_ = GL_NEAREST; + break; + case libcamera::formats::SRGGB10_CSI2P: + firstRed_.setX(0.0); + firstRed_.setY(0.0); + fragmentShaderDefines_.append("#define BPP_X 1.25"); + fragmentShaderFile_ = ":bayer_1x_packed.frag"; + textureMinMagFilters_ = GL_NEAREST; + break; default: ret = false; qWarning() << "[ViewFinderGL]:" @@ -290,6 +326,8 @@ bool ViewFinderGL::createFragmentShader() textureUniformU_ = shaderProgram_.uniformLocation("tex_u"); textureUniformV_ = shaderProgram_.uniformLocation("tex_v"); textureUniformStep_ = shaderProgram_.uniformLocation("tex_step"); + textureUniformSize_ = shaderProgram_.uniformLocation("tex_size"); + textureUniformBayerFirstRed_ = shaderProgram_.uniformLocation("tex_bayer_first_red"); /* Create the textures. */ for (std::unique_ptr &texture : textures_) { @@ -306,8 +344,10 @@ bool ViewFinderGL::createFragmentShader() void ViewFinderGL::configureTexture(QOpenGLTexture &texture) { glBindTexture(GL_TEXTURE_2D, texture.textureId()); - glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR); - glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR); + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, + textureMinMagFilters_); + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, + textureMinMagFilters_); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE); } @@ -547,6 +587,39 @@ void ViewFinderGL::doRender() shaderProgram_.setUniformValue(textureUniformY_, 0); break; + case libcamera::formats::SBGGR10_CSI2P: + case libcamera::formats::SGBRG10_CSI2P: + case libcamera::formats::SGRBG10_CSI2P: + case libcamera::formats::SRGGB10_CSI2P: + /* + * Packed raw Bayer 10-bit formats are stored in GL_RED texture. + * The texture width is 10/8 of the image width. + * TODO: account for padding bytes at the end of the line. + */ + glActiveTexture(GL_TEXTURE0); + configureTexture(*textures_[0]); + glTexImage2D(GL_TEXTURE_2D, + 0, + GL_RED, + size_.width() * 5 / 4, + size_.height(), + 0, + GL_RED, + GL_UNSIGNED_BYTE, + data_); + shaderProgram_.setUniformValue(textureUniformY_, 0); + shaderProgram_.setUniformValue(textureUniformBayerFirstRed_, + firstRed_); + shaderProgram_.setUniformValue(textureUniformSize_, + size_.width() * 5 / 4, + size_.height(), + size_.width(), + size_.height()); + shaderProgram_.setUniformValue(textureUniformStep_, + 1.0f / (size_.width() * 5 / 4 - 1), + 1.0f / (size_.height() - 1)); + break; + default: break; }; diff --git a/src/qcam/viewfinder_gl.h b/src/qcam/viewfinder_gl.h index 1b1faa91..337718e3 100644 --- a/src/qcam/viewfinder_gl.h +++ b/src/qcam/viewfinder_gl.h @@ -81,13 +81,19 @@ private: /* Textures */ std::array, 3> textures_; + /* Common texture parameters */ + GLuint textureMinMagFilters_; /* YUV texture parameters */ GLuint textureUniformU_; GLuint textureUniformV_; GLuint textureUniformY_; + GLuint textureUniformSize_; GLuint textureUniformStep_; unsigned int horzSubSample_; unsigned int vertSubSample_; + /* Raw Bayer texture parameters */ + GLuint textureUniformBayerFirstRed_; + QPointF firstRed_; QMutex mutex_; /* Prevent concurrent access to image_ */ };