From patchwork Fri Jan 1 17:15:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 10801 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id EEF69C0F1A for ; Fri, 1 Jan 2021 17:16:27 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id B846A615D2; Fri, 1 Jan 2021 18:16:27 +0100 (CET) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="Dp6qHITx"; dkim-atps=neutral Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [IPv6:2a00:1450:4864:20::134]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9D7476031C for ; Fri, 1 Jan 2021 18:16:25 +0100 (CET) Received: by mail-lf1-x134.google.com with SMTP id o13so49862371lfr.3 for ; Fri, 01 Jan 2021 09:16:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9IyTWCw2utGO0KRGIo3WDP0dk63K2gFAGl/Yab4UDtM=; b=Dp6qHITxEOw3j067VMliEfo/stR4OdrDYwQ4jP82Asco0HQA7qQibL/XQXkH6oQLQH I7CKqbsMALUAdHygArKq6+K9GK9iXT54Y2bqcBDKyAQQNyy49bwDSf5pCdcB5UnzGAmH kjZC9rBmpSFcI9fpY0VTbevuxVzzbOY0qi6A6ahWXXzPguHkyrsZdzFz7iFLWsbMTJgj F4VZ8qEcUACEKFK8jvDdNW2bLnkAtluORBqXRqkkbssZmbQQ2mEphQOy53eNtcbBixVt +dH2Gmz1GGk2Rr75kV6dxD9+y2fTY6Llxb4sL4eiwfHax5Zw2b2wcejK0fl1j+k5FrC8 FSXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9IyTWCw2utGO0KRGIo3WDP0dk63K2gFAGl/Yab4UDtM=; b=QXZhm/Ga1n3RhZD5yI+wCVrqMpiYvmrTIYzFpKXEQqvwxbjspq6wy58M7zv7tC9dmy tCc4m0EzSJpUMjZIsiYSLVukPKM4SiqpPrVnep8KKbv6Z1hv5qipHmO0u2ovDYCNAw3d BYNFqXvhkjymh264CGzP7c2g28OW/w8pzjRGfKsJvRd1RJ/jD9ntytoge9uwuhJD6SDE I6Mf8FeiJpjCbn6mJXjZrl0rtcS2la2NBK2MEkKjmm6uGpi4h0O4QEEb5ys7IW8i3fRV vtMG1buLT0tbyjZeR6FLO8RqIeBoILystSvoKdkBo3hhXhd7FRpogY1QEi4JhBlYFHgJ /BEA== X-Gm-Message-State: AOAM533eeDbTM6amqIFrmFZV26AF7zUJoM/1w/o2vSmHROsmh+1/UAtr EQz+nwqwaPl+goIwG6YyvMPzwzFiqu5G2/Pf X-Google-Smtp-Source: ABdhPJyio7hslykYyuv08mWDFweA3kOZM6/1dIgzrcp3r3KINnJ6V+B5NnugomTAw55g0jiP594A7A== X-Received: by 2002:a19:8985:: with SMTP id l127mr14216946lfd.617.1609521384506; Fri, 01 Jan 2021 09:16:24 -0800 (PST) Received: from localhost.localdomain ([85.249.43.69]) by smtp.googlemail.com with ESMTPSA id m12sm7873213lji.110.2021.01.01.09.16.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Jan 2021 09:16:23 -0800 (PST) From: Andrey Konovalov To: libcamera-devel@lists.libcamera.org, laurent.pinchart@ideasonboard.com Date: Fri, 1 Jan 2021 20:15:53 +0300 Message-Id: <20210101171553.32727-3-andrey.konovalov@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210101171553.32727-1-andrey.konovalov@linaro.org> References: <20210101171553.32727-1-andrey.konovalov@linaro.org> Subject: [libcamera-devel] [PATCH][RFC v2 2/2] qcam: viewfinder_gl: Add shader to render packed RAW12 formats X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: morgan@casual-effects.com MIME-Version: 1.0 Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The shader supports all 4 packed RAW12 variants. Simple bi-linear filtering is implemented. The 4 LS bits of the 12-bit colour values are dropped as the RGBA format we convert into has only 8 bits per colour. The texture coordinates passed to the fragment shader are ajusted to point to the nearest pixel in the image. This prevents artifacts when the image is scaled from the frame resolution to the window size. Signed-off-by: Andrey Konovalov --- src/qcam/assets/shader/bayer_12_packed.frag | 196 ++++++++++++++++++++ src/qcam/assets/shader/shaders.qrc | 1 + src/qcam/viewfinder_gl.cpp | 72 ++++++- src/qcam/viewfinder_gl.h | 6 + 4 files changed, 273 insertions(+), 2 deletions(-) create mode 100644 src/qcam/assets/shader/bayer_12_packed.frag diff --git a/src/qcam/assets/shader/bayer_12_packed.frag b/src/qcam/assets/shader/bayer_12_packed.frag new file mode 100644 index 00000000..9de7e802 --- /dev/null +++ b/src/qcam/assets/shader/bayer_12_packed.frag @@ -0,0 +1,196 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* + * Based on the code from http://jgt.akpeters.com/papers/McGuire08/ + * + * Efficient, High-Quality Bayer Demosaic Filtering on GPUs + * + * Morgan McGuire + * + * This paper appears in issue Volume 13, Number 4. + * --------------------------------------------------------- + * Copyright (c) 2008, Morgan McGuire. All rights reserved. + * + * + * Modified by Linaro Ltd for 12-bit packed vs 8-bit raw Bayer format, + * and for simpler demosaic algorithm. + * Copyright (C) 2020, Linaro + * + * bayer_12_packed.frag - Fragment shader code for raw Bayer 12-bit packed + * format + */ + +#ifdef GL_ES +precision mediump float; +#endif + +varying vec2 textureOut; + +uniform vec2 tex_size; +uniform vec2 tex_step; +uniform vec2 tex_bayer_first_red; + +uniform sampler2D tex_raw; + +void main(void) +{ + vec3 rgb; + + /* + * center holds the coordinates of the pixel being sampled. + * center.xy are on the [0, 1] range, + * center.zw are in pixels: [0, frame width - 1] and + * [0, frame height - 1] respectively. + */ + vec4 center; + /* + * x-positions of the adjacent pixels on the [0, 1] range. + */ + vec2 xcoords; + /* + * y-positions of the adjacent pixels on the [0, 1] range. + */ + vec2 ycoords; + + /* + * The coordinates passed to the shader in textureOut may point + * to a place in between the pixels if the viewfinder window is scaled + * from the original captured frame size. Align them to the nearest + * texel (center.xy) and pixel (center.zw). + */ + center.xy = textureOut * tex_size; /* [0, texture size) */ + center.z = floor(center.x * 2.0); + center.xy = floor(center.xy); + center.w = center.y; + center.xy *= tex_step; /* on the [0, 1] range */ + + vec2 alternate = mod(center.zw + tex_bayer_first_red, 2.0); + bool even_col = alternate.x < 1.0; + bool even_raw = alternate.y < 1.0; + + xcoords = center.x + vec2(-tex_step.x, tex_step.x); + ycoords = center.y + vec2(-tex_step.y, tex_step.y); + + /* + * We need to sample the central pixel and the ones with offset + * of -1 to +1 pixel in both X and Y directions. Let's name these + * pixels as below, where C is the central pixel: + * +----+----+----+----+ + * | \ x| | | | + * |y \ | -1 | 0 | +1 | + * +----+----+----+----+ + * | +1 | D2 | A1 | D3 | + * +----+----+----+----+ + * | 0 | B0 | C | B1 | + * +----+----+----+----+ + * | -1 | D0 | A0 | D1 | + * +----+----+----+----+ + * As we use RGB texture, and take only 8 MS bits of every pixel value, + * the blue component is never used. + * In the below equations (0,-1).r means "r component of the texel + * shifted by -tex_step.y from the center.xy one" etc. + * In the even raw / even column (EE) case the colour values are: + * R = C = (0,0).r, + * G = (A0 + A1 + B0 + B1) / 4.0 = + * ( (0,-1).r + (0,1).r + (-1,0).g + (0,0).g ) / 4.0, + * B = (D0 + D1 + D2 + D3) / 4.0 = + * ( (-1,-1).g + (0,-1).g + (-1,1).g + (0,1).g ) / 4.0 + * For even raw / odd column (EO): + * R = (B0 + B1) / 2.0 = ( (0,0).r + (1,0).r ) / 2.0, + * G = C = (0,0).g, + * B = (A0 + A1) / 2.0 = ( (0,-1).g + (0,1).g ) / 2.0 + * For odd raw / even column (OE): + * R = (A0 + A1) / 2.0 = ( (0,-1).r + (0,1).r ) / 2.0, + * G = C = (0,0).r, + * B = (B0 + B1) / 2.0 = ( (-1,0).g + (0,0).g ) / 2.0 + * For odd raw / odd column (OO): + * R = (D0 + D1 + D2 + D3) / 4.0 = + * ( (0,-1).r + (1,-1).r + (0,1).r + (1,1).r ) / 4.0, + * G = (A0 + A1 + B0 + B1) / 4.0 = + * ( (0,-1).g + (0,1).g + (0,0).r + (1,0).r ) / 4.0, + * B = C = (0,0).g + */ + + /* Fetch all the values we might need */ + + /* .xy = (0,-1).rg, zw = (0, 1).rg - all Ax, and some of Dx values */ + vec4 vals_AD = vec4( + texture2D(tex_raw, vec2(center.x, ycoords[0])).rg, + texture2D(tex_raw, vec2(center.x, ycoords[1])).rg); + /* + * .xy = (0,0).rg, .z = (-1,0).g, .w = (1,0).r + * - all Bx and C, some of Dx values + */ + vec4 vals_BCD = vec4( + texture2D(tex_raw, center.xy).rg, + texture2D(tex_raw, vec2(xcoords[0], center.y)).g, + texture2D(tex_raw, vec2(xcoords[1], center.y)).r); + /* + * .x = (-1,-1).g, .y = (-1,1).g, .z = (1,-1).r, .w = (1,1).r + * - the remaining Dx values + */ + vec4 vals_D = vec4( + texture2D(tex_raw, vec2(xcoords[0], ycoords[0])).g, + texture2D(tex_raw, vec2(xcoords[0], ycoords[1])).g, + texture2D(tex_raw, vec2(xcoords[1], ycoords[0])).r, + texture2D(tex_raw, vec2(xcoords[1], ycoords[1])).r); + + /* + * Let: + * E = ( (0,-1).r + (0, 1).r ) / 2.0 + * F = ( (0,-1).g + (0, 1).g ) / 2.0 + * G = ( (0,0).g + (-1,0).g ) / 2.0 + * H = ( (0,0).r + (1,0).r ) / 2.0 + * J = ( (-1,-1).g + (-1,1).g ) / 2.0 + * K = ( (1,-1).r + (1,1).r ) / 2.0 + * Then: + * Even raw, even column case (EE): + * Red = (0,0).r, + * Green = (E + G) / 2.0, + * Blue = (F + J) / 2.0 + * EO: + * Red = H, + * Green = (0,0).g, + * Blue = F + * OE: + * Red = E, + * Green = (0,0).r, + * Blue = G + * OO: + * R = (E + K) / 2.0, + * G = (F + H ) / 2.0, + * B = (0,0).g + * Then if we let: + * L = (E + G) / 2.0 + * M = (F + H) / 2.0 + * and modify J and K like this: + * J = (F + J) / 2.0 + * K = (E + K) / 2.0 + * the Red/Green/Blue output values can be obtained from EFGH and JKLM + * with swizzles thus eliminating the branches. + */ + + vec4 EFGH = vec4( + vals_AD.x + vals_AD.z, + vals_AD.y + vals_AD.w, + vals_BCD.y + vals_BCD.z, + vals_BCD.x + vals_BCD.w + ) / 2.0; + vec4 JKLM = vec4( + vals_D.x + vals_D.y, + vals_D.z + vals_D.w, + EFGH.x + EFGH.z, /* E + G */ + EFGH.y + EFGH.w /* F + H */ + ) / 2.0; + + JKLM.xy = (JKLM.xy + EFGH.yx) / 2.0; + + rgb = (even_col) ? + ((even_raw) ? + vec3(vals_BCD.x, JKLM.z, JKLM.x) : + vec3(EFGH.x, vals_BCD.x, EFGH.z)) : + ((even_raw) ? + vec3(EFGH.w, vals_BCD.y, EFGH.y) : + vec3(JKLM.y, JKLM.w, vals_BCD.y)); + + gl_FragColor = vec4(rgb, 1.0); +} diff --git a/src/qcam/assets/shader/shaders.qrc b/src/qcam/assets/shader/shaders.qrc index 8a8f9de1..2650e0e3 100644 --- a/src/qcam/assets/shader/shaders.qrc +++ b/src/qcam/assets/shader/shaders.qrc @@ -5,6 +5,7 @@ YUV_2_planes.frag YUV_3_planes.frag YUV_packed.frag + bayer_12_packed.frag identity.vert diff --git a/src/qcam/viewfinder_gl.cpp b/src/qcam/viewfinder_gl.cpp index ff719418..89e70730 100644 --- a/src/qcam/viewfinder_gl.cpp +++ b/src/qcam/viewfinder_gl.cpp @@ -36,6 +36,11 @@ static const QList supportedFormats{ libcamera::formats::RGBA8888, libcamera::formats::BGR888, libcamera::formats::RGB888, + /* Raw Bayer 12-bit packed */ + libcamera::formats::SBGGR12_CSI2P, + libcamera::formats::SGBRG12_CSI2P, + libcamera::formats::SGRBG12_CSI2P, + libcamera::formats::SRGGB12_CSI2P, }; ViewFinderGL::ViewFinderGL(QWidget *parent) @@ -114,6 +119,9 @@ bool ViewFinderGL::selectFormat(const libcamera::PixelFormat &format) { bool ret = true; + /* Set min/mag filters to GL_LINEAR by default. */ + textureMinMagFilters_ = GL_LINEAR; + fragmentShaderDefines_.clear(); switch (format) { @@ -203,6 +211,30 @@ bool ViewFinderGL::selectFormat(const libcamera::PixelFormat &format) fragmentShaderDefines_.append("#define RGB_PATTERN bgr"); fragmentShaderFile_ = ":RGB.frag"; break; + case libcamera::formats::SBGGR12_CSI2P: + firstRed_.setX(1.0); + firstRed_.setY(1.0); + fragmentShaderFile_ = ":bayer_12_packed.frag"; + textureMinMagFilters_ = GL_NEAREST; + break; + case libcamera::formats::SGBRG12_CSI2P: + firstRed_.setX(0.0); + firstRed_.setY(1.0); + fragmentShaderFile_ = ":bayer_12_packed.frag"; + textureMinMagFilters_ = GL_NEAREST; + break; + case libcamera::formats::SGRBG12_CSI2P: + firstRed_.setX(1.0); + firstRed_.setY(0.0); + fragmentShaderFile_ = ":bayer_12_packed.frag"; + textureMinMagFilters_ = GL_NEAREST; + break; + case libcamera::formats::SRGGB12_CSI2P: + firstRed_.setX(0.0); + firstRed_.setY(0.0); + fragmentShaderFile_ = ":bayer_12_packed.frag"; + textureMinMagFilters_ = GL_NEAREST; + break; default: ret = false; qWarning() << "[ViewFinderGL]:" @@ -290,6 +322,8 @@ bool ViewFinderGL::createFragmentShader() textureUniformU_ = shaderProgram_.uniformLocation("tex_u"); textureUniformV_ = shaderProgram_.uniformLocation("tex_v"); textureUniformStep_ = shaderProgram_.uniformLocation("tex_step"); + textureUniformSize_ = shaderProgram_.uniformLocation("tex_size"); + textureUniformBayerFirstRed_ = shaderProgram_.uniformLocation("tex_bayer_first_red"); /* Create the textures. */ for (std::unique_ptr &texture : textures_) { @@ -306,8 +340,10 @@ bool ViewFinderGL::createFragmentShader() void ViewFinderGL::configureTexture(QOpenGLTexture &texture) { glBindTexture(GL_TEXTURE_2D, texture.textureId()); - glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR); - glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR); + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, + textureMinMagFilters_); + glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, + textureMinMagFilters_); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE); } @@ -547,6 +583,38 @@ void ViewFinderGL::doRender() shaderProgram_.setUniformValue(textureUniformY_, 0); break; + case libcamera::formats::SBGGR12_CSI2P: + case libcamera::formats::SGBRG12_CSI2P: + case libcamera::formats::SGRBG12_CSI2P: + case libcamera::formats::SRGGB12_CSI2P: + /* + * Packed raw Bayer 12-bit formats are stored in RGB texture + * to match the OpenGL texel size with the 3 bytes repeating + * pattern in RAW12P. + * The texture width is thus half of the image with. + */ + glActiveTexture(GL_TEXTURE0); + configureTexture(*textures_[0]); + glTexImage2D(GL_TEXTURE_2D, + 0, + GL_RGB, + size_.width() / 2, + size_.height(), + 0, + GL_RGB, + GL_UNSIGNED_BYTE, + data_); + shaderProgram_.setUniformValue(textureUniformY_, 0); + shaderProgram_.setUniformValue(textureUniformBayerFirstRed_, + firstRed_); + shaderProgram_.setUniformValue(textureUniformSize_, + size_.width() / 2, + size_.height()); + shaderProgram_.setUniformValue(textureUniformStep_, + 1.0f / (size_.width() / 2 - 1), + 1.0f / (size_.height() - 1)); + break; + default: break; }; diff --git a/src/qcam/viewfinder_gl.h b/src/qcam/viewfinder_gl.h index 1b1faa91..337718e3 100644 --- a/src/qcam/viewfinder_gl.h +++ b/src/qcam/viewfinder_gl.h @@ -81,13 +81,19 @@ private: /* Textures */ std::array, 3> textures_; + /* Common texture parameters */ + GLuint textureMinMagFilters_; /* YUV texture parameters */ GLuint textureUniformU_; GLuint textureUniformV_; GLuint textureUniformY_; + GLuint textureUniformSize_; GLuint textureUniformStep_; unsigned int horzSubSample_; unsigned int vertSubSample_; + /* Raw Bayer texture parameters */ + GLuint textureUniformBayerFirstRed_; + QPointF firstRed_; QMutex mutex_; /* Prevent concurrent access to image_ */ };