From patchwork Thu May 27 10:55:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12452 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 0A04BC3203 for ; Thu, 27 May 2021 10:55:37 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C0459602AF; Thu, 27 May 2021 12:55:36 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="k7epmvCC"; dkim-atps=neutral Received: from mail-lj1-x231.google.com (mail-lj1-x231.google.com [IPv6:2a00:1450:4864:20::231]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6CCEC602AB for ; Thu, 27 May 2021 12:55:30 +0200 (CEST) Received: by mail-lj1-x231.google.com with SMTP id p20so188729ljj.8 for ; Thu, 27 May 2021 03:55:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lV47W5pBA/MVL8Hi2QxGJiTUjZ70nJNpg0cFlxhb3TU=; b=k7epmvCCbyy6JABrmX9oev+cCxwVonACdqfbTzLTVUp5ZKEuLPkgiNQC+h46r1CLtv xkOm8rWfpt58iZddQ0Qo9aWNDllxnrgo7tw6zTLjQ+rzIUcMr2o8a24kCnUKUUdwJRcc nsQlJWIxPFFUv18B7zJ8qA+3qZohyVkuKntMNE4DupHc2gpWvFekovauaIOwLbEeHNQh qgUeK77KHUXWNdDG+EaIYTytNNBIoegEXBthdo/Vn2NVBsGfFsIunFyWwpZ+AeyfyWst 4AEEEV0wdL6Hrg0dOv9StBvQYrgHkwqM1MCU+QGb5LjXn6WsobR1vyd7xNUKGAH23P5p +ZRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lV47W5pBA/MVL8Hi2QxGJiTUjZ70nJNpg0cFlxhb3TU=; b=QW2mZd1KGiKmmTNC6qC7s2c9rX9LCOMfWwpcq8RkbE3KzGplewDOUIl3Y6ATSnmY79 fBiwpjgfVoMHG4K3Fj7UbJ3O1MA+Ol/vzoAouxSqOFzU2E0Zw3o5kd1BM/P+zyo/OHls 4YB58ARXDyTzHbwKD8B3vHZwJ1EAuScgLdNj3AbAEyoLhNeFMvFT4/5y+lvnDRLjWmA5 uinjWnzsLI7LqsT+uKAoJnKDmVb8wv3KdevJqyILQY6mUxAdLJQdI7G4pMrTlM69jbD/ /+ikfT5HfiaNeMzJGxlQd4FQZSgRBLP1hCg15UugLWUHZkVhcS5IhM/MR7KqphAI4bFu aRAw== X-Gm-Message-State: AOAM5313CYIfiGCgmy60JuYD+eigXdVdUwbxUN58dvvt2s8Lm+G5LrY3 MDT9A76s4H3U8cAt/ReiDBT0S3KZ6X9Qiw== X-Google-Smtp-Source: ABdhPJxc4uarbFcFAcfn9Qi93ttCO8ac+0RDj+e2Fs68QxdlZnHcbmemA0hlpLI2l1hJX8tiGkd/iQ== X-Received: by 2002:a2e:a78c:: with SMTP id c12mr2027444ljf.421.1622112929690; Thu, 27 May 2021 03:55:29 -0700 (PDT) Received: from localhost.localdomain ([85.249.41.56]) by smtp.googlemail.com with ESMTPSA id h23sm170148lfc.52.2021.05.27.03.55.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 May 2021 03:55:28 -0700 (PDT) From: Andrey Konovalov To: libcamera-devel@lists.libcamera.org, laurent.pinchart@ideasonboard.com Date: Thu, 27 May 2021 13:55:11 +0300 Message-Id: <20210527105511.447089-6-andrey.konovalov@linaro.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210527105511.447089-1-andrey.konovalov@linaro.org> References: <20210527105511.447089-1-andrey.konovalov@linaro.org> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 5/5] qcam: viewfinder_gl: Add support for RAW8 Bayer formats X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: morgan@casual-effects.com Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" All the four Bayer orders are supported. The texture coordinates passed to the fragment shader are ajusted to point to the nearest pixel in the image. This prevents artifacts when the image is scaled from the frame resolution to the window size. Signed-off-by: Andrey Konovalov --- src/qcam/assets/shader/bayer_8.frag | 136 ++++++++++++++++++++++++++++ src/qcam/assets/shader/bayer_8.vert | 72 +++++++++++++++ src/qcam/assets/shader/shaders.qrc | 1 + src/qcam/viewfinder_gl.cpp | 37 +++++++- 4 files changed, 244 insertions(+), 2 deletions(-) create mode 100644 src/qcam/assets/shader/bayer_8.frag create mode 100644 src/qcam/assets/shader/bayer_8.vert diff --git a/src/qcam/assets/shader/bayer_8.frag b/src/qcam/assets/shader/bayer_8.frag new file mode 100644 index 00000000..d93ef1da --- /dev/null +++ b/src/qcam/assets/shader/bayer_8.frag @@ -0,0 +1,136 @@ +/* SPDX-License-Identifier: BSD-2-Clause */ +/* +From http://jgt.akpeters.com/papers/McGuire08/ + +Efficient, High-Quality Bayer Demosaic Filtering on GPUs + +Morgan McGuire + +This paper appears in issue Volume 13, Number 4. +--------------------------------------------------------- +Copyright (c) 2008, Morgan McGuire. All rights reserved. + + +Modified by Linaro Ltd to integrate it into libcamera, and to +fix the artifacts due to pixel coordinates interpolation. +Copyright (C) 2021, Linaro +*/ + +//Pixel Shader + +varying vec2 textureOut; + +/* The texture size: tex_size.xy is in bytes, tex_size.zw is in pixels */ +uniform vec4 tex_size; +uniform vec2 tex_step; + +/** Pixel position of the first red pixel in the */ +/** Bayer pattern. [{0,1}, {0, 1}]*/ +uniform vec2 tex_bayer_first_red; + +/** Monochrome RGBA or GL_LUMINANCE Bayer encoded texture.*/ +uniform sampler2D tex_raw; + +void main(void) { + #define fetch(x, y) texture2D(tex_raw, vec2(x, y)).r + + /** .xy = Pixel being sampled in the fragment shader on the range [0, 1] + .zw = ...on the range [0, sourceSize], offset by firstRed */ + vec4 center; + + /** center.x + (-2/w, -1/w, 1/w, 2/w); These are the x-positions */ + /** of the adjacent pixels.*/ + vec4 xCoord; + + /** center.y + (-2/h, -1/h, 1/h, 2/h); These are the y-positions */ + /** of the adjacent pixels.*/ + vec4 yCoord; + + /* Align the center coordinates to the nearest pixel */ + center.zw = floor(textureOut * tex_size.zw); + center.xy = center.zw * tex_step; + center.zw += tex_bayer_first_red; + + xCoord = center.x + vec4(-2.0 * tex_step.x, + -tex_step.x, tex_step.x, 2.0 * tex_step.x); + yCoord = center.y + vec4(-2.0 * tex_step.y, + -tex_step.y, tex_step.y, 2.0 * tex_step.y); + + float C = texture2D(tex_raw, center.xy).r; // ( 0, 0) + const vec4 kC = vec4( 4.0, 6.0, 5.0, 5.0) / 8.0; + + // Determine which of four types of pixels we are on. + vec2 alternate = mod(floor(center.zw), 2.0); + + vec4 Dvec = vec4( + fetch(xCoord[1], yCoord[1]), // (-1,-1) + fetch(xCoord[1], yCoord[2]), // (-1, 1) + fetch(xCoord[2], yCoord[1]), // ( 1,-1) + fetch(xCoord[2], yCoord[2])); // ( 1, 1) + + vec4 PATTERN = (kC.xyz * C).xyzz; + + // Can also be a dot product with (1,1,1,1) on hardware where that is + // specially optimized. + // Equivalent to: D = Dvec[0] + Dvec[1] + Dvec[2] + Dvec[3]; + Dvec.xy += Dvec.zw; + Dvec.x += Dvec.y; + + vec4 value = vec4( + fetch(center.x, yCoord[0]), // ( 0,-2) + fetch(center.x, yCoord[1]), // ( 0,-1) + fetch(xCoord[0], center.y), // (-2, 0) + fetch(xCoord[1], center.y)); // (-1, 0) + + vec4 temp = vec4( + fetch(center.x, yCoord[3]), // ( 0, 2) + fetch(center.x, yCoord[2]), // ( 0, 1) + fetch(xCoord[3], center.y), // ( 2, 0) + fetch(xCoord[2], center.y)); // ( 1, 0) + + // Even the simplest compilers should be able to constant-fold these to + // avoid the division. + // Note that on scalar processors these constants force computation of some + // identical products twice. + const vec4 kA = vec4(-1.0, -1.5, 0.5, -1.0) / 8.0; + const vec4 kB = vec4( 2.0, 0.0, 0.0, 4.0) / 8.0; + const vec4 kD = vec4( 0.0, 2.0, -1.0, -1.0) / 8.0; + + // Conserve constant registers and take advantage of free swizzle on load + #define kE (kA.xywz) + #define kF (kB.xywz) + + value += temp; + + // There are five filter patterns (identity, cross, checker, + // theta, phi). Precompute the terms from all of them and then + // use swizzles to assign to color channels. + // + // Channel Matches + // x cross (e.g., EE G) + // y checker (e.g., EE B) + // z theta (e.g., EO R) + // w phi (e.g., EO R) + #define A (value[0]) + #define B (value[1]) + #define D (Dvec.x) + #define E (value[2]) + #define F (value[3]) + + // Avoid zero elements. On a scalar processor this saves two MADDs + // and it has no effect on a vector processor. + PATTERN.yzw += (kD.yz * D).xyy; + + PATTERN += (kA.xyz * A).xyzx + (kE.xyw * E).xyxz; + PATTERN.xw += kB.xw * B; + PATTERN.xz += kF.xz * F; + + vec3 rgb = (alternate.y == 0.0) ? + ((alternate.x == 0.0) ? + vec3(C, PATTERN.xy) : + vec3(PATTERN.z, C, PATTERN.w)) : + ((alternate.x == 0.0) ? + vec3(PATTERN.w, C, PATTERN.z) : + vec3(PATTERN.yx, C)); + gl_FragColor = vec4(rgb, 1.0); +} diff --git a/src/qcam/assets/shader/bayer_8.vert b/src/qcam/assets/shader/bayer_8.vert new file mode 100644 index 00000000..26525db7 --- /dev/null +++ b/src/qcam/assets/shader/bayer_8.vert @@ -0,0 +1,72 @@ +/* +From http://jgt.akpeters.com/papers/McGuire08/ + +Efficient, High-Quality Bayer Demosaic Filtering on GPUs + +Morgan McGuire + +This paper appears in issue Volume 13, Number 4. +--------------------------------------------------------- +Copyright (c) 2008, Morgan McGuire. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided + with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +*/ + +//Vertex Shader + +attribute vec4 vertexIn; +attribute vec2 textureIn; + +/* The texture size: tex_size.xy is in bytes, tex_size.zw is in pixels */ +uniform vec4 tex_size; +uniform vec2 tex_step; + +/** Pixel position of the first red pixel in the */ +/** Bayer pattern. [{0,1}, {0, 1}]*/ +uniform vec2 tex_bayer_first_red; + +/** .xy = Pixel being sampled in the fragment shader on the range [0, 1] + .zw = ...on the range [0, sourceSize], offset by firstRed */ +varying vec4 center; + +/** center.x + (-2/w, -1/w, 1/w, 2/w); These are the x-positions */ +/** of the adjacent pixels.*/ +varying vec4 xCoord; + +/** center.y + (-2/h, -1/h, 1/h, 2/h); These are the y-positions */ +/** of the adjacent pixels.*/ +varying vec4 yCoord; + +void main(void) { + center.xy = textureIn; + center.zw = textureIn * tex_size.zw + tex_bayer_first_red; + + xCoord = center.x + vec4(-2.0 * tex_step.x, + -tex_step.x, tex_step.x, 2.0 * tex_step.x); + yCoord = center.y + vec4(-2.0 * tex_step.y, + -tex_step.y, tex_step.y, 2.0 * tex_step.y); + + gl_Position = vertexIn; +} diff --git a/src/qcam/assets/shader/shaders.qrc b/src/qcam/assets/shader/shaders.qrc index d76d65c5..79f44a30 100644 --- a/src/qcam/assets/shader/shaders.qrc +++ b/src/qcam/assets/shader/shaders.qrc @@ -5,6 +5,7 @@ YUV_2_planes.frag YUV_3_planes.frag YUV_packed.frag + bayer_8.frag bayer_1x_packed.frag identity.vert diff --git a/src/qcam/viewfinder_gl.cpp b/src/qcam/viewfinder_gl.cpp index 13ab31aa..ef85e93d 100644 --- a/src/qcam/viewfinder_gl.cpp +++ b/src/qcam/viewfinder_gl.cpp @@ -36,6 +36,11 @@ static const QList supportedFormats{ libcamera::formats::RGBA8888, libcamera::formats::BGR888, libcamera::formats::RGB888, + /* Raw Bayer 8-bit */ + libcamera::formats::SBGGR8, + libcamera::formats::SGBRG8, + libcamera::formats::SGRBG8, + libcamera::formats::SRGGB8, /* Raw Bayer 10-bit packed */ libcamera::formats::SBGGR10_CSI2P, libcamera::formats::SGBRG10_CSI2P, @@ -220,6 +225,30 @@ bool ViewFinderGL::selectFormat(const libcamera::PixelFormat &format) fragmentShaderDefines_.append("#define RGB_PATTERN bgr"); fragmentShaderFile_ = ":RGB.frag"; break; + case libcamera::formats::SBGGR8: + firstRed_.setX(1.0); + firstRed_.setY(1.0); + fragmentShaderFile_ = ":bayer_8.frag"; + textureMinMagFilters_ = GL_NEAREST; + break; + case libcamera::formats::SGBRG8: + firstRed_.setX(0.0); + firstRed_.setY(1.0); + fragmentShaderFile_ = ":bayer_8.frag"; + textureMinMagFilters_ = GL_NEAREST; + break; + case libcamera::formats::SGRBG8: + firstRed_.setX(1.0); + firstRed_.setY(0.0); + fragmentShaderFile_ = ":bayer_8.frag"; + textureMinMagFilters_ = GL_NEAREST; + break; + case libcamera::formats::SRGGB8: + firstRed_.setX(0.0); + firstRed_.setY(0.0); + fragmentShaderFile_ = ":bayer_8.frag"; + textureMinMagFilters_ = GL_NEAREST; + break; case libcamera::formats::SBGGR10_CSI2P: firstRed_.setX(1.0); firstRed_.setY(1.0); @@ -624,6 +653,10 @@ void ViewFinderGL::doRender() shaderProgram_.setUniformValue(textureUniformY_, 0); break; + case libcamera::formats::SBGGR8: + case libcamera::formats::SGBRG8: + case libcamera::formats::SGRBG8: + case libcamera::formats::SRGGB8: case libcamera::formats::SBGGR10_CSI2P: case libcamera::formats::SGBRG10_CSI2P: case libcamera::formats::SGRBG10_CSI2P: @@ -633,8 +666,8 @@ void ViewFinderGL::doRender() case libcamera::formats::SGRBG12_CSI2P: case libcamera::formats::SRGGB12_CSI2P: /* - * Packed raw Bayer 10-bit and 12-bit formats are stored in - * GL_RED texture. + * Raw Bayer 8-bit, and packed raw Bayer 10-bit/12-bit formats + * are stored in GL_RED texture. */ glActiveTexture(GL_TEXTURE0); configureTexture(*textures_[0]);