{"id":10399,"url":"https://patchwork.libcamera.org/api/1.1/patches/10399/?format=json","web_url":"https://patchwork.libcamera.org/patch/10399/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20201109213411.30987-3-andrey.konovalov@linaro.org>","date":"2020-11-09T21:34:11","name":"[libcamera-devel,RFC,2/2] qcam: viewfinder_gl: Add shader to render packed RAW12 formats","commit_ref":null,"pull_url":null,"state":"changes-requested","archived":false,"hash":"2238ee6102157ec4682b6e3cfd2305f2b0ac9789","submitter":{"id":25,"url":"https://patchwork.libcamera.org/api/1.1/people/25/?format=json","name":"Andrey Konovalov","email":"andrey.konovalov@linaro.org"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/10399/mbox/","series":[{"id":1453,"url":"https://patchwork.libcamera.org/api/1.1/series/1453/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=1453","date":"2020-11-09T21:34:09","name":"qcam: viewfinder_gl: add RAW12P format support","version":1,"mbox":"https://patchwork.libcamera.org/series/1453/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/10399/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/10399/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 561B4BE082\n\tfor ;\n\tMon, 9 Nov 2020 21:34:39 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 251AE63092;\n\tMon, 9 Nov 2020 22:34:39 +0100 (CET)","from mail-lj1-x236.google.com (mail-lj1-x236.google.com\n\t[IPv6:2a00:1450:4864:20::236])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 41D896308E\n\tfor ;\n\tMon, 9 Nov 2020 22:34:37 +0100 (CET)","by mail-lj1-x236.google.com with SMTP id h23so7743556ljg.13\n\tfor ;\n\tMon, 09 Nov 2020 13:34:37 -0800 (PST)","from localhost.localdomain ([85.249.41.73])\n\tby smtp.googlemail.com with ESMTPSA id\n\tc1sm1375986lfj.222.2020.11.09.13.34.35\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tMon, 09 Nov 2020 13:34:35 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=linaro.org header.i=@linaro.org\n\theader.b=\"tJv5tCfh\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google;\n\th=from:to:cc:subject:date:message-id:in-reply-to:references;\n\tbh=waZ1r9rpLnabALA0VgcsP+5ckvQMLi/sijirHcmOdz4=;\n\tb=tJv5tCfhirJqXEIUudJfAPgHABr6JsjaZaCJAngjHP8f4c9iBu1LoK85b3MRTOwkNE\n\tDVP59F6uxzGT/OMd9kjg582caQ9G3AEqGKZIPhoJnK1ROVtaNHhIpwfsATRsqhKPQCCY\n\tL75a6ZCqkAekihoCngnHjHIhXRHqEHDskjx+RksQuud2vWH0Keyi1ET747hhWNlLn849\n\t/pk7AJ7wycf+M6pUydLdyhWlZ9c4JK/9acGFJmgQpMhgPCGbPwJe4Ew9uADr35VHaQi+\n\tCORgB9gzrdoBvzti8XNfZ7sXzGlOd17PqYImr/jYsYSNMr1IeYLV6BnxSCImo6F+SyP3\n\tRELw==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to\n\t:references;\n\tbh=waZ1r9rpLnabALA0VgcsP+5ckvQMLi/sijirHcmOdz4=;\n\tb=obTRazrJkvvB6pSW3WoyQyHq89McKdX0mYBpG19aEhojWL5p90nrZI6/7+Qa/c4++d\n\tHYXl3QRhyzNiP+tU4aJK/acKcxa6DTIxsRt9rQsbgXg4UxOPW9o2Jyel/1bCtKVYkMJI\n\tHrS3oQaGpXo0QXhR3g9IuaVlewxImtOApNSVW7DD4NLzQfsu0ujedBPcRol3EazRks6A\n\tGjGHy7VUNcdF1UmcuufQCdNjtsk9mWzL1mQxIrVJdWP1uFX7H5jClv8W8RXnIUgqCOUL\n\tQ1hTaw+jrFlLzEG+AfRxh8zMupvf/9/YUL7TMOEpqP0WJTjWtVQ0UyrkHakANoveNDpS\n\tSLqQ==","X-Gm-Message-State":"AOAM5310G5EHBVBtUZpDbIFB1jY/vXt43OOoRGW5eONpUCEltxMp2y77\n\tNnsYO0VeeTsXC/Ttyaa6HlJ6FsH9nR9CNg==","X-Google-Smtp-Source":"ABdhPJy5xpkS5loF43aJuPnuo3XtBVG8fa2DA0tX91asK2J3YwV5DWcVQw5TaGmIlpVwelQseGakAQ==","X-Received":"by 2002:a2e:8216:: with SMTP id\n\tw22mr6503132ljg.358.1604957676046; \n\tMon, 09 Nov 2020 13:34:36 -0800 (PST)","From":"Andrey Konovalov ","To":"libcamera-devel@lists.libcamera.org","Date":"Tue, 10 Nov 2020 00:34:11 +0300","Message-Id":"<20201109213411.30987-3-andrey.konovalov@linaro.org>","X-Mailer":"git-send-email 2.17.1","In-Reply-To":"<20201109213411.30987-1-andrey.konovalov@linaro.org>","References":"<20201109213411.30987-1-andrey.konovalov@linaro.org>","Subject":"[libcamera-devel] [PATCH][RFC 2/2] qcam: viewfinder_gl: Add shader\n\tto render packed RAW12 formats","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"The shader supports all 4 packed RAW12 variants.\nSimple bi-linear filtering is implemented.\nThe 4 LS bits of the 12-bit colour values are dropped as the RGBA\nformat we convert into has only 8 bits per colour.\n\nSigned-off-by: Andrey Konovalov \n---\n src/qcam/assets/shader/bayer.vert | 28 ++++++++\n src/qcam/assets/shader/bayer_12_packed.frag | 68 +++++++++++++++++++\n src/qcam/assets/shader/shaders.qrc | 2 +\n src/qcam/viewfinder_gl.cpp | 74 ++++++++++++++++++++-\n src/qcam/viewfinder_gl.h | 7 ++\n 5 files changed, 177 insertions(+), 2 deletions(-)\n create mode 100644 src/qcam/assets/shader/bayer.vert\n create mode 100644 src/qcam/assets/shader/bayer_12_packed.frag","diff":"diff --git a/src/qcam/assets/shader/bayer.vert b/src/qcam/assets/shader/bayer.vert\nnew file mode 100644\nindex 00000000..375e0b60\n--- /dev/null\n+++ b/src/qcam/assets/shader/bayer.vert\n@@ -0,0 +1,28 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2020, Linaro\n+ *\n+ * bayer.vert - Vertex shader for raw Bayer to RGB conversion\n+ */\n+\n+attribute vec4 vertexIn;\n+attribute vec2 textureIn;\n+\n+uniform vec4 srcSize;\n+uniform vec2 firstRed;\n+\n+varying vec4 center;\n+varying vec2 xcoords;\n+varying vec2 ycoords;\n+\n+void main(void)\n+{\n+\tcenter.xy = textureIn;\n+\tcenter.zw = textureIn * srcSize.xy + firstRed;\n+\n+\tvec2 invSize = srcSize.zw;\n+\txcoords = center.x + vec2(-invSize.x, invSize.x);\n+\tycoords = center.y + vec2(-invSize.y, invSize.y);\n+\n+\tgl_Position = vertexIn;\n+}\ndiff --git a/src/qcam/assets/shader/bayer_12_packed.frag b/src/qcam/assets/shader/bayer_12_packed.frag\nnew file mode 100644\nindex 00000000..eb822b85\n--- /dev/null\n+++ b/src/qcam/assets/shader/bayer_12_packed.frag\n@@ -0,0 +1,68 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2020, Linaro\n+ *\n+ * bayer_12_packed.frag - Fragment shader code for raw Bayer 12-bit packed\n+ * format\n+ */\n+\n+#ifdef GL_ES\n+precision mediump float;\n+#endif\n+\n+varying vec4 center;\n+varying vec2 xcoords;\n+varying vec2 ycoords;\n+\n+uniform sampler2D tex_raw;\n+\n+void main(void)\n+{\n+\tvec3 rgb;\n+\n+\tvec2 alternate = mod(center.zw, 2.0);\n+\n+\tbool even_col = alternate.x < 1.0;\n+\tbool even_raw = alternate.y < 1.0;\n+\n+\t/* .xy = (0,-1).rg, zw = (0, 1).rg */\n+\tvec4 vals_AD = vec4(\n+\t\ttexture2D(tex_raw, vec2(center.x, ycoords[0])).rg,\n+\t\ttexture2D(tex_raw, vec2(center.x, ycoords[1])).rg);\n+\t/* .xy = (0,0).rg, .z = (-1,0).g, .w = (1,0).r */\n+\tvec4 vals_BCD = vec4(\n+\t\ttexture2D(tex_raw, center.xy).rg,\n+\t\ttexture2D(tex_raw, vec2(xcoords[0], center.y)).g,\n+\t\ttexture2D(tex_raw, vec2(xcoords[1], center.y)).r);\n+\t/* .x = (-1,-1).g, .y = (-1,1).g, .z = (1,-1).r, .w = (1,1).r */\n+\tvec4 vals_D = vec4(\n+\t\ttexture2D(tex_raw, vec2(xcoords[0], ycoords[0])).g,\n+\t\ttexture2D(tex_raw, vec2(xcoords[0], ycoords[1])).g,\n+\t\ttexture2D(tex_raw, vec2(xcoords[1], ycoords[0])).r,\n+\t\ttexture2D(tex_raw, vec2(xcoords[1], ycoords[1])).r);\n+\n+\tvec4 EFGH = vec4(\n+\t\tvals_AD.x + vals_AD.z,\t\t/* 2*E = (0,-1).r + (0, 1).r */\n+\t\tvals_AD.y + vals_AD.w, \t\t/* 2*F = (0,-1).g + (0, 1).g */\n+\t\tvals_BCD.y + vals_BCD.z,\t/* 2*G = (0,0).g + (-1,0).g */\n+\t\tvals_BCD.x + vals_BCD.w\t\t/* 2*H = (0,0).r + (1,0).r */\n+\t\t) / 2.0;\n+\tvec2 JK = vec2(\n+\t\tvals_D.x + vals_D.y,\t\t/* 2*J = (-1,-1).g + (-1,1).g */\n+\t\tvals_D.z + vals_D.w\t\t/* 2*K = (1,-1).r + (1,1).r */\n+\t\t) / 2.0;\n+\n+\tif (even_col) {\n+\t\trgb = (even_raw) ?\n+\t\t\tvec3(vals_BCD.x, (EFGH.x + EFGH.z) / 2.0,\n+\t\t\t (EFGH.y + JK.x) / 2.0) :\n+\t\t\tvec3(EFGH.x, vals_BCD.x, EFGH.z);\n+\t} else {\n+\t\trgb = (even_raw) ?\n+\t\t\tvec3(EFGH.w, vals_BCD.y, EFGH.y) :\n+\t\t\tvec3((EFGH.x + JK.y) / 2.0,\n+\t\t\t (EFGH.y + EFGH.w) / 2.0,\n+\t\t\t vals_BCD.y);\n+\t}\n+\tgl_FragColor = vec4(rgb, 1.0);\n+}\ndiff --git a/src/qcam/assets/shader/shaders.qrc b/src/qcam/assets/shader/shaders.qrc\nindex 8a8f9de1..7bb18033 100644\n--- a/src/qcam/assets/shader/shaders.qrc\n+++ b/src/qcam/assets/shader/shaders.qrc\n@@ -5,6 +5,8 @@\n \tYUV_2_planes.frag\n \tYUV_3_planes.frag\n \tYUV_packed.frag\n+\tbayer_12_packed.frag\n+\tbayer.vert\n \tidentity.vert\n \n \ndiff --git a/src/qcam/viewfinder_gl.cpp b/src/qcam/viewfinder_gl.cpp\nindex c74ce77b..e504a6c2 100644\n--- a/src/qcam/viewfinder_gl.cpp\n+++ b/src/qcam/viewfinder_gl.cpp\n@@ -36,6 +36,11 @@ static const QList supportedFormats{\n \tlibcamera::formats::RGBA8888,\n \tlibcamera::formats::BGR888,\n \tlibcamera::formats::RGB888,\n+\t/* Raw Bayer 12-bit packed */\n+\tlibcamera::formats::SBGGR12_CSI2P,\n+\tlibcamera::formats::SGBRG12_CSI2P,\n+\tlibcamera::formats::SGRBG12_CSI2P,\n+\tlibcamera::formats::SRGGB12_CSI2P,\n };\n \n ViewFinderGL::ViewFinderGL(QWidget *parent)\n@@ -115,6 +120,7 @@ bool ViewFinderGL::selectFormat(const libcamera::PixelFormat &format)\n \tbool ret = true;\n \n \tvertexShaderFile_ = \":identity.vert\";\n+\ttextureMinMaxFilters_ = GL_LINEAR;\n \n \tfragmentShaderDefines_.clear();\n \n@@ -205,6 +211,34 @@ bool ViewFinderGL::selectFormat(const libcamera::PixelFormat &format)\n \t\tfragmentShaderDefines_.append(\"#define RGB_PATTERN bgr\");\n \t\tfragmentShaderFile_ = \":RGB.frag\";\n \t\tbreak;\n+\tcase libcamera::formats::SBGGR12_CSI2P:\n+\t\tfirstRed_[0] = 1.0;\n+\t\tfirstRed_[1] = 1.0;\n+\t\tfragmentShaderFile_ = \":bayer_12_packed.frag\";\n+\t\tvertexShaderFile_ = \":bayer.vert\";\n+\t\ttextureMinMaxFilters_ = GL_NEAREST;\n+\t\tbreak;\n+\tcase libcamera::formats::SGBRG12_CSI2P:\n+\t\tfirstRed_[0] = 0.0;\n+\t\tfirstRed_[1] = 1.0;\n+\t\tfragmentShaderFile_ = \":bayer_12_packed.frag\";\n+\t\tvertexShaderFile_ = \":bayer.vert\";\n+\t\ttextureMinMaxFilters_ = GL_NEAREST;\n+\t\tbreak;\n+\tcase libcamera::formats::SGRBG12_CSI2P:\n+\t\tfirstRed_[0] = 1.0;\n+\t\tfirstRed_[1] = 0.0;\n+\t\tfragmentShaderFile_ = \":bayer_12_packed.frag\";\n+\t\tvertexShaderFile_ = \":bayer.vert\";\n+\t\ttextureMinMaxFilters_ = GL_NEAREST;\n+\t\tbreak;\n+\tcase libcamera::formats::SRGGB12_CSI2P:\n+\t\tfirstRed_[0] = 0.0;\n+\t\tfirstRed_[1] = 0.0;\n+\t\tfragmentShaderFile_ = \":bayer_12_packed.frag\";\n+\t\tvertexShaderFile_ = \":bayer.vert\";\n+\t\ttextureMinMaxFilters_ = GL_NEAREST;\n+\t\tbreak;\n \tdefault:\n \t\tret = false;\n \t\tqWarning() << \"[ViewFinderGL]:\"\n@@ -292,6 +326,9 @@ bool ViewFinderGL::createFragmentShader()\n \ttextureUniformU_ = shaderProgram_.uniformLocation(\"tex_u\");\n \ttextureUniformV_ = shaderProgram_.uniformLocation(\"tex_v\");\n \ttextureUniformStepX_ = shaderProgram_.uniformLocation(\"tex_stepx\");\n+\ttextureUniformRaw_ = shaderProgram_.uniformLocation(\"tex_raw\");\n+\ttextureUniformSrcSize_ = shaderProgram_.uniformLocation(\"srcSize\");\n+\ttextureUniformFirstRed_ = shaderProgram_.uniformLocation(\"firstRed\");\n \n \t/* Create the textures. */\n \tfor (std::unique_ptr &texture : textures_) {\n@@ -308,8 +345,10 @@ bool ViewFinderGL::createFragmentShader()\n void ViewFinderGL::configureTexture(QOpenGLTexture &texture)\n {\n \tglBindTexture(GL_TEXTURE_2D, texture.textureId());\n-\tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);\n-\tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);\n+\tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER,\n+\t\t\ttextureMinMaxFilters_);\n+\tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER,\n+\t\t\ttextureMinMaxFilters_);\n \tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);\n \tglTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);\n }\n@@ -548,6 +587,37 @@ void ViewFinderGL::doRender()\n \t\tshaderProgram_.setUniformValue(textureUniformY_, 0);\n \t\tbreak;\n \n+\tcase libcamera::formats::SBGGR12_CSI2P:\n+\tcase libcamera::formats::SGBRG12_CSI2P:\n+\tcase libcamera::formats::SGRBG12_CSI2P:\n+\tcase libcamera::formats::SRGGB12_CSI2P:\n+\t\t/*\n+\t\t * Packed raw Bayer 12-bit foramts are stored in RGB texture\n+\t\t * to match the OpenGL texel size with the 3 bytes repeating\n+\t\t * pattern in RAW12P.\n+\t\t * The texture width is thus half of the image with.\n+\t\t */\n+\t\tglActiveTexture(GL_TEXTURE0);\n+\t\tconfigureTexture(*textures_[0]);\n+\t\tglTexImage2D(GL_TEXTURE_2D,\n+\t\t\t 0,\n+\t\t\t GL_RGB,\n+\t\t\t size_.width() / 2,\n+\t\t\t size_.height(),\n+\t\t\t 0,\n+\t\t\t GL_RGB,\n+\t\t\t GL_UNSIGNED_BYTE,\n+\t\t\t data_);\n+\t\tshaderProgram_.setUniformValue(textureUniformRaw_, 0);\n+\t\tshaderProgram_.setUniformValue(textureUniformFirstRed_,\n+\t\t\t\t\t firstRed_[0], firstRed_[1]);\n+\t\tshaderProgram_.setUniformValue(textureUniformSrcSize_,\n+\t\t\t\t\t size_.width(),\n+\t\t\t\t\t size_.height(),\n+\t\t\t\t\t 1.0f / (size_.width() / 2 - 1),\n+\t\t\t\t\t 1.0f / (size_.height() - 1));\n+\t\tbreak;\n+\n \tdefault:\n \t\tbreak;\n \t};\ndiff --git a/src/qcam/viewfinder_gl.h b/src/qcam/viewfinder_gl.h\nindex 6cf8f347..186492f3 100644\n--- a/src/qcam/viewfinder_gl.h\n+++ b/src/qcam/viewfinder_gl.h\n@@ -82,6 +82,8 @@ private:\n \t/* Textures */\n \tstd::array, 3> textures_;\n \n+\t/* Common texture parameters */\n+\tGLuint textureMinMaxFilters_;\n \t/* YUV texture parameters */\n \tGLuint textureUniformU_;\n \tGLuint textureUniformV_;\n@@ -89,6 +91,11 @@ private:\n \tGLuint textureUniformStepX_;\n \tunsigned int horzSubSample_;\n \tunsigned int vertSubSample_;\n+\t/* Raw Bayer texture parameters */\n+\tGLuint textureUniformRaw_;\n+\tGLuint textureUniformSrcSize_;\n+\tGLuint textureUniformFirstRed_;\n+\tGLfloat firstRed_[2];\n \n \tQMutex mutex_; /* Prevent concurrent access to image_ */\n };\n","prefixes":["libcamera-devel","RFC","2/2"]}