{"id":9643,"url":"https://patchwork.libcamera.org/api/patches/9643/?format=json","web_url":"https://patchwork.libcamera.org/patch/9643/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20200916145254.1644-8-laurent.pinchart@ideasonboard.com>","date":"2020-09-16T14:52:54","name":"[libcamera-devel,7/7] qcam: viewfinder_gl: Add shader to render packed YUV formats","commit_ref":"dcc47ff715ef1a8b97df53c810ec0c4b069b06e5","pull_url":null,"state":"accepted","archived":false,"hash":"0ff097f8f8d3c57f433c24b6876959b8dc602b94","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/?format=json","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/9643/mbox/","series":[{"id":1293,"url":"https://patchwork.libcamera.org/api/series/1293/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=1293","date":"2020-09-16T14:52:47","name":"qcam: Accelerate packed YUV rendering with OpenGL","version":1,"mbox":"https://patchwork.libcamera.org/series/1293/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/9643/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/9643/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 9B819BF01C\n\tfor ;\n\tWed, 16 Sep 2020 14:55:05 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 64ED262EDD;\n\tWed, 16 Sep 2020 16:55:05 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 552FB60533\n\tfor ;\n\tWed, 16 Sep 2020 16:55:03 +0200 (CEST)","from pendragon.lan (62-78-145-57.bb.dnainternet.fi [62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 8AA7326B;\n\tWed, 16 Sep 2020 16:54:46 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"M/morLns\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1600268094;\n\tbh=Swn/Ir4eReUebL8j0ylBOlhA+RLunAwFJu3LZ7HFe8M=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n\tb=M/morLnsgD30ia36aN12fX89YqwDZxZwP9/8QOhoQkjAKBx7LzsW6x+NXfo2UyGaD\n\t8YzC4KoTXK2BtMN2GcJXNsadmJJ7g+2WChL+hvcjRYOde+nibuhnN4f2I6oRVY5ck7\n\tsLQ1A/ATlkADdC0BI3zAKMHy2Ij+1L562co7g5CE=","From":"Laurent Pinchart ","To":"libcamera-devel@lists.libcamera.org","Date":"Wed, 16 Sep 2020 17:52:54 +0300","Message-Id":"<20200916145254.1644-8-laurent.pinchart@ideasonboard.com>","X-Mailer":"git-send-email 2.27.0","In-Reply-To":"<20200916145254.1644-1-laurent.pinchart@ideasonboard.com>","References":"<20200916145254.1644-1-laurent.pinchart@ideasonboard.com>","MIME-Version":"1.0","Subject":"[libcamera-devel] [PATCH 7/7] qcam: viewfinder_gl: Add shader to\n\trender packed YUV formats","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"The shader supports all 4 packed 8-bit YUV variants.\n\nSigned-off-by: Laurent Pinchart \n---\n src/qcam/assets/shader/YUV_packed.frag | 82 ++++++++++++++++++++++++++\n src/qcam/assets/shader/shaders.qrc | 1 +\n src/qcam/viewfinder_gl.cpp | 56 ++++++++++++++++++\n src/qcam/viewfinder_gl.h | 1 +\n 4 files changed, 140 insertions(+)\n create mode 100644 src/qcam/assets/shader/YUV_packed.frag","diff":"diff --git a/src/qcam/assets/shader/YUV_packed.frag b/src/qcam/assets/shader/YUV_packed.frag\nnew file mode 100644\nindex 000000000000..224dfafe383e\n--- /dev/null\n+++ b/src/qcam/assets/shader/YUV_packed.frag\n@@ -0,0 +1,82 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2020, Laurent Pinchart \n+ *\n+ * YUV_packed.frag - Fragment shader code for YUYV packed formats\n+ */\n+\n+#ifdef GL_ES\n+precision mediump float;\n+#endif\n+\n+varying vec2 textureOut;\n+\n+uniform sampler2D tex_y;\n+uniform float tex_stepx;\n+\n+void main(void)\n+{\n+\tmat3 yuv2rgb_bt601_mat = mat3(\n+\t\tvec3(1.164, 1.164, 1.164),\n+\t\tvec3(0.000, -0.392, 2.017),\n+\t\tvec3(1.596, -0.813, 0.000)\n+\t);\n+\tvec3 yuv2rgb_bt601_offset = vec3(0.063, 0.500, 0.500);\n+\n+\t/*\n+\t * The sampler won't interpolate the texture correctly along the X axis,\n+\t * as each RGBA pixel effectively stores two pixels. We thus need to\n+\t * interpolate manually.\n+\t *\n+\t * In integer texture coordinates, the Y values are layed out in the\n+\t * texture memory as follows:\n+\t *\n+\t * ...| Y U Y V | Y U Y V | Y U Y V |...\n+\t * ...| R G B A | R G B A | R G B A |...\n+\t * ^ ^ ^ ^ ^ ^\n+\t * | | | | | |\n+\t * n-1 n-0.5 n n+0.5 n+1 n+1.5\n+\t *\n+\t * For a texture location x in the interval [n, n+1[, sample the left\n+\t * and right pixels at n and n+1, and interpolate them with\n+\t *\n+\t * left.r * (1 - a) + left.b * a\tif fract(x) < 0.5\n+\t * left.b * (1 - a) + right.r * a\tif fract(x) >= 0.5\n+\t *\n+\t * with a = fract(x * 2) which can also be written\n+\t *\n+\t * a = fract(x) * 2\t\t\tif fract(x) < 0.5\n+\t * a = fract(x) * 2 - 1\t\t\tif fract(x) >= 0.5\n+\t */\n+\tvec2 pos = textureOut;\n+\tfloat f_x = fract(pos.x / tex_stepx);\n+\n+\tvec4 left = texture2D(tex_y, vec2(pos.x - f_x * tex_stepx, pos.y));\n+\tvec4 right = texture2D(tex_y, vec2(pos.x + (1.0 - f_x) * tex_stepx , pos.y));\n+\n+#if defined(YUV_PATTERN_UYVY)\n+\tfloat y_left = mix(left.g, left.a, f_x * 2.0);\n+\tfloat y_right = mix(left.a, right.g, f_x * 2.0 - 1.0);\n+\tvec2 uv = mix(left.rb, right.rb, f_x);\n+#elif defined(YUV_PATTERN_VYUY)\n+\tfloat y_left = mix(left.g, left.a, f_x * 2.0);\n+\tfloat y_right = mix(left.a, right.g, f_x * 2.0 - 1.0);\n+\tvec2 uv = mix(left.br, right.br, f_x);\n+#elif defined(YUV_PATTERN_YUYV)\n+\tfloat y_left = mix(left.r, left.b, f_x * 2.0);\n+\tfloat y_right = mix(left.b, right.r, f_x * 2.0 - 1.0);\n+\tvec2 uv = mix(left.ga, right.ga, f_x);\n+#elif defined(YUV_PATTERN_YVYU)\n+\tfloat y_left = mix(left.r, left.b, f_x * 2.0);\n+\tfloat y_right = mix(left.b, right.r, f_x * 2.0 - 1.0);\n+\tvec2 uv = mix(left.ag, right.ag, f_x);\n+#else\n+#error Invalid pattern\n+#endif\n+\n+\tfloat y = mix(y_left, y_right, step(0.5, f_x));\n+\n+\tvec3 rgb = yuv2rgb_bt601_mat * (vec3(y, uv) - yuv2rgb_bt601_offset);\n+\n+\tgl_FragColor = vec4(rgb, 1.0);\n+}\ndiff --git a/src/qcam/assets/shader/shaders.qrc b/src/qcam/assets/shader/shaders.qrc\nindex 7010d8433c9b..857ed9fd5cde 100644\n--- a/src/qcam/assets/shader/shaders.qrc\n+++ b/src/qcam/assets/shader/shaders.qrc\n@@ -4,5 +4,6 @@\n \tYUV.vert\n \tYUV_2_planes.frag\n \tYUV_3_planes.frag\n+\tYUV_packed.frag\n \n \ndiff --git a/src/qcam/viewfinder_gl.cpp b/src/qcam/viewfinder_gl.cpp\nindex b8a4827267c3..0b5c942658cd 100644\n--- a/src/qcam/viewfinder_gl.cpp\n+++ b/src/qcam/viewfinder_gl.cpp\n@@ -14,12 +14,19 @@\n #include \n \n static const QList supportedFormats{\n+\t/* Packed (single plane) */\n+\tlibcamera::formats::UYVY,\n+\tlibcamera::formats::VYUY,\n+\tlibcamera::formats::YUYV,\n+\tlibcamera::formats::YVYU,\n+\t/* Semi planar (two planes) */\n \tlibcamera::formats::NV12,\n \tlibcamera::formats::NV21,\n \tlibcamera::formats::NV16,\n \tlibcamera::formats::NV61,\n \tlibcamera::formats::NV24,\n \tlibcamera::formats::NV42,\n+\t/* Fully planar (three planes) */\n \tlibcamera::formats::YUV420,\n \tlibcamera::formats::YVU420,\n };\n@@ -149,6 +156,22 @@ bool ViewFinderGL::selectFormat(const libcamera::PixelFormat &format)\n \t\tvertSubSample_ = 2;\n \t\tfragmentShaderFile_ = \":YUV_3_planes.frag\";\n \t\tbreak;\n+\tcase libcamera::formats::UYVY:\n+\t\tfragmentShaderDefines_.append(\"#define YUV_PATTERN_UYVY\");\n+\t\tfragmentShaderFile_ = \":YUV_packed.frag\";\n+\t\tbreak;\n+\tcase libcamera::formats::VYUY:\n+\t\tfragmentShaderDefines_.append(\"#define YUV_PATTERN_VYUY\");\n+\t\tfragmentShaderFile_ = \":YUV_packed.frag\";\n+\t\tbreak;\n+\tcase libcamera::formats::YUYV:\n+\t\tfragmentShaderDefines_.append(\"#define YUV_PATTERN_YUYV\");\n+\t\tfragmentShaderFile_ = \":YUV_packed.frag\";\n+\t\tbreak;\n+\tcase libcamera::formats::YVYU:\n+\t\tfragmentShaderDefines_.append(\"#define YUV_PATTERN_YVYU\");\n+\t\tfragmentShaderFile_ = \":YUV_packed.frag\";\n+\t\tbreak;\n \tdefault:\n \t\tret = false;\n \t\tqWarning() << \"[ViewFinderGL]:\"\n@@ -235,6 +258,7 @@ bool ViewFinderGL::createFragmentShader()\n \ttextureUniformY_ = shaderProgram_.uniformLocation(\"tex_y\");\n \ttextureUniformU_ = shaderProgram_.uniformLocation(\"tex_u\");\n \ttextureUniformV_ = shaderProgram_.uniformLocation(\"tex_v\");\n+\ttextureUniformStepX_ = shaderProgram_.uniformLocation(\"tex_stepx\");\n \n \tif (!textureY_.isCreated())\n \t\ttextureY_.create();\n@@ -431,6 +455,38 @@ void ViewFinderGL::doRender()\n \t\tshaderProgram_.setUniformValue(textureUniformU_, 1);\n \t\tbreak;\n \n+\tcase libcamera::formats::UYVY:\n+\tcase libcamera::formats::VYUY:\n+\tcase libcamera::formats::YUYV:\n+\tcase libcamera::formats::YVYU:\n+\t\t/*\n+\t\t * Packed YUV formats are stored in a RGBA texture to match the\n+\t\t * OpenGL texel size with the 4 bytes repeating pattern in YUV.\n+\t\t * The texture width is thus half of the image with.\n+\t\t */\n+\t\tglActiveTexture(GL_TEXTURE0);\n+\t\tconfigureTexture(textureY_);\n+\t\tglTexImage2D(GL_TEXTURE_2D,\n+\t\t\t 0,\n+\t\t\t GL_RGBA,\n+\t\t\t size_.width() / 2,\n+\t\t\t size_.height(),\n+\t\t\t 0,\n+\t\t\t GL_RGBA,\n+\t\t\t GL_UNSIGNED_BYTE,\n+\t\t\t yuvData_);\n+\t\tshaderProgram_.setUniformValue(textureUniformY_, 0);\n+\n+\t\t/*\n+\t\t * The shader needs the step between two texture pixels in the\n+\t\t * horizontal direction, expressed in texture coordinate units\n+\t\t * ([0, 1]). There are exactly width - 1 steps between the\n+\t\t * leftmost and rightmost texels.\n+\t\t */\n+\t\tshaderProgram_.setUniformValue(textureUniformStepX_,\n+\t\t\t\t\t 1.0f / (size_.width() / 2 - 1));\n+\t\tbreak;\n+\n \tdefault:\n \t\tbreak;\n \t};\ndiff --git a/src/qcam/viewfinder_gl.h b/src/qcam/viewfinder_gl.h\nindex 53424dc10bc5..ad1e195e45c7 100644\n--- a/src/qcam/viewfinder_gl.h\n+++ b/src/qcam/viewfinder_gl.h\n@@ -79,6 +79,7 @@ private:\n \tGLuint textureUniformU_;\n \tGLuint textureUniformV_;\n \tGLuint textureUniformY_;\n+\tGLuint textureUniformStepX_;\n \tQOpenGLTexture textureU_;\n \tQOpenGLTexture textureV_;\n \tQOpenGLTexture textureY_;\n","prefixes":["libcamera-devel","7/7"]}